[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 438039866 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2880952K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003190] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008320] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009041] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014064] *** VALIDATE selinux *** [ 0.022322] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026809] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029150] *** VALIDATE tmpfs *** [ 0.031341] *** VALIDATE proc *** [ 0.032284] *** VALIDATE cgroup *** [ 0.033014] *** VALIDATE cgroup2 *** [ 0.034280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036049] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037038] Spectre V2 : User space: Vulnerable [ 0.038015] Speculative Store Bypass: Vulnerable [ 0.041680] debug: unmapping init [mem 0xffffffffb5a59000-0xffffffffb5a60fff] [ 0.043193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044764] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.047018] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049018] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052323] rcu: Hierarchical SRCU implementation. [ 0.054518] smp: Bringing up secondary CPUs ... [ 0.055570] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.059752] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142259] node 0 deferred pages initialised in 77ms [ 0.146303] devtmpfs: initialized [ 0.147225] x86/mm: Memory block size: 128MB [ 0.149848] gcov: version magic: 0x41383552 [ 0.151352] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153313] pinctrl core: initialized pinctrl subsystem [ 0.154211] [ 0.154792] ************************************************************* [ 0.155014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156016] ** ** [ 0.157010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158013] ** ** [ 0.159013] ** This means that this kernel is built to expose internal ** [ 0.160013] ** IOMMU data structures, which may compromise security on ** [ 0.161019] ** your system. ** [ 0.162014] ** ** [ 0.163014] ** If you see this message and you are not debugging the ** [ 0.164012] ** kernel, report this immediately to your vendor! ** [ 0.165013] ** ** [ 0.166015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167012] ************************************************************* [ 0.168767] NET: Registered protocol family 16 [ 0.169466] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173046] cpuidle: using governor menu [ 0.174775] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177450] PCI: Using configuration type 1 for base access [ 0.180138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190180] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195297] cryptd: max_cpu_qlen set to 1000 [ 0.197406] ACPI: Added _OSI(Module Device) [ 0.198010] ACPI: Added _OSI(Processor Device) [ 0.199008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204205] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.208563] ACPI: Interpreter enabled [ 0.209084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210011] ACPI: Using IOAPIC for interrupt routing [ 0.211093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212448] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222099] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.225077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227416] acpiphp: Slot [2] registered [ 0.228125] acpiphp: Slot [5] registered [ 0.229107] acpiphp: Slot [6] registered [ 0.230120] acpiphp: Slot [3] registered [ 0.231101] acpiphp: Slot [4] registered [ 0.232088] acpiphp: Slot [7] registered [ 0.233127] acpiphp: Slot [8] registered [ 0.234103] acpiphp: Slot [9] registered [ 0.235106] acpiphp: Slot [10] registered [ 0.236116] acpiphp: Slot [11] registered [ 0.238119] acpiphp: Slot [12] registered [ 0.240121] acpiphp: Slot [13] registered [ 0.242099] acpiphp: Slot [14] registered [ 0.243107] acpiphp: Slot [15] registered [ 0.244089] acpiphp: Slot [16] registered [ 0.246122] acpiphp: Slot [17] registered [ 0.248098] acpiphp: Slot [18] registered [ 0.249114] acpiphp: Slot [19] registered [ 0.251144] acpiphp: Slot [20] registered [ 0.252063] acpiphp: Slot [21] registered [ 0.253217] acpiphp: Slot [22] registered [ 0.255098] acpiphp: Slot [23] registered [ 0.256110] acpiphp: Slot [24] registered [ 0.258136] acpiphp: Slot [25] registered [ 0.260121] acpiphp: Slot [26] registered [ 0.261091] acpiphp: Slot [27] registered [ 0.262070] acpiphp: Slot [28] registered [ 0.264105] acpiphp: Slot [29] registered [ 0.265202] acpiphp: Slot [30] registered [ 0.267255] acpiphp: Slot [31] registered [ 0.269079] PCI host bridge to bus 0000:00 [ 0.271026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.279025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.282027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288369] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291375] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.295169] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.302053] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.306061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.309020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.312019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316618] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.319778] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.322048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.325673] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329988] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.340662] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.344013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.350424] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.371031] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.379015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.396017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.408193] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.416015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.423018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.435019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.445832] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.448433] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.450332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.451286] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.453236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.459062] iommu: Default domain type: Passthrough [ 0.460332] SCSI subsystem initialized [ 0.462118] ACPI: bus type USB registered [ 0.463159] usbcore: registered new interface driver usbfs [ 0.465072] usbcore: registered new interface driver hub [ 0.466065] usbcore: registered new device driver usb [ 0.467149] pps_core: LinuxPPS API ver. 1 registered [ 0.469007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.471053] PTP clock support registered [ 0.472125] EDAC MC: Ver: 3.0.0 [ 0.474155] PCI: Using ACPI for IRQ routing [ 0.475826] NetLabel: Initializing [ 0.477008] NetLabel: domain hash size = 128 [ 0.478008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.479075] NetLabel: unlabeled traffic allowed by default [ 0.481262] vgaarb: loaded [ 0.482282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.483012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.493000] clocksource: Switched to clocksource kvm-clock [ 0.599294] VFS: Disk quotas dquot_6.6.0 [ 0.601679] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.605258] *** VALIDATE ramfs *** [ 0.606446] *** VALIDATE hugetlbfs *** [ 0.608440] pnp: PnP ACPI init [ 0.611278] pnp: PnP ACPI: found 6 devices [ 0.646365] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.649128] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.651215] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.653842] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.656795] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.659506] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.662800] NET: Registered protocol family 2 [ 0.665504] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.671511] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.675195] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.680721] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.684154] TCP: Hash tables configured (established 65536 bind 65536) [ 0.687138] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.690364] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.693386] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.696632] NET: Registered protocol family 1 [ 0.699753] RPC: Registered named UNIX socket transport module. [ 0.702094] RPC: Registered udp transport module. [ 0.703813] RPC: Registered tcp transport module. [ 0.705466] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.707979] NET: Registered protocol family 44 [ 0.709480] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.711669] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.713859] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.716083] PCI: CLS 0 bytes, default 64 [ 0.717554] Unpacking initramfs... [ 2.118477] debug: unmapping init [mem 0xffff8d72fcc64000-0xffff8d72fffcffff] [ 2.122575] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.124961] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.127956] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.655788] Initialise system trusted keyrings [ 2.657379] Key type blacklist registered [ 2.659295] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.670085] zbud: loaded [ 2.673344] *** VALIDATE nfs *** [ 2.674536] *** VALIDATE nfs4 *** [ 2.675806] pstore: using deflate compression [ 2.679152] Platform Keyring initialized [ 2.782787] NET: Registered protocol family 38 [ 2.784340] Key type asymmetric registered [ 2.785580] Asymmetric key parser 'x509' registered [ 2.786875] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.789263] io scheduler mq-deadline registered [ 2.790678] io scheduler kyber registered [ 2.792367] io scheduler bfq registered [ 2.793931] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.796398] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.798498] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.800954] ACPI: Power Button [PWRF] [ 2.806089] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.810694] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.820144] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.847947] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.874835] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.879948] Non-volatile memory driver v1.3 [ 2.881035] Linux agpgart interface v0.103 [ 2.905411] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.908083] vda: detected capacity change from 0 to 74702848 [ 2.923548] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.926996] vdb: detected capacity change from 0 to 1073741824 [ 2.934132] libphy: Fixed MDIO Bus: probed [ 2.956134] usbcore: registered new interface driver usbserial_generic [ 2.958960] usbserial: USB Serial support registered for generic [ 2.961763] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.966786] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.968694] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.971176] mousedev: PS/2 mouse device common for all mice [ 2.973830] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.975412] rtc_cmos 00:05: RTC can wake from S4 [ 2.982654] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.985186] rtc_cmos 00:05: registered as rtc0 [ 2.991709] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.993692] intel_pstate: CPU model not supported [ 2.997765] hid: raw HID events driver (C) Jiri Kosina [ 2.999788] usbcore: registered new interface driver usbhid [ 3.001403] usbhid: USB HID core driver [ 3.002822] drop_monitor: Initializing network drop monitor service [ 3.004454] Initializing XFRM netlink socket [ 3.004715] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.007498] NET: Registered protocol family 10 [ 3.011558] Segment Routing with IPv6 [ 3.015251] NET: Registered protocol family 17 [ 3.018725] mpls_gso: MPLS GSO support [ 3.024846] RAS: Correctable Errors collector initialized. [ 3.028218] AVX version of gcm_enc/dec engaged. [ 3.031475] AES CTR mode by8 optimization enabled [ 3.096130] sched_clock: Marking stable (3096103963, 0)->(4083164283, -987060320) [ 3.101551] registered taskstats version 1 [ 3.104831] Loading compiled-in X.509 certificates [ 3.107997] zswap: loaded using pool lzo/zbud [ 3.129588] Key type big_key registered [ 3.144732] Key type encrypted registered [ 3.147182] ima: No TPM chip found, activating TPM-bypass! [ 3.151057] ima: Allocated hash algorithm: sha1 [ 3.152658] ima: No architecture policies found [ 3.154714] evm: Initialising EVM extended attributes: [ 3.156955] evm: security.selinux [ 3.158343] evm: security.ima [ 3.159680] evm: security.capability [ 3.161239] evm: HMAC attrs: 0x1 [ 3.164107] rtc_cmos 00:05: setting system clock to 2026-08-15 06:38:18 UTC (1786775898) [ 3.171248] debug: unmapping init [mem 0xffffffffb6a03000-0xffffffffb6bfffff] [ 3.174286] debug: unmapping init [mem 0xffffffffb5782000-0xffffffffb5a58fff] [ 3.181189] Write protecting the kernel read-only data: 28672k [ 3.184837] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.187659] debug: unmapping init [mem 0xffffffffb4714000-0xffffffffb47fffff] [ 3.220263] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.229585] systemd[1]: Detected virtualization kvm. [ 3.232154] systemd[1]: Detected architecture x86-64. [ 3.234490] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.263726] systemd[1]: No hostname configured. [ 3.265684] systemd[1]: Set hostname to . [ 3.268189] random: systemd: uninitialized urandom read (16 bytes read) [ 3.270853] systemd[1]: Initializing machine ID from random generator. [ 3.430211] random: systemd: uninitialized urandom read (16 bytes read) [ 3.432993] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.438103] random: systemd: uninitialized urandom read (16 bytes read) [ 3.440482] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.446820] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] [ 3.728535] random: fast init done Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.151411] device-mapper: uevent: version 1.0.3 [ 4.155122] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.971444] virtio_net virtio0 ens2: renamed from eth0 [ 5.126835] scsi host0: ata_piix [ 5.177391] scsi host1: ata_piix [ 5.179554] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.182817] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.769348] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.666307] random: crng init done [ 9.667846] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.232233] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.397917] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.730377] SELinux: Disabled at runtime. [ 11.789695] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.798414] systemd[1]: Detected virtualization kvm. [ 11.800232] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.265147] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.267967] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.272157] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.276264] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.280156] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.289283] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.293757] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.429750] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.746221] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.083951] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.124885] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.184574] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.203810] EDAC sbridge: Ver: 1.1.2 [ 14.304991] Key type dns_resolver registered [ 14.604325] NFS: Registering the id_resolver key type [ 14.606286] Key type id_resolver registered [ 14.607983] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg114-client login: [ 57.973242] libcfs: loading out-of-tree module taints kernel. [ 58.223190] Key type ._llcrypt registered [ 58.226215] Key type .llcrypt registered [ 58.671803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 58.681480] alg: No test for adler32 (adler32-zlib) [ 60.023109] Lustre: Lustre: Build Version: 2.17.57_1_g05abe65 [ 60.594591] LNet: Added LNI 192.168.201.14@tcp [8/256/0/180] [ 62.320282] Key type lgssc registered [ 63.968223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.686210] hrtimer: interrupt took 8615806 ns [ 232.232421] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 236.816982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.610396] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing check_logdir /tmp/testlogs/ [ 256.220394] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing yml_node [ 258.018367] Lustre: lustre-OST0000-osc-ffff8d73433a1000: disconnect after 24s idle [ 261.115444] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 263.647562] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 266.107706] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 267.659356] Lustre: DEBUG MARKER: -----============= acceptance-small: recovery-small ============----- Sat Aug 15 02:42:41 EDT 2026 [ 284.436619] Lustre: DEBUG MARKER: excepting tests: 136 [ 285.839513] Lustre: DEBUG MARKER: === recovery-small: start setup 02:43:00 (1786776180) === [ 290.642451] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing check_config_client /mnt/lustre [ 308.632765] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 321.574259] Lustre: DEBUG MARKER: === recovery-small: finish setup 02:43:35 (1786776215) === [ 323.323570] Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 02:43:37 (1786776217) [ 339.936167] Lustre: 10875:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776219/real 1786776219] req@ffff8d7343fce680 x1873570383804800/t0(0) o700->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:30/10 lens 264/248 e 0 to 1 dl 1786776235 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 339.975140] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 340.001771] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 357.856541] Lustre: 10895:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776237/real 1786776237] req@ffff8d73603e1c00 x1873570383807104/t0(0) o36->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:12/10 lens 520/576 e 0 to 1 dl 1786776253 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 357.879663] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 357.925772] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 376.288423] Lustre: 10921:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776255/real 1786776255] req@ffff8d73603e0000 x1873570383808896/t0(0) o101->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786776271 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:0 [ 376.325104] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 376.368563] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 395.232220] Lustre: 10942:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776274/real 1786776274] req@ffff8d7343fe7b80 x1873570383811712/t0(0) o36->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:12/10 lens 488/512 e 0 to 1 dl 1786776290 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:4294967295 [ 395.243338] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 395.300885] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 414.688173] Lustre: 10967:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776293/real 1786776293] req@ffff8d7343fce680 x1873570383813504/t0(0) o34->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:12/10 lens 472/728 e 0 to 1 dl 1786776309 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 414.735274] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 414.797326] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 433.633226] Lustre: 10988:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776312/real 1786776312] req@ffff8d734cfa6300 x1873570383815296/t0(0) o34->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:12/10 lens 472/728 e 0 to 1 dl 1786776328 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 433.679486] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 433.728412] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 441.545093] Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep ======= 02:45:35 (1786776335) [ 459.232326] Lustre: 11602:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776338/real 1786776338] req@ffff8d73603e0a80 x1873570383818240/t0(0) o101->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786776354 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 [ 459.254820] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 459.303065] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 485.893338] Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ===== 02:46:19 (1786776379) [ 502.752277] Lustre: 12229:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776382/real 1786776382] req@ffff8d7343fcea00 x1873570383825664/t0(0) o101->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786776398 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'mv.0' uid:0 gid:0 projid:0 [ 502.781972] Lustre: 12229:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 502.795950] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 502.809166] Lustre: Skipped 1 previous similar message [ 502.853526] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 502.861539] Lustre: Skipped 1 previous similar message [ 527.892898] Lustre: DEBUG MARKER: == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 02:47:02 (1786776422) [ 550.368811] Lustre: lustre-OST0001-osc-ffff8d73433a1000: disconnect after 22s idle [ 550.386530] Lustre: Skipped 1 previous similar message [ 585.186667] Lustre: 12934:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776464/real 1786776464] req@ffff8d7343fcd500 x1873570383841408/t0(0) o36->lustre-MDT0000-mdc-ffff8d73433a1000@192.168.201.114@tcp:12/10 lens 504/624 e 0 to 1 dl 1786776480 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'unlink.0' uid:0 gid:0 projid:4294967295 [ 585.207528] Lustre: 12934:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 585.215531] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 585.231777] Lustre: Skipped 3 previous similar messages [ 585.274304] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 585.280550] Lustre: Skipped 3 previous similar messages [ 592.056562] Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ===== 02:48:06 (1786776486) [ 614.984111] Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) === 02:48:29 (1786776509) [ 629.681507] Lustre: DEBUG MARKER: == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 02:48:43 (1786776523) [ 630.004157] Lustre: *** cfs_fail_loc=305, val=0*** [ 645.613774] Lustre: *** cfs_fail_loc=305, val=0*** [ 649.715941] Lustre: lustre-OST0001-osc-ffff8d73433a1000: disconnect after 21s idle [ 661.950135] Lustre: *** cfs_fail_loc=305, val=0*** [ 677.350366] Lustre: *** cfs_fail_loc=305, val=0*** [ 693.706035] Lustre: *** cfs_fail_loc=305, val=0*** [ 710.092896] Lustre: *** cfs_fail_loc=305, val=0*** [ 725.427516] Lustre: *** cfs_fail_loc=305, val=0*** [ 742.305427] LustreError: lustre-MDT0000-mdc-ffff8d73433a1000: operation ldlm_enqueue to node 192.168.201.114@tcp failed: rc = -107 [ 742.309726] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 742.319386] Lustre: Skipped 1 previous similar message [ 742.334500] LustreError: lustre-MDT0000-mdc-ffff8d73433a1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 742.382170] Lustre: lustre-MDT0000-mdc-ffff8d73433a1000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 742.405144] Lustre: Skipped 1 previous similar message [ 751.986767] Lustre: DEBUG MARKER: == recovery-small test 10b: re-send BL AST =============== 02:50:46 (1786776646) [ 777.092913] Lustre: DEBUG MARKER: == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 02:51:11 (1786776671) [ 777.778790] Lustre: *** cfs_fail_loc=305, val=0*** [ 777.781547] Lustre: Skipped 1 previous similar message [ 785.282553] Lustre: DEBUG MARKER: == recovery-small test 10d: test failed blocking ast ===== 02:51:19 (1786776679) [ 788.378438] Lustre: Unmounted lustre-client [ 788.842754] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 790.381046] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: operation ldlm_enqueue to node 192.168.201.114@tcp failed: rc = -107 [ 790.399317] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 790.420803] Lustre: 2404:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.114@tcp:/lustre/fid: [0x200000404:0x1:0x0]/ may get corrupted (rc -108) [ 793.179748] Lustre: Unmounted lustre-client [ 799.631195] Lustre: DEBUG MARKER: == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 02:51:33 (1786776693) [ 801.369895] Lustre: DEBUG MARKER: SKIP: recovery-small test_10e need two clients [ 803.671355] Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 02:51:37 (1786776697) [ 829.476397] Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 02:52:03 (1786776723) [ 829.562820] Lustre: DEBUG MARKER: multiop /mnt/lustre/f12.recovery-small OS_c [ 845.792227] Lustre: 18589:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776725/real 1786776725] req@ffff8d7343fe4380 x1873570383916928/t0(0) o35->lustre-MDT0000-mdc-ffff8d734c6b9000@192.168.201.114@tcp:23/10 lens 392/624 e 0 to 1 dl 1786776741 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 845.835622] Lustre: 18589:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 873.289647] Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 02:52:47 (1786776767) [ 900.284854] Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 02:53:14 (1786776794) [ 907.891275] Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ========= 02:53:22 (1786776802) [ 916.073095] Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 02:53:30 (1786776810) [ 962.571539] Lustre: DEBUG MARKER: == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 02:54:16 (1786776856) [ 1015.643221] Lustre: DEBUG MARKER: == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 02:55:09 (1786776909) [ 1017.014077] Lustre: DEBUG MARKER: SKIP: recovery-small test_17b Needs multiple clients [ 1018.620632] Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 02:55:12 (1786776912) [ 1019.489293] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1019.558700] Lustre: lustre-OST0001-osc-ffff8d734c6b9000: Connection to lustre-OST0001 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1019.569180] Lustre: Skipped 5 previous similar messages [ 1019.586743] LustreError: lustre-OST0001-osc-ffff8d734c6b9000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1019.603103] Lustre: lustre-OST0001-osc-ffff8d734c6b9000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 1019.609695] Lustre: Skipped 4 previous similar messages [ 1025.589928] Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 02:55:19 (1786776919) [ 1031.693168] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1056.386325] Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 02:55:50 (1786776950) [ 1060.701113] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: operation ost_statfs to node 192.168.201.114@tcp failed: rc = -107 [ 1071.101606] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.114@tcp) after server handle changed from 0x244602f7286a64a2 to 0x244602f7286a6607 [ 1071.108772] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1080.244225] Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 02:56:14 (1786776974) [ 1080.893199] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1080.895592] Lustre: Skipped 1 previous similar message [ 1114.080283] Lustre: 2405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786776993/real 1786776993] req@ffff8d7343fe6a00 x1873570383982208/t0(0) o103->lustre-MDT0000-mdc-ffff8d734c6b9000@192.168.201.114@tcp:17/18 lens 328/224 e 0 to 1 dl 1786777009 ref 1 fl Rpc:XQr/202/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 [ 1114.107986] Lustre: 2405:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1183.745362] LustreError: lustre-MDT0000-mdc-ffff8d734c6b9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1186.399067] Lustre: Unmounted lustre-client [ 1192.736351] Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 02:58:06 (1786777086) [ 1193.437804] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1300.998326] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1302.405243] Lustre: Unmounted lustre-client [ 1309.624770] Lustre: DEBUG MARKER: == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 03:00:03 (1786777203) [ 1310.252596] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1314.737219] Lustre: Unmounted lustre-client [ 1325.168842] Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 03:00:19 (1786777219) [ 1326.622391] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: operation ldlm_enqueue to node 192.168.201.114@tcp failed: rc = -12 [ 1333.628775] Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 03:00:27 (1786777227) [ 1335.167476] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: operation ldlm_enqueue to node 192.168.201.114@tcp failed: rc = -12 [ 1343.997982] Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 03:00:37 (1786777237) [ 1373.702610] Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 03:01:07 (1786777267) [ 1519.545549] Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 03:03:33 (1786777413) [ 1542.624224] Lustre: lustre-MDT0000-mdc-ffff8d734c6b9000: Connection to lustre-MDT0000 (at 192.168.201.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1542.638395] Lustre: Skipped 18 previous similar messages [ 1542.687083] Lustre: lustre-MDT0000-mdc-ffff8d734c6b9000: Connection restored to 192.168.201.114@tcp (at 192.168.201.114@tcp) [ 1542.695695] Lustre: Skipped 16 previous similar messages [ 1550.967577] Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 03:04:05 (1786777445) [ 1563.493478] Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 03:04:17 (1786777457) [ 1698.784660] Lustre: 30760:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786777459/real 1786777459] req@ffff8d7343fcea00 x1873570384112000/t0(0) o36->lustre-MDT0000-mdc-ffff8d734c6b9000@192.168.201.114@tcp:12/10 lens 488/512 e 0 to 1 dl 1786777594 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 1698.812976] Lustre: 30760:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 1701.062777] Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 03:06:34 (1786777594) [ 1729.636268] Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 03:07:03 (1786777623) [ 1755.277956] Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 03:07:29 (1786777649) [ 1781.779329] Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod ========================================================== 03:07:55 (1786777675) [ 1806.012225] Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ========================================================== 03:08:20 (1786777700) [ 1834.977967] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 192.168.201.114@tcp) was lost; in progress operations using this service will fail [ 1835.038949] Lustre: Evicted from MGS (at 192.168.201.114@tcp) after server handle changed from 0x244602f7286a5b10 to 0x244602f7286a82a8 [ 1847.986265] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1849.496980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1857.107298] Lustre: DEBUG MARKER: == recovery-small test 24a: fsync error (should return error) ========================================================== 03:09:11 (1786777751) [ 1859.020057] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: operation ost_write to node 192.168.201.114@tcp failed: rc = -107 [ 1859.036900] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1859.047080] Lustre: 2407:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.114@tcp:/lustre/fid: [0x240000402:0x8:0x0]// may get corrupted (rc -5) [ 1859.062510] LustreError: 35298:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8d734c6b9000: namespace resource [0x280000400:0x4:0x0].0x0 (ffff8d7353613200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1865.546478] Lustre: DEBUG MARKER: == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 03:09:19 (1786777759) [ 1867.450875] LustreError: lustre-OST0000-osc-ffff8d734c6b9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1867.477315] Lustre: 2407:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.114@tcp:/lustre/fid: [0x200000405:0x2f:0x0]// may get corrupted (rc -108) [ 1873.051636] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 03:09:27 (1786777767) [ 1875.057268] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node [ 1876.937681] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 03:09:30 (1786777770) [ 1878.456289] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node [ 1880.364080] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 03:09:34 (1786777774) [ 1884.507339] LustreError: lustre-MDT0000-mdc-ffff8d734c6b9000: operation mds_close to node 192.168.201.114@tcp failed: rc = -19 [ 1884.525192] LustreError: Skipped 1 previous similar message [ 1902.560579] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 192.168.201.114@tcp) was lost; in progress operations using this service will fail [ 1911.785601] Lustre: Evicted from MGS (at 192.168.201.114@tcp) after server handle changed from 0x244602f7286a82a8 to 0x244602f7286a9b60 [ 1913.668711] BUG: spinlock recursion on CPU#2, ptlrpcd_rcv/2403 [ 1913.676172] lock: 0xffff8d73433a52e0, .magic: dead4ead, .owner: ptlrpcd_rcv/2403, .owner_cpu: 2 [ 1913.679639] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1913.699821] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1913.706455] Call Trace: [ 1913.715734] ? dump_stack+0xbb/0x10e [ 1913.717967] ? spin_dump.cold.3+0x32/0x65 [ 1913.722797] ? do_raw_spin_lock+0x10f/0x190 [ 1913.731515] ? _raw_spin_lock+0x21/0x30 [ 1913.733492] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1913.737968] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 1913.739335] ? mdc_commit_open+0x91/0xb0 [mdc] [ 1913.743823] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1913.750605] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1913.755844] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 1913.761878] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1913.766217] ? __next_timer_interrupt+0x160/0x160 [ 1913.768730] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1913.771588] ? do_wait_intr+0xf0/0xf0 [ 1913.773400] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1913.776830] ? kthread+0x1d1/0x200 [ 1913.779672] ? set_kthread_struct+0x70/0x70 [ 1913.781764] ? ret_from_fork+0x1f/0x30 [ 1940.054182] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 1940.057024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 1940.059716] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 1940.059729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1940.059809] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1940.059814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1940.059817] RIP: 0010:native_safe_halt+0x2e/0x50 [ 1940.059830] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 1940.059834] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 1940.059839] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1940.059841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 1940.059843] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1940.059845] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 1940.059848] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 1940.059851] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 1940.059854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1940.059856] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 1940.059865] Call Trace: [ 1940.059871] [ 1940.059877] ? show_regs.cold.9+0x22/0x2f [ 1940.059890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1940.059896] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1940.059902] ? watchdog+0x40/0x40 [ 1940.059906] ? hrtimer_interrupt+0x128/0x2c0 [ 1940.059910] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1940.059917] ? apic_timer_interrupt+0xf/0x20 [ 1940.059921] [ 1940.059924] ? native_safe_halt+0x2e/0x50 [ 1940.059928] kvm_wait+0xcd/0xe0 [ 1940.059936] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1940.059946] do_raw_spin_lock+0x177/0x190 [ 1940.059951] _raw_spin_lock+0x21/0x30 [ 1940.059955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1940.060401] ? do_raw_spin_unlock+0x75/0x190 [ 1940.060407] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1940.061363] ? do_wait_intr+0xf0/0xf0 [ 1940.061376] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1940.061723] mdc_close+0x53b/0xd70 [mdc] [ 1940.061952] lmv_close+0x18d/0x490 [lmv] [ 1940.062371] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 1940.062378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1940.062426] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1940.062430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1940.062432] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1940.062441] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 1940.062444] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1940.062447] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1940.062449] RDX: 0000000000007761 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 1940.062450] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 1940.062452] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 1940.062453] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 1940.062455] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 1940.062457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1940.062459] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 1940.063209] Call Trace: [ 1940.063217] [ 1940.063220] ? show_regs.cold.9+0x22/0x2f [ 1940.063230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1940.063236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1940.063240] ? watchdog+0x40/0x40 [ 1940.063244] ? hrtimer_interrupt+0x128/0x2c0 [ 1940.063247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1940.063257] ? apic_timer_interrupt+0xf/0x20 [ 1940.063261] [ 1940.063264] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1940.063269] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 1940.063273] do_raw_spin_lock+0x177/0x190 [ 1940.063277] _raw_spin_lock+0x21/0x30 [ 1940.063283] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1940.061992] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 1940.064333] ll_md_real_close+0xa5/0x2a0 [lustre] [ 1940.064624] Modules linked in: lustre(O) [ 1940.064597] obd_mod_free+0x46/0x140 [ptlrpc] [ 1940.065154] mdc_commit_open+0x91/0xb0 [mdc] [ 1940.065190] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1940.064413] ll_md_close+0x274/0x800 [lustre] [ 1940.065521] ll_file_release+0x128/0xc50 [lustre] [ 1940.065609] __fput+0x10d/0x440 [ 1940.065619] ____fput+0x12/0x20 [ 1940.065624] task_work_run+0xc6/0x110 [ 1940.065632] exit_to_usermode_loop+0x1dd/0x1f0 [ 1940.065640] do_syscall_64+0x3d6/0x440 [ 1940.065646] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1940.065654] RIP: 0033:0x7fa749552cd8 [ 1940.065662] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 1940.065666] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 1940.065671] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 1940.065674] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 1940.065676] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 1940.065678] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 1940.065680] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 1940.066033] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1940.066210] after_reply+0xe2c/0x1780 [ptlrpc] [ 1940.067467] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1940.067670] ? __next_timer_interrupt+0x160/0x160 [ 1940.067679] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1940.067834] ? do_wait_intr+0xf0/0xf0 [ 1940.067841] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1940.068033] kthread+0x1d1/0x200 [ 1940.068038] ? set_kthread_struct+0x70/0x70 [ 1940.068042] ret_from_fork+0x1f/0x30 [ 1940.074832] Modules linked in: [ 1940.076295] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1940.076385] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1940.076393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1940.076397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1940.076416] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1940.104144] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 1940.109482] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1940.119406] auth_rpcgss [ 1940.122347] RAX: 0000000000007c8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1940.122362] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 1940.122366] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 1940.135088] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1940.135151] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1940.135159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1940.135168] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 1940.135489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1940.135502] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1940.135507] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1940.135512] RAX: 000000000000737e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1940.135514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 1940.135516] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1940.135518] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 1940.135521] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 1940.135524] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 1940.135527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1940.135530] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 1940.135539] Call Trace: [ 1940.135545] [ 1940.135553] ? show_regs.cold.9+0x22/0x2f [ 1940.135565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1940.135573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1940.135578] ? watchdog+0x40/0x40 [ 1940.135582] ? hrtimer_interrupt+0x128/0x2c0 [ 1940.135586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1940.135597] ? apic_timer_interrupt+0xf/0x20 [ 1940.135602] [ 1940.135605] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1940.135609] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1940.135614] do_raw_spin_lock+0x177/0x190 [ 1940.135619] _raw_spin_lock+0x21/0x30 [ 1940.135623] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 1940.135833] process_one_work+0x2c8/0x700 [ 1940.135840] worker_thread+0x296/0x6e0 [ 1940.135845] ? rescuer_thread+0x570/0x570 [ 1940.135849] kthread+0x1d1/0x200 [ 1940.135853] ? set_kthread_struct+0x70/0x70 [ 1940.135857] ret_from_fork+0x1f/0x30 [ 1940.733925] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 1940.736826] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 1940.741903] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 1940.750904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1940.754201] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 1940.756694] Call Trace: [ 1940.760792] [ 1940.762078] ? show_regs.cold.9+0x22/0x2f [ 1940.764408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1940.768913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1940.773130] ? watchdog+0x40/0x40 [ 1940.775940] ? hrtimer_interrupt+0x128/0x2c0 [ 1940.780429] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1940.785229] ? apic_timer_interrupt+0xf/0x20 [ 1940.787662] [ 1940.788822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1940.794412] do_raw_spin_lock+0x177/0x190 [ 1940.798227] _raw_spin_lock+0x21/0x30 [ 1940.801532] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1940.803601] ? do_raw_spin_unlock+0x75/0x190 [ 1940.814626] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1940.819424] ? do_wait_intr+0xf0/0xf0 [ 1940.827163] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1940.832203] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 1940.840601] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 1940.848437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1940.862421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 1940.864390] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 1940.874348] mdc_intent_lock+0x353/0x7e0 [mdc] [ 1940.880476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1940.885589] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 1940.896975] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 1940.902743] lmv_intent_open+0x338/0x1150 [lmv] [ 1940.904087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1940.910418] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1940.912458] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 1940.915015] ll_intent_lock+0xc5/0xaa0 [lustre] [ 1940.917841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1940.920888] ll_lookup_it+0xe0e/0x2270 [lustre] [ 1940.923063] ? do_raw_spin_unlock+0x75/0x190 [ 1940.925547] ? _raw_spin_unlock+0x12/0x30 [ 1940.927167] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 1940.929631] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 1940.932054] ll_atomic_open+0x36a/0xd70 [lustre] [ 1940.933981] lookup_open+0x356/0xb40 [ 1940.935641] do_last+0x313/0xfc0 [ 1940.937174] ? nd_jump_root+0xe5/0x160 [ 1940.938600] ? path_init+0x437/0x520 [ 1940.940064] path_openat+0xf7/0x500 [ 1940.941364] ? do_raw_spin_unlock+0x75/0x190 [ 1940.943010] do_filp_open+0x99/0x140 [ 1940.944409] ? getname_flags+0x6e/0x330 [ 1940.945932] ? __check_object_size+0xff/0x256 [ 1940.947588] ? do_raw_spin_unlock+0x75/0x190 [ 1940.949285] ? _raw_spin_unlock+0x12/0x30 [ 1940.950846] do_sys_openat2+0x2b4/0x410 [ 1940.952613] do_sys_open+0x73/0xa0 [ 1940.954098] __x64_sys_openat+0x24/0x30 [ 1940.955569] do_syscall_64+0xc1/0x440 [ 1940.956975] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1940.959388] RIP: 0033:0x7fa749552332 [ 1940.960811] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 1940.967865] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 1940.970808] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 1940.973495] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 1940.976230] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 1940.978953] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 1940.981573] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 1968.057020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 1968.059521] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 1968.060417] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 1968.060425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1968.060511] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1968.060517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1968.060519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1968.060534] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 1968.060539] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1968.060543] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1968.060545] RDX: 0000000000003981 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 1968.060548] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 1968.060550] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 1968.060552] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 1968.060555] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 1968.060557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1968.060559] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 1968.060568] Call Trace: [ 1968.060575] [ 1968.060580] ? show_regs.cold.9+0x22/0x2f [ 1968.060591] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1968.060598] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1968.060601] ? watchdog+0x40/0x40 [ 1968.060606] ? hrtimer_interrupt+0x128/0x2c0 [ 1968.060610] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1968.060619] ? apic_timer_interrupt+0xf/0x20 [ 1968.060624] [ 1968.060627] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1968.060632] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 1968.060637] do_raw_spin_lock+0x177/0x190 [ 1968.060643] _raw_spin_lock+0x21/0x30 [ 1968.060648] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1968.060934] obd_mod_free+0x46/0x140 [ptlrpc] [ 1968.063647] mdc_commit_open+0x91/0xb0 [mdc] [ 1968.063686] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1968.063887] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1968.064766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1968.064838] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1968.064843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1968.064850] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 1968.066583] after_reply+0xe2c/0x1780 [ptlrpc] [ 1968.066782] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1968.067459] [ 1968.067462] RIP: 0010:native_safe_halt+0x2e/0x50 [ 1968.067473] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 1968.067477] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1968.067492] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1968.067495] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 1968.067497] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1968.067499] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 1968.067502] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 1968.067505] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 1968.067507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1968.067509] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 1968.067517] Call Trace: [ 1968.067521] [ 1968.067524] ? show_regs.cold.9+0x22/0x2f [ 1968.067533] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1968.067541] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1968.067545] ? watchdog+0x40/0x40 [ 1968.067549] ? hrtimer_interrupt+0x128/0x2c0 [ 1968.067553] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1968.067559] ? apic_timer_interrupt+0xf/0x20 [ 1968.067563] [ 1968.067566] ? native_safe_halt+0x2e/0x50 [ 1968.067570] kvm_wait+0xcd/0xe0 [ 1968.067578] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1968.067586] do_raw_spin_lock+0x177/0x190 [ 1968.067590] _raw_spin_lock+0x21/0x30 [ 1968.067594] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 1968.067801] process_one_work+0x2c8/0x700 [ 1968.067808] worker_thread+0x296/0x6e0 [ 1968.067812] ? rescuer_thread+0x570/0x570 [ 1968.067815] kthread+0x1d1/0x200 [ 1968.067818] ? set_kthread_struct+0x70/0x70 [ 1968.067822] ret_from_fork+0x1f/0x30 [ 1968.066978] ? __next_timer_interrupt+0x160/0x160 [ 1968.069445] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1968.069710] ? do_wait_intr+0xf0/0xf0 [ 1968.069720] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1968.069919] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 1968.069930] Modules linked in: [ 1968.069929] kthread+0x1d1/0x200 [ 1968.069933] lustre(O) [ 1968.069940] ? set_kthread_struct+0x70/0x70 [ 1968.069941] osp(O) [ 1968.069947] ret_from_fork+0x1f/0x30 [ 1968.069955] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1968.070033] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1968.070038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1968.070041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1968.070051] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1968.070055] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 1968.070059] RAX: 0000000000006a87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1968.070062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 1968.070064] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 1968.070066] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 1968.070068] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 1968.070072] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 1968.070075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1968.070077] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 1968.070086] Call Trace: [ 1968.070090] [ 1968.070092] ? show_regs.cold.9+0x22/0x2f [ 1968.070101] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1968.070107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1968.070111] ? watchdog+0x40/0x40 [ 1968.070116] ? hrtimer_interrupt+0x128/0x2c0 [ 1968.070120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1968.070126] ? apic_timer_interrupt+0xf/0x20 [ 1968.070132] [ 1968.070134] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1968.070141] do_raw_spin_lock+0x177/0x190 [ 1968.070145] _raw_spin_lock+0x21/0x30 [ 1968.070150] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1968.070384] ? do_raw_spin_unlock+0x75/0x190 [ 1968.070390] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1968.070601] ? do_wait_intr+0xf0/0xf0 [ 1968.070608] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1968.070807] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 1968.071016] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 1968.071226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1968.071335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 1968.071374] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 1968.071405] mdc_intent_lock+0x353/0x7e0 [mdc] [ 1968.071431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1968.071528] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 1968.071743] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 1968.071772] lmv_intent_open+0x338/0x1150 [lmv] [ 1968.071803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1968.071887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1968.071969] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 1968.071990] ll_intent_lock+0xc5/0xaa0 [lustre] [ 1968.072073] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1968.072152] ll_lookup_it+0xe0e/0x2270 [lustre] [ 1968.072232] ? do_raw_spin_unlock+0x75/0x190 [ 1968.072237] ? _raw_spin_unlock+0x12/0x30 [ 1968.072243] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 1968.072267] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 1968.072286] ll_atomic_open+0x36a/0xd70 [lustre] [ 1968.072368] lookup_open+0x356/0xb40 [ 1968.072377] do_last+0x313/0xfc0 [ 1968.072382] ? nd_jump_root+0xe5/0x160 [ 1968.072386] ? path_init+0x437/0x520 [ 1968.072391] path_openat+0xf7/0x500 [ 1968.072395] ? do_raw_spin_unlock+0x75/0x190 [ 1968.072400] do_filp_open+0x99/0x140 [ 1968.072407] ? getname_flags+0x6e/0x330 [ 1968.072412] ? __check_object_size+0xff/0x256 [ 1968.072419] ? do_raw_spin_unlock+0x75/0x190 [ 1968.072423] ? _raw_spin_unlock+0x12/0x30 [ 1968.072428] do_sys_openat2+0x2b4/0x410 [ 1968.072432] do_sys_open+0x73/0xa0 [ 1968.072436] __x64_sys_openat+0x24/0x30 [ 1968.072440] do_syscall_64+0xc1/0x440 [ 1968.072446] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1968.072452] RIP: 0033:0x7fa749552332 [ 1968.072459] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 1968.072463] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 1968.072468] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 1968.072471] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 1968.072473] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 1968.072475] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 1968.072477] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 1968.787852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1968.812739] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1968.818171] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1968.822536] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1968.825371] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1968.834094] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1968.837881] RAX: 0000000000006a04 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1968.842330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 1968.847164] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1968.850366] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 1968.854605] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 1968.858290] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 1968.862635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1968.865729] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 1968.871406] Call Trace: [ 1968.872530] [ 1968.873153] ? show_regs.cold.9+0x22/0x2f [ 1968.875387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1968.877458] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1968.879139] ? watchdog+0x40/0x40 [ 1968.880290] ? hrtimer_interrupt+0x128/0x2c0 [ 1968.882144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1968.884985] ? apic_timer_interrupt+0xf/0x20 [ 1968.887351] [ 1968.888381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1968.892458] do_raw_spin_lock+0x177/0x190 [ 1968.894855] _raw_spin_lock+0x21/0x30 [ 1968.896592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1968.900633] ? do_raw_spin_unlock+0x75/0x190 [ 1968.905429] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1968.913531] ? do_wait_intr+0xf0/0xf0 [ 1968.916594] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1968.921631] mdc_close+0x53b/0xd70 [mdc] [ 1968.925622] lmv_close+0x18d/0x490 [lmv] [ 1968.926978] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 1968.930644] ll_md_real_close+0xa5/0x2a0 [lustre] [ 1968.933669] ll_md_close+0x274/0x800 [lustre] [ 1968.935819] ll_file_release+0x128/0xc50 [lustre] [ 1968.938411] __fput+0x10d/0x440 [ 1968.939869] ____fput+0x12/0x20 [ 1968.941431] task_work_run+0xc6/0x110 [ 1968.943326] exit_to_usermode_loop+0x1dd/0x1f0 [ 1968.945687] do_syscall_64+0x3d6/0x440 [ 1968.947439] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1968.949729] RIP: 0033:0x7fa749552cd8 [ 1968.951619] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 1968.963665] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 1968.973155] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 1968.978794] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 1968.990490] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 1968.999722] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 1969.005371] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 1973.220567] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 56s! [ 1973.232647] Showing busy workqueues and worker pools: [ 1973.239754] workqueue events: flags=0x0 [ 1973.247284] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.254324] pending: cpu_latency_work [ptlrpc] [ 1973.263348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 1973.269371] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 1973.273959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.276990] pending: kfree_rcu_monitor [ 1973.279482] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 1973.283027] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 1973.286859] workqueue events_long: flags=0x0 [ 1973.290374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.294195] pending: lu_objects_flush [obdclass] [ 1973.297305] workqueue events_unbound: flags=0x2 [ 1973.299715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 1973.302992] pending: flush_memcg_stats_dwork [ 1973.305829] workqueue events_power_efficient: flags=0x80 [ 1973.309092] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.317214] pending: neigh_periodic_work [ 1973.318803] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.329892] pending: neigh_periodic_work [ 1973.333209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.337085] pending: do_cache_clean [ 1973.339490] workqueue mm_percpu_wq: flags=0x8 [ 1973.342408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.347458] pending: vmstat_update [ 1973.348498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.351864] pending: vmstat_update [ 1973.355977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1973.360712] pending: vmstat_update [ 1973.364286] workqueue writeback: flags=0x4a [ 1973.366924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 1973.371746] pending: wb_workfn, wb_workfn [ 1973.378255] workqueue xprtiod: flags=0xa [ 1973.380264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 1973.387905] pending: xs_stream_data_receive_workfn [ 1973.393243] workqueue knbd0-recv: flags=0x1a [ 1973.394306] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 1973.399451] in-flight: 651:recv_work [ 1973.401536] workqueue ptlrpc_pinger: flags=0x2 [ 1973.410933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 1973.422954] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 1973.434468] pool 8: cpus=0-3 flags=0x4 nice=0 hung=56s workers=4 idle: 37 31846 28897 [ 1973.445063] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=1963s workers=2 idle: 653 [ 1973.689024] rcu: INFO: rcu_sched self-detected stall on CPU [ 1973.693208] rcu: 2-...!: (35160 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 1973.696906] (t=60000 jiffies g=258701 q=1419) [ 1973.698605] rcu: rcu_sched kthread starved for 56795 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 1973.705594] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 1973.709519] rcu: RCU grace-period kthread stack dump: [ 1973.711679] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 1973.715277] Call Trace: [ 1973.715951] __schedule+0x351/0xcb0 [ 1973.717357] schedule+0xc0/0x180 [ 1973.718601] schedule_timeout+0xb4/0x190 [ 1973.720214] ? do_raw_spin_unlock+0x75/0x190 [ 1973.721840] ? __next_timer_interrupt+0x160/0x160 [ 1973.723762] ? prepare_to_swait_event+0xf3/0x140 [ 1973.725520] rcu_gp_kthread+0x8dd/0x15b0 [ 1973.726893] ? force_qs_rnp+0x2c0/0x2c0 [ 1973.728256] kthread+0x1d1/0x200 [ 1973.729344] ? set_kthread_struct+0x70/0x70 [ 1973.730983] ret_from_fork+0x1f/0x30 [ 1973.732203] rcu: Stack dump where RCU GP kthread last ran: [ 1973.733825] Sending NMI from CPU 2 to CPUs 1: [ 1973.735173] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 1973.735426] NMI backtrace for cpu 2 [ 1973.738808] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1973.742717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1973.745431] Call Trace: [ 1973.746372] [ 1973.746993] ? dump_stack+0xbb/0x10e [ 1973.748369] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 1973.750187] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 1973.752460] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 1973.755936] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 1973.759391] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 1973.761565] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 1973.763953] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 1973.768190] ? update_process_times+0x6d/0xc0 [ 1973.770443] ? tick_sched_handle.isra.18+0x27/0x90 [ 1973.772892] ? tick_sched_timer+0x53/0xb0 [ 1973.775943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1973.778375] ? tick_sched_handle.isra.18+0x90/0x90 [ 1973.781524] ? hrtimer_interrupt+0x128/0x2c0 [ 1973.783595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1973.785657] ? apic_timer_interrupt+0xf/0x20 [ 1973.787708] [ 1973.788640] ? native_safe_halt+0x2e/0x50 [ 1973.790443] ? kvm_wait+0xcd/0xe0 [ 1973.791751] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 1973.793906] ? do_raw_spin_lock+0x177/0x190 [ 1973.795816] ? _raw_spin_lock+0x21/0x30 [ 1973.797499] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1973.800400] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 1973.802878] ? mdc_commit_open+0x91/0xb0 [mdc] [ 1973.810866] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1973.814743] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1973.818863] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 1973.821601] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1973.825808] ? __next_timer_interrupt+0x160/0x160 [ 1973.828206] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1973.830614] ? do_wait_intr+0xf0/0xf0 [ 1973.832544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1973.835460] ? kthread+0x1d1/0x200 [ 1973.836970] ? set_kthread_struct+0x70/0x70 [ 1973.839206] ? ret_from_fork+0x1f/0x30 [ 1996.059207] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 1996.062932] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1996.080333] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1996.083597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1996.087792] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1996.090005] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1996.097501] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1996.100456] RAX: 0000000000001049 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 1996.103225] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 1996.106509] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1996.109530] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 1996.112120] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 1996.115163] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 1996.118136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1996.120357] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 1996.122689] Call Trace: [ 1996.123850] [ 1996.124546] ? show_regs.cold.9+0x22/0x2f [ 1996.126185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1996.127962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1996.129786] ? watchdog+0x40/0x40 [ 1996.131003] ? hrtimer_interrupt+0x128/0x2c0 [ 1996.132556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1996.134892] ? apic_timer_interrupt+0xf/0x20 [ 1996.136563] [ 1996.137417] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1996.139577] do_raw_spin_lock+0x177/0x190 [ 1996.141247] _raw_spin_lock+0x21/0x30 [ 1996.142516] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1996.144956] ? do_raw_spin_unlock+0x75/0x190 [ 1996.146569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1996.148873] ? do_wait_intr+0xf0/0xf0 [ 1996.150174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1996.152147] mdc_close+0x53b/0xd70 [mdc] [ 1996.153762] lmv_close+0x18d/0x490 [lmv] [ 1996.155310] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 1996.157560] ll_md_real_close+0xa5/0x2a0 [lustre] [ 1996.160007] ll_md_close+0x274/0x800 [lustre] [ 1996.161879] ll_file_release+0x128/0xc50 [lustre] [ 1996.164166] __fput+0x10d/0x440 [ 1996.165389] ____fput+0x12/0x20 [ 1996.166559] task_work_run+0xc6/0x110 [ 1996.168068] exit_to_usermode_loop+0x1dd/0x1f0 [ 1996.169859] do_syscall_64+0x3d6/0x440 [ 1996.171295] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1996.173582] RIP: 0033:0x7fa749552cd8 [ 1996.175070] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 1996.182281] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 1996.185440] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 1996.188257] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 1996.191214] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 1996.194297] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 1996.197159] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2000.054024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2000.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2000.057046] Modules linked in: [ 2000.058020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2000.058027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2000.058106] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2000.058113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2000.058116] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2000.058131] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2000.058136] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2000.058141] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2000.058143] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2000.058145] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2000.058147] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2000.058150] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2000.058153] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2000.058156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2000.058158] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2000.058168] Call Trace: [ 2000.058174] [ 2000.058179] ? show_regs.cold.9+0x22/0x2f [ 2000.058192] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2000.058198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2000.058203] ? watchdog+0x40/0x40 [ 2000.058207] ? hrtimer_interrupt+0x128/0x2c0 [ 2000.058211] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2000.058218] ? apic_timer_interrupt+0xf/0x20 [ 2000.058224] [ 2000.058227] ? native_safe_halt+0x2e/0x50 [ 2000.058232] kvm_wait+0xcd/0xe0 [ 2000.058240] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2000.058247] do_raw_spin_lock+0x177/0x190 [ 2000.058252] _raw_spin_lock+0x21/0x30 [ 2000.058257] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2000.058520] obd_mod_free+0x46/0x140 [ptlrpc] [ 2000.058763] mdc_commit_open+0x91/0xb0 [mdc] [ 2000.058800] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2000.059008] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2000.059206] after_reply+0xe2c/0x1780 [ptlrpc] [ 2000.059404] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2000.059613] ? __next_timer_interrupt+0x160/0x160 [ 2000.059622] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2000.059840] ? do_wait_intr+0xf0/0xf0 [ 2000.059849] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2000.060072] kthread+0x1d1/0x200 [ 2000.060078] ? set_kthread_struct+0x70/0x70 [ 2000.060082] ret_from_fork+0x1f/0x30 [ 2000.065019] Modules linked in: [ 2000.066267] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2000.066338] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2000.066345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2000.066351] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2000.066363] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2000.066368] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2000.066376] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2000.066380] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2000.066382] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2000.066384] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2000.066387] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2000.066390] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2000.066393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2000.066395] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2000.066404] Call Trace: [ 2000.066410] [ 2000.066415] ? show_regs.cold.9+0x22/0x2f [ 2000.066430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2000.066440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2000.066444] ? watchdog+0x40/0x40 [ 2000.066449] ? hrtimer_interrupt+0x128/0x2c0 [ 2000.066454] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2000.066460] ? apic_timer_interrupt+0xf/0x20 [ 2000.066465] [ 2000.066468] ? native_safe_halt+0x2e/0x50 [ 2000.066472] kvm_wait+0xcd/0xe0 [ 2000.066483] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2000.066492] do_raw_spin_lock+0x177/0x190 [ 2000.066497] _raw_spin_lock+0x21/0x30 [ 2000.066505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2000.066928] ? do_raw_spin_unlock+0x75/0x190 [ 2000.066936] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2000.067251] ? do_wait_intr+0xf0/0xf0 [ 2000.067264] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2000.067506] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2000.068021] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2000.068257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2000.068381] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2000.068421] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2000.068452] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2000.068485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2000.068588] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2000.068946] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2000.068983] lmv_intent_open+0x338/0x1150 [lmv] [ 2000.069012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2000.069113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2000.069259] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2000.069288] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2000.069377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2000.069471] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2000.076447] lustre(O) [ 2000.097455] ? do_raw_spin_unlock+0x75/0x190 [ 2000.097474] ? _raw_spin_unlock+0x12/0x30 [ 2000.097482] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2000.097511] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2000.097532] ll_atomic_open+0x36a/0xd70 [lustre] [ 2000.097707] lookup_open+0x356/0xb40 [ 2000.097716] do_last+0x313/0xfc0 [ 2000.097726] ? nd_jump_root+0xe5/0x160 [ 2000.097731] ? path_init+0x437/0x520 [ 2000.097736] path_openat+0xf7/0x500 [ 2000.097740] ? do_raw_spin_unlock+0x75/0x190 [ 2000.097745] do_filp_open+0x99/0x140 [ 2000.097752] ? getname_flags+0x6e/0x330 [ 2000.097757] ? __check_object_size+0xff/0x256 [ 2000.097764] ? do_raw_spin_unlock+0x75/0x190 [ 2000.097768] ? _raw_spin_unlock+0x12/0x30 [ 2000.097772] do_sys_openat2+0x2b4/0x410 [ 2000.097777] do_sys_open+0x73/0xa0 [ 2000.097781] __x64_sys_openat+0x24/0x30 [ 2000.097784] do_syscall_64+0xc1/0x440 [ 2000.097792] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2000.097798] RIP: 0033:0x7fa749552332 [ 2000.097849] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2000.097854] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2000.097860] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2000.097862] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2000.097864] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2000.097867] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2000.097869] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2000.545872] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2000.559897] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2000.563423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2000.566251] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2000.568593] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2000.570113] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2000.575975] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2000.578816] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2000.581283] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2000.583733] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2000.586147] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2000.588803] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2000.591203] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2000.594205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2000.596034] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2000.598379] Call Trace: [ 2000.599248] [ 2000.599863] ? show_regs.cold.9+0x22/0x2f [ 2000.601183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2000.602783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2000.604355] ? watchdog+0x40/0x40 [ 2000.605582] ? hrtimer_interrupt+0x128/0x2c0 [ 2000.606909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2000.608722] ? apic_timer_interrupt+0xf/0x20 [ 2000.610138] [ 2000.610862] ? native_safe_halt+0x2e/0x50 [ 2000.612380] kvm_wait+0xcd/0xe0 [ 2000.613504] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2000.615313] do_raw_spin_lock+0x177/0x190 [ 2000.616583] _raw_spin_lock+0x21/0x30 [ 2000.617858] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2000.619873] process_one_work+0x2c8/0x700 [ 2000.621107] worker_thread+0x296/0x6e0 [ 2000.622282] ? rescuer_thread+0x570/0x570 [ 2000.623706] kthread+0x1d1/0x200 [ 2000.624731] ? set_kthread_struct+0x70/0x70 [ 2000.626230] ret_from_fork+0x1f/0x30 [ 2003.937077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 86s! [ 2003.940801] Showing busy workqueues and worker pools: [ 2003.942449] workqueue events: flags=0x0 [ 2003.945318] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2003.947901] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc] [ 2003.953281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2003.959349] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2003.965490] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2003.973454] pending: kfree_rcu_monitor [ 2003.978846] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2003.984198] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2003.987987] workqueue events_long: flags=0x0 [ 2003.990234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2003.993962] pending: lu_objects_flush [obdclass] [ 2003.997051] workqueue events_unbound: flags=0x2 [ 2003.999280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2004.002929] pending: flush_memcg_stats_dwork [ 2004.005326] workqueue events_power_efficient: flags=0x80 [ 2004.009428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2004.013642] pending: check_lifetime [ 2004.016136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2004.019895] pending: neigh_periodic_work [ 2004.022174] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2004.025629] pending: neigh_periodic_work [ 2004.027966] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2004.031544] pending: do_cache_clean [ 2004.033812] workqueue mm_percpu_wq: flags=0x8 [ 2004.036718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2004.041322] pending: vmstat_update [ 2004.043282] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2004.047395] pending: vmstat_update [ 2004.049957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2004.053868] pending: vmstat_update [ 2004.056468] workqueue writeback: flags=0x4a [ 2004.059086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2004.062462] pending: wb_workfn, wb_workfn [ 2004.065284] workqueue xprtiod: flags=0xa [ 2004.067734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2004.071540] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2004.074743] workqueue ipv6_addrconf: flags=0x40008 [ 2004.076995] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2004.084455] pending: addrconf_verify_work [ 2004.086478] workqueue knbd0-recv: flags=0x1a [ 2004.089830] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2004.093369] in-flight: 651:recv_work [ 2004.095262] workqueue ptlrpc_pinger: flags=0x2 [ 2004.097316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2004.100270] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2004.103169] pool 8: cpus=0-3 flags=0x4 nice=0 hung=86s workers=4 idle: 37 31846 28897 [ 2004.106515] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=1994s workers=2 idle: 653 [ 2024.060726] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2024.068820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2024.089722] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2024.095145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2024.101493] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2024.104852] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2024.111829] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2024.114435] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2024.118432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2024.120986] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2024.127472] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2024.131416] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2024.136392] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2024.142101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2024.144363] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2024.147491] Call Trace: [ 2024.148386] [ 2024.151630] ? show_regs.cold.9+0x22/0x2f [ 2024.155579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2024.157190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2024.159107] ? watchdog+0x40/0x40 [ 2024.163424] ? hrtimer_interrupt+0x128/0x2c0 [ 2024.167477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2024.169122] ? apic_timer_interrupt+0xf/0x20 [ 2024.171398] [ 2024.172075] ? native_safe_halt+0x2e/0x50 [ 2024.175290] kvm_wait+0xcd/0xe0 [ 2024.176408] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2024.178454] do_raw_spin_lock+0x177/0x190 [ 2024.179886] _raw_spin_lock+0x21/0x30 [ 2024.181775] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2024.183625] ? do_raw_spin_unlock+0x75/0x190 [ 2024.186657] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2024.188919] ? do_wait_intr+0xf0/0xf0 [ 2024.190325] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2024.194195] mdc_close+0x53b/0xd70 [mdc] [ 2024.195199] lmv_close+0x18d/0x490 [lmv] [ 2024.196183] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2024.198533] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2024.200128] ll_md_close+0x274/0x800 [lustre] [ 2024.202306] ll_file_release+0x128/0xc50 [lustre] [ 2024.203915] __fput+0x10d/0x440 [ 2024.205105] ____fput+0x12/0x20 [ 2024.206358] task_work_run+0xc6/0x110 [ 2024.208158] exit_to_usermode_loop+0x1dd/0x1f0 [ 2024.211684] do_syscall_64+0x3d6/0x440 [ 2024.212962] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2024.214403] RIP: 0033:0x7fa749552cd8 [ 2024.220976] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2024.245137] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2024.249415] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2024.254913] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2024.259796] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2024.265255] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2024.273592] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2028.058047] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2028.063361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2028.065598] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2028.101669] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2028.101684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2028.101687] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2028.101700] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2028.101704] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2028.101710] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2028.101712] RDX: 00000000000016b1 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2028.101715] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2028.101716] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2028.101718] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2028.101721] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2028.101724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2028.101726] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2028.101734] Call Trace: [ 2028.101740] [ 2028.101746] ? show_regs.cold.9+0x22/0x2f [ 2028.101756] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2028.101764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2028.101769] ? watchdog+0x40/0x40 [ 2028.101774] ? hrtimer_interrupt+0x128/0x2c0 [ 2028.101777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2028.101784] ? apic_timer_interrupt+0xf/0x20 [ 2028.101797] [ 2028.101800] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2028.101804] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2028.101813] do_raw_spin_lock+0x177/0x190 [ 2028.101822] _raw_spin_lock+0x21/0x30 [ 2028.101829] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2028.102290] obd_mod_free+0x46/0x140 [ptlrpc] [ 2028.105515] mdc_commit_open+0x91/0xb0 [mdc] [ 2028.109848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2028.109927] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2028.109934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2028.109941] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2028.126677] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2028.126978] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2028.127202] after_reply+0xe2c/0x1780 [ptlrpc] [ 2028.127399] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2028.138610] [ 2028.138619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2028.138636] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2028.138640] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2028.138644] RAX: 0000000000006d2c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2028.138647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2028.138649] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2028.138651] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2028.138653] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2028.138656] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2028.138659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2028.138661] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2028.138669] Call Trace: [ 2028.138674] [ 2028.138680] ? show_regs.cold.9+0x22/0x2f [ 2028.138690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2028.138696] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2028.138701] ? watchdog+0x40/0x40 [ 2028.138705] ? hrtimer_interrupt+0x128/0x2c0 [ 2028.138709] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2028.138716] ? apic_timer_interrupt+0xf/0x20 [ 2028.138721] [ 2028.138723] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2028.138728] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2028.138735] do_raw_spin_lock+0x177/0x190 [ 2028.138739] _raw_spin_lock+0x21/0x30 [ 2028.138744] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2028.139492] process_one_work+0x2c8/0x700 [ 2028.139499] worker_thread+0x296/0x6e0 [ 2028.139504] ? rescuer_thread+0x570/0x570 [ 2028.139507] kthread+0x1d1/0x200 [ 2028.139512] ? set_kthread_struct+0x70/0x70 [ 2028.139516] ret_from_fork+0x1f/0x30 [ 2028.347046] ? __next_timer_interrupt+0x160/0x160 [ 2028.349152] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2028.351585] ? do_wait_intr+0xf0/0xf0 [ 2028.353052] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2028.355460] kthread+0x1d1/0x200 [ 2028.356612] ? set_kthread_struct+0x70/0x70 [ 2028.358279] ret_from_fork+0x1f/0x30 [ 2032.054016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2032.057898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2032.081993] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2032.087522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2032.091681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2032.094214] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2032.102950] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2032.106960] RAX: 0000000000004f8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2032.110263] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2032.113329] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2032.116987] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2032.119267] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2032.122880] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2032.126689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2032.129272] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2032.131925] Call Trace: [ 2032.133444] [ 2032.134893] ? show_regs.cold.9+0x22/0x2f [ 2032.136616] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2032.138837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2032.140912] ? watchdog+0x40/0x40 [ 2032.142421] ? hrtimer_interrupt+0x128/0x2c0 [ 2032.144432] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2032.146684] ? apic_timer_interrupt+0xf/0x20 [ 2032.148672] [ 2032.149678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2032.152211] do_raw_spin_lock+0x177/0x190 [ 2032.154184] _raw_spin_lock+0x21/0x30 [ 2032.156010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2032.159287] ? do_raw_spin_unlock+0x75/0x190 [ 2032.161350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2032.163626] ? do_wait_intr+0xf0/0xf0 [ 2032.165426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2032.167842] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2032.170365] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2032.173345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2032.176251] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2032.178578] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2032.180806] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2032.183061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2032.185482] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2032.188924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2032.191629] lmv_intent_open+0x338/0x1150 [lmv] [ 2032.193777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2032.196788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2032.199213] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2032.201195] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2032.204301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2032.206826] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2032.209024] ? do_raw_spin_unlock+0x75/0x190 [ 2032.211020] ? _raw_spin_unlock+0x12/0x30 [ 2032.212744] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2032.215202] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2032.217504] ll_atomic_open+0x36a/0xd70 [lustre] [ 2032.219781] lookup_open+0x356/0xb40 [ 2032.221316] do_last+0x313/0xfc0 [ 2032.222831] ? nd_jump_root+0xe5/0x160 [ 2032.224295] ? path_init+0x437/0x520 [ 2032.225904] path_openat+0xf7/0x500 [ 2032.227602] ? do_raw_spin_unlock+0x75/0x190 [ 2032.229385] do_filp_open+0x99/0x140 [ 2032.231011] ? getname_flags+0x6e/0x330 [ 2032.233106] ? __check_object_size+0xff/0x256 [ 2032.235131] ? do_raw_spin_unlock+0x75/0x190 [ 2032.237236] ? _raw_spin_unlock+0x12/0x30 [ 2032.239403] do_sys_openat2+0x2b4/0x410 [ 2032.241000] do_sys_open+0x73/0xa0 [ 2032.242634] __x64_sys_openat+0x24/0x30 [ 2032.244284] do_syscall_64+0xc1/0x440 [ 2032.245831] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2032.248553] RIP: 0033:0x7fa749552332 [ 2032.250047] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2032.258330] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2032.261735] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2032.265010] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2032.268281] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2032.271630] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2032.274784] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2034.145204] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 116s! [ 2034.149842] Showing busy workqueues and worker pools: [ 2034.152590] workqueue events: flags=0x0 [ 2034.155012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2034.158874] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc] [ 2034.164854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2034.168424] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2034.174350] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.178438] pending: kfree_rcu_monitor [ 2034.181432] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2034.185544] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2034.189292] workqueue events_long: flags=0x0 [ 2034.191574] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.195379] pending: lu_objects_flush [obdclass] [ 2034.198637] workqueue events_unbound: flags=0x2 [ 2034.202012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2034.206773] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2034.211709] workqueue events_power_efficient: flags=0x80 [ 2034.215821] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.220363] pending: check_lifetime [ 2034.223967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.228966] pending: neigh_periodic_work [ 2034.231946] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.236839] pending: neigh_periodic_work [ 2034.239087] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.243790] pending: do_cache_clean [ 2034.246489] workqueue mm_percpu_wq: flags=0x8 [ 2034.248624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.252769] pending: vmstat_update [ 2034.255364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.259961] pending: vmstat_update [ 2034.262014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2034.265632] pending: vmstat_update [ 2034.268049] workqueue writeback: flags=0x4a [ 2034.269829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2034.272709] pending: wb_workfn, wb_workfn [ 2034.274845] workqueue xprtiod: flags=0xa [ 2034.276606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2034.279765] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2034.282773] workqueue ipv6_addrconf: flags=0x40008 [ 2034.284869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2034.287409] pending: addrconf_verify_work [ 2034.289608] workqueue knbd0-recv: flags=0x1a [ 2034.291293] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2034.293979] in-flight: 651:recv_work [ 2034.295611] workqueue ptlrpc_pinger: flags=0x2 [ 2034.297304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2034.299889] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2034.302849] pool 8: cpus=0-3 flags=0x4 nice=0 hung=117s workers=4 idle: 37 31846 28897 [ 2034.309961] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2024s workers=2 idle: 653 [ 2052.060128] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2052.079752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2052.123131] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2052.129228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2052.140266] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2052.148912] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2052.170589] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2052.177118] RAX: 0000000000007a3f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2052.183865] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2052.195598] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2052.207488] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2052.216699] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2052.220712] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2052.225692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2052.231547] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2052.240900] Call Trace: [ 2052.242471] [ 2052.246368] ? show_regs.cold.9+0x22/0x2f [ 2052.251736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2052.254347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2052.257667] ? watchdog+0x40/0x40 [ 2052.261683] ? hrtimer_interrupt+0x128/0x2c0 [ 2052.266591] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2052.269873] ? apic_timer_interrupt+0xf/0x20 [ 2052.274126] [ 2052.275857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2052.278538] do_raw_spin_lock+0x177/0x190 [ 2052.282259] _raw_spin_lock+0x21/0x30 [ 2052.284020] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2052.290490] ? do_raw_spin_unlock+0x75/0x190 [ 2052.294336] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2052.298549] ? do_wait_intr+0xf0/0xf0 [ 2052.299565] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2052.302528] mdc_close+0x53b/0xd70 [mdc] [ 2052.304613] lmv_close+0x18d/0x490 [lmv] [ 2052.307759] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2052.317724] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2052.321820] ll_md_close+0x274/0x800 [lustre] [ 2052.328527] ll_file_release+0x128/0xc50 [lustre] [ 2052.335672] __fput+0x10d/0x440 [ 2052.336692] ____fput+0x12/0x20 [ 2052.337589] task_work_run+0xc6/0x110 [ 2052.338686] exit_to_usermode_loop+0x1dd/0x1f0 [ 2052.346657] do_syscall_64+0x3d6/0x440 [ 2052.350631] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2052.356543] RIP: 0033:0x7fa749552cd8 [ 2052.359589] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2052.371649] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2052.375224] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2052.378439] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2052.381251] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2052.384323] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2052.387773] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2056.060618] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2056.062825] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2056.066459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2056.066538] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2056.066545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2056.066548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2056.066564] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2056.066568] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2056.066573] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2056.066575] RDX: 00000000000015a3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2056.066577] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2056.066580] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2056.066581] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2056.066584] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2056.066587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2056.066589] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2056.066598] Call Trace: [ 2056.066603] [ 2056.066609] ? show_regs.cold.9+0x22/0x2f [ 2056.066619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2056.066626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2056.066630] ? watchdog+0x40/0x40 [ 2056.066634] ? hrtimer_interrupt+0x128/0x2c0 [ 2056.066638] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2056.066648] ? apic_timer_interrupt+0xf/0x20 [ 2056.066652] [ 2056.066654] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2056.066658] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2056.066662] do_raw_spin_lock+0x177/0x190 [ 2056.066666] _raw_spin_lock+0x21/0x30 [ 2056.066672] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2056.066931] obd_mod_free+0x46/0x140 [ptlrpc] [ 2056.067146] mdc_commit_open+0x91/0xb0 [mdc] [ 2056.067186] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2056.067378] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2056.067572] after_reply+0xe2c/0x1780 [ptlrpc] [ 2056.067765] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2056.072023] ? __next_timer_interrupt+0x160/0x160 [ 2056.072037] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2056.083037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2056.083126] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2056.083133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2056.083139] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2056.084693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2056.084706] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2056.084710] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2056.084715] RAX: 0000000000007f0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2056.084718] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2056.084720] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2056.084723] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2056.084725] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2056.084729] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2056.084731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2056.084733] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2056.084742] Call Trace: [ 2056.084746] [ 2056.084752] ? show_regs.cold.9+0x22/0x2f [ 2056.084760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2056.084766] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2056.084771] ? watchdog+0x40/0x40 [ 2056.084775] ? hrtimer_interrupt+0x128/0x2c0 [ 2056.084779] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2056.084788] ? apic_timer_interrupt+0xf/0x20 [ 2056.084793] [ 2056.084796] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2056.084800] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2056.084805] do_raw_spin_lock+0x177/0x190 [ 2056.084809] _raw_spin_lock+0x21/0x30 [ 2056.084814] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2056.087189] process_one_work+0x2c8/0x700 [ 2056.087201] worker_thread+0x296/0x6e0 [ 2056.087206] ? rescuer_thread+0x570/0x570 [ 2056.087210] kthread+0x1d1/0x200 [ 2056.087325] ? set_kthread_struct+0x70/0x70 [ 2056.087330] ret_from_fork+0x1f/0x30 [ 2056.514653] ? do_wait_intr+0xf0/0xf0 [ 2056.516120] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2056.520395] kthread+0x1d1/0x200 [ 2056.521616] ? set_kthread_struct+0x70/0x70 [ 2056.532137] ret_from_fork+0x1f/0x30 [ 2060.060736] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2060.064432] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2060.138358] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2060.154734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2060.162356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2060.168199] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2060.190280] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2060.197467] RAX: 0000000000007485 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2060.203502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2060.207151] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2060.225495] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2060.235571] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2060.252287] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2060.265200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2060.272059] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2060.284932] Call Trace: [ 2060.288653] [ 2060.289763] ? show_regs.cold.9+0x22/0x2f [ 2060.294544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2060.297254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2060.306688] ? watchdog+0x40/0x40 [ 2060.309085] ? hrtimer_interrupt+0x128/0x2c0 [ 2060.316486] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2060.318338] ? apic_timer_interrupt+0xf/0x20 [ 2060.325368] [ 2060.328131] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2060.332815] do_raw_spin_lock+0x177/0x190 [ 2060.341439] _raw_spin_lock+0x21/0x30 [ 2060.345021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2060.358518] ? do_raw_spin_unlock+0x75/0x190 [ 2060.366770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2060.377932] ? do_wait_intr+0xf0/0xf0 [ 2060.383571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2060.391266] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2060.397464] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2060.407022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2060.419716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2060.423347] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2060.432393] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2060.438075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2060.441263] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2060.447190] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2060.451671] lmv_intent_open+0x338/0x1150 [lmv] [ 2060.454094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2060.457788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2060.463256] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2060.471427] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2060.481641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2060.492734] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2060.498613] ? do_raw_spin_unlock+0x75/0x190 [ 2060.506972] ? _raw_spin_unlock+0x12/0x30 [ 2060.510103] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2060.522594] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2060.532544] ll_atomic_open+0x36a/0xd70 [lustre] [ 2060.536693] lookup_open+0x356/0xb40 [ 2060.538277] do_last+0x313/0xfc0 [ 2060.539998] ? nd_jump_root+0xe5/0x160 [ 2060.541617] ? path_init+0x437/0x520 [ 2060.544467] path_openat+0xf7/0x500 [ 2060.546481] ? do_raw_spin_unlock+0x75/0x190 [ 2060.553471] do_filp_open+0x99/0x140 [ 2060.566069] ? getname_flags+0x6e/0x330 [ 2060.573450] ? __check_object_size+0xff/0x256 [ 2060.576557] ? do_raw_spin_unlock+0x75/0x190 [ 2060.580496] ? _raw_spin_unlock+0x12/0x30 [ 2060.589617] do_sys_openat2+0x2b4/0x410 [ 2060.591011] do_sys_open+0x73/0xa0 [ 2060.599051] __x64_sys_openat+0x24/0x30 [ 2060.602738] do_syscall_64+0xc1/0x440 [ 2060.605655] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2060.613812] RIP: 0033:0x7fa749552332 [ 2060.622337] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2060.646540] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2060.663713] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2060.673736] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2060.688241] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2060.704793] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2060.718984] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2064.352088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 147s! [ 2064.356163] Showing busy workqueues and worker pools: [ 2064.359335] workqueue events: flags=0x0 [ 2064.362303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2064.368763] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc] [ 2064.382284] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2064.386374] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2064.395251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.401532] pending: kfree_rcu_monitor [ 2064.403284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2064.407778] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2064.417320] workqueue events_long: flags=0x0 [ 2064.421837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.429621] pending: lu_objects_flush [obdclass] [ 2064.434803] workqueue events_unbound: flags=0x2 [ 2064.437451] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2064.440171] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2064.445191] workqueue events_power_efficient: flags=0x80 [ 2064.447491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.456761] pending: check_lifetime [ 2064.463261] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.470850] pending: neigh_periodic_work [ 2064.480650] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.491292] pending: neigh_periodic_work [ 2064.497081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.504349] pending: do_cache_clean [ 2064.506478] workqueue mm_percpu_wq: flags=0x8 [ 2064.511436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.516481] pending: vmstat_update [ 2064.525706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.536616] pending: vmstat_update [ 2064.544248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2064.552600] pending: vmstat_update [ 2064.557409] workqueue writeback: flags=0x4a [ 2064.560692] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2064.564317] pending: wb_workfn, wb_workfn [ 2064.566213] workqueue xprtiod: flags=0xa [ 2064.569260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2064.573445] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2064.577418] workqueue ipv6_addrconf: flags=0x40008 [ 2064.580560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2064.584284] pending: addrconf_verify_work [ 2064.588361] workqueue knbd0-recv: flags=0x1a [ 2064.591641] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2064.600953] in-flight: 651:recv_work [ 2064.602265] workqueue ptlrpc_pinger: flags=0x2 [ 2064.608660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2064.618414] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2064.625568] pool 8: cpus=0-3 flags=0x4 nice=0 hung=147s workers=4 idle: 37 31846 28897 [ 2064.634495] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2054s workers=2 idle: 653 [ 2080.059104] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2080.061666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2080.089931] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2080.100070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2080.107892] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2080.110702] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2080.131776] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2080.138058] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2080.142508] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2080.145598] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2080.148580] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2080.154030] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2080.157164] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2080.161140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2080.171446] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2080.180443] Call Trace: [ 2080.185457] [ 2080.186237] ? show_regs.cold.9+0x22/0x2f [ 2080.192179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2080.195647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2080.201668] ? watchdog+0x40/0x40 [ 2080.205639] ? hrtimer_interrupt+0x128/0x2c0 [ 2080.208687] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2080.213424] ? apic_timer_interrupt+0xf/0x20 [ 2080.217775] [ 2080.218581] ? native_safe_halt+0x2e/0x50 [ 2080.221141] kvm_wait+0xcd/0xe0 [ 2080.225839] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2080.228921] do_raw_spin_lock+0x177/0x190 [ 2080.230397] _raw_spin_lock+0x21/0x30 [ 2080.236383] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2080.241983] ? do_raw_spin_unlock+0x75/0x190 [ 2080.244494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2080.251193] ? do_wait_intr+0xf0/0xf0 [ 2080.255942] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2080.259470] mdc_close+0x53b/0xd70 [mdc] [ 2080.262972] lmv_close+0x18d/0x490 [lmv] [ 2080.264447] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2080.273088] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2080.277194] ll_md_close+0x274/0x800 [lustre] [ 2080.282011] ll_file_release+0x128/0xc50 [lustre] [ 2080.285890] __fput+0x10d/0x440 [ 2080.288142] ____fput+0x12/0x20 [ 2080.289686] task_work_run+0xc6/0x110 [ 2080.291443] exit_to_usermode_loop+0x1dd/0x1f0 [ 2080.293818] do_syscall_64+0x3d6/0x440 [ 2080.300473] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2080.305864] RIP: 0033:0x7fa749552cd8 [ 2080.311964] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2080.331361] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2080.339425] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2080.344817] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2080.352590] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2080.359900] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2080.362333] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2084.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2084.058025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2084.060257] Modules linked in: lustre(O) [ 2084.063334] Modules linked in: [ 2084.063336] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2084.063386] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2084.063391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2084.063396] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2084.064966] lustre(O) [ 2084.065827] [ 2084.065831] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2084.082881] osp(O) [ 2084.086346] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2084.086367] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2084.086373] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2084.086375] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2084.086377] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2084.086379] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2084.086380] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2084.086383] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2084.086385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2084.086387] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2084.086393] Call Trace: [ 2084.086398] [ 2084.086404] ? show_regs.cold.9+0x22/0x2f [ 2084.086416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2084.086422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2084.086425] ? watchdog+0x40/0x40 [ 2084.090059] ofd(O) [ 2084.092755] ? hrtimer_interrupt+0x128/0x2c0 [ 2084.093444] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2084.093520] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2084.093526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2084.093529] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2084.093539] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2084.093543] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2084.093547] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2084.093548] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2084.093550] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2084.093552] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2084.093554] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2084.093556] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2084.093558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2084.093560] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2084.093568] Call Trace: [ 2084.093573] [ 2084.093576] ? show_regs.cold.9+0x22/0x2f [ 2084.093585] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2084.093590] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2084.093594] ? watchdog+0x40/0x40 [ 2084.093598] ? hrtimer_interrupt+0x128/0x2c0 [ 2084.093602] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2084.093608] ? apic_timer_interrupt+0xf/0x20 [ 2084.093612] [ 2084.093614] ? native_safe_halt+0x2e/0x50 [ 2084.093618] kvm_wait+0xcd/0xe0 [ 2084.093625] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2084.093631] do_raw_spin_lock+0x177/0x190 [ 2084.093634] _raw_spin_lock+0x21/0x30 [ 2084.093639] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2084.094860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2084.096882] obd_mod_free+0x46/0x140 [ptlrpc] [ 2084.097923] ? apic_timer_interrupt+0xf/0x20 [ 2084.105733] mdc_commit_open+0x91/0xb0 [mdc] [ 2084.109406] [ 2084.112230] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2084.116316] ? native_safe_halt+0x2e/0x50 [ 2084.119592] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2084.123894] kvm_wait+0xcd/0xe0 [ 2084.126829] after_reply+0xe2c/0x1780 [ptlrpc] [ 2084.127115] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2084.127482] ? __next_timer_interrupt+0x160/0x160 [ 2084.127493] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2084.127772] ? do_wait_intr+0xf0/0xf0 [ 2084.127782] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2084.128007] kthread+0x1d1/0x200 [ 2084.128012] ? set_kthread_struct+0x70/0x70 [ 2084.128016] ret_from_fork+0x1f/0x30 [ 2084.416042] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2084.420026] do_raw_spin_lock+0x177/0x190 [ 2084.422788] _raw_spin_lock+0x21/0x30 [ 2084.424947] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2084.428714] process_one_work+0x2c8/0x700 [ 2084.429835] worker_thread+0x296/0x6e0 [ 2084.430779] ? rescuer_thread+0x570/0x570 [ 2084.431906] kthread+0x1d1/0x200 [ 2084.433039] ? set_kthread_struct+0x70/0x70 [ 2084.434536] ret_from_fork+0x1f/0x30 [ 2092.054197] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2092.057411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2092.093380] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2092.114661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2092.125671] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2092.129854] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2092.163114] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2092.168709] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2092.173941] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2092.184612] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2092.187865] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2092.196906] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2092.203950] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2092.207359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.211333] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2092.216949] Call Trace: [ 2092.221470] [ 2092.224241] ? show_regs.cold.9+0x22/0x2f [ 2092.227838] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2092.230889] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2092.233567] ? watchdog+0x40/0x40 [ 2092.236695] ? hrtimer_interrupt+0x128/0x2c0 [ 2092.238184] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2092.239639] ? apic_timer_interrupt+0xf/0x20 [ 2092.241132] [ 2092.244567] ? native_safe_halt+0x2e/0x50 [ 2092.247666] kvm_wait+0xcd/0xe0 [ 2092.248532] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2092.255160] do_raw_spin_lock+0x177/0x190 [ 2092.262788] _raw_spin_lock+0x21/0x30 [ 2092.268592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2092.270462] ? do_raw_spin_unlock+0x75/0x190 [ 2092.273537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2092.277581] ? do_wait_intr+0xf0/0xf0 [ 2092.280445] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2092.297415] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2092.305163] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2092.322673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.333279] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2092.348571] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2092.355567] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2092.358426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.362620] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2092.365492] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2092.373886] lmv_intent_open+0x338/0x1150 [lmv] [ 2092.377299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.379591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.381379] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2092.382988] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2092.389718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.398741] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2092.406128] ? do_raw_spin_unlock+0x75/0x190 [ 2092.408885] ? _raw_spin_unlock+0x12/0x30 [ 2092.411674] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2092.427514] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2092.429104] ll_atomic_open+0x36a/0xd70 [lustre] [ 2092.436256] lookup_open+0x356/0xb40 [ 2092.437569] do_last+0x313/0xfc0 [ 2092.438712] ? nd_jump_root+0xe5/0x160 [ 2092.448791] ? path_init+0x437/0x520 [ 2092.452666] path_openat+0xf7/0x500 [ 2092.466663] ? do_raw_spin_unlock+0x75/0x190 [ 2092.468100] do_filp_open+0x99/0x140 [ 2092.474200] ? getname_flags+0x6e/0x330 [ 2092.476669] ? __check_object_size+0xff/0x256 [ 2092.479824] ? do_raw_spin_unlock+0x75/0x190 [ 2092.481438] ? _raw_spin_unlock+0x12/0x30 [ 2092.482909] do_sys_openat2+0x2b4/0x410 [ 2092.484393] do_sys_open+0x73/0xa0 [ 2092.485680] __x64_sys_openat+0x24/0x30 [ 2092.487145] do_syscall_64+0xc1/0x440 [ 2092.488514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2092.490390] RIP: 0033:0x7fa749552332 [ 2092.491722] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2092.499886] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2092.505404] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2092.508756] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2092.514180] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2092.520593] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2092.523575] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2095.073077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 177s! [ 2095.081017] Showing busy workqueues and worker pools: [ 2095.087283] workqueue events: flags=0x0 [ 2095.092281] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2095.101311] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2095.117313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2095.130652] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2095.140613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.143801] pending: kfree_rcu_monitor [ 2095.145088] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2095.150676] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2095.154809] workqueue events_long: flags=0x0 [ 2095.159945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.163135] pending: lu_objects_flush [obdclass] [ 2095.165852] workqueue events_unbound: flags=0x2 [ 2095.167934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2095.174424] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2095.181863] workqueue events_power_efficient: flags=0x80 [ 2095.184207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.187607] pending: check_lifetime [ 2095.195218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.200290] pending: neigh_periodic_work [ 2095.202066] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.206733] pending: neigh_periodic_work [ 2095.208234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.218741] pending: do_cache_clean [ 2095.225046] workqueue mm_percpu_wq: flags=0x8 [ 2095.230365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.240927] pending: vmstat_update [ 2095.248181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.262015] pending: vmstat_update [ 2095.264620] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2095.281499] pending: vmstat_update [ 2095.282728] workqueue writeback: flags=0x4a [ 2095.284079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2095.291122] pending: wb_workfn, wb_workfn [ 2095.308739] workqueue xprtiod: flags=0xa [ 2095.310024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2095.313854] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2095.320862] workqueue ipv6_addrconf: flags=0x40008 [ 2095.324426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2095.327372] pending: addrconf_verify_work [ 2095.329371] workqueue knbd0-recv: flags=0x1a [ 2095.331336] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2095.334271] in-flight: 651:recv_work [ 2095.336120] workqueue ptlrpc_pinger: flags=0x2 [ 2095.337912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2095.340799] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2095.348111] pool 8: cpus=0-3 flags=0x4 nice=0 hung=178s workers=4 idle: 37 31846 28897 [ 2095.354191] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2085s workers=2 idle: 653 [ 2108.062316] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2108.066352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2108.090248] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2108.093938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2108.097626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2108.100147] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2108.114819] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2108.118692] RAX: 00000000000077a2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2108.122297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2108.127120] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2108.132722] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2108.136529] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2108.140351] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2108.143721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2108.145865] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2108.149383] Call Trace: [ 2108.150354] [ 2108.154328] ? show_regs.cold.9+0x22/0x2f [ 2108.157037] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2108.160020] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2108.162313] ? watchdog+0x40/0x40 [ 2108.164545] ? hrtimer_interrupt+0x128/0x2c0 [ 2108.167163] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2108.171723] ? apic_timer_interrupt+0xf/0x20 [ 2108.174593] [ 2108.175405] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2108.177930] do_raw_spin_lock+0x177/0x190 [ 2108.183369] _raw_spin_lock+0x21/0x30 [ 2108.184890] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2108.186866] ? do_raw_spin_unlock+0x75/0x190 [ 2108.205087] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2108.210929] ? do_wait_intr+0xf0/0xf0 [ 2108.212354] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2108.220693] mdc_close+0x53b/0xd70 [mdc] [ 2108.222700] lmv_close+0x18d/0x490 [lmv] [ 2108.225099] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2108.230487] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2108.233499] ll_md_close+0x274/0x800 [lustre] [ 2108.235534] ll_file_release+0x128/0xc50 [lustre] [ 2108.240410] __fput+0x10d/0x440 [ 2108.242344] ____fput+0x12/0x20 [ 2108.248449] task_work_run+0xc6/0x110 [ 2108.252205] exit_to_usermode_loop+0x1dd/0x1f0 [ 2108.253592] do_syscall_64+0x3d6/0x440 [ 2108.254788] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2108.260682] RIP: 0033:0x7fa749552cd8 [ 2108.261879] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2108.276913] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2108.280697] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2108.283176] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2108.286867] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2108.289849] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2108.299088] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2112.057013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2112.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2112.065843] Modules linked in: [ 2112.070754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2112.070835] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2112.070843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2112.070846] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2112.070859] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2112.070864] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2112.070868] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2112.070871] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2112.070873] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2112.070875] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2112.070877] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2112.070880] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2112.070883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2112.070886] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2112.070894] Call Trace: [ 2112.070899] [ 2112.070904] ? show_regs.cold.9+0x22/0x2f [ 2112.070914] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2112.070921] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2112.070925] ? watchdog+0x40/0x40 [ 2112.070929] ? hrtimer_interrupt+0x128/0x2c0 [ 2112.070933] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2112.070940] ? apic_timer_interrupt+0xf/0x20 [ 2112.070944] [ 2112.070947] ? native_safe_halt+0x2e/0x50 [ 2112.070951] kvm_wait+0xcd/0xe0 [ 2112.070969] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2112.070977] do_raw_spin_lock+0x177/0x190 [ 2112.070981] _raw_spin_lock+0x21/0x30 [ 2112.070987] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2112.071262] obd_mod_free+0x46/0x140 [ptlrpc] [ 2112.071471] mdc_commit_open+0x91/0xb0 [mdc] [ 2112.073030] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2112.073095] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2112.073101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2112.073107] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2112.095741] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2112.096029] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2112.096359] after_reply+0xe2c/0x1780 [ptlrpc] [ 2112.096563] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2112.096757] ? __next_timer_interrupt+0x160/0x160 [ 2112.096768] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2112.096993] ? do_wait_intr+0xf0/0xf0 [ 2112.097003] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2112.097201] kthread+0x1d1/0x200 [ 2112.097207] ? set_kthread_struct+0x70/0x70 [ 2112.097210] ret_from_fork+0x1f/0x30 [ 2112.362303] [ 2112.362908] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2112.366078] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2112.379986] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2112.386926] RAX: 0000000000003be7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2112.393005] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2112.395285] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2112.401819] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2112.407038] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2112.414140] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2112.420990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2112.425246] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2112.432652] Call Trace: [ 2112.433466] [ 2112.434414] ? show_regs.cold.9+0x22/0x2f [ 2112.435777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2112.439353] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2112.442915] ? watchdog+0x40/0x40 [ 2112.445873] ? hrtimer_interrupt+0x128/0x2c0 [ 2112.449237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2112.453592] ? apic_timer_interrupt+0xf/0x20 [ 2112.456902] [ 2112.459442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2112.463151] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2112.467971] do_raw_spin_lock+0x177/0x190 [ 2112.471595] _raw_spin_lock+0x21/0x30 [ 2112.473986] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2112.478150] process_one_work+0x2c8/0x700 [ 2112.480969] worker_thread+0x296/0x6e0 [ 2112.484548] ? rescuer_thread+0x570/0x570 [ 2112.487568] kthread+0x1d1/0x200 [ 2112.488669] ? set_kthread_struct+0x70/0x70 [ 2112.489904] ret_from_fork+0x1f/0x30 [ 2120.062338] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2120.070328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.095172] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.100510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.103887] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.106315] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2120.129044] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2120.134913] RAX: 0000000000007e99 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2120.146213] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2120.156445] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2120.163807] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2120.170406] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2120.177833] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2120.182086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.185422] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2120.189953] Call Trace: [ 2120.192982] [ 2120.194023] ? show_regs.cold.9+0x22/0x2f [ 2120.195626] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.203810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.209127] ? watchdog+0x40/0x40 [ 2120.210414] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.211929] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.214921] ? apic_timer_interrupt+0xf/0x20 [ 2120.216825] [ 2120.217718] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.219956] do_raw_spin_lock+0x177/0x190 [ 2120.221681] _raw_spin_lock+0x21/0x30 [ 2120.222989] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2120.225263] ? do_raw_spin_unlock+0x75/0x190 [ 2120.226712] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2120.229019] ? do_wait_intr+0xf0/0xf0 [ 2120.233119] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2120.236780] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2120.239308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2120.242285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.244801] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2120.247094] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2120.248926] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2120.250898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.253348] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2120.255887] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2120.258307] lmv_intent_open+0x338/0x1150 [lmv] [ 2120.259868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.261643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.263264] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2120.264782] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2120.266303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.268321] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2120.270266] ? do_raw_spin_unlock+0x75/0x190 [ 2120.272044] ? _raw_spin_unlock+0x12/0x30 [ 2120.273783] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2120.276337] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2120.278873] ll_atomic_open+0x36a/0xd70 [lustre] [ 2120.281298] lookup_open+0x356/0xb40 [ 2120.282857] do_last+0x313/0xfc0 [ 2120.283977] ? nd_jump_root+0xe5/0x160 [ 2120.285445] ? path_init+0x437/0x520 [ 2120.286913] path_openat+0xf7/0x500 [ 2120.288380] ? do_raw_spin_unlock+0x75/0x190 [ 2120.290696] do_filp_open+0x99/0x140 [ 2120.292235] ? getname_flags+0x6e/0x330 [ 2120.293981] ? __check_object_size+0xff/0x256 [ 2120.295826] ? do_raw_spin_unlock+0x75/0x190 [ 2120.297711] ? _raw_spin_unlock+0x12/0x30 [ 2120.299412] do_sys_openat2+0x2b4/0x410 [ 2120.302348] do_sys_open+0x73/0xa0 [ 2120.303890] __x64_sys_openat+0x24/0x30 [ 2120.305664] do_syscall_64+0xc1/0x440 [ 2120.307269] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2120.309696] RIP: 0033:0x7fa749552332 [ 2120.316803] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2120.332777] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2120.337759] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2120.341786] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2120.346960] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2120.354519] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2120.359597] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2125.798259] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 208s! [ 2125.803391] Showing busy workqueues and worker pools: [ 2125.806243] workqueue events: flags=0x0 [ 2125.808091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2125.811087] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2125.816893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2125.820493] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2125.824914] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.827949] pending: kfree_rcu_monitor [ 2125.829703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2125.832657] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2125.835790] workqueue events_long: flags=0x0 [ 2125.837645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.840993] pending: lu_objects_flush [obdclass] [ 2125.843900] workqueue events_unbound: flags=0x2 [ 2125.845489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2125.848377] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2125.851632] workqueue events_power_efficient: flags=0x80 [ 2125.853782] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.856899] pending: check_lifetime [ 2125.858327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.861469] pending: neigh_periodic_work [ 2125.863079] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.866071] pending: neigh_periodic_work [ 2125.867728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.870835] pending: do_cache_clean [ 2125.872325] workqueue mm_percpu_wq: flags=0x8 [ 2125.873947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.876947] pending: vmstat_update [ 2125.878935] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.881272] pending: vmstat_update [ 2125.882821] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2125.885459] pending: vmstat_update [ 2125.887518] workqueue writeback: flags=0x4a [ 2125.889576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2125.892481] pending: wb_workfn, wb_workfn [ 2125.894369] workqueue xprtiod: flags=0xa [ 2125.895793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2125.898906] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2125.901991] workqueue ipv6_addrconf: flags=0x40008 [ 2125.904149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2125.907097] pending: addrconf_verify_work [ 2125.908923] workqueue knbd0-recv: flags=0x1a [ 2125.910883] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2125.913727] in-flight: 651:recv_work [ 2125.915651] workqueue ptlrpc_pinger: flags=0x2 [ 2125.917506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2125.920065] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2125.922357] pool 8: cpus=0-3 flags=0x4 nice=0 hung=208s workers=4 idle: 37 31846 28897 [ 2125.925258] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2115s workers=2 idle: 653 [ 2136.059009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2136.063579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2136.085489] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2136.089917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2136.094002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2136.096863] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2136.104672] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2136.107918] RAX: 0000000000002177 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2136.110876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2136.113193] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2136.117175] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2136.121131] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2136.131097] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2136.137035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2136.140446] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2136.148009] Call Trace: [ 2136.149175] [ 2136.150141] ? show_regs.cold.9+0x22/0x2f [ 2136.152997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2136.159628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2136.162877] ? watchdog+0x40/0x40 [ 2136.164091] ? hrtimer_interrupt+0x128/0x2c0 [ 2136.168642] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2136.173598] ? apic_timer_interrupt+0xf/0x20 [ 2136.176097] [ 2136.176882] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2136.179071] do_raw_spin_lock+0x177/0x190 [ 2136.180414] _raw_spin_lock+0x21/0x30 [ 2136.181591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2136.184612] ? do_raw_spin_unlock+0x75/0x190 [ 2136.188596] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2136.190666] ? do_wait_intr+0xf0/0xf0 [ 2136.192587] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2136.198926] mdc_close+0x53b/0xd70 [mdc] [ 2136.200225] lmv_close+0x18d/0x490 [lmv] [ 2136.203988] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2136.206123] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2136.207664] ll_md_close+0x274/0x800 [lustre] [ 2136.209053] ll_file_release+0x128/0xc50 [lustre] [ 2136.210510] __fput+0x10d/0x440 [ 2136.211404] ____fput+0x12/0x20 [ 2136.212342] task_work_run+0xc6/0x110 [ 2136.213403] exit_to_usermode_loop+0x1dd/0x1f0 [ 2136.214955] do_syscall_64+0x3d6/0x440 [ 2136.216056] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2136.217824] RIP: 0033:0x7fa749552cd8 [ 2136.218876] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2136.224202] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2136.226408] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2136.228875] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2136.230919] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2136.233165] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2136.235483] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2140.057291] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2140.058048] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2140.059734] Modules linked in: [ 2140.061840] Modules linked in: [ 2140.061842] lustre(O) osp(O) [ 2140.062780] lustre(O) [ 2140.063716] ofd(O) [ 2140.064674] osp(O) ofd(O) [ 2140.065584] lod(O) [ 2140.066224] lod(O) [ 2140.066911] mdt(O) [ 2140.067714] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2140.067774] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2140.067779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2140.067781] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2140.067794] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2140.067798] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2140.067803] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2140.067805] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2140.067807] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2140.067809] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2140.067811] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2140.067814] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2140.067816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2140.067818] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2140.067827] Call Trace: [ 2140.067833] [ 2140.067839] ? show_regs.cold.9+0x22/0x2f [ 2140.068703] mdd(O) [ 2140.069437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2140.069448] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2140.069452] ? watchdog+0x40/0x40 [ 2140.069456] ? hrtimer_interrupt+0x128/0x2c0 [ 2140.069459] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2140.069479] ? apic_timer_interrupt+0xf/0x20 [ 2140.069483] [ 2140.069487] ? native_safe_halt+0x2e/0x50 [ 2140.069491] kvm_wait+0xcd/0xe0 [ 2140.069499] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2140.069507] do_raw_spin_lock+0x177/0x190 [ 2140.069511] _raw_spin_lock+0x21/0x30 [ 2140.069517] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2140.091129] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [ 2140.094292] obd_mod_free+0x46/0x140 [ptlrpc] [ 2140.097532] i2c_piix4 [ 2140.098813] mdc_commit_open+0x91/0xb0 [mdc] [ 2140.098858] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2140.105602] pcspkr [ 2140.107902] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2140.110473] squashfs [ 2140.112518] after_reply+0xe2c/0x1780 [ptlrpc] [ 2140.117295] ata_generic [ 2140.119626] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2140.123689] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2140.123718] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2140.123724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2140.123730] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2140.129548] ? __next_timer_interrupt+0x160/0x160 [ 2140.132273] [ 2140.135412] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2140.136447] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2140.137101] ? do_wait_intr+0xf0/0xf0 [ 2140.137235] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2140.140940] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2140.141819] kthread+0x1d1/0x200 [ 2140.143912] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 [ 2140.145692] ? set_kthread_struct+0x70/0x70 [ 2140.146948] ORIG_RAX: ffffffffffffff13 [ 2140.148310] ret_from_fork+0x1f/0x30 [ 2140.151324] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2140.239163] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2140.245302] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2140.248366] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2140.251861] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2140.255418] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2140.260233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2140.263023] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2140.266491] Call Trace: [ 2140.267816] [ 2140.268608] ? show_regs.cold.9+0x22/0x2f [ 2140.270013] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2140.272479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2140.274692] ? watchdog+0x40/0x40 [ 2140.276405] ? hrtimer_interrupt+0x128/0x2c0 [ 2140.278970] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2140.281350] ? apic_timer_interrupt+0xf/0x20 [ 2140.284004] [ 2140.285424] ? native_safe_halt+0x2e/0x50 [ 2140.287478] kvm_wait+0xcd/0xe0 [ 2140.291829] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2140.294964] do_raw_spin_lock+0x177/0x190 [ 2140.298308] _raw_spin_lock+0x21/0x30 [ 2140.299943] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2140.302630] process_one_work+0x2c8/0x700 [ 2140.306944] worker_thread+0x296/0x6e0 [ 2140.308894] ? rescuer_thread+0x570/0x570 [ 2140.311677] kthread+0x1d1/0x200 [ 2140.313595] ? set_kthread_struct+0x70/0x70 [ 2140.316907] ret_from_fork+0x1f/0x30 [ 2152.056339] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2152.067223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2152.094290] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2152.105908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2152.109081] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2152.120560] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2152.132079] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2152.136776] RAX: 0000000000007184 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2152.140432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2152.154655] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2152.159600] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2152.166833] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2152.170153] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2152.182709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2152.187953] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2152.196267] Call Trace: [ 2152.199903] [ 2152.201471] ? show_regs.cold.9+0x22/0x2f [ 2152.203898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2152.209754] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2152.212214] ? watchdog+0x40/0x40 [ 2152.213955] ? hrtimer_interrupt+0x128/0x2c0 [ 2152.218281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2152.225393] ? apic_timer_interrupt+0xf/0x20 [ 2152.236170] [ 2152.237818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2152.246752] do_raw_spin_lock+0x177/0x190 [ 2152.254248] _raw_spin_lock+0x21/0x30 [ 2152.256590] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2152.262163] ? do_raw_spin_unlock+0x75/0x190 [ 2152.272344] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2152.282121] ? do_wait_intr+0xf0/0xf0 [ 2152.288907] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2152.293560] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2152.295608] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2152.297817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2152.299704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2152.301830] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2152.303224] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2152.304608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2152.306221] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2152.308423] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2152.310022] lmv_intent_open+0x338/0x1150 [lmv] [ 2152.311353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2152.313166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2152.314556] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2152.315885] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2152.317220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2152.318537] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2152.319854] ? do_raw_spin_unlock+0x75/0x190 [ 2152.320903] ? _raw_spin_unlock+0x12/0x30 [ 2152.322084] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2152.323494] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2152.324830] ll_atomic_open+0x36a/0xd70 [lustre] [ 2152.326143] lookup_open+0x356/0xb40 [ 2152.327245] do_last+0x313/0xfc0 [ 2152.328083] ? nd_jump_root+0xe5/0x160 [ 2152.328934] ? path_init+0x437/0x520 [ 2152.329948] path_openat+0xf7/0x500 [ 2152.330870] ? do_raw_spin_unlock+0x75/0x190 [ 2152.331910] do_filp_open+0x99/0x140 [ 2152.332897] ? getname_flags+0x6e/0x330 [ 2152.333881] ? __check_object_size+0xff/0x256 [ 2152.335209] ? do_raw_spin_unlock+0x75/0x190 [ 2152.336369] ? _raw_spin_unlock+0x12/0x30 [ 2152.337361] do_sys_openat2+0x2b4/0x410 [ 2152.338324] do_sys_open+0x73/0xa0 [ 2152.339189] __x64_sys_openat+0x24/0x30 [ 2152.340511] do_syscall_64+0xc1/0x440 [ 2152.342126] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2152.344595] RIP: 0033:0x7fa749552332 [ 2152.345819] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2152.353360] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2152.355723] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2152.357479] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2152.364287] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2152.369218] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2152.372586] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2153.695893] rcu: INFO: rcu_sched self-detected stall on CPU [ 2153.706636] rcu: 2-...!: (140559 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 2153.720928] (t=240006 jiffies g=258701 q=1442) [ 2153.724445] rcu: rcu_sched kthread starved for 236801 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 2153.730249] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2153.746592] rcu: RCU grace-period kthread stack dump: [ 2153.758309] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2153.765864] Call Trace: [ 2153.766619] __schedule+0x351/0xcb0 [ 2153.771963] schedule+0xc0/0x180 [ 2153.774564] schedule_timeout+0xb4/0x190 [ 2153.784061] ? do_raw_spin_unlock+0x75/0x190 [ 2153.787847] ? __next_timer_interrupt+0x160/0x160 [ 2153.797424] ? prepare_to_swait_event+0xf3/0x140 [ 2153.805334] rcu_gp_kthread+0x8dd/0x15b0 [ 2153.807954] ? force_qs_rnp+0x2c0/0x2c0 [ 2153.810140] kthread+0x1d1/0x200 [ 2153.818177] ? set_kthread_struct+0x70/0x70 [ 2153.819732] ret_from_fork+0x1f/0x30 [ 2153.829461] rcu: Stack dump where RCU GP kthread last ran: [ 2153.836765] Sending NMI from CPU 2 to CPUs 3: [ 2153.844359] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.804 msecs [ 2153.848149] NMI backtrace for cpu 3 [ 2153.848157] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2153.848159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2153.848171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2153.848173] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2153.848174] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 [ 2153.848177] RAX: 0000000000003f38 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2153.848178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2153.848179] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2153.848180] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2153.848181] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2153.848183] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2153.848184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2153.848185] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2153.848186] Call Trace: [ 2153.848187] [ 2153.848188] ? show_regs.cold.9+0x22/0x2f [ 2153.848189] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2153.848190] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2153.848191] ? nmi_handle+0xbd/0x210 [ 2153.848192] ? default_do_nmi+0x4b/0x1b0 [ 2153.848193] ? do_nmi+0x280/0x380 [ 2153.848194] ? end_repeat_nmi+0x16/0x1f [ 2153.848195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2153.848196] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2153.848197] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2153.848198] [ 2153.848199] do_raw_spin_lock+0x177/0x190 [ 2153.848200] _raw_spin_lock+0x21/0x30 [ 2153.848201] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2153.848202] ? do_raw_spin_unlock+0x75/0x190 [ 2153.848203] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2153.848204] ? do_wait_intr+0xf0/0xf0 [ 2153.848204] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2153.848205] mdc_close+0x53b/0xd70 [mdc] [ 2153.848206] lmv_close+0x18d/0x490 [lmv] [ 2153.848207] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2153.848208] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2153.848209] ll_md_close+0x274/0x800 [lustre] [ 2153.848210] ll_file_release+0x128/0xc50 [lustre] [ 2153.848211] __fput+0x10d/0x440 [ 2153.848212] ____fput+0x12/0x20 [ 2153.848983] task_work_run+0xc6/0x110 [ 2153.848987] exit_to_usermode_loop+0x1dd/0x1f0 [ 2153.848988] do_syscall_64+0x3d6/0x440 [ 2153.848989] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2153.848990] RIP: 0033:0x7fa749552cd8 [ 2153.848991] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2153.848993] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2153.848996] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2153.848997] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2153.848999] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2153.849000] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2153.849001] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2153.849007] NMI backtrace for cpu 2 [ 2154.233606] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2154.250844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2154.269571] Call Trace: [ 2154.270544] [ 2154.271346] ? dump_stack+0xbb/0x10e [ 2154.280493] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2154.285445] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2154.289859] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2154.292932] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2154.296870] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2154.302949] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2154.312441] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2154.316428] ? update_process_times+0x6d/0xc0 [ 2154.321979] ? tick_sched_handle.isra.18+0x27/0x90 [ 2154.329877] ? tick_sched_timer+0x53/0xb0 [ 2154.337537] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2154.345170] ? tick_sched_handle.isra.18+0x90/0x90 [ 2154.349048] ? hrtimer_interrupt+0x128/0x2c0 [ 2154.351684] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2154.356841] ? apic_timer_interrupt+0xf/0x20 [ 2154.363058] [ 2154.363966] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2154.368622] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2154.376628] ? do_raw_spin_lock+0x177/0x190 [ 2154.383580] ? _raw_spin_lock+0x21/0x30 [ 2154.386126] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2154.392565] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2154.409052] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2154.420755] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2154.423047] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2154.428796] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2154.434037] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2154.437969] ? __next_timer_interrupt+0x160/0x160 [ 2154.441550] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2154.445031] ? do_wait_intr+0xf0/0xf0 [ 2154.449118] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2154.463106] ? kthread+0x1d1/0x200 [ 2154.472089] ? set_kthread_struct+0x70/0x70 [ 2154.475831] ? ret_from_fork+0x1f/0x30 [ 2156.004058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 238s! [ 2156.007782] Showing busy workqueues and worker pools: [ 2156.010538] workqueue events: flags=0x0 [ 2156.012585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2156.016532] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2156.022134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2156.025760] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2156.033377] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.037814] pending: kfree_rcu_monitor [ 2156.040494] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2156.045571] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2156.047960] workqueue events_long: flags=0x0 [ 2156.051505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.055356] pending: lu_objects_flush [obdclass] [ 2156.058088] workqueue events_unbound: flags=0x2 [ 2156.060230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2156.063174] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2156.067603] workqueue events_power_efficient: flags=0x80 [ 2156.070204] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.073746] pending: check_lifetime [ 2156.075700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.079331] pending: neigh_periodic_work [ 2156.081413] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.085163] pending: neigh_periodic_work [ 2156.087417] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.091088] pending: do_cache_clean [ 2156.093361] workqueue mm_percpu_wq: flags=0x8 [ 2156.095991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.100005] pending: vmstat_update [ 2156.102026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.105911] pending: vmstat_update [ 2156.107912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2156.111340] pending: vmstat_update [ 2156.113263] workqueue writeback: flags=0x4a [ 2156.114996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2156.118170] pending: wb_workfn, wb_workfn [ 2156.120304] workqueue xprtiod: flags=0xa [ 2156.121742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2156.124645] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2156.127440] workqueue ipv6_addrconf: flags=0x40008 [ 2156.129890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2156.133319] pending: addrconf_verify_work [ 2156.135543] workqueue knbd0-recv: flags=0x1a [ 2156.137424] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2156.140621] in-flight: 651:recv_work [ 2156.143374] workqueue ptlrpc_pinger: flags=0x2 [ 2156.145887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2156.148602] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2156.151550] pool 8: cpus=0-3 flags=0x4 nice=0 hung=238s workers=4 idle: 37 31846 28897 [ 2156.155354] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2146s workers=2 idle: 653 [ 2168.062390] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2168.074572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2168.144754] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2168.154141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2168.159524] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2168.164354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2168.168083] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2168.179662] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2168.185456] RAX: 0000000000000715 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2168.196771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2168.206538] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2168.210343] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2168.217549] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2168.226378] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2168.233495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2168.241464] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2168.252985] Call Trace: [ 2168.256611] [ 2168.260065] ? show_regs.cold.9+0x22/0x2f [ 2168.265629] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2168.270576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2168.272739] ? watchdog+0x40/0x40 [ 2168.274559] ? hrtimer_interrupt+0x128/0x2c0 [ 2168.276363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2168.278537] ? apic_timer_interrupt+0xf/0x20 [ 2168.279940] [ 2168.280628] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2168.283367] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2168.286208] do_raw_spin_lock+0x177/0x190 [ 2168.287895] _raw_spin_lock+0x21/0x30 [ 2168.289561] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2168.292544] process_one_work+0x2c8/0x700 [ 2168.294429] worker_thread+0x296/0x6e0 [ 2168.296284] ? rescuer_thread+0x570/0x570 [ 2168.298333] kthread+0x1d1/0x200 [ 2168.299866] ? set_kthread_struct+0x70/0x70 [ 2168.301855] ret_from_fork+0x1f/0x30 [ 2180.058792] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2180.062123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2180.062946] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2180.114444] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2180.114458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2180.114461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2180.114476] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2180.114480] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2180.114485] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2180.114489] RDX: 000000000000289f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2180.114491] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2180.114493] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2180.114496] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2180.114498] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2180.114501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.114503] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2180.114511] Call Trace: [ 2180.114517] [ 2180.114522] ? show_regs.cold.9+0x22/0x2f [ 2180.114532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2180.114538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2180.114542] ? watchdog+0x40/0x40 [ 2180.114546] ? hrtimer_interrupt+0x128/0x2c0 [ 2180.114552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2180.114559] ? apic_timer_interrupt+0xf/0x20 [ 2180.114563] [ 2180.114566] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2180.114570] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2180.114575] do_raw_spin_lock+0x177/0x190 [ 2180.114579] _raw_spin_lock+0x21/0x30 [ 2180.114586] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2180.116067] obd_mod_free+0x46/0x140 [ptlrpc] [ 2180.116271] mdc_commit_open+0x91/0xb0 [mdc] [ 2180.116308] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2180.116526] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2180.116722] after_reply+0xe2c/0x1780 [ptlrpc] [ 2180.116930] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2180.117124] ? __next_timer_interrupt+0x160/0x160 [ 2180.117133] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2180.117335] ? do_wait_intr+0xf0/0xf0 [ 2180.117344] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2180.121259] Modules linked in: [ 2180.137996] kthread+0x1d1/0x200 [ 2180.138016] ? set_kthread_struct+0x70/0x70 [ 2180.138022] ret_from_fork+0x1f/0x30 [ 2180.351438] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2180.391270] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2180.404935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2180.417303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2180.430357] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2180.448933] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2180.454268] RAX: 0000000000006c8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2180.458994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2180.464424] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2180.471612] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2180.477778] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2180.482071] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2180.485370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.488937] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2180.495492] Call Trace: [ 2180.496352] [ 2180.497074] ? show_regs.cold.9+0x22/0x2f [ 2180.500330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2180.502133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2180.504825] ? watchdog+0x40/0x40 [ 2180.505464] ? hrtimer_interrupt+0x128/0x2c0 [ 2180.506845] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2180.508280] ? apic_timer_interrupt+0xf/0x20 [ 2180.509773] [ 2180.510462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2180.525218] do_raw_spin_lock+0x177/0x190 [ 2180.534592] _raw_spin_lock+0x21/0x30 [ 2180.535726] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2180.542943] ? do_raw_spin_unlock+0x75/0x190 [ 2180.549568] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2180.551954] ? do_wait_intr+0xf0/0xf0 [ 2180.554187] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2180.556532] mdc_close+0x53b/0xd70 [mdc] [ 2180.558332] lmv_close+0x18d/0x490 [lmv] [ 2180.559977] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2180.562564] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2180.564971] ll_md_close+0x274/0x800 [lustre] [ 2180.566964] ll_file_release+0x128/0xc50 [lustre] [ 2180.569597] __fput+0x10d/0x440 [ 2180.570904] ____fput+0x12/0x20 [ 2180.576389] task_work_run+0xc6/0x110 [ 2180.578148] exit_to_usermode_loop+0x1dd/0x1f0 [ 2180.580267] do_syscall_64+0x3d6/0x440 [ 2180.581968] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2180.584319] RIP: 0033:0x7fa749552cd8 [ 2180.585947] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2180.594942] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2180.598416] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2180.603186] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2180.612435] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2180.619617] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2180.625692] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2184.054060] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2184.066068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2184.109454] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2184.119052] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2184.127584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2184.132144] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2184.149483] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2184.153057] RAX: 000000000000128e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2184.159179] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2184.164979] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2184.171009] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2184.177192] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2184.185789] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2184.193220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2184.198283] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2184.204975] Call Trace: [ 2184.207423] [ 2184.208428] ? show_regs.cold.9+0x22/0x2f [ 2184.211371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2184.215364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2184.217779] ? watchdog+0x40/0x40 [ 2184.219603] ? hrtimer_interrupt+0x128/0x2c0 [ 2184.222159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2184.227251] ? apic_timer_interrupt+0xf/0x20 [ 2184.230185] [ 2184.231371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2184.234381] do_raw_spin_lock+0x177/0x190 [ 2184.238380] _raw_spin_lock+0x21/0x30 [ 2184.241197] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2184.246807] ? do_raw_spin_unlock+0x75/0x190 [ 2184.249539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2184.254853] ? do_wait_intr+0xf0/0xf0 [ 2184.256254] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2184.261258] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2184.267869] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2184.271368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.275461] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2184.280810] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2184.284248] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2184.288058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.291467] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2184.295324] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2184.302054] lmv_intent_open+0x338/0x1150 [lmv] [ 2184.305724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.309792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.315777] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2184.319490] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2184.324160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.329147] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2184.331185] ? do_raw_spin_unlock+0x75/0x190 [ 2184.334334] ? _raw_spin_unlock+0x12/0x30 [ 2184.338751] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2184.341560] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2184.344115] ll_atomic_open+0x36a/0xd70 [lustre] [ 2184.347982] lookup_open+0x356/0xb40 [ 2184.351835] do_last+0x313/0xfc0 [ 2184.353352] ? nd_jump_root+0xe5/0x160 [ 2184.356192] ? path_init+0x437/0x520 [ 2184.360993] path_openat+0xf7/0x500 [ 2184.362406] ? do_raw_spin_unlock+0x75/0x190 [ 2184.365088] do_filp_open+0x99/0x140 [ 2184.366836] ? getname_flags+0x6e/0x330 [ 2184.370011] ? __check_object_size+0xff/0x256 [ 2184.372034] ? do_raw_spin_unlock+0x75/0x190 [ 2184.373502] ? _raw_spin_unlock+0x12/0x30 [ 2184.374741] do_sys_openat2+0x2b4/0x410 [ 2184.377732] do_sys_open+0x73/0xa0 [ 2184.379359] __x64_sys_openat+0x24/0x30 [ 2184.381759] do_syscall_64+0xc1/0x440 [ 2184.383835] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2184.387259] RIP: 0033:0x7fa749552332 [ 2184.389168] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2184.401221] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2184.404725] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2184.408268] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2184.412382] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2184.417889] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2184.421856] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2186.209627] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 269s! [ 2186.213649] Showing busy workqueues and worker pools: [ 2186.220310] workqueue events: flags=0x0 [ 2186.221898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2186.226337] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2186.234146] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2186.240251] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2186.244832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.249982] pending: kfree_rcu_monitor [ 2186.253012] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2186.262179] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2186.267796] workqueue events_long: flags=0x0 [ 2186.272578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.276156] pending: lu_objects_flush [obdclass] [ 2186.279915] workqueue events_unbound: flags=0x2 [ 2186.283797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2186.287251] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2186.292705] workqueue events_power_efficient: flags=0x80 [ 2186.295252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.299206] pending: check_lifetime [ 2186.302183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.305675] pending: neigh_periodic_work [ 2186.311437] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.315158] pending: neigh_periodic_work [ 2186.317830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.323759] pending: do_cache_clean [ 2186.325570] workqueue mm_percpu_wq: flags=0x8 [ 2186.329545] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.346044] pending: vmstat_update [ 2186.352058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.359112] pending: vmstat_update [ 2186.364068] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2186.374651] pending: vmstat_update [ 2186.376037] workqueue writeback: flags=0x4a [ 2186.377560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2186.385192] pending: wb_workfn, wb_workfn [ 2186.387960] workqueue xprtiod: flags=0xa [ 2186.391810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2186.396687] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2186.402201] workqueue ipv6_addrconf: flags=0x40008 [ 2186.411620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2186.424597] pending: addrconf_verify_work [ 2186.432909] workqueue knbd0-recv: flags=0x1a [ 2186.434392] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2186.444983] in-flight: 651:recv_work [ 2186.447126] workqueue ptlrpc_pinger: flags=0x2 [ 2186.452854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2186.461703] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2186.469762] pool 8: cpus=0-3 flags=0x4 nice=0 hung=269s workers=4 idle: 37 31846 28897 [ 2186.481517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2176s workers=2 idle: 653 [ 2196.064199] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2196.081146] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2196.144010] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2196.154640] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2196.164148] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2196.167154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2196.175626] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2196.195932] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2196.205698] RAX: 0000000000007e07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2196.236233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2196.248026] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2196.259721] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2196.281548] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2196.311222] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2196.333496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2196.352496] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2196.373814] Call Trace: [ 2196.374661] [ 2196.375318] ? show_regs.cold.9+0x22/0x2f [ 2196.383444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2196.388586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2196.396003] ? watchdog+0x40/0x40 [ 2196.404949] ? hrtimer_interrupt+0x128/0x2c0 [ 2196.410612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2196.412312] ? apic_timer_interrupt+0xf/0x20 [ 2196.432063] [ 2196.439298] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2196.448750] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2196.459454] do_raw_spin_lock+0x177/0x190 [ 2196.463312] _raw_spin_lock+0x21/0x30 [ 2196.475547] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2196.481634] process_one_work+0x2c8/0x700 [ 2196.483008] worker_thread+0x296/0x6e0 [ 2196.493017] ? rescuer_thread+0x570/0x570 [ 2196.497655] kthread+0x1d1/0x200 [ 2196.507511] ? set_kthread_struct+0x70/0x70 [ 2196.509012] ret_from_fork+0x1f/0x30 [ 2208.058223] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2208.059610] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2208.061450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2208.061531] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2208.061539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2208.061542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2208.061556] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2208.061560] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2208.061565] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2208.061567] RDX: 00000000000034b3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2208.061569] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2208.061571] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2208.061574] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2208.061577] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2208.061580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.061582] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2208.061590] Call Trace: [ 2208.061596] [ 2208.061602] ? show_regs.cold.9+0x22/0x2f [ 2208.061611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2208.061617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2208.061621] ? watchdog+0x40/0x40 [ 2208.061626] ? hrtimer_interrupt+0x128/0x2c0 [ 2208.061631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2208.061639] ? apic_timer_interrupt+0xf/0x20 [ 2208.061643] [ 2208.061646] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2208.061652] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2208.061656] do_raw_spin_lock+0x177/0x190 [ 2208.061661] _raw_spin_lock+0x21/0x30 [ 2208.061667] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2208.063253] obd_mod_free+0x46/0x140 [ptlrpc] [ 2208.064370] mdc_commit_open+0x91/0xb0 [mdc] [ 2208.064409] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2208.087042] Modules linked in: [ 2208.109156] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2208.109528] after_reply+0xe2c/0x1780 [ptlrpc] [ 2208.109744] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2208.122141] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2208.122220] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2208.122228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2208.122231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.122245] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2208.122249] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2208.122254] RAX: 0000000000006aaa RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2208.122257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2208.122259] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2208.122261] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2208.122264] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2208.122267] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2208.122282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.122285] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2208.122292] Call Trace: [ 2208.122298] [ 2208.122304] ? show_regs.cold.9+0x22/0x2f [ 2208.122313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2208.122321] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2208.122390] ? watchdog+0x40/0x40 [ 2208.122395] ? hrtimer_interrupt+0x128/0x2c0 [ 2208.122399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2208.122406] ? apic_timer_interrupt+0xf/0x20 [ 2208.122411] [ 2208.122414] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.122421] do_raw_spin_lock+0x177/0x190 [ 2208.122425] _raw_spin_lock+0x21/0x30 [ 2208.122429] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2208.122736] ? do_raw_spin_unlock+0x75/0x190 [ 2208.122742] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2208.124115] ? do_wait_intr+0xf0/0xf0 [ 2208.124125] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2208.124420] mdc_close+0x53b/0xd70 [mdc] [ 2208.124458] lmv_close+0x18d/0x490 [lmv] [ 2208.124489] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2208.124615] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2208.124704] ll_md_close+0x274/0x800 [lustre] [ 2208.124790] ll_file_release+0x128/0xc50 [lustre] [ 2208.124877] __fput+0x10d/0x440 [ 2208.124886] ____fput+0x12/0x20 [ 2208.124891] task_work_run+0xc6/0x110 [ 2208.124898] exit_to_usermode_loop+0x1dd/0x1f0 [ 2208.124905] do_syscall_64+0x3d6/0x440 [ 2208.124910] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2208.125123] RIP: 0033:0x7fa749552cd8 [ 2208.125131] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2208.125135] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2208.125140] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2208.125142] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2208.125145] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2208.125147] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2208.125149] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2208.462079] ? __next_timer_interrupt+0x160/0x160 [ 2208.465833] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2208.469811] ? do_wait_intr+0xf0/0xf0 [ 2208.471607] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2208.475513] kthread+0x1d1/0x200 [ 2208.477570] ? set_kthread_struct+0x70/0x70 [ 2208.480674] ret_from_fork+0x1f/0x30 [ 2212.054015] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2212.057102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2212.078665] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2212.084420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2212.088307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2212.091199] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2212.099083] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2212.102632] RAX: 0000000000005b1b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2212.106071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2212.109391] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2212.112295] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2212.115266] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2212.118581] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2212.121965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2212.124886] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2212.127861] Call Trace: [ 2212.128873] [ 2212.129724] ? show_regs.cold.9+0x22/0x2f [ 2212.131577] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2212.133995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2212.136211] ? watchdog+0x40/0x40 [ 2212.138501] ? hrtimer_interrupt+0x128/0x2c0 [ 2212.140534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2212.142629] ? apic_timer_interrupt+0xf/0x20 [ 2212.144668] [ 2212.145822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2212.148423] do_raw_spin_lock+0x177/0x190 [ 2212.150076] _raw_spin_lock+0x21/0x30 [ 2212.151830] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2212.155034] ? do_raw_spin_unlock+0x75/0x190 [ 2212.157050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2212.159780] ? do_wait_intr+0xf0/0xf0 [ 2212.161294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2212.163692] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2212.166198] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2212.169001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.171484] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2212.174227] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2212.176388] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2212.178424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.180645] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2212.183456] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2212.185811] lmv_intent_open+0x338/0x1150 [lmv] [ 2212.187208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.189215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.191103] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2212.193068] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2212.195175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.197682] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2212.199570] ? do_raw_spin_unlock+0x75/0x190 [ 2212.201342] ? _raw_spin_unlock+0x12/0x30 [ 2212.203244] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2212.205502] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2212.208062] ll_atomic_open+0x36a/0xd70 [lustre] [ 2212.210740] lookup_open+0x356/0xb40 [ 2212.212363] do_last+0x313/0xfc0 [ 2212.213692] ? nd_jump_root+0xe5/0x160 [ 2212.215622] ? path_init+0x437/0x520 [ 2212.217417] path_openat+0xf7/0x500 [ 2212.219017] ? do_raw_spin_unlock+0x75/0x190 [ 2212.220862] do_filp_open+0x99/0x140 [ 2212.222744] ? getname_flags+0x6e/0x330 [ 2212.224784] ? __check_object_size+0xff/0x256 [ 2212.227347] ? do_raw_spin_unlock+0x75/0x190 [ 2212.229604] ? _raw_spin_unlock+0x12/0x30 [ 2212.231305] do_sys_openat2+0x2b4/0x410 [ 2212.233263] do_sys_open+0x73/0xa0 [ 2212.235222] __x64_sys_openat+0x24/0x30 [ 2212.237338] do_syscall_64+0xc1/0x440 [ 2212.239342] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2212.241883] RIP: 0033:0x7fa749552332 [ 2212.244161] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2212.254695] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2212.260057] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2212.263656] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2212.267714] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2212.271427] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2212.275408] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2216.928162] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 299s! [ 2216.932186] Showing busy workqueues and worker pools: [ 2216.934850] workqueue events: flags=0x0 [ 2216.936949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2216.940513] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2216.946403] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2216.949870] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2216.955033] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2216.958766] pending: kfree_rcu_monitor [ 2216.961127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2216.964246] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2216.967566] workqueue events_long: flags=0x0 [ 2216.969879] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2216.974564] pending: lu_objects_flush [obdclass] [ 2216.977309] workqueue events_unbound: flags=0x2 [ 2216.980285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2216.984124] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2216.989137] workqueue events_power_efficient: flags=0x80 [ 2216.992186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2216.995257] pending: check_lifetime [ 2216.997358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2217.000653] pending: neigh_periodic_work [ 2217.002919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2217.006453] pending: neigh_periodic_work [ 2217.008586] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2217.012028] pending: do_cache_clean [ 2217.013804] workqueue mm_percpu_wq: flags=0x8 [ 2217.015976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2217.019466] pending: vmstat_update [ 2217.021322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2217.024732] pending: vmstat_update [ 2217.026696] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2217.029408] pending: vmstat_update [ 2217.031020] workqueue writeback: flags=0x4a [ 2217.032227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2217.035378] pending: wb_workfn, wb_workfn [ 2217.037433] workqueue xprtiod: flags=0xa [ 2217.039429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2217.042055] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2217.045218] workqueue ipv6_addrconf: flags=0x40008 [ 2217.047484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2217.051348] pending: addrconf_verify_work [ 2217.053628] workqueue knbd0-recv: flags=0x1a [ 2217.056152] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2217.060124] in-flight: 651:recv_work [ 2217.062439] workqueue ptlrpc_pinger: flags=0x2 [ 2217.064852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2217.068165] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2217.072473] pool 8: cpus=0-3 flags=0x4 nice=0 hung=299s workers=4 idle: 37 31846 28897 [ 2217.078567] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2206s workers=2 idle: 653 [ 2224.059230] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2224.067316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2224.139927] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2224.153207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2224.168032] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2224.171591] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2224.175955] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2224.189883] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2224.195400] RAX: 0000000000007f05 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2224.203106] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2224.207098] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2224.210843] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2224.216666] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2224.218363] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2224.223438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2224.233440] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2224.238442] Call Trace: [ 2224.239642] [ 2224.241083] ? show_regs.cold.9+0x22/0x2f [ 2224.243230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2224.245164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2224.249345] ? watchdog+0x40/0x40 [ 2224.251416] ? hrtimer_interrupt+0x128/0x2c0 [ 2224.255588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2224.260318] ? apic_timer_interrupt+0xf/0x20 [ 2224.263526] [ 2224.264775] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2224.269641] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2224.275900] do_raw_spin_lock+0x177/0x190 [ 2224.278273] _raw_spin_lock+0x21/0x30 [ 2224.282841] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2224.285279] process_one_work+0x2c8/0x700 [ 2224.286670] worker_thread+0x296/0x6e0 [ 2224.290329] ? rescuer_thread+0x570/0x570 [ 2224.295255] kthread+0x1d1/0x200 [ 2224.299239] ? set_kthread_struct+0x70/0x70 [ 2224.302293] ret_from_fork+0x1f/0x30 [ 2236.058877] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2236.059062] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2236.062186] Modules linked in: [ 2236.065213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2236.065392] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2236.065399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2236.065402] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2236.065413] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2236.065418] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2236.065422] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2236.065425] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2236.065427] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2236.065429] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2236.065431] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2236.065435] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2236.065437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.065440] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2236.065447] Call Trace: [ 2236.065451] [ 2236.065457] ? show_regs.cold.9+0x22/0x2f [ 2236.065465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2236.065472] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2236.065477] ? watchdog+0x40/0x40 [ 2236.065481] ? hrtimer_interrupt+0x128/0x2c0 [ 2236.065485] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2236.065491] ? apic_timer_interrupt+0xf/0x20 [ 2236.066960] lustre(O) [ 2236.088478] [ 2236.088490] ? native_safe_halt+0x2e/0x50 [ 2236.088562] kvm_wait+0xcd/0xe0 [ 2236.088571] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2236.088581] do_raw_spin_lock+0x177/0x190 [ 2236.088586] _raw_spin_lock+0x21/0x30 [ 2236.088591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2236.089082] ? do_raw_spin_unlock+0x75/0x190 [ 2236.089089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2236.089294] ? do_wait_intr+0xf0/0xf0 [ 2236.089301] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2236.089655] mdc_close+0x53b/0xd70 [mdc] [ 2236.089697] lmv_close+0x18d/0x490 [lmv] [ 2236.089727] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2236.089850] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2236.094696] osp(O) [ 2236.098660] ll_md_close+0x274/0x800 [lustre] [ 2236.098791] ll_file_release+0x128/0xc50 [lustre] [ 2236.099120] __fput+0x10d/0x440 [ 2236.099129] ____fput+0x12/0x20 [ 2236.099133] task_work_run+0xc6/0x110 [ 2236.099141] exit_to_usermode_loop+0x1dd/0x1f0 [ 2236.099148] do_syscall_64+0x3d6/0x440 [ 2236.099153] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2236.099161] RIP: 0033:0x7fa749552cd8 [ 2236.099167] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2236.099171] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2236.099176] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2236.099178] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2236.099180] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2236.099182] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2236.099184] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2236.344120] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2236.384038] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2236.396500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2236.400715] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2236.403976] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2236.423742] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2236.434018] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2236.440658] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2236.448469] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2236.455915] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2236.462969] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2236.471952] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2236.481080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.493321] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2236.507552] Call Trace: [ 2236.509897] [ 2236.513882] ? show_regs.cold.9+0x22/0x2f [ 2236.521086] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2236.524630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2236.527016] ? watchdog+0x40/0x40 [ 2236.529135] ? hrtimer_interrupt+0x128/0x2c0 [ 2236.531229] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2236.534612] ? apic_timer_interrupt+0xf/0x20 [ 2236.538154] [ 2236.540647] ? native_safe_halt+0x2e/0x50 [ 2236.542754] kvm_wait+0xcd/0xe0 [ 2236.544880] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2236.548576] do_raw_spin_lock+0x177/0x190 [ 2236.550872] _raw_spin_lock+0x21/0x30 [ 2236.553160] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2236.557175] obd_mod_free+0x46/0x140 [ptlrpc] [ 2236.560178] mdc_commit_open+0x91/0xb0 [mdc] [ 2236.563934] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2236.571645] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2236.573837] after_reply+0xe2c/0x1780 [ptlrpc] [ 2236.586702] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2236.594852] ? __next_timer_interrupt+0x160/0x160 [ 2236.596409] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2236.607753] ? do_wait_intr+0xf0/0xf0 [ 2236.615695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2236.621550] kthread+0x1d1/0x200 [ 2236.623050] ? set_kthread_struct+0x70/0x70 [ 2236.631225] ret_from_fork+0x1f/0x30 [ 2244.054097] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2244.062193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2244.099680] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2244.110297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2244.121743] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2244.123907] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2244.132238] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2244.138183] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2244.145295] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2244.150896] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2244.154634] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2244.161927] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2244.170174] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2244.174690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2244.181102] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2244.185665] Call Trace: [ 2244.187803] [ 2244.188701] ? show_regs.cold.9+0x22/0x2f [ 2244.192336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2244.198297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2244.201241] ? watchdog+0x40/0x40 [ 2244.203154] ? hrtimer_interrupt+0x128/0x2c0 [ 2244.205935] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2244.210771] ? apic_timer_interrupt+0xf/0x20 [ 2244.212928] [ 2244.213901] ? native_safe_halt+0x2e/0x50 [ 2244.216779] kvm_wait+0xcd/0xe0 [ 2244.217969] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2244.221247] do_raw_spin_lock+0x177/0x190 [ 2244.225699] _raw_spin_lock+0x21/0x30 [ 2244.227243] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2244.231229] ? do_raw_spin_unlock+0x75/0x190 [ 2244.236213] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2244.242318] ? do_wait_intr+0xf0/0xf0 [ 2244.250380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2244.258542] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2244.262575] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2244.269709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2244.273189] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2244.277764] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2244.281784] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2244.285237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2244.290769] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2244.295229] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2244.302143] lmv_intent_open+0x338/0x1150 [lmv] [ 2244.304983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2244.308218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2244.311792] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2244.313916] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2244.316182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2244.324513] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2244.325985] ? do_raw_spin_unlock+0x75/0x190 [ 2244.327141] ? _raw_spin_unlock+0x12/0x30 [ 2244.334126] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2244.338730] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2244.342874] ll_atomic_open+0x36a/0xd70 [lustre] [ 2244.346814] lookup_open+0x356/0xb40 [ 2244.349212] do_last+0x313/0xfc0 [ 2244.352310] ? nd_jump_root+0xe5/0x160 [ 2244.354725] ? path_init+0x437/0x520 [ 2244.356330] path_openat+0xf7/0x500 [ 2244.358284] ? do_raw_spin_unlock+0x75/0x190 [ 2244.360945] do_filp_open+0x99/0x140 [ 2244.362347] ? getname_flags+0x6e/0x330 [ 2244.364297] ? __check_object_size+0xff/0x256 [ 2244.366409] ? do_raw_spin_unlock+0x75/0x190 [ 2244.369299] ? _raw_spin_unlock+0x12/0x30 [ 2244.372245] do_sys_openat2+0x2b4/0x410 [ 2244.379096] do_sys_open+0x73/0xa0 [ 2244.383804] __x64_sys_openat+0x24/0x30 [ 2244.386939] do_syscall_64+0xc1/0x440 [ 2244.390235] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2244.403885] RIP: 0033:0x7fa749552332 [ 2244.406200] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2244.423969] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2244.428344] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2244.433200] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2244.436025] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2244.440746] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2244.446246] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2247.140069] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 329s! [ 2247.143995] Showing busy workqueues and worker pools: [ 2247.147212] workqueue events: flags=0x0 [ 2247.151971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2247.158543] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2247.169613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2247.174808] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2247.182406] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.185012] pending: kfree_rcu_monitor [ 2247.188382] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2247.196467] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2247.201850] workqueue events_long: flags=0x0 [ 2247.204915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.211003] pending: lu_objects_flush [obdclass] [ 2247.219407] workqueue events_unbound: flags=0x2 [ 2247.222899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2247.231824] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2247.244938] workqueue events_power_efficient: flags=0x80 [ 2247.246948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.251494] pending: check_lifetime [ 2247.254488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.257773] pending: neigh_periodic_work [ 2247.259329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.263419] pending: neigh_periodic_work [ 2247.265512] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.268550] pending: do_cache_clean [ 2247.271789] workqueue mm_percpu_wq: flags=0x8 [ 2247.275196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.280674] pending: vmstat_update [ 2247.283706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.287533] pending: vmstat_update [ 2247.289000] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2247.291656] pending: vmstat_update [ 2247.292968] workqueue writeback: flags=0x4a [ 2247.295354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2247.296902] pending: wb_workfn, wb_workfn [ 2247.298762] workqueue xprtiod: flags=0xa [ 2247.300493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2247.308022] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2247.311731] workqueue ipv6_addrconf: flags=0x40008 [ 2247.321182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2247.334869] pending: addrconf_verify_work [ 2247.342889] workqueue knbd0-recv: flags=0x1a [ 2247.347893] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2247.357567] in-flight: 651:recv_work [ 2247.361406] workqueue ptlrpc_pinger: flags=0x2 [ 2247.364512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2247.368165] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2247.380407] pool 8: cpus=0-3 flags=0x4 nice=0 hung=330s workers=4 idle: 37 31846 28897 [ 2247.382874] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2237s workers=2 idle: 653 [ 2252.057159] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2252.064553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2252.118546] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2252.125415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2252.129604] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2252.132718] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2252.135814] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2252.145583] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2252.153112] RAX: 000000000000778e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2252.154984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2252.159852] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2252.163044] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2252.166610] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2252.169669] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2252.173080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2252.180912] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2252.184432] Call Trace: [ 2252.185774] [ 2252.186964] ? show_regs.cold.9+0x22/0x2f [ 2252.188821] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2252.194148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2252.201141] ? watchdog+0x40/0x40 [ 2252.208661] ? hrtimer_interrupt+0x128/0x2c0 [ 2252.210207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2252.211551] ? apic_timer_interrupt+0xf/0x20 [ 2252.212884] [ 2252.213959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2252.215813] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2252.217968] do_raw_spin_lock+0x177/0x190 [ 2252.219281] _raw_spin_lock+0x21/0x30 [ 2252.220440] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2252.223389] process_one_work+0x2c8/0x700 [ 2252.224899] worker_thread+0x296/0x6e0 [ 2252.226322] ? rescuer_thread+0x570/0x570 [ 2252.227783] kthread+0x1d1/0x200 [ 2252.229041] ? set_kthread_struct+0x70/0x70 [ 2252.230408] ret_from_fork+0x1f/0x30 [ 2264.058274] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2264.059113] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2264.062152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2264.062235] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2264.062243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2264.062246] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2264.062261] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2264.062266] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2264.062272] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2264.062275] RDX: 00000000000012ed RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2264.062278] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2264.062280] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2264.062282] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2264.062285] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2264.062288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2264.062302] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2264.062310] Call Trace: [ 2264.062315] [ 2264.062321] ? show_regs.cold.9+0x22/0x2f [ 2264.062330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2264.062336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2264.062340] ? watchdog+0x40/0x40 [ 2264.062343] ? hrtimer_interrupt+0x128/0x2c0 [ 2264.062347] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2264.062354] ? apic_timer_interrupt+0xf/0x20 [ 2264.062358] [ 2264.062360] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2264.062364] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2264.062368] do_raw_spin_lock+0x177/0x190 [ 2264.062372] _raw_spin_lock+0x21/0x30 [ 2264.062377] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2264.062654] obd_mod_free+0x46/0x140 [ptlrpc] [ 2264.062848] mdc_commit_open+0x91/0xb0 [mdc] [ 2264.062884] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2264.063079] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2264.063272] after_reply+0xe2c/0x1780 [ptlrpc] [ 2264.063472] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2264.063666] ? __next_timer_interrupt+0x160/0x160 [ 2264.063677] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2264.063884] ? do_wait_intr+0xf0/0xf0 [ 2264.063892] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2264.064092] kthread+0x1d1/0x200 [ 2264.064099] ? set_kthread_struct+0x70/0x70 [ 2264.064102] ret_from_fork+0x1f/0x30 [ 2264.324389] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2264.355881] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2264.362533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2264.368196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2264.370951] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2264.382641] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2264.388800] RAX: 0000000000007e93 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2264.392730] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2264.399529] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2264.405696] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2264.409221] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2264.412830] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2264.416315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2264.419802] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2264.425349] Call Trace: [ 2264.426602] [ 2264.432348] ? show_regs.cold.9+0x22/0x2f [ 2264.434390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2264.438260] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2264.439881] ? watchdog+0x40/0x40 [ 2264.442948] ? hrtimer_interrupt+0x128/0x2c0 [ 2264.445724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2264.449160] ? apic_timer_interrupt+0xf/0x20 [ 2264.452376] [ 2264.453507] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2264.457394] do_raw_spin_lock+0x177/0x190 [ 2264.459098] _raw_spin_lock+0x21/0x30 [ 2264.461265] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2264.463800] ? do_raw_spin_unlock+0x75/0x190 [ 2264.469090] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2264.472548] ? do_wait_intr+0xf0/0xf0 [ 2264.475420] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2264.478593] mdc_close+0x53b/0xd70 [mdc] [ 2264.479862] lmv_close+0x18d/0x490 [lmv] [ 2264.483650] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2264.486439] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2264.490498] ll_md_close+0x274/0x800 [lustre] [ 2264.492139] ll_file_release+0x128/0xc50 [lustre] [ 2264.498203] __fput+0x10d/0x440 [ 2264.501645] ____fput+0x12/0x20 [ 2264.504489] task_work_run+0xc6/0x110 [ 2264.507786] exit_to_usermode_loop+0x1dd/0x1f0 [ 2264.510816] do_syscall_64+0x3d6/0x440 [ 2264.512610] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2264.514965] RIP: 0033:0x7fa749552cd8 [ 2264.521754] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2264.544977] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2264.551200] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2264.556455] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2264.561767] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2264.567373] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2264.571882] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2272.054020] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2272.058392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2272.082168] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2272.088206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2272.094496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2272.098482] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2272.111707] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2272.119494] RAX: 000000000000229d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2272.130301] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2272.137816] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2272.146418] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2272.153823] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2272.159534] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2272.167030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2272.171947] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2272.175700] Call Trace: [ 2272.176934] [ 2272.177926] ? show_regs.cold.9+0x22/0x2f [ 2272.180217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2272.182678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2272.185079] ? watchdog+0x40/0x40 [ 2272.187055] ? hrtimer_interrupt+0x128/0x2c0 [ 2272.189335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2272.191655] ? apic_timer_interrupt+0xf/0x20 [ 2272.194317] [ 2272.195590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2272.199035] do_raw_spin_lock+0x177/0x190 [ 2272.205406] _raw_spin_lock+0x21/0x30 [ 2272.208375] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2272.213474] ? do_raw_spin_unlock+0x75/0x190 [ 2272.215542] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2272.218667] ? do_wait_intr+0xf0/0xf0 [ 2272.220244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2272.224834] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2272.235325] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2272.248314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2272.262198] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2272.270560] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2272.275223] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2272.282315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2272.289313] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2272.294644] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2272.302301] lmv_intent_open+0x338/0x1150 [lmv] [ 2272.307201] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2272.322038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2272.325724] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2272.327362] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2272.334648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2272.343575] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2272.350064] ? do_raw_spin_unlock+0x75/0x190 [ 2272.354978] ? _raw_spin_unlock+0x12/0x30 [ 2272.362642] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2272.367295] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2272.375461] ll_atomic_open+0x36a/0xd70 [lustre] [ 2272.378326] lookup_open+0x356/0xb40 [ 2272.383681] do_last+0x313/0xfc0 [ 2272.388470] ? nd_jump_root+0xe5/0x160 [ 2272.390560] ? path_init+0x437/0x520 [ 2272.398385] path_openat+0xf7/0x500 [ 2272.400237] ? do_raw_spin_unlock+0x75/0x190 [ 2272.403835] do_filp_open+0x99/0x140 [ 2272.411250] ? getname_flags+0x6e/0x330 [ 2272.417811] ? __check_object_size+0xff/0x256 [ 2272.423252] ? do_raw_spin_unlock+0x75/0x190 [ 2272.430459] ? _raw_spin_unlock+0x12/0x30 [ 2272.437483] do_sys_openat2+0x2b4/0x410 [ 2272.441744] do_sys_open+0x73/0xa0 [ 2272.446621] __x64_sys_openat+0x24/0x30 [ 2272.453281] do_syscall_64+0xc1/0x440 [ 2272.458020] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2272.464310] RIP: 0033:0x7fa749552332 [ 2272.468592] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2272.498922] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2272.504209] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2272.520283] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2272.529218] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2272.536925] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2272.542573] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2277.860853] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 360s! [ 2277.865407] Showing busy workqueues and worker pools: [ 2277.877219] workqueue events: flags=0x0 [ 2277.880848] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2277.890891] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2277.902242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2277.905476] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2277.914520] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2277.920543] pending: kfree_rcu_monitor [ 2277.922912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2277.929003] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2277.939479] workqueue events_long: flags=0x0 [ 2277.946278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2277.952975] pending: lu_objects_flush [obdclass] [ 2277.968823] workqueue events_unbound: flags=0x2 [ 2277.971371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2277.973721] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2277.982373] workqueue events_power_efficient: flags=0x80 [ 2277.987510] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2277.992306] pending: check_lifetime [ 2277.994184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2277.998319] pending: neigh_periodic_work [ 2277.999580] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2278.005577] pending: neigh_periodic_work [ 2278.008515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2278.011880] pending: do_cache_clean [ 2278.012964] workqueue mm_percpu_wq: flags=0x8 [ 2278.014261] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2278.017946] pending: vmstat_update [ 2278.018937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2278.021335] pending: vmstat_update [ 2278.022394] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2278.024350] pending: vmstat_update [ 2278.025717] workqueue writeback: flags=0x4a [ 2278.026885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2278.035099] pending: wb_workfn, wb_workfn [ 2278.037432] workqueue xprtiod: flags=0xa [ 2278.040280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2278.042288] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2278.046164] workqueue ipv6_addrconf: flags=0x40008 [ 2278.047934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2278.049935] pending: addrconf_verify_work [ 2278.051030] workqueue knbd0-recv: flags=0x1a [ 2278.052232] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2278.054465] in-flight: 651:recv_work [ 2278.056735] workqueue ptlrpc_pinger: flags=0x2 [ 2278.059230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2278.061127] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2278.063339] pool 8: cpus=0-3 flags=0x4 nice=0 hung=360s workers=4 idle: 37 31846 28897 [ 2278.066084] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2267s workers=2 idle: 653 [ 2280.057148] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2280.060065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2280.076850] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2280.082159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2280.085812] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2280.089401] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2280.092205] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2280.101058] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2280.103760] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2280.107864] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2280.111388] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2280.115837] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2280.118288] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2280.122401] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2280.125911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2280.128173] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2280.131425] Call Trace: [ 2280.132466] [ 2280.133387] ? show_regs.cold.9+0x22/0x2f [ 2280.135110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2280.137209] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2280.139160] ? watchdog+0x40/0x40 [ 2280.140515] ? hrtimer_interrupt+0x128/0x2c0 [ 2280.142219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2280.144532] ? apic_timer_interrupt+0xf/0x20 [ 2280.146239] [ 2280.147226] ? native_safe_halt+0x2e/0x50 [ 2280.148978] kvm_wait+0xcd/0xe0 [ 2280.150294] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2280.152497] do_raw_spin_lock+0x177/0x190 [ 2280.154109] _raw_spin_lock+0x21/0x30 [ 2280.155661] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2280.157791] process_one_work+0x2c8/0x700 [ 2280.159192] worker_thread+0x296/0x6e0 [ 2280.160571] ? rescuer_thread+0x570/0x570 [ 2280.162513] kthread+0x1d1/0x200 [ 2280.163992] ? set_kthread_struct+0x70/0x70 [ 2280.165694] ret_from_fork+0x1f/0x30 [ 2292.060276] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2292.060441] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2292.073561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2292.074177] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2292.074185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2292.074188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2292.074204] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2292.074208] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2292.074212] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2292.074214] RDX: 0000000000005bab RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2292.074217] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2292.074219] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2292.074221] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2292.074225] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2292.074228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2292.074230] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2292.074238] Call Trace: [ 2292.074243] [ 2292.074249] ? show_regs.cold.9+0x22/0x2f [ 2292.074256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2292.075390] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2292.075395] ? watchdog+0x40/0x40 [ 2292.075399] ? hrtimer_interrupt+0x128/0x2c0 [ 2292.075404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2292.075414] ? apic_timer_interrupt+0xf/0x20 [ 2292.075418] [ 2292.075422] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2292.075497] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2292.075502] do_raw_spin_lock+0x177/0x190 [ 2292.075507] _raw_spin_lock+0x21/0x30 [ 2292.075586] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2292.080655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2292.080736] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2292.080742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2292.080745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2292.080759] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2292.080764] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2292.080768] RAX: 0000000000007d89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2292.080771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2292.080773] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2292.080775] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2292.080778] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2292.080782] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2292.080785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2292.080787] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2292.080795] Call Trace: [ 2292.080809] [ 2292.080815] ? show_regs.cold.9+0x22/0x2f [ 2292.080825] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2292.080831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2292.080840] ? watchdog+0x40/0x40 [ 2292.080844] ? hrtimer_interrupt+0x128/0x2c0 [ 2292.080848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2292.080856] ? apic_timer_interrupt+0xf/0x20 [ 2292.080861] [ 2292.080863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2292.080870] do_raw_spin_lock+0x177/0x190 [ 2292.080874] _raw_spin_lock+0x21/0x30 [ 2292.080878] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2292.081171] ? do_raw_spin_unlock+0x75/0x190 [ 2292.081177] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2292.081377] ? do_wait_intr+0xf0/0xf0 [ 2292.081384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2292.081580] mdc_close+0x53b/0xd70 [mdc] [ 2292.081618] lmv_close+0x18d/0x490 [lmv] [ 2292.081647] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2292.082429] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2292.083747] ll_md_close+0x274/0x800 [lustre] [ 2292.083857] ll_file_release+0x128/0xc50 [lustre] [ 2292.083937] __fput+0x10d/0x440 [ 2292.083945] ____fput+0x12/0x20 [ 2292.083950] task_work_run+0xc6/0x110 [ 2292.083957] exit_to_usermode_loop+0x1dd/0x1f0 [ 2292.083964] do_syscall_64+0x3d6/0x440 [ 2292.083968] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2292.083976] RIP: 0033:0x7fa749552cd8 [ 2292.083983] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2292.083987] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2292.083992] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2292.083994] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2292.083996] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2292.083997] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2292.083999] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2292.498769] obd_mod_free+0x46/0x140 [ptlrpc] [ 2292.500946] mdc_commit_open+0x91/0xb0 [mdc] [ 2292.502752] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2292.504996] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2292.507600] after_reply+0xe2c/0x1780 [ptlrpc] [ 2292.510020] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2292.512373] ? __next_timer_interrupt+0x160/0x160 [ 2292.514628] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2292.516703] ? do_wait_intr+0xf0/0xf0 [ 2292.518242] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2292.520927] kthread+0x1d1/0x200 [ 2292.522215] ? set_kthread_struct+0x70/0x70 [ 2292.524444] ret_from_fork+0x1f/0x30 [ 2304.054086] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2304.057995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2304.081103] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2304.086413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2304.090213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2304.092823] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2304.101374] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2304.105239] RAX: 00000000000048c2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2304.108203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2304.111051] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2304.114470] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2304.117648] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2304.120722] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2304.125215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.127918] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2304.130897] Call Trace: [ 2304.131549] [ 2304.132345] ? show_regs.cold.9+0x22/0x2f [ 2304.134006] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2304.136350] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2304.138569] ? watchdog+0x40/0x40 [ 2304.140135] ? hrtimer_interrupt+0x128/0x2c0 [ 2304.142212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2304.144727] ? apic_timer_interrupt+0xf/0x20 [ 2304.147806] [ 2304.148842] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2304.151973] do_raw_spin_lock+0x177/0x190 [ 2304.153926] _raw_spin_lock+0x21/0x30 [ 2304.156185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2304.159134] ? do_raw_spin_unlock+0x75/0x190 [ 2304.161435] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2304.165105] ? do_wait_intr+0xf0/0xf0 [ 2304.170003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2304.172643] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2304.175931] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2304.178827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2304.181898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2304.186533] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2304.189195] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2304.195632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2304.198427] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2304.203724] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2304.209092] lmv_intent_open+0x338/0x1150 [lmv] [ 2304.218349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2304.224632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2304.232350] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2304.238273] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2304.247477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2304.254278] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2304.256470] ? do_raw_spin_unlock+0x75/0x190 [ 2304.258664] ? _raw_spin_unlock+0x12/0x30 [ 2304.260700] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2304.263545] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2304.266123] ll_atomic_open+0x36a/0xd70 [lustre] [ 2304.272937] lookup_open+0x356/0xb40 [ 2304.275025] do_last+0x313/0xfc0 [ 2304.279026] ? nd_jump_root+0xe5/0x160 [ 2304.280676] ? path_init+0x437/0x520 [ 2304.285903] path_openat+0xf7/0x500 [ 2304.288699] ? do_raw_spin_unlock+0x75/0x190 [ 2304.294673] do_filp_open+0x99/0x140 [ 2304.296458] ? getname_flags+0x6e/0x330 [ 2304.298955] ? __check_object_size+0xff/0x256 [ 2304.301248] ? do_raw_spin_unlock+0x75/0x190 [ 2304.307401] ? _raw_spin_unlock+0x12/0x30 [ 2304.311929] do_sys_openat2+0x2b4/0x410 [ 2304.315939] do_sys_open+0x73/0xa0 [ 2304.319677] __x64_sys_openat+0x24/0x30 [ 2304.322092] do_syscall_64+0xc1/0x440 [ 2304.323533] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2304.325250] RIP: 0033:0x7fa749552332 [ 2304.326292] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2304.332958] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2304.336464] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2304.339725] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2304.343198] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2304.346616] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2304.349948] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2308.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2308.060578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2308.081773] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2308.086894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2308.091162] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2308.094624] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2308.096791] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2308.104565] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2308.108479] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2308.111589] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2308.115529] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2308.118934] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2308.122196] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2308.125732] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2308.129710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2308.132314] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2308.135810] Call Trace: [ 2308.136899] [ 2308.137750] ? show_regs.cold.9+0x22/0x2f [ 2308.139329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2308.141162] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2308.143652] ? watchdog+0x40/0x40 [ 2308.145224] ? hrtimer_interrupt+0x128/0x2c0 [ 2308.147630] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2308.149441] ? apic_timer_interrupt+0xf/0x20 [ 2308.151818] [ 2308.153119] ? native_safe_halt+0x2e/0x50 [ 2308.155153] kvm_wait+0xcd/0xe0 [ 2308.156700] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2308.159305] do_raw_spin_lock+0x177/0x190 [ 2308.161496] _raw_spin_lock+0x21/0x30 [ 2308.163505] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2308.167540] process_one_work+0x2c8/0x700 [ 2308.169718] worker_thread+0x296/0x6e0 [ 2308.171552] ? rescuer_thread+0x570/0x570 [ 2308.173556] kthread+0x1d1/0x200 [ 2308.175328] ? set_kthread_struct+0x70/0x70 [ 2308.177354] ret_from_fork+0x1f/0x30 [ 2308.578341] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 391s! [ 2308.584925] Showing busy workqueues and worker pools: [ 2308.598400] workqueue events: flags=0x0 [ 2308.599749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2308.613442] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2308.635600] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2308.645152] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2308.666010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.678047] pending: kfree_rcu_monitor [ 2308.683621] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2308.691057] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2308.703800] workqueue events_long: flags=0x0 [ 2308.713722] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.722067] pending: lu_objects_flush [obdclass] [ 2308.724498] workqueue events_unbound: flags=0x2 [ 2308.725878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2308.741008] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2308.749542] workqueue events_power_efficient: flags=0x80 [ 2308.760602] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.780923] pending: check_lifetime [ 2308.783203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.788095] pending: neigh_periodic_work [ 2308.793482] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.800719] pending: neigh_periodic_work [ 2308.805447] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.810907] pending: do_cache_clean [ 2308.818479] workqueue mm_percpu_wq: flags=0x8 [ 2308.828025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.832911] pending: vmstat_update [ 2308.839691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.847607] pending: vmstat_update [ 2308.861187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2308.866640] pending: vmstat_update [ 2308.871130] workqueue writeback: flags=0x4a [ 2308.874645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2308.882036] pending: wb_workfn, wb_workfn [ 2308.886086] workqueue xprtiod: flags=0xa [ 2308.894031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2308.897939] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2308.919373] workqueue ipv6_addrconf: flags=0x40008 [ 2308.921384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2308.934425] pending: addrconf_verify_work [ 2308.937882] workqueue knbd0-recv: flags=0x1a [ 2308.941350] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2308.945775] in-flight: 651:recv_work [ 2308.951463] workqueue ptlrpc_pinger: flags=0x2 [ 2308.957824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2308.961825] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2308.970374] pool 8: cpus=0-3 flags=0x4 nice=0 hung=391s workers=4 idle: 37 31846 28897 [ 2308.975658] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2298s workers=2 idle: 653 [ 2320.060646] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2320.064298] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2320.077816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2320.077893] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2320.077900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2320.077902] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2320.077919] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2320.077923] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2320.077927] RAX: 0000000000005664 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2320.077930] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2320.077932] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2320.077933] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2320.077936] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2320.077938] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2320.077941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2320.077943] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2320.077952] Call Trace: [ 2320.077958] [ 2320.077963] ? show_regs.cold.9+0x22/0x2f [ 2320.077973] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2320.077980] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2320.077984] ? watchdog+0x40/0x40 [ 2320.077988] ? hrtimer_interrupt+0x128/0x2c0 [ 2320.077992] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2320.078000] ? apic_timer_interrupt+0xf/0x20 [ 2320.078005] [ 2320.078007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2320.078015] do_raw_spin_lock+0x177/0x190 [ 2320.078033] _raw_spin_lock+0x21/0x30 [ 2320.078039] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2320.078344] ? do_raw_spin_unlock+0x75/0x190 [ 2320.078351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2320.080993] ? do_wait_intr+0xf0/0xf0 [ 2320.081005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2320.090582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2320.090845] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2320.090857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2320.090859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2320.090873] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2320.090877] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2320.090881] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2320.090883] RDX: 0000000000005b85 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2320.090886] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2320.090888] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2320.090890] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2320.090893] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2320.090896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2320.090898] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2320.090907] Call Trace: [ 2320.090912] [ 2320.090917] ? show_regs.cold.9+0x22/0x2f [ 2320.090926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2320.090938] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2320.090942] ? watchdog+0x40/0x40 [ 2320.090946] ? hrtimer_interrupt+0x128/0x2c0 [ 2320.090950] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2320.090958] ? apic_timer_interrupt+0xf/0x20 [ 2320.090963] [ 2320.090966] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2320.090972] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2320.090977] do_raw_spin_lock+0x177/0x190 [ 2320.090981] _raw_spin_lock+0x21/0x30 [ 2320.090988] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2320.091904] obd_mod_free+0x46/0x140 [ptlrpc] [ 2320.092511] mdc_commit_open+0x91/0xb0 [mdc] [ 2320.092829] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2320.094896] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2320.095423] after_reply+0xe2c/0x1780 [ptlrpc] [ 2320.095649] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2320.095861] ? __next_timer_interrupt+0x160/0x160 [ 2320.095871] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2320.133642] mdc_close+0x53b/0xd70 [mdc] [ 2320.133688] lmv_close+0x18d/0x490 [lmv] [ 2320.133718] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2320.133822] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2320.133903] ll_md_close+0x274/0x800 [lustre] [ 2320.150166] ? do_wait_intr+0xf0/0xf0 [ 2320.150189] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2320.150857] kthread+0x1d1/0x200 [ 2320.150864] ? set_kthread_struct+0x70/0x70 [ 2320.150868] ret_from_fork+0x1f/0x30 [ 2320.763872] ll_file_release+0x128/0xc50 [lustre] [ 2320.771389] __fput+0x10d/0x440 [ 2320.773353] ____fput+0x12/0x20 [ 2320.774912] task_work_run+0xc6/0x110 [ 2320.776198] exit_to_usermode_loop+0x1dd/0x1f0 [ 2320.786991] do_syscall_64+0x3d6/0x440 [ 2320.795807] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2320.799393] RIP: 0033:0x7fa749552cd8 [ 2320.803343] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2320.820694] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2320.826265] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2320.831267] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2320.834810] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2320.841586] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2320.849537] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2333.698212] rcu: INFO: rcu_sched self-detected stall on CPU [ 2333.701284] rcu: 2-...!: (235991 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 2333.709417] (t=420009 jiffies g=258701 q=1527) [ 2333.711737] rcu: rcu_sched kthread starved for 416804 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2333.718255] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2333.723456] rcu: RCU grace-period kthread stack dump: [ 2333.725404] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2333.731514] Call Trace: [ 2333.732931] __schedule+0x351/0xcb0 [ 2333.734696] schedule+0xc0/0x180 [ 2333.736610] schedule_timeout+0xb4/0x190 [ 2333.738695] ? do_raw_spin_unlock+0x75/0x190 [ 2333.740994] ? __next_timer_interrupt+0x160/0x160 [ 2333.743505] ? prepare_to_swait_event+0xf3/0x140 [ 2333.745684] rcu_gp_kthread+0x8dd/0x15b0 [ 2333.747901] ? force_qs_rnp+0x2c0/0x2c0 [ 2333.749777] kthread+0x1d1/0x200 [ 2333.751395] ? set_kthread_struct+0x70/0x70 [ 2333.753315] ret_from_fork+0x1f/0x30 [ 2333.755291] rcu: Stack dump where RCU GP kthread last ran: [ 2333.758472] NMI backtrace for cpu 2 [ 2333.760306] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2333.767188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2333.772056] Call Trace: [ 2333.773874] [ 2333.775256] ? dump_stack+0xbb/0x10e [ 2333.776894] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2333.779507] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2333.781852] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2333.784524] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2333.787597] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2333.789926] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2333.792534] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 2333.794858] ? update_process_times+0x6d/0xc0 [ 2333.796872] ? tick_sched_handle.isra.18+0x27/0x90 [ 2333.799364] ? tick_sched_timer+0x53/0xb0 [ 2333.801588] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2333.803403] ? tick_sched_handle.isra.18+0x90/0x90 [ 2333.805877] ? hrtimer_interrupt+0x128/0x2c0 [ 2333.807895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2333.810178] ? apic_timer_interrupt+0xf/0x20 [ 2333.812205] [ 2333.813134] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2333.815680] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2333.818001] ? do_raw_spin_lock+0x177/0x190 [ 2333.819679] ? _raw_spin_lock+0x21/0x30 [ 2333.821055] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2333.824044] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2333.826866] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2333.829017] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2333.832049] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2333.835454] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2333.838455] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2333.841628] ? __next_timer_interrupt+0x160/0x160 [ 2333.844276] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2333.846495] ? do_wait_intr+0xf0/0xf0 [ 2333.848332] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2333.851215] ? kthread+0x1d1/0x200 [ 2333.852885] ? set_kthread_struct+0x70/0x70 [ 2333.854672] ? ret_from_fork+0x1f/0x30 [ 2333.856396] NMI backtrace for cpu 2 [ 2333.857855] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2333.863012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2333.867973] Call Trace: [ 2333.869356] [ 2333.870325] ? dump_stack+0xbb/0x10e [ 2333.872076] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2333.874342] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2333.876752] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2333.879275] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2333.881828] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2333.884816] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2333.886928] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2333.889429] ? update_process_times+0x6d/0xc0 [ 2333.892578] ? tick_sched_handle.isra.18+0x27/0x90 [ 2333.894704] ? tick_sched_timer+0x53/0xb0 [ 2333.897057] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2333.898949] ? tick_sched_handle.isra.18+0x90/0x90 [ 2333.900405] ? hrtimer_interrupt+0x128/0x2c0 [ 2333.902223] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2333.904664] ? apic_timer_interrupt+0xf/0x20 [ 2333.906907] [ 2333.908205] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2333.910319] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2333.913234] ? do_raw_spin_lock+0x177/0x190 [ 2333.915378] ? _raw_spin_lock+0x21/0x30 [ 2333.917392] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2333.920921] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2333.923346] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2333.925876] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2333.930492] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2333.934743] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2333.937491] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2333.940675] ? __next_timer_interrupt+0x160/0x160 [ 2333.943061] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2333.945910] ? do_wait_intr+0xf0/0xf0 [ 2333.947693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2333.950744] ? kthread+0x1d1/0x200 [ 2333.952380] ? set_kthread_struct+0x70/0x70 [ 2333.954416] ? ret_from_fork+0x1f/0x30 [ 2336.060397] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2336.061891] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2336.066075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2336.066332] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2336.066339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2336.066342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2336.066355] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2336.066359] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2336.066363] RAX: 0000000000003188 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2336.066365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2336.066367] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2336.066369] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2336.066371] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2336.066374] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2336.066377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2336.066379] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2336.066387] Call Trace: [ 2336.066393] [ 2336.066398] ? show_regs.cold.9+0x22/0x2f [ 2336.066408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2336.066414] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2336.066418] ? watchdog+0x40/0x40 [ 2336.066422] ? hrtimer_interrupt+0x128/0x2c0 [ 2336.066426] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2336.066434] ? apic_timer_interrupt+0xf/0x20 [ 2336.066438] [ 2336.066441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2336.066447] do_raw_spin_lock+0x177/0x190 [ 2336.066451] _raw_spin_lock+0x21/0x30 [ 2336.066456] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2336.081940] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2336.082022] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2336.083390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2336.083396] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2336.150392] ? do_raw_spin_unlock+0x75/0x190 [ 2336.150411] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2336.151233] ? do_wait_intr+0xf0/0xf0 [ 2336.151242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2336.151597] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2336.151884] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2336.152208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2336.167532] [ 2336.167541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2336.167560] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2336.167564] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2336.167569] RAX: 0000000000005890 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2336.167572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2336.167574] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2336.167576] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2336.167578] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2336.167582] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2336.167584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2336.167586] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2336.167595] Call Trace: [ 2336.167600] [ 2336.167606] ? show_regs.cold.9+0x22/0x2f [ 2336.167616] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2336.167622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2336.167626] ? watchdog+0x40/0x40 [ 2336.167631] ? hrtimer_interrupt+0x128/0x2c0 [ 2336.167635] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2336.167642] ? apic_timer_interrupt+0xf/0x20 [ 2336.167647] [ 2336.167649] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2336.167654] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2336.167661] do_raw_spin_lock+0x177/0x190 [ 2336.167678] _raw_spin_lock+0x21/0x30 [ 2336.167684] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2336.169422] process_one_work+0x2c8/0x700 [ 2336.169431] worker_thread+0x296/0x6e0 [ 2336.169436] ? rescuer_thread+0x570/0x570 [ 2336.169441] kthread+0x1d1/0x200 [ 2336.169446] ? set_kthread_struct+0x70/0x70 [ 2336.169449] ret_from_fork+0x1f/0x30 [ 2336.617837] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2336.623081] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2336.628411] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2336.631127] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2336.634821] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2336.638430] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2336.641583] lmv_intent_open+0x338/0x1150 [lmv] [ 2336.645032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2336.651051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2336.654338] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2336.656724] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2336.659984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2336.663276] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2336.666315] ? do_raw_spin_unlock+0x75/0x190 [ 2336.668533] ? _raw_spin_unlock+0x12/0x30 [ 2336.670581] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2336.673977] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2336.676547] ll_atomic_open+0x36a/0xd70 [lustre] [ 2336.678637] lookup_open+0x356/0xb40 [ 2336.680885] do_last+0x313/0xfc0 [ 2336.683053] ? nd_jump_root+0xe5/0x160 [ 2336.685051] ? path_init+0x437/0x520 [ 2336.687069] path_openat+0xf7/0x500 [ 2336.689890] ? do_raw_spin_unlock+0x75/0x190 [ 2336.693624] do_filp_open+0x99/0x140 [ 2336.699595] ? getname_flags+0x6e/0x330 [ 2336.701808] ? __check_object_size+0xff/0x256 [ 2336.707554] ? do_raw_spin_unlock+0x75/0x190 [ 2336.714358] ? _raw_spin_unlock+0x12/0x30 [ 2336.722443] do_sys_openat2+0x2b4/0x410 [ 2336.732129] do_sys_open+0x73/0xa0 [ 2336.736725] __x64_sys_openat+0x24/0x30 [ 2336.742405] do_syscall_64+0xc1/0x440 [ 2336.745065] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2336.748138] RIP: 0033:0x7fa749552332 [ 2336.750301] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2336.767303] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2336.774407] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2336.783758] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2336.792514] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2336.800152] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2336.810039] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2339.304378] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 422s! [ 2339.307200] Showing busy workqueues and worker pools: [ 2339.315606] workqueue events: flags=0x0 [ 2339.319139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2339.328039] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2339.338854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2339.346003] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2339.351439] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.358909] pending: kfree_rcu_monitor [ 2339.363927] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2339.373950] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2339.381050] workqueue events_long: flags=0x0 [ 2339.383874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.388666] pending: lu_objects_flush [obdclass] [ 2339.392963] workqueue events_unbound: flags=0x2 [ 2339.399375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2339.404471] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2339.415085] workqueue events_power_efficient: flags=0x80 [ 2339.421841] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.430789] pending: check_lifetime [ 2339.435672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.442049] pending: neigh_periodic_work [ 2339.447108] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.449774] pending: neigh_periodic_work [ 2339.456642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.465571] pending: do_cache_clean [ 2339.470473] workqueue mm_percpu_wq: flags=0x8 [ 2339.476288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.484918] pending: vmstat_update [ 2339.489083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.497750] pending: vmstat_update [ 2339.502308] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2339.514814] pending: vmstat_update [ 2339.521095] workqueue writeback: flags=0x4a [ 2339.528318] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2339.534471] pending: wb_workfn, wb_workfn [ 2339.538257] workqueue xprtiod: flags=0xa [ 2339.540801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2339.549004] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2339.555881] workqueue ipv6_addrconf: flags=0x40008 [ 2339.560409] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2339.565824] pending: addrconf_verify_work [ 2339.568847] workqueue knbd0-recv: flags=0x1a [ 2339.573888] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2339.582119] in-flight: 651:recv_work [ 2339.587336] workqueue ptlrpc_pinger: flags=0x2 [ 2339.592457] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2339.597844] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2339.605923] pool 8: cpus=0-3 flags=0x4 nice=0 hung=422s workers=4 idle: 37 31846 28897 [ 2339.611099] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2329s workers=2 idle: 653 [ 2348.059779] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2348.064564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2348.107467] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2348.119650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2348.126425] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2348.131335] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2348.158643] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2348.168674] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2348.174638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2348.182350] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2348.189937] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2348.194752] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2348.207905] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2348.215683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2348.220017] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2348.230654] Call Trace: [ 2348.234276] [ 2348.235776] ? show_regs.cold.9+0x22/0x2f [ 2348.241211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2348.249712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2348.251807] ? watchdog+0x40/0x40 [ 2348.253145] ? hrtimer_interrupt+0x128/0x2c0 [ 2348.257306] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2348.265975] ? apic_timer_interrupt+0xf/0x20 [ 2348.269801] [ 2348.271035] ? native_safe_halt+0x2e/0x50 [ 2348.275585] kvm_wait+0xcd/0xe0 [ 2348.278053] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2348.281018] do_raw_spin_lock+0x177/0x190 [ 2348.282394] _raw_spin_lock+0x21/0x30 [ 2348.284102] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2348.288334] ? do_raw_spin_unlock+0x75/0x190 [ 2348.290899] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2348.295268] ? do_wait_intr+0xf0/0xf0 [ 2348.297249] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2348.299885] mdc_close+0x53b/0xd70 [mdc] [ 2348.302023] lmv_close+0x18d/0x490 [lmv] [ 2348.304101] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2348.308126] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2348.312007] ll_md_close+0x274/0x800 [lustre] [ 2348.315274] ll_file_release+0x128/0xc50 [lustre] [ 2348.317297] __fput+0x10d/0x440 [ 2348.319948] ____fput+0x12/0x20 [ 2348.324200] task_work_run+0xc6/0x110 [ 2348.328215] exit_to_usermode_loop+0x1dd/0x1f0 [ 2348.330251] do_syscall_64+0x3d6/0x440 [ 2348.333211] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2348.338186] RIP: 0033:0x7fa749552cd8 [ 2348.343839] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2348.355203] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2348.359676] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2348.362688] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2348.365365] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2348.369355] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2348.375312] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2360.058015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2360.066715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.135341] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.161117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.166796] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2360.170129] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2360.194181] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2360.202284] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2360.206093] RDX: 00000000000069ba RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2360.209263] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2360.213079] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2360.216783] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2360.220142] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2360.225816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.230346] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2360.236904] Call Trace: [ 2360.240080] [ 2360.243758] ? show_regs.cold.9+0x22/0x2f [ 2360.245824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.252737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.256672] ? watchdog+0x40/0x40 [ 2360.258776] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.267176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.270970] ? apic_timer_interrupt+0xf/0x20 [ 2360.273804] [ 2360.275945] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2360.280808] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2360.285192] do_raw_spin_lock+0x177/0x190 [ 2360.287884] _raw_spin_lock+0x21/0x30 [ 2360.289783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2360.292499] obd_mod_free+0x46/0x140 [ptlrpc] [ 2360.295233] mdc_commit_open+0x91/0xb0 [mdc] [ 2360.297805] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2360.301060] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2360.305016] after_reply+0xe2c/0x1780 [ptlrpc] [ 2360.307732] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2360.311008] ? __next_timer_interrupt+0x160/0x160 [ 2360.312971] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2360.315794] ? do_wait_intr+0xf0/0xf0 [ 2360.317291] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2360.319717] kthread+0x1d1/0x200 [ 2360.321740] ? set_kthread_struct+0x70/0x70 [ 2360.326617] ret_from_fork+0x1f/0x30 [ 2364.057677] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2364.061856] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2364.063997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2364.064071] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2364.064077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2364.064084] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2364.064407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2364.064417] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2364.064421] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2364.064424] RAX: 0000000000005b83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2364.064426] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2364.064428] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2364.064430] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2364.064432] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2364.064434] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2364.064437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2364.064438] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2364.064446] Call Trace: [ 2364.064450] [ 2364.064455] ? show_regs.cold.9+0x22/0x2f [ 2364.064464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2364.064469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2364.064473] ? watchdog+0x40/0x40 [ 2364.064477] ? hrtimer_interrupt+0x128/0x2c0 [ 2364.064480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2364.064486] ? apic_timer_interrupt+0xf/0x20 [ 2364.064490] [ 2364.064493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2364.064496] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2364.064500] do_raw_spin_lock+0x177/0x190 [ 2364.064504] _raw_spin_lock+0x21/0x30 [ 2364.064508] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2364.064789] process_one_work+0x2c8/0x700 [ 2364.064795] worker_thread+0x296/0x6e0 [ 2364.064798] ? rescuer_thread+0x570/0x570 [ 2364.064801] kthread+0x1d1/0x200 [ 2364.064805] ? set_kthread_struct+0x70/0x70 [ 2364.064808] ret_from_fork+0x1f/0x30 [ 2364.352083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2364.393497] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2364.406248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2364.416501] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2364.423754] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2364.448429] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2364.456090] RAX: 0000000000003e10 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2364.464096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2364.469807] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2364.477255] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2364.482463] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2364.489213] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2364.498016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2364.504498] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2364.519481] Call Trace: [ 2364.520225] [ 2364.521731] ? show_regs.cold.9+0x22/0x2f [ 2364.531720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2364.537658] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2364.541283] ? watchdog+0x40/0x40 [ 2364.545649] ? hrtimer_interrupt+0x128/0x2c0 [ 2364.547374] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2364.551859] ? apic_timer_interrupt+0xf/0x20 [ 2364.557164] [ 2364.560751] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2364.564044] do_raw_spin_lock+0x177/0x190 [ 2364.569046] _raw_spin_lock+0x21/0x30 [ 2364.570983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2364.577576] ? do_raw_spin_unlock+0x75/0x190 [ 2364.580247] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2364.587814] ? do_wait_intr+0xf0/0xf0 [ 2364.596696] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2364.606915] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2364.614287] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2364.619796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2364.625686] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2364.631453] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2364.638501] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2364.643794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2364.651768] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2364.660164] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2364.663235] lmv_intent_open+0x338/0x1150 [lmv] [ 2364.665873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2364.669138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2364.672825] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2364.674997] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2364.680468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2364.684696] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2364.689863] ? do_raw_spin_unlock+0x75/0x190 [ 2364.692929] ? _raw_spin_unlock+0x12/0x30 [ 2364.695862] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2364.698724] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2364.710918] ll_atomic_open+0x36a/0xd70 [lustre] [ 2364.715136] lookup_open+0x356/0xb40 [ 2364.718461] do_last+0x313/0xfc0 [ 2364.720601] ? nd_jump_root+0xe5/0x160 [ 2364.726097] ? path_init+0x437/0x520 [ 2364.730249] path_openat+0xf7/0x500 [ 2364.735787] ? do_raw_spin_unlock+0x75/0x190 [ 2364.739428] do_filp_open+0x99/0x140 [ 2364.744405] ? getname_flags+0x6e/0x330 [ 2364.745877] ? __check_object_size+0xff/0x256 [ 2364.747462] ? do_raw_spin_unlock+0x75/0x190 [ 2364.751790] ? _raw_spin_unlock+0x12/0x30 [ 2364.755169] do_sys_openat2+0x2b4/0x410 [ 2364.759075] do_sys_open+0x73/0xa0 [ 2364.769039] __x64_sys_openat+0x24/0x30 [ 2364.778005] do_syscall_64+0xc1/0x440 [ 2364.783871] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2364.794961] RIP: 0033:0x7fa749552332 [ 2364.796271] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2364.819470] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2364.834760] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2364.846141] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2364.855541] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2364.860186] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2364.868541] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2370.018577] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 452s! [ 2370.032973] Showing busy workqueues and worker pools: [ 2370.038945] workqueue events: flags=0x0 [ 2370.043541] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2370.049891] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2370.055595] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2370.059211] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2370.063420] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.066879] pending: kfree_rcu_monitor [ 2370.068726] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2370.072392] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2370.075141] workqueue events_long: flags=0x0 [ 2370.076840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.078759] pending: lu_objects_flush [obdclass] [ 2370.081037] workqueue events_unbound: flags=0x2 [ 2370.083074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2370.085706] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2370.090493] workqueue events_power_efficient: flags=0x80 [ 2370.095600] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.098334] pending: check_lifetime [ 2370.099615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.117937] pending: neigh_periodic_work [ 2370.130978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.140285] pending: neigh_periodic_work [ 2370.148843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.160114] pending: do_cache_clean [ 2370.162936] workqueue mm_percpu_wq: flags=0x8 [ 2370.173084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.179427] pending: vmstat_update [ 2370.185299] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.191230] pending: vmstat_update [ 2370.193370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2370.198164] pending: vmstat_update [ 2370.208412] workqueue writeback: flags=0x4a [ 2370.216476] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2370.223715] pending: wb_workfn, wb_workfn [ 2370.226547] workqueue xprtiod: flags=0xa [ 2370.228869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2370.243463] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2370.256111] workqueue ipv6_addrconf: flags=0x40008 [ 2370.264838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2370.270653] pending: addrconf_verify_work [ 2370.273287] workqueue knbd0-recv: flags=0x1a [ 2370.275617] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2370.283439] in-flight: 651:recv_work [ 2370.285593] workqueue ptlrpc_pinger: flags=0x2 [ 2370.287931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2370.299269] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2370.305049] pool 8: cpus=0-3 flags=0x4 nice=0 hung=453s workers=4 idle: 37 31846 28897 [ 2370.319481] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2360s workers=2 idle: 653 [ 2376.060400] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2376.081864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2376.114208] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2376.121848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2376.132342] RIP: 0010:load_balance+0xabe/0x1090 [ 2376.136833] Code: 8b 40 10 c7 40 28 01 00 00 00 8b 84 24 98 00 00 00 a8 01 0f 85 c4 04 00 00 8b 74 24 10 85 f6 0f 85 fe 02 00 00 83 7c 24 40 02 <0f> 84 0c 04 00 00 48 8b 44 24 20 48 83 05 47 b6 dd 02 01 83 40 64 [ 2376.148244] RSP: 0018:ffffb893c0d8cde0 EFLAGS: 00000297 ORIG_RAX: ffffffffffffff13 [ 2376.151967] RAX: 0000000000000000 RBX: ffff8d7382133380 RCX: dead000000000200 [ 2376.156397] RDX: ffff8d73591100b0 RSI: 0000000000000000 RDI: 0000000000000202 [ 2376.159654] RBP: ffff8d73591100b0 R08: ffff8d73591100b0 R09: 0000000000012da5 [ 2376.169031] R10: 000000000001b800 R11: 000000000000b928 R12: ffff8d73412dd030 [ 2376.173378] R13: ffff8d7382133e08 R14: 0000000000000000 R15: ffff8d7382133380 [ 2376.179100] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2376.198592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2376.205286] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2376.210631] Call Trace: [ 2376.212453] [ 2376.213432] ? show_regs.cold.9+0x22/0x2f [ 2376.214971] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2376.218717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2376.220845] ? watchdog+0x40/0x40 [ 2376.222719] ? hrtimer_interrupt+0x128/0x2c0 [ 2376.224899] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2376.226868] ? apic_timer_interrupt+0xf/0x20 [ 2376.229026] ? apic_timer_interrupt+0xa/0x20 [ 2376.230796] ? load_balance+0xabe/0x1090 [ 2376.232452] rebalance_domains+0x348/0x580 [ 2376.234153] ? kvm_clock_get_cycles+0x2c/0x50 [ 2376.236137] run_rebalance_domains+0x68/0xa0 [ 2376.237923] __do_softirq+0x134/0x4e8 [ 2376.239550] irq_exit_rcu+0x1c6/0x1e0 [ 2376.241275] irq_exit+0x12/0x30 [ 2376.242796] smp_apic_timer_interrupt+0xe0/0x270 [ 2376.244861] apic_timer_interrupt+0xf/0x20 [ 2376.246965] [ 2376.247664] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2376.256782] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2376.265730] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2376.269113] RAX: 0000000000007928 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2376.277571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2376.282928] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2376.290903] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2376.294234] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2376.297427] do_raw_spin_lock+0x177/0x190 [ 2376.298958] _raw_spin_lock+0x21/0x30 [ 2376.303181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2376.305975] ? do_raw_spin_unlock+0x75/0x190 [ 2376.307996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2376.310312] ? do_wait_intr+0xf0/0xf0 [ 2376.313179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2376.315152] mdc_close+0x53b/0xd70 [mdc] [ 2376.318975] lmv_close+0x18d/0x490 [lmv] [ 2376.320209] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2376.326287] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2376.328944] ll_md_close+0x274/0x800 [lustre] [ 2376.330915] ll_file_release+0x128/0xc50 [lustre] [ 2376.333017] __fput+0x10d/0x440 [ 2376.334317] ____fput+0x12/0x20 [ 2376.335729] task_work_run+0xc6/0x110 [ 2376.337531] exit_to_usermode_loop+0x1dd/0x1f0 [ 2376.339655] do_syscall_64+0x3d6/0x440 [ 2376.341225] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2376.343443] RIP: 0033:0x7fa749552cd8 [ 2376.346025] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2376.355446] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2376.358961] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2376.362137] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2376.372067] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2376.380162] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2376.383250] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2388.058446] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2388.070612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2388.107800] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2388.112242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2388.115655] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2388.117210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2388.123701] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2388.126731] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2388.129447] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2388.132428] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2388.135171] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2388.139069] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2388.141902] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2388.145835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.147808] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2388.150566] Call Trace: [ 2388.152133] [ 2388.152959] ? show_regs.cold.9+0x22/0x2f [ 2388.154441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2388.156257] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2388.158006] ? watchdog+0x40/0x40 [ 2388.159258] ? hrtimer_interrupt+0x128/0x2c0 [ 2388.160898] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2388.162680] ? apic_timer_interrupt+0xf/0x20 [ 2388.164208] [ 2388.164918] ? native_safe_halt+0x2e/0x50 [ 2388.166805] kvm_wait+0xcd/0xe0 [ 2388.167996] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2388.170277] do_raw_spin_lock+0x177/0x190 [ 2388.171629] _raw_spin_lock+0x21/0x30 [ 2388.173158] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2388.174826] obd_mod_free+0x46/0x140 [ptlrpc] [ 2388.177062] mdc_commit_open+0x91/0xb0 [mdc] [ 2388.178392] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2388.180320] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2388.182402] after_reply+0xe2c/0x1780 [ptlrpc] [ 2388.184268] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2388.186224] ? __next_timer_interrupt+0x160/0x160 [ 2388.187768] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2388.190251] ? do_wait_intr+0xf0/0xf0 [ 2388.191632] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2388.193588] kthread+0x1d1/0x200 [ 2388.194484] ? set_kthread_struct+0x70/0x70 [ 2388.195933] ret_from_fork+0x1f/0x30 [ 2392.067155] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2392.079855] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2392.127360] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2392.133945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2392.140037] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2392.143338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2392.146685] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2392.155234] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2392.158679] RAX: 00000000000078b8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2392.161946] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2392.165331] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2392.168477] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2392.171632] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2392.175383] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2392.181084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2392.184266] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2392.188115] Call Trace: [ 2392.189233] [ 2392.190311] ? show_regs.cold.9+0x22/0x2f [ 2392.192211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2392.194466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2392.196828] ? watchdog+0x40/0x40 [ 2392.198570] ? hrtimer_interrupt+0x128/0x2c0 [ 2392.201153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2392.203950] ? apic_timer_interrupt+0xf/0x20 [ 2392.206028] [ 2392.207436] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2392.210006] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2392.213528] do_raw_spin_lock+0x177/0x190 [ 2392.215982] _raw_spin_lock+0x21/0x30 [ 2392.218069] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2392.220897] process_one_work+0x2c8/0x700 [ 2392.222615] worker_thread+0x296/0x6e0 [ 2392.224209] ? rescuer_thread+0x570/0x570 [ 2392.226097] kthread+0x1d1/0x200 [ 2392.228176] ? set_kthread_struct+0x70/0x70 [ 2392.230475] ret_from_fork+0x1f/0x30 [ 2396.054060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2396.057669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2396.091325] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2396.096565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2396.100274] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2396.104238] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2396.117971] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2396.126374] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2396.134685] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2396.141779] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2396.145534] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2396.156912] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2396.161010] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2396.168370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2396.172411] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2396.177291] Call Trace: [ 2396.179849] [ 2396.180848] ? show_regs.cold.9+0x22/0x2f [ 2396.182741] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2396.185190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2396.188189] ? watchdog+0x40/0x40 [ 2396.190182] ? hrtimer_interrupt+0x128/0x2c0 [ 2396.192869] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2396.195807] ? apic_timer_interrupt+0xf/0x20 [ 2396.198323] [ 2396.199743] ? native_safe_halt+0x2e/0x50 [ 2396.202099] kvm_wait+0xcd/0xe0 [ 2396.203949] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2396.207758] do_raw_spin_lock+0x177/0x190 [ 2396.210234] _raw_spin_lock+0x21/0x30 [ 2396.212434] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2396.216228] ? do_raw_spin_unlock+0x75/0x190 [ 2396.218652] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2396.221255] ? do_wait_intr+0xf0/0xf0 [ 2396.223815] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2396.227014] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2396.229953] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2396.232792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2396.236256] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2396.239209] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2396.241692] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2396.245705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2396.254610] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2396.259524] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2396.263229] lmv_intent_open+0x338/0x1150 [lmv] [ 2396.265784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2396.268756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2396.271826] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2396.274818] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2396.277674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2396.280805] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2396.284265] ? do_raw_spin_unlock+0x75/0x190 [ 2396.287576] ? _raw_spin_unlock+0x12/0x30 [ 2396.290354] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2396.293415] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2396.296550] ll_atomic_open+0x36a/0xd70 [lustre] [ 2396.298919] lookup_open+0x356/0xb40 [ 2396.299877] do_last+0x313/0xfc0 [ 2396.301590] ? nd_jump_root+0xe5/0x160 [ 2396.302946] ? path_init+0x437/0x520 [ 2396.304402] path_openat+0xf7/0x500 [ 2396.306223] ? do_raw_spin_unlock+0x75/0x190 [ 2396.308680] do_filp_open+0x99/0x140 [ 2396.313609] ? getname_flags+0x6e/0x330 [ 2396.316593] ? __check_object_size+0xff/0x256 [ 2396.318325] ? do_raw_spin_unlock+0x75/0x190 [ 2396.321846] ? _raw_spin_unlock+0x12/0x30 [ 2396.323425] do_sys_openat2+0x2b4/0x410 [ 2396.330417] do_sys_open+0x73/0xa0 [ 2396.337123] __x64_sys_openat+0x24/0x30 [ 2396.342121] do_syscall_64+0xc1/0x440 [ 2396.344190] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2396.352333] RIP: 0033:0x7fa749552332 [ 2396.360379] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2396.384059] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2396.392275] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2396.396155] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2396.404676] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2396.413232] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2396.419544] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2400.738025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 483s! [ 2400.744824] Showing busy workqueues and worker pools: [ 2400.755961] workqueue events: flags=0x0 [ 2400.757338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2400.767359] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2400.789348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2400.804117] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2400.822988] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2400.831351] pending: kfree_rcu_monitor [ 2400.835441] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2400.843312] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2400.849370] workqueue events_long: flags=0x0 [ 2400.856811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2400.864384] pending: lu_objects_flush [obdclass] [ 2400.874141] workqueue events_unbound: flags=0x2 [ 2400.880912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2400.892282] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2400.907210] workqueue events_power_efficient: flags=0x80 [ 2400.915964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2400.924422] pending: check_lifetime [ 2400.929989] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2400.937275] pending: neigh_periodic_work [ 2400.945148] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2400.955397] pending: neigh_periodic_work [ 2400.959943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2400.964730] pending: do_cache_clean [ 2400.971373] workqueue mm_percpu_wq: flags=0x8 [ 2400.975212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2400.982775] pending: vmstat_update [ 2400.985348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2400.993136] pending: vmstat_update [ 2400.996337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2401.003309] pending: vmstat_update [ 2401.009355] workqueue writeback: flags=0x4a [ 2401.012449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2401.021161] pending: wb_workfn, wb_workfn [ 2401.025942] workqueue xprtiod: flags=0xa [ 2401.030148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2401.038348] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2401.044522] workqueue ipv6_addrconf: flags=0x40008 [ 2401.049349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2401.062313] pending: addrconf_verify_work [ 2401.069807] workqueue knbd0-recv: flags=0x1a [ 2401.078488] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2401.080615] in-flight: 651:recv_work [ 2401.091710] workqueue ptlrpc_pinger: flags=0x2 [ 2401.093149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2401.111193] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2401.123129] pool 8: cpus=0-3 flags=0x4 nice=0 hung=483s workers=4 idle: 37 31846 28897 [ 2401.125796] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2391s workers=2 idle: 653 [ 2404.059457] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2404.064175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2404.110054] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2404.117104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2404.125560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2404.130473] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2404.146808] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2404.155513] RAX: 0000000000006d13 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2404.160376] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2404.165218] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2404.178274] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2404.187414] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2404.198327] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2404.212481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2404.219152] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2404.224879] Call Trace: [ 2404.229029] [ 2404.234111] ? show_regs.cold.9+0x22/0x2f [ 2404.235472] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2404.243606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2404.250829] ? watchdog+0x40/0x40 [ 2404.251962] ? hrtimer_interrupt+0x128/0x2c0 [ 2404.253430] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2404.265567] ? apic_timer_interrupt+0xf/0x20 [ 2404.271854] [ 2404.274600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2404.278227] do_raw_spin_lock+0x177/0x190 [ 2404.286528] _raw_spin_lock+0x21/0x30 [ 2404.297341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2404.307929] ? do_raw_spin_unlock+0x75/0x190 [ 2404.320313] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2404.331487] ? do_wait_intr+0xf0/0xf0 [ 2404.337005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2404.342517] mdc_close+0x53b/0xd70 [mdc] [ 2404.345905] lmv_close+0x18d/0x490 [lmv] [ 2404.349000] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2404.352944] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2404.356094] ll_md_close+0x274/0x800 [lustre] [ 2404.360999] ll_file_release+0x128/0xc50 [lustre] [ 2404.362399] __fput+0x10d/0x440 [ 2404.368208] ____fput+0x12/0x20 [ 2404.374119] task_work_run+0xc6/0x110 [ 2404.378303] exit_to_usermode_loop+0x1dd/0x1f0 [ 2404.387821] do_syscall_64+0x3d6/0x440 [ 2404.393493] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2404.401583] RIP: 0033:0x7fa749552cd8 [ 2404.411349] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2404.433939] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2404.444249] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2404.450912] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2404.454960] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2404.466796] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2404.472238] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2416.068315] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2416.086163] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2416.115260] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2416.120098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2416.124367] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2416.128030] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2416.137917] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2416.142587] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2416.148565] RDX: 0000000000007f2f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2416.171322] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2416.176047] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2416.179312] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2416.185316] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2416.199044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2416.204549] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2416.214061] Call Trace: [ 2416.217314] [ 2416.218991] ? show_regs.cold.9+0x22/0x2f [ 2416.223012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2416.226755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2416.229721] ? watchdog+0x40/0x40 [ 2416.232719] ? hrtimer_interrupt+0x128/0x2c0 [ 2416.234982] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2416.238706] ? apic_timer_interrupt+0xf/0x20 [ 2416.242961] [ 2416.246033] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2416.259935] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2416.265101] do_raw_spin_lock+0x177/0x190 [ 2416.269972] _raw_spin_lock+0x21/0x30 [ 2416.273937] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2416.289047] obd_mod_free+0x46/0x140 [ptlrpc] [ 2416.296853] mdc_commit_open+0x91/0xb0 [mdc] [ 2416.302913] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2416.308555] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2416.313017] after_reply+0xe2c/0x1780 [ptlrpc] [ 2416.320213] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2416.331593] ? __next_timer_interrupt+0x160/0x160 [ 2416.335679] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2416.345169] ? do_wait_intr+0xf0/0xf0 [ 2416.347365] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2416.351004] kthread+0x1d1/0x200 [ 2416.354831] ? set_kthread_struct+0x70/0x70 [ 2416.357157] ret_from_fork+0x1f/0x30 [ 2420.057193] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2420.067142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2420.106679] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2420.121084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2420.127027] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2420.132225] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2420.135711] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2420.153828] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2420.158789] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2420.163580] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2420.172809] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2420.183388] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2420.189848] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2420.197838] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2420.204523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2420.207798] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2420.216729] Call Trace: [ 2420.220973] [ 2420.222200] ? show_regs.cold.9+0x22/0x2f [ 2420.226029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2420.229430] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2420.235733] ? watchdog+0x40/0x40 [ 2420.239530] ? hrtimer_interrupt+0x128/0x2c0 [ 2420.244721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2420.249787] ? apic_timer_interrupt+0xf/0x20 [ 2420.252472] [ 2420.253534] ? native_safe_halt+0x2e/0x50 [ 2420.257895] kvm_wait+0xcd/0xe0 [ 2420.260547] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2420.265281] do_raw_spin_lock+0x177/0x190 [ 2420.271048] _raw_spin_lock+0x21/0x30 [ 2420.273900] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2420.278923] process_one_work+0x2c8/0x700 [ 2420.281493] worker_thread+0x296/0x6e0 [ 2420.286534] ? rescuer_thread+0x570/0x570 [ 2420.292776] kthread+0x1d1/0x200 [ 2420.296936] ? set_kthread_struct+0x70/0x70 [ 2420.303588] ret_from_fork+0x1f/0x30 [ 2428.057504] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2428.063119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.104832] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.112300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.116299] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.118470] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.125296] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.127966] RAX: 0000000000007b1d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2428.130728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2428.133533] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2428.136013] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2428.138659] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2428.141388] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2428.144304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.145948] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2428.148036] Call Trace: [ 2428.148698] [ 2428.149311] ? show_regs.cold.9+0x22/0x2f [ 2428.150335] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.151560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.152663] ? watchdog+0x40/0x40 [ 2428.153570] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.154653] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.155821] ? apic_timer_interrupt+0xf/0x20 [ 2428.156885] [ 2428.157551] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.158853] do_raw_spin_lock+0x177/0x190 [ 2428.159875] _raw_spin_lock+0x21/0x30 [ 2428.160822] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2428.162409] ? do_raw_spin_unlock+0x75/0x190 [ 2428.163364] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2428.164746] ? do_wait_intr+0xf0/0xf0 [ 2428.165596] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2428.167437] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2428.169626] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2428.171661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.173763] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2428.175868] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2428.177696] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2428.179453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.181519] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2428.183908] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2428.186084] lmv_intent_open+0x338/0x1150 [lmv] [ 2428.187891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.189952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.192024] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2428.193771] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2428.195610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.198023] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2428.200020] ? do_raw_spin_unlock+0x75/0x190 [ 2428.201712] ? _raw_spin_unlock+0x12/0x30 [ 2428.203286] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2428.205291] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2428.207187] ll_atomic_open+0x36a/0xd70 [lustre] [ 2428.208916] lookup_open+0x356/0xb40 [ 2428.210346] do_last+0x313/0xfc0 [ 2428.211619] ? nd_jump_root+0xe5/0x160 [ 2428.213060] ? path_init+0x437/0x520 [ 2428.214474] path_openat+0xf7/0x500 [ 2428.215755] ? do_raw_spin_unlock+0x75/0x190 [ 2428.217421] do_filp_open+0x99/0x140 [ 2428.218759] ? getname_flags+0x6e/0x330 [ 2428.220295] ? __check_object_size+0xff/0x256 [ 2428.221980] ? do_raw_spin_unlock+0x75/0x190 [ 2428.223636] ? _raw_spin_unlock+0x12/0x30 [ 2428.225211] do_sys_openat2+0x2b4/0x410 [ 2428.226798] do_sys_open+0x73/0xa0 [ 2428.228183] __x64_sys_openat+0x24/0x30 [ 2428.229732] do_syscall_64+0xc1/0x440 [ 2428.231084] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2428.233017] RIP: 0033:0x7fa749552332 [ 2428.234503] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2428.242061] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2428.246702] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2428.250306] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2428.253449] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2428.255998] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2428.259971] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2431.456051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 514s! [ 2431.459381] Showing busy workqueues and worker pools: [ 2431.461756] workqueue events: flags=0x0 [ 2431.463409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2431.466646] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2431.472537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2431.475703] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2431.479741] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.482719] pending: kfree_rcu_monitor [ 2431.484682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2431.488222] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2431.491197] workqueue events_long: flags=0x0 [ 2431.492773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.496327] pending: lu_objects_flush [obdclass] [ 2431.498861] workqueue events_unbound: flags=0x2 [ 2431.501258] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2431.503838] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2431.508326] workqueue events_power_efficient: flags=0x80 [ 2431.510956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.514306] pending: check_lifetime [ 2431.515758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.518599] pending: neigh_periodic_work [ 2431.520824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.523567] pending: neigh_periodic_work [ 2431.525560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.528728] pending: do_cache_clean [ 2431.530636] workqueue mm_percpu_wq: flags=0x8 [ 2431.532569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.534976] pending: vmstat_update [ 2431.536731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.540291] pending: vmstat_update [ 2431.541799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2431.545614] pending: vmstat_update [ 2431.550364] workqueue writeback: flags=0x4a [ 2431.552256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2431.555319] pending: wb_workfn, wb_workfn [ 2431.557503] workqueue xprtiod: flags=0xa [ 2431.559585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2431.562836] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2431.566560] workqueue ipv6_addrconf: flags=0x40008 [ 2431.568790] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2431.571957] pending: addrconf_verify_work [ 2431.573949] workqueue knbd0-recv: flags=0x1a [ 2431.576278] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2431.579487] in-flight: 651:recv_work [ 2431.581481] workqueue ptlrpc_pinger: flags=0x2 [ 2431.583536] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2431.586541] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2431.590269] pool 8: cpus=0-3 flags=0x4 nice=0 hung=514s workers=4 idle: 37 31846 28897 [ 2431.594476] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2421s workers=2 idle: 653 [ 2432.061268] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2432.082461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2432.169955] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2432.197728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2432.220629] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2432.233877] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2432.266075] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2432.276251] RAX: 0000000000007d06 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2432.291319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2432.297660] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2432.314544] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2432.318022] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2432.326686] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2432.330616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2432.333466] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2432.341667] Call Trace: [ 2432.342454] [ 2432.344634] ? show_regs.cold.9+0x22/0x2f [ 2432.353715] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2432.364152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2432.365893] ? watchdog+0x40/0x40 [ 2432.367141] ? hrtimer_interrupt+0x128/0x2c0 [ 2432.381000] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2432.386111] ? apic_timer_interrupt+0xf/0x20 [ 2432.391969] [ 2432.394660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2432.400323] do_raw_spin_lock+0x177/0x190 [ 2432.405015] _raw_spin_lock+0x21/0x30 [ 2432.409644] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2432.413946] ? do_raw_spin_unlock+0x75/0x190 [ 2432.415877] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2432.419355] ? do_wait_intr+0xf0/0xf0 [ 2432.421365] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2432.427973] mdc_close+0x53b/0xd70 [mdc] [ 2432.432921] lmv_close+0x18d/0x490 [lmv] [ 2432.446546] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2432.460215] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2432.464690] ll_md_close+0x274/0x800 [lustre] [ 2432.466431] ll_file_release+0x128/0xc50 [lustre] [ 2432.484793] __fput+0x10d/0x440 [ 2432.493696] ____fput+0x12/0x20 [ 2432.498158] task_work_run+0xc6/0x110 [ 2432.503987] exit_to_usermode_loop+0x1dd/0x1f0 [ 2432.508401] do_syscall_64+0x3d6/0x440 [ 2432.509801] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2432.520872] RIP: 0033:0x7fa749552cd8 [ 2432.523963] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2432.546124] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2432.551306] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2432.555036] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2432.559142] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2432.566956] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2432.571371] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2444.058239] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2444.061493] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2444.079954] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2444.084096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2444.087757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2444.090313] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2444.098288] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2444.101371] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2444.104279] RDX: 0000000000007d13 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2444.107274] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2444.110179] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2444.112637] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2444.115601] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2444.118857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2444.121302] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2444.124175] Call Trace: [ 2444.125253] [ 2444.125962] ? show_regs.cold.9+0x22/0x2f [ 2444.127366] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2444.129366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2444.131361] ? watchdog+0x40/0x40 [ 2444.132642] ? hrtimer_interrupt+0x128/0x2c0 [ 2444.134467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2444.136593] ? apic_timer_interrupt+0xf/0x20 [ 2444.138637] [ 2444.139600] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2444.142077] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2444.144174] do_raw_spin_lock+0x177/0x190 [ 2444.145792] _raw_spin_lock+0x21/0x30 [ 2444.147473] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2444.150009] obd_mod_free+0x46/0x140 [ptlrpc] [ 2444.152304] mdc_commit_open+0x91/0xb0 [mdc] [ 2444.154515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2444.156591] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2444.158828] after_reply+0xe2c/0x1780 [ptlrpc] [ 2444.160490] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2444.163173] ? __next_timer_interrupt+0x160/0x160 [ 2444.165219] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2444.167263] ? do_wait_intr+0xf0/0xf0 [ 2444.168687] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2444.171254] kthread+0x1d1/0x200 [ 2444.172450] ? set_kthread_struct+0x70/0x70 [ 2444.174235] ret_from_fork+0x1f/0x30 [ 2448.057029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2448.064038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2448.085027] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2448.095338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2448.104310] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2448.111130] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2448.112975] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2448.121172] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2448.127830] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2448.140720] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2448.150119] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2448.155630] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2448.162635] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2448.169198] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2448.174918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2448.178326] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2448.182159] Call Trace: [ 2448.185338] [ 2448.186198] ? show_regs.cold.9+0x22/0x2f [ 2448.189883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2448.192632] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2448.196996] ? watchdog+0x40/0x40 [ 2448.199236] ? hrtimer_interrupt+0x128/0x2c0 [ 2448.201421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2448.204436] ? apic_timer_interrupt+0xf/0x20 [ 2448.207609] [ 2448.210129] ? native_safe_halt+0x2e/0x50 [ 2448.217696] kvm_wait+0xcd/0xe0 [ 2448.220344] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2448.222390] do_raw_spin_lock+0x177/0x190 [ 2448.234171] _raw_spin_lock+0x21/0x30 [ 2448.236148] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2448.238413] process_one_work+0x2c8/0x700 [ 2448.240937] worker_thread+0x296/0x6e0 [ 2448.242949] ? rescuer_thread+0x570/0x570 [ 2448.244707] kthread+0x1d1/0x200 [ 2448.246072] ? set_kthread_struct+0x70/0x70 [ 2448.247847] ret_from_fork+0x1f/0x30 [ 2456.056511] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2456.062349] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.082544] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.087541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.092401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.094851] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2456.104954] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2456.108181] RAX: 0000000000002783 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2456.110890] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2456.114402] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2456.118493] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2456.122352] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2456.126294] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2456.129496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.132040] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2456.137296] Call Trace: [ 2456.138389] [ 2456.139498] ? show_regs.cold.9+0x22/0x2f [ 2456.141477] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.143850] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.145973] ? watchdog+0x40/0x40 [ 2456.147494] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.149414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.151898] ? apic_timer_interrupt+0xf/0x20 [ 2456.154478] [ 2456.155522] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.157912] do_raw_spin_lock+0x177/0x190 [ 2456.159700] _raw_spin_lock+0x21/0x30 [ 2456.161437] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2456.167211] ? do_raw_spin_unlock+0x75/0x190 [ 2456.168718] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2456.172845] ? do_wait_intr+0xf0/0xf0 [ 2456.174455] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2456.176994] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2456.179816] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2456.182587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.185152] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2456.187773] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2456.189808] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2456.192605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.195821] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2456.199226] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2456.202056] lmv_intent_open+0x338/0x1150 [lmv] [ 2456.203998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.206471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.209282] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2456.211172] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2456.215112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.218033] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2456.219867] ? do_raw_spin_unlock+0x75/0x190 [ 2456.224601] ? _raw_spin_unlock+0x12/0x30 [ 2456.227372] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2456.230081] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2456.233540] ll_atomic_open+0x36a/0xd70 [lustre] [ 2456.236641] lookup_open+0x356/0xb40 [ 2456.241975] do_last+0x313/0xfc0 [ 2456.243552] ? nd_jump_root+0xe5/0x160 [ 2456.245722] ? path_init+0x437/0x520 [ 2456.247726] path_openat+0xf7/0x500 [ 2456.249460] ? do_raw_spin_unlock+0x75/0x190 [ 2456.252079] do_filp_open+0x99/0x140 [ 2456.254296] ? getname_flags+0x6e/0x330 [ 2456.257485] ? __check_object_size+0xff/0x256 [ 2456.259620] ? do_raw_spin_unlock+0x75/0x190 [ 2456.261440] ? _raw_spin_unlock+0x12/0x30 [ 2456.262944] do_sys_openat2+0x2b4/0x410 [ 2456.266590] do_sys_open+0x73/0xa0 [ 2456.270192] __x64_sys_openat+0x24/0x30 [ 2456.272580] do_syscall_64+0xc1/0x440 [ 2456.275592] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2456.279043] RIP: 0033:0x7fa749552332 [ 2456.281124] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2456.293491] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2456.296907] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2456.300803] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2456.304282] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2456.308571] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2456.312031] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2460.059774] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2460.067449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2460.122029] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2460.161336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2460.173019] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2460.178467] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2460.204311] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2460.214505] RAX: 0000000000004983 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2460.222220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2460.224734] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2460.239647] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2460.253499] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2460.262107] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2460.277866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2460.286485] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2460.292182] Call Trace: [ 2460.300865] [ 2460.303011] ? show_regs.cold.9+0x22/0x2f [ 2460.306659] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2460.309297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2460.312296] ? watchdog+0x40/0x40 [ 2460.317336] ? hrtimer_interrupt+0x128/0x2c0 [ 2460.321200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2460.326444] ? apic_timer_interrupt+0xf/0x20 [ 2460.331187] [ 2460.334042] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2460.340084] do_raw_spin_lock+0x177/0x190 [ 2460.345204] _raw_spin_lock+0x21/0x30 [ 2460.353533] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2460.365254] ? do_raw_spin_unlock+0x75/0x190 [ 2460.376258] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2460.382356] ? do_wait_intr+0xf0/0xf0 [ 2460.397204] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2460.407374] mdc_close+0x53b/0xd70 [mdc] [ 2460.414942] lmv_close+0x18d/0x490 [lmv] [ 2460.421589] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2460.427340] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2460.435650] ll_md_close+0x274/0x800 [lustre] [ 2460.441828] ll_file_release+0x128/0xc50 [lustre] [ 2460.443544] __fput+0x10d/0x440 [ 2460.446642] ____fput+0x12/0x20 [ 2460.450183] task_work_run+0xc6/0x110 [ 2460.452518] exit_to_usermode_loop+0x1dd/0x1f0 [ 2460.463807] do_syscall_64+0x3d6/0x440 [ 2460.468687] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2460.470468] RIP: 0033:0x7fa749552cd8 [ 2460.481034] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2460.502652] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2460.511151] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2460.516878] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2460.522629] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2460.528981] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2460.532235] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2461.666201] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 544s! [ 2461.677893] Showing busy workqueues and worker pools: [ 2461.680652] workqueue events: flags=0x0 [ 2461.682554] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2461.686222] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2461.693763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2461.709486] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2461.722254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.729136] pending: kfree_rcu_monitor [ 2461.730596] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2461.737523] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2461.745925] workqueue events_long: flags=0x0 [ 2461.750298] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.757184] pending: lu_objects_flush [obdclass] [ 2461.764310] workqueue events_unbound: flags=0x2 [ 2461.766553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2461.772186] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2461.778083] workqueue events_power_efficient: flags=0x80 [ 2461.781121] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.784531] pending: check_lifetime [ 2461.786536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.789959] pending: neigh_periodic_work [ 2461.792178] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.794982] pending: neigh_periodic_work [ 2461.796799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.799916] pending: do_cache_clean [ 2461.802056] workqueue mm_percpu_wq: flags=0x8 [ 2461.804131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.807314] pending: vmstat_update [ 2461.809715] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.812786] pending: vmstat_update [ 2461.814487] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2461.817858] pending: vmstat_update [ 2461.819985] workqueue writeback: flags=0x4a [ 2461.822407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2461.825731] pending: wb_workfn, wb_workfn [ 2461.827569] workqueue xprtiod: flags=0xa [ 2461.829175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2461.832165] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2461.835758] workqueue ipv6_addrconf: flags=0x40008 [ 2461.837831] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2461.841252] pending: addrconf_verify_work [ 2461.843453] workqueue knbd0-recv: flags=0x1a [ 2461.845258] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2461.848779] in-flight: 651:recv_work [ 2461.850703] workqueue ptlrpc_pinger: flags=0x2 [ 2461.852870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2461.856047] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2461.858425] pool 8: cpus=0-3 flags=0x4 nice=0 hung=544s workers=4 idle: 37 31846 28897 [ 2461.862223] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2451s workers=2 idle: 653 [ 2472.058052] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2472.062126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2472.105493] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2472.111364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2472.119979] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2472.125979] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2472.138672] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2472.143161] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2472.147374] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2472.153858] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2472.158905] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2472.167165] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2472.172668] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2472.179505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2472.181423] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2472.190013] Call Trace: [ 2472.191583] [ 2472.192575] ? show_regs.cold.9+0x22/0x2f [ 2472.194876] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2472.197826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2472.201552] ? watchdog+0x40/0x40 [ 2472.204622] ? hrtimer_interrupt+0x128/0x2c0 [ 2472.210334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2472.213521] ? apic_timer_interrupt+0xf/0x20 [ 2472.218034] [ 2472.218840] ? native_safe_halt+0x2e/0x50 [ 2472.220224] kvm_wait+0xcd/0xe0 [ 2472.227881] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2472.232320] do_raw_spin_lock+0x177/0x190 [ 2472.234880] _raw_spin_lock+0x21/0x30 [ 2472.239154] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2472.246145] obd_mod_free+0x46/0x140 [ptlrpc] [ 2472.249151] mdc_commit_open+0x91/0xb0 [mdc] [ 2472.252581] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2472.258881] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2472.266407] after_reply+0xe2c/0x1780 [ptlrpc] [ 2472.268062] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2472.270803] ? __next_timer_interrupt+0x160/0x160 [ 2472.273727] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2472.277538] ? do_wait_intr+0xf0/0xf0 [ 2472.281014] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2472.283850] kthread+0x1d1/0x200 [ 2472.289142] ? set_kthread_struct+0x70/0x70 [ 2472.291471] ret_from_fork+0x1f/0x30 [ 2476.064710] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2476.080405] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2476.148238] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2476.167563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2476.174492] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2476.183449] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2476.192088] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2476.221087] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2476.228420] RAX: 0000000000004709 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2476.233133] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2476.238652] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2476.249939] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2476.258368] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2476.272206] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2476.276775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2476.278532] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2476.289387] Call Trace: [ 2476.290234] [ 2476.302268] ? show_regs.cold.9+0x22/0x2f [ 2476.303685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2476.305316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2476.320765] ? watchdog+0x40/0x40 [ 2476.329524] ? hrtimer_interrupt+0x128/0x2c0 [ 2476.335738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2476.337415] ? apic_timer_interrupt+0xf/0x20 [ 2476.349267] [ 2476.355735] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2476.365886] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2476.369201] do_raw_spin_lock+0x177/0x190 [ 2476.373898] _raw_spin_lock+0x21/0x30 [ 2476.376019] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2476.382878] process_one_work+0x2c8/0x700 [ 2476.386111] worker_thread+0x296/0x6e0 [ 2476.389064] ? rescuer_thread+0x570/0x570 [ 2476.391042] kthread+0x1d1/0x200 [ 2476.397013] ? set_kthread_struct+0x70/0x70 [ 2476.401349] ret_from_fork+0x1f/0x30 [ 2488.054082] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2488.059030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2488.063412] Modules linked in: [ 2488.066315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2488.066407] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2488.066414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2488.066417] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2488.066429] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2488.066433] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2488.066438] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2488.066440] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2488.066442] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2488.066444] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2488.066447] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2488.066450] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2488.066453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2488.066455] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2488.066463] Call Trace: [ 2488.066468] [ 2488.066474] ? show_regs.cold.9+0x22/0x2f [ 2488.066482] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2488.066488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2488.066492] ? watchdog+0x40/0x40 [ 2488.066496] ? hrtimer_interrupt+0x128/0x2c0 [ 2488.066500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2488.066507] ? apic_timer_interrupt+0xf/0x20 [ 2488.066511] [ 2488.066515] ? native_safe_halt+0x2e/0x50 [ 2488.066518] kvm_wait+0xcd/0xe0 [ 2488.066526] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2488.066535] do_raw_spin_lock+0x177/0x190 [ 2488.066550] _raw_spin_lock+0x21/0x30 [ 2488.066554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2488.066860] ? do_raw_spin_unlock+0x75/0x190 [ 2488.066866] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2488.067203] ? do_wait_intr+0xf0/0xf0 [ 2488.067211] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2488.067431] mdc_close+0x53b/0xd70 [mdc] [ 2488.067474] lmv_close+0x18d/0x490 [lmv] [ 2488.067504] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2488.067689] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2488.067783] ll_md_close+0x274/0x800 [lustre] [ 2488.067879] ll_file_release+0x128/0xc50 [lustre] [ 2488.068034] __fput+0x10d/0x440 [ 2488.068043] ____fput+0x12/0x20 [ 2488.068047] task_work_run+0xc6/0x110 [ 2488.068054] exit_to_usermode_loop+0x1dd/0x1f0 [ 2488.068061] do_syscall_64+0x3d6/0x440 [ 2488.068065] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2488.068072] RIP: 0033:0x7fa749552cd8 [ 2488.068079] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2488.068082] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2488.068088] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2488.068090] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2488.068092] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2488.068094] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2488.068097] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2488.331143] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2488.358215] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2488.362555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2488.367945] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2488.370648] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2488.379682] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2488.384746] RAX: 000000000000471c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2488.390814] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2488.393551] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2488.395689] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2488.400388] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2488.402814] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2488.405759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2488.408062] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2488.410424] Call Trace: [ 2488.411016] [ 2488.412295] ? show_regs.cold.9+0x22/0x2f [ 2488.413491] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2488.414858] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2488.420457] ? watchdog+0x40/0x40 [ 2488.421857] ? hrtimer_interrupt+0x128/0x2c0 [ 2488.422983] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2488.427990] ? apic_timer_interrupt+0xf/0x20 [ 2488.432955] [ 2488.433840] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2488.436347] do_raw_spin_lock+0x177/0x190 [ 2488.437544] _raw_spin_lock+0x21/0x30 [ 2488.440816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2488.444552] ? do_raw_spin_unlock+0x75/0x190 [ 2488.453781] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2488.455599] ? do_wait_intr+0xf0/0xf0 [ 2488.457891] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2488.460324] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2488.463396] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2488.468533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2488.472053] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2488.479556] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2488.482919] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2488.487739] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2488.498515] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2488.500780] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2488.504047] lmv_intent_open+0x338/0x1150 [lmv] [ 2488.507751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2488.517203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2488.521727] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2488.524021] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2488.526084] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2488.534768] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2488.539030] ? do_raw_spin_unlock+0x75/0x190 [ 2488.540363] ? _raw_spin_unlock+0x12/0x30 [ 2488.543954] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2488.550629] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2488.562536] ll_atomic_open+0x36a/0xd70 [lustre] [ 2488.568891] lookup_open+0x356/0xb40 [ 2488.571755] do_last+0x313/0xfc0 [ 2488.577465] ? nd_jump_root+0xe5/0x160 [ 2488.580890] ? path_init+0x437/0x520 [ 2488.583905] path_openat+0xf7/0x500 [ 2488.585917] ? do_raw_spin_unlock+0x75/0x190 [ 2488.596403] do_filp_open+0x99/0x140 [ 2488.599425] ? getname_flags+0x6e/0x330 [ 2488.601831] ? __check_object_size+0xff/0x256 [ 2488.603796] ? do_raw_spin_unlock+0x75/0x190 [ 2488.607723] ? _raw_spin_unlock+0x12/0x30 [ 2488.611925] do_sys_openat2+0x2b4/0x410 [ 2488.615510] do_sys_open+0x73/0xa0 [ 2488.619862] __x64_sys_openat+0x24/0x30 [ 2488.623819] do_syscall_64+0xc1/0x440 [ 2488.628071] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2488.635002] RIP: 0033:0x7fa749552332 [ 2488.639577] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2488.653720] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2488.658386] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2488.661688] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2488.665515] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2488.669415] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2488.672786] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2491.872678] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 574s! [ 2491.877364] Showing busy workqueues and worker pools: [ 2491.880437] workqueue events: flags=0x0 [ 2491.882413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2491.887462] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2491.894414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2491.900455] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2491.906865] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2491.912201] pending: kfree_rcu_monitor [ 2491.914503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2491.920860] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2491.923512] workqueue events_long: flags=0x0 [ 2491.925670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2491.928345] pending: lu_objects_flush [obdclass] [ 2491.934818] workqueue events_unbound: flags=0x2 [ 2491.940120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2491.943328] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2491.955711] workqueue events_power_efficient: flags=0x80 [ 2491.961982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2491.969129] pending: check_lifetime [ 2491.973875] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2491.979652] pending: neigh_periodic_work [ 2491.981221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2491.988094] pending: neigh_periodic_work [ 2491.990261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2492.000090] pending: do_cache_clean [ 2492.005571] workqueue mm_percpu_wq: flags=0x8 [ 2492.012983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2492.022112] pending: vmstat_update [ 2492.026553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2492.032907] pending: vmstat_update [ 2492.034936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2492.047127] pending: vmstat_update [ 2492.051487] workqueue writeback: flags=0x4a [ 2492.057504] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2492.060313] pending: wb_workfn, wb_workfn [ 2492.062473] workqueue xprtiod: flags=0xa [ 2492.067534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2492.069978] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2492.075458] workqueue ipv6_addrconf: flags=0x40008 [ 2492.079041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2492.081790] pending: addrconf_verify_work [ 2492.084410] workqueue knbd0-recv: flags=0x1a [ 2492.085784] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2492.095526] in-flight: 651:recv_work [ 2492.096532] workqueue ptlrpc_pinger: flags=0x2 [ 2492.105616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2492.110527] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2492.118976] pool 8: cpus=0-3 flags=0x4 nice=0 hung=574s workers=4 idle: 37 31846 28897 [ 2492.127639] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2482s workers=2 idle: 653 [ 2500.063618] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2500.067096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2500.085937] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2500.090532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2500.094035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2500.097243] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2500.114359] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2500.126032] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2500.141253] RDX: 0000000000001bad RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2500.147021] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2500.154497] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2500.161681] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2500.165654] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2500.171882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2500.181778] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2500.192857] Call Trace: [ 2500.196091] [ 2500.198361] ? show_regs.cold.9+0x22/0x2f [ 2500.200247] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2500.201868] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2500.204644] ? watchdog+0x40/0x40 [ 2500.205880] ? hrtimer_interrupt+0x128/0x2c0 [ 2500.208140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2500.210223] ? apic_timer_interrupt+0xf/0x20 [ 2500.212470] [ 2500.213371] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2500.215454] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2500.218166] do_raw_spin_lock+0x177/0x190 [ 2500.220030] _raw_spin_lock+0x21/0x30 [ 2500.221696] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2500.224221] obd_mod_free+0x46/0x140 [ptlrpc] [ 2500.226379] mdc_commit_open+0x91/0xb0 [mdc] [ 2500.228096] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2500.230092] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2500.232173] after_reply+0xe2c/0x1780 [ptlrpc] [ 2500.234264] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2500.236483] ? __next_timer_interrupt+0x160/0x160 [ 2500.238165] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2500.239767] ? do_wait_intr+0xf0/0xf0 [ 2500.241039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2500.243028] kthread+0x1d1/0x200 [ 2500.244412] ? set_kthread_struct+0x70/0x70 [ 2500.245855] ret_from_fork+0x1f/0x30 [ 2504.057140] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2504.059152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2504.092717] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2504.104754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2504.110748] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2504.116011] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2504.117637] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2504.125419] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2504.128533] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2504.131231] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2504.134130] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2504.137121] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2504.139927] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2504.142907] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2504.146098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2504.148393] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2504.151406] Call Trace: [ 2504.152361] [ 2504.153333] ? show_regs.cold.9+0x22/0x2f [ 2504.154887] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2504.157054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2504.158640] ? watchdog+0x40/0x40 [ 2504.160044] ? hrtimer_interrupt+0x128/0x2c0 [ 2504.161556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2504.163448] ? apic_timer_interrupt+0xf/0x20 [ 2504.165135] [ 2504.165895] ? native_safe_halt+0x2e/0x50 [ 2504.167528] kvm_wait+0xcd/0xe0 [ 2504.168777] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2504.171064] do_raw_spin_lock+0x177/0x190 [ 2504.172572] _raw_spin_lock+0x21/0x30 [ 2504.174134] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2504.176453] process_one_work+0x2c8/0x700 [ 2504.178110] worker_thread+0x296/0x6e0 [ 2504.179646] ? rescuer_thread+0x570/0x570 [ 2504.181312] kthread+0x1d1/0x200 [ 2504.182442] ? set_kthread_struct+0x70/0x70 [ 2504.183536] ret_from_fork+0x1f/0x30 [ 2513.701203] rcu: INFO: rcu_sched self-detected stall on CPU [ 2513.705273] rcu: 2-...!: (330897 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 2513.717329] (t=600012 jiffies g=258701 q=1624) [ 2513.722099] rcu: rcu_sched kthread starved for 596807 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 2513.729589] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2513.739256] rcu: RCU grace-period kthread stack dump: [ 2513.744779] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2513.752851] Call Trace: [ 2513.754957] __schedule+0x351/0xcb0 [ 2513.758591] schedule+0xc0/0x180 [ 2513.760391] schedule_timeout+0xb4/0x190 [ 2513.762128] ? do_raw_spin_unlock+0x75/0x190 [ 2513.764557] ? __next_timer_interrupt+0x160/0x160 [ 2513.766049] ? prepare_to_swait_event+0xf3/0x140 [ 2513.771846] rcu_gp_kthread+0x8dd/0x15b0 [ 2513.774007] ? force_qs_rnp+0x2c0/0x2c0 [ 2513.781559] kthread+0x1d1/0x200 [ 2513.782887] ? set_kthread_struct+0x70/0x70 [ 2513.787747] ret_from_fork+0x1f/0x30 [ 2513.789437] rcu: Stack dump where RCU GP kthread last ran: [ 2513.791960] Sending NMI from CPU 2 to CPUs 3: [ 2513.799140] NMI backtrace for cpu 3 [ 2513.799146] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2513.799148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2513.799150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2513.799151] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2513.799153] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 [ 2513.799156] RAX: 0000000000006659 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2513.799157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2513.799158] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2513.799159] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2513.799160] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2513.799161] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2513.799162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2513.799163] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2513.799164] Call Trace: [ 2513.799165] [ 2513.799166] ? show_regs.cold.9+0x22/0x2f [ 2513.799166] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2513.799167] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2513.799168] ? nmi_handle+0xbd/0x210 [ 2513.799169] ? default_do_nmi+0x4b/0x1b0 [ 2513.799170] ? do_nmi+0x280/0x380 [ 2513.799170] ? end_repeat_nmi+0x16/0x1f [ 2513.799171] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2513.799172] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2513.799173] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2513.799174] [ 2513.799175] do_raw_spin_lock+0x177/0x190 [ 2513.799176] _raw_spin_lock+0x21/0x30 [ 2513.799177] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2513.799178] ? do_raw_spin_unlock+0x75/0x190 [ 2513.799179] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2513.799180] ? do_wait_intr+0xf0/0xf0 [ 2513.799180] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2513.799181] mdc_close+0x53b/0xd70 [mdc] [ 2513.799182] lmv_close+0x18d/0x490 [lmv] [ 2513.799183] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2513.799184] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2513.799185] ll_md_close+0x274/0x800 [lustre] [ 2513.799186] ll_file_release+0x128/0xc50 [lustre] [ 2513.799187] __fput+0x10d/0x440 [ 2513.799188] ____fput+0x12/0x20 [ 2513.799189] task_work_run+0xc6/0x110 [ 2513.799190] exit_to_usermode_loop+0x1dd/0x1f0 [ 2513.799191] do_syscall_64+0x3d6/0x440 [ 2513.799192] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2513.799193] RIP: 0033:0x7fa749552cd8 [ 2513.799194] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2513.799195] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2513.799197] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2513.799198] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2513.799199] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2513.799200] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2513.799201] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2513.801424] NMI backtrace for cpu 2 [ 2514.002603] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2514.009116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2514.013257] Call Trace: [ 2514.014357] [ 2514.017746] ? dump_stack+0xbb/0x10e [ 2514.019344] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2514.025513] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2514.027934] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2514.030373] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2514.034244] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2514.038357] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2514.050046] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2514.051930] ? update_process_times+0x6d/0xc0 [ 2514.055865] ? tick_sched_handle.isra.18+0x27/0x90 [ 2514.058465] ? tick_sched_timer+0x53/0xb0 [ 2514.060458] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2514.062502] ? tick_sched_handle.isra.18+0x90/0x90 [ 2514.066815] ? hrtimer_interrupt+0x128/0x2c0 [ 2514.072377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2514.077325] ? apic_timer_interrupt+0xf/0x20 [ 2514.079310] [ 2514.080431] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2514.084871] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2514.087475] ? do_raw_spin_lock+0x177/0x190 [ 2514.092342] ? _raw_spin_lock+0x21/0x30 [ 2514.095917] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2514.100778] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2514.108820] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2514.114396] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2514.118083] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2514.127087] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2514.129465] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2514.135341] ? __next_timer_interrupt+0x160/0x160 [ 2514.139521] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2514.143504] ? do_wait_intr+0xf0/0xf0 [ 2514.145277] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2514.152029] ? kthread+0x1d1/0x200 [ 2514.153391] ? set_kthread_struct+0x70/0x70 [ 2514.157233] ? ret_from_fork+0x1f/0x30 [ 2520.054585] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2520.060118] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2520.098183] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2520.106439] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2520.115938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2520.122104] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2520.143097] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2520.147345] RAX: 0000000000007a9d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2520.150559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2520.154250] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2520.157437] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2520.160661] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2520.164419] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2520.168001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2520.170419] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2520.173464] Call Trace: [ 2520.174758] [ 2520.175489] ? show_regs.cold.9+0x22/0x2f [ 2520.177396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2520.179578] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2520.183248] ? watchdog+0x40/0x40 [ 2520.185873] ? hrtimer_interrupt+0x128/0x2c0 [ 2520.187561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2520.190480] ? apic_timer_interrupt+0xf/0x20 [ 2520.193219] [ 2520.194161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2520.196508] do_raw_spin_lock+0x177/0x190 [ 2520.198789] _raw_spin_lock+0x21/0x30 [ 2520.201425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2520.204090] ? do_raw_spin_unlock+0x75/0x190 [ 2520.205934] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2520.208291] ? do_wait_intr+0xf0/0xf0 [ 2520.211322] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2520.215397] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2520.220877] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2520.224982] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.228711] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2520.234262] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2520.239294] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2520.241711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.245034] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2520.249335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2520.252029] lmv_intent_open+0x338/0x1150 [lmv] [ 2520.254324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.256338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.260614] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2520.262389] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2520.266251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.268860] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2520.273127] ? do_raw_spin_unlock+0x75/0x190 [ 2520.279786] ? _raw_spin_unlock+0x12/0x30 [ 2520.281638] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2520.286709] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2520.290979] ll_atomic_open+0x36a/0xd70 [lustre] [ 2520.293506] lookup_open+0x356/0xb40 [ 2520.294847] do_last+0x313/0xfc0 [ 2520.297005] ? nd_jump_root+0xe5/0x160 [ 2520.298919] ? path_init+0x437/0x520 [ 2520.300269] path_openat+0xf7/0x500 [ 2520.302389] ? do_raw_spin_unlock+0x75/0x190 [ 2520.303699] do_filp_open+0x99/0x140 [ 2520.305637] ? getname_flags+0x6e/0x330 [ 2520.307580] ? __check_object_size+0xff/0x256 [ 2520.309237] ? do_raw_spin_unlock+0x75/0x190 [ 2520.310971] ? _raw_spin_unlock+0x12/0x30 [ 2520.312305] do_sys_openat2+0x2b4/0x410 [ 2520.313929] do_sys_open+0x73/0xa0 [ 2520.315265] __x64_sys_openat+0x24/0x30 [ 2520.316979] do_syscall_64+0xc1/0x440 [ 2520.318493] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2520.320918] RIP: 0033:0x7fa749552332 [ 2520.322442] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2520.334980] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2520.338612] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2520.344384] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2520.350776] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2520.356869] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2520.360283] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2522.592699] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 605s! [ 2522.603303] Showing busy workqueues and worker pools: [ 2522.607825] workqueue events: flags=0x0 [ 2522.609322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2522.621377] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2522.639290] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2522.646802] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2522.659251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.666981] pending: kfree_rcu_monitor [ 2522.670105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2522.679398] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2522.686741] workqueue events_long: flags=0x0 [ 2522.693433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.700813] pending: lu_objects_flush [obdclass] [ 2522.706379] workqueue events_unbound: flags=0x2 [ 2522.717748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2522.731831] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2522.740117] workqueue events_power_efficient: flags=0x80 [ 2522.744257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.750434] pending: check_lifetime [ 2522.753112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.762063] pending: neigh_periodic_work [ 2522.767690] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.779092] pending: neigh_periodic_work [ 2522.784034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.796659] pending: do_cache_clean [ 2522.804497] workqueue mm_percpu_wq: flags=0x8 [ 2522.810744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.819734] pending: vmstat_update [ 2522.823374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.839089] pending: vmstat_update [ 2522.843740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2522.849327] pending: vmstat_update [ 2522.851817] workqueue writeback: flags=0x4a [ 2522.853385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2522.856451] pending: wb_workfn, wb_workfn [ 2522.858639] workqueue xprtiod: flags=0xa [ 2522.862451] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2522.873981] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2522.884948] workqueue ipv6_addrconf: flags=0x40008 [ 2522.895731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2522.904801] pending: addrconf_verify_work [ 2522.911438] workqueue knbd0-recv: flags=0x1a [ 2522.916064] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2522.921763] in-flight: 651:recv_work [ 2522.927785] workqueue ptlrpc_pinger: flags=0x2 [ 2522.930658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2522.933974] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2522.939445] pool 8: cpus=0-3 flags=0x4 nice=0 hung=605s workers=4 idle: 37 31846 28897 [ 2522.945831] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2512s workers=2 idle: 653 [ 2532.057096] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2532.060445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2532.109491] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2532.120436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2532.125438] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2532.129439] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2532.133019] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2532.140050] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2532.143380] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2532.146750] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2532.150158] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2532.153436] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2532.156791] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2532.159665] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2532.167274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2532.170056] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2532.179208] Call Trace: [ 2532.180709] [ 2532.182250] ? show_regs.cold.9+0x22/0x2f [ 2532.185335] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2532.187266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2532.193745] ? watchdog+0x40/0x40 [ 2532.195363] ? hrtimer_interrupt+0x128/0x2c0 [ 2532.200057] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2532.205726] ? apic_timer_interrupt+0xf/0x20 [ 2532.208488] [ 2532.209714] ? native_safe_halt+0x2e/0x50 [ 2532.212277] kvm_wait+0xcd/0xe0 [ 2532.213910] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2532.217388] do_raw_spin_lock+0x177/0x190 [ 2532.219665] _raw_spin_lock+0x21/0x30 [ 2532.221994] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2532.227018] process_one_work+0x2c8/0x700 [ 2532.230756] worker_thread+0x296/0x6e0 [ 2532.238087] ? rescuer_thread+0x570/0x570 [ 2532.242708] kthread+0x1d1/0x200 [ 2532.245221] ? set_kthread_struct+0x70/0x70 [ 2532.248014] ret_from_fork+0x1f/0x30 [ 2540.059086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2540.063988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.068515] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2540.081351] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.081364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.081366] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2540.081380] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2540.081384] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2540.081390] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2540.081392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2540.081394] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2540.081397] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2540.081399] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2540.081403] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2540.081406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.081408] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2540.081415] Call Trace: [ 2540.081420] [ 2540.081425] ? show_regs.cold.9+0x22/0x2f [ 2540.081433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.081439] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.081443] ? watchdog+0x40/0x40 [ 2540.081447] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.081463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.081471] ? apic_timer_interrupt+0xf/0x20 [ 2540.081475] [ 2540.081478] ? native_safe_halt+0x2e/0x50 [ 2540.081483] kvm_wait+0xcd/0xe0 [ 2540.081490] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2540.081500] do_raw_spin_lock+0x177/0x190 [ 2540.081505] _raw_spin_lock+0x21/0x30 [ 2540.081510] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2540.081771] ? do_raw_spin_unlock+0x75/0x190 [ 2540.081776] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2540.082104] ? do_wait_intr+0xf0/0xf0 [ 2540.082112] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2540.082295] mdc_close+0x53b/0xd70 [mdc] [ 2540.082323] lmv_close+0x18d/0x490 [lmv] [ 2540.082344] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2540.082676] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2540.082750] ll_md_close+0x274/0x800 [lustre] [ 2540.082916] ll_file_release+0x128/0xc50 [lustre] [ 2540.083003] __fput+0x10d/0x440 [ 2540.083012] ____fput+0x12/0x20 [ 2540.083017] task_work_run+0xc6/0x110 [ 2540.083022] exit_to_usermode_loop+0x1dd/0x1f0 [ 2540.083028] do_syscall_64+0x3d6/0x440 [ 2540.083032] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2540.083038] RIP: 0033:0x7fa749552cd8 [ 2540.083043] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2540.083046] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2540.083049] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2540.083051] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2540.083052] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2540.083054] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2540.083055] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2540.243960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.266740] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.273965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.277266] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2540.279093] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2540.298398] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.305509] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2540.310666] RDX: 0000000000001f6e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2540.316655] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2540.322744] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2540.327005] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2540.333608] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2540.342583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.347128] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2540.354634] Call Trace: [ 2540.356862] [ 2540.357923] ? show_regs.cold.9+0x22/0x2f [ 2540.361878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.365659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.375865] ? watchdog+0x40/0x40 [ 2540.381899] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.388594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.390188] ? apic_timer_interrupt+0xf/0x20 [ 2540.392252] [ 2540.393332] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2540.396133] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2540.398962] do_raw_spin_lock+0x177/0x190 [ 2540.401271] _raw_spin_lock+0x21/0x30 [ 2540.403243] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2540.406352] obd_mod_free+0x46/0x140 [ptlrpc] [ 2540.409390] mdc_commit_open+0x91/0xb0 [mdc] [ 2540.411759] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2540.415889] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2540.419306] after_reply+0xe2c/0x1780 [ptlrpc] [ 2540.422274] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2540.425486] ? __next_timer_interrupt+0x160/0x160 [ 2540.428213] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2540.430608] ? do_wait_intr+0xf0/0xf0 [ 2540.439351] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2540.442119] kthread+0x1d1/0x200 [ 2540.443599] ? set_kthread_struct+0x70/0x70 [ 2540.445639] ret_from_fork+0x1f/0x30 [ 2548.054027] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2548.056780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2548.142200] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2548.156226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2548.173210] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2548.178990] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2548.204365] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2548.212461] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2548.217267] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2548.226441] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2548.236646] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2548.255951] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2548.272684] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2548.294987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2548.306867] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2548.317371] Call Trace: [ 2548.319334] [ 2548.320976] ? show_regs.cold.9+0x22/0x2f [ 2548.324969] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2548.333625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2548.340136] ? watchdog+0x40/0x40 [ 2548.341217] ? hrtimer_interrupt+0x128/0x2c0 [ 2548.352371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2548.357888] ? apic_timer_interrupt+0xf/0x20 [ 2548.362479] [ 2548.364899] ? native_safe_halt+0x2e/0x50 [ 2548.371868] kvm_wait+0xcd/0xe0 [ 2548.378864] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2548.383785] do_raw_spin_lock+0x177/0x190 [ 2548.392770] _raw_spin_lock+0x21/0x30 [ 2548.398570] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2548.402379] ? do_raw_spin_unlock+0x75/0x190 [ 2548.403885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2548.410598] ? do_wait_intr+0xf0/0xf0 [ 2548.414652] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2548.419916] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2548.425621] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2548.429876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.434082] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2548.444795] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2548.455709] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2548.458174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.463010] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2548.471825] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2548.479213] lmv_intent_open+0x338/0x1150 [lmv] [ 2548.485881] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.487795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.494469] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2548.498536] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2548.507480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.509826] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2548.515525] ? do_raw_spin_unlock+0x75/0x190 [ 2548.524386] ? _raw_spin_unlock+0x12/0x30 [ 2548.529662] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2548.536519] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2548.539631] ll_atomic_open+0x36a/0xd70 [lustre] [ 2548.549888] lookup_open+0x356/0xb40 [ 2548.555613] do_last+0x313/0xfc0 [ 2548.559346] ? nd_jump_root+0xe5/0x160 [ 2548.567431] ? path_init+0x437/0x520 [ 2548.568816] path_openat+0xf7/0x500 [ 2548.579367] ? do_raw_spin_unlock+0x75/0x190 [ 2548.583357] do_filp_open+0x99/0x140 [ 2548.595519] ? getname_flags+0x6e/0x330 [ 2548.596944] ? __check_object_size+0xff/0x256 [ 2548.605340] ? do_raw_spin_unlock+0x75/0x190 [ 2548.610001] ? _raw_spin_unlock+0x12/0x30 [ 2548.612074] do_sys_openat2+0x2b4/0x410 [ 2548.614277] do_sys_open+0x73/0xa0 [ 2548.615353] __x64_sys_openat+0x24/0x30 [ 2548.616831] do_syscall_64+0xc1/0x440 [ 2548.619769] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2548.627522] RIP: 0033:0x7fa749552332 [ 2548.632616] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2548.654298] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2548.663657] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2548.672065] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2548.677440] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2548.681586] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2548.690367] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2553.322143] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 636s! [ 2553.330605] Showing busy workqueues and worker pools: [ 2553.332152] workqueue events: flags=0x0 [ 2553.333326] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2553.335377] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2553.343522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2553.349695] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2553.358016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.368887] pending: kfree_rcu_monitor [ 2553.370018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2553.375253] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2553.378344] workqueue events_long: flags=0x0 [ 2553.380055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.382203] pending: lu_objects_flush [obdclass] [ 2553.383829] workqueue events_unbound: flags=0x2 [ 2553.387169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2553.389875] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2553.393812] workqueue events_power_efficient: flags=0x80 [ 2553.395929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.399415] pending: check_lifetime [ 2553.401091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.403668] pending: neigh_periodic_work [ 2553.406323] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.409239] pending: neigh_periodic_work [ 2553.411405] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.414826] pending: do_cache_clean [ 2553.416799] workqueue mm_percpu_wq: flags=0x8 [ 2553.418776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.421796] pending: vmstat_update [ 2553.423345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.426509] pending: vmstat_update [ 2553.428412] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2553.431501] pending: vmstat_update [ 2553.433045] workqueue writeback: flags=0x4a [ 2553.435196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2553.437677] pending: wb_workfn, wb_workfn [ 2553.439345] workqueue xprtiod: flags=0xa [ 2553.441151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2553.444374] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2553.447583] workqueue ipv6_addrconf: flags=0x40008 [ 2553.449471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2553.452493] pending: addrconf_verify_work [ 2553.454454] workqueue knbd0-recv: flags=0x1a [ 2553.456164] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2553.459059] in-flight: 651:recv_work [ 2553.460696] workqueue ptlrpc_pinger: flags=0x2 [ 2553.462265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2553.465635] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2553.468040] pool 8: cpus=0-3 flags=0x4 nice=0 hung=636s workers=4 idle: 37 31846 28897 [ 2553.471546] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2543s workers=2 idle: 653 [ 2560.057022] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2560.062088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2560.092653] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2560.100822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2560.109297] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2560.119141] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2560.122981] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2560.132167] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2560.135628] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2560.140524] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2560.143910] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2560.149221] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2560.153936] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2560.157668] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2560.160926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2560.163177] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2560.165582] Call Trace: [ 2560.166951] [ 2560.167998] ? show_regs.cold.9+0x22/0x2f [ 2560.170516] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2560.172434] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2560.175440] ? watchdog+0x40/0x40 [ 2560.178709] ? hrtimer_interrupt+0x128/0x2c0 [ 2560.183597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2560.186638] ? apic_timer_interrupt+0xf/0x20 [ 2560.188871] [ 2560.189461] ? native_safe_halt+0x2e/0x50 [ 2560.190987] kvm_wait+0xcd/0xe0 [ 2560.192921] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2560.194781] do_raw_spin_lock+0x177/0x190 [ 2560.201138] _raw_spin_lock+0x21/0x30 [ 2560.204483] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2560.207211] process_one_work+0x2c8/0x700 [ 2560.210908] worker_thread+0x296/0x6e0 [ 2560.213216] ? rescuer_thread+0x570/0x570 [ 2560.216504] kthread+0x1d1/0x200 [ 2560.217634] ? set_kthread_struct+0x70/0x70 [ 2560.219874] ret_from_fork+0x1f/0x30 [ 2568.058085] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2568.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2568.065643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.065881] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.065889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.065892] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2568.065908] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2568.065912] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2568.065917] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2568.065919] RDX: 00000000000074f2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2568.065922] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2568.065924] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2568.065926] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2568.065929] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2568.065932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.065934] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2568.065943] Call Trace: [ 2568.065950] [ 2568.065956] ? show_regs.cold.9+0x22/0x2f [ 2568.065986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.065994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.065999] ? watchdog+0x40/0x40 [ 2568.066004] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.066008] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.066014] ? apic_timer_interrupt+0xf/0x20 [ 2568.066020] [ 2568.066023] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2568.066029] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2568.066035] do_raw_spin_lock+0x177/0x190 [ 2568.066041] _raw_spin_lock+0x21/0x30 [ 2568.066048] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2568.066316] obd_mod_free+0x46/0x140 [ptlrpc] [ 2568.070143] Modules linked in: lustre(O) [ 2568.131662] mdc_commit_open+0x91/0xb0 [mdc] [ 2568.131781] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2568.132176] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2568.142640] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.142763] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.142770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.142773] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.142786] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2568.142790] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2568.142795] RAX: 0000000000007890 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2568.142797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2568.142799] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2568.142801] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2568.142804] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2568.142807] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2568.142810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.142812] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2568.142820] Call Trace: [ 2568.142825] [ 2568.142830] ? show_regs.cold.9+0x22/0x2f [ 2568.143456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.143463] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.143467] ? watchdog+0x40/0x40 [ 2568.143471] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.143475] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.143483] ? apic_timer_interrupt+0xf/0x20 [ 2568.143488] [ 2568.143491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.143499] do_raw_spin_lock+0x177/0x190 [ 2568.143504] _raw_spin_lock+0x21/0x30 [ 2568.143508] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2568.151677] after_reply+0xe2c/0x1780 [ptlrpc] [ 2568.152127] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2568.152666] ? __next_timer_interrupt+0x160/0x160 [ 2568.152798] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2568.153015] ? do_wait_intr+0xf0/0xf0 [ 2568.153024] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2568.153232] kthread+0x1d1/0x200 [ 2568.153238] ? set_kthread_struct+0x70/0x70 [ 2568.153242] ret_from_fork+0x1f/0x30 [ 2568.642473] ? do_raw_spin_unlock+0x75/0x190 [ 2568.644967] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2568.648474] ? do_wait_intr+0xf0/0xf0 [ 2568.650566] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2568.656019] mdc_close+0x53b/0xd70 [mdc] [ 2568.659006] lmv_close+0x18d/0x490 [lmv] [ 2568.661027] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2568.668647] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2568.674014] ll_md_close+0x274/0x800 [lustre] [ 2568.677808] ll_file_release+0x128/0xc50 [lustre] [ 2568.681176] __fput+0x10d/0x440 [ 2568.683396] ____fput+0x12/0x20 [ 2568.685111] task_work_run+0xc6/0x110 [ 2568.687568] exit_to_usermode_loop+0x1dd/0x1f0 [ 2568.690860] do_syscall_64+0x3d6/0x440 [ 2568.694541] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2568.700190] RIP: 0033:0x7fa749552cd8 [ 2568.703650] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2568.734159] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2568.736702] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2568.752039] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2568.761430] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2568.766056] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2568.778925] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2580.054022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2580.057318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2580.085008] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2580.094942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2580.106342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2580.113136] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2580.125345] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2580.134972] RAX: 0000000000002584 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2580.142403] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2580.148457] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2580.155804] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2580.157569] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2580.163227] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2580.170789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2580.174147] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2580.176732] Call Trace: [ 2580.178265] [ 2580.179083] ? show_regs.cold.9+0x22/0x2f [ 2580.183260] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2580.187765] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2580.195736] ? watchdog+0x40/0x40 [ 2580.201149] ? hrtimer_interrupt+0x128/0x2c0 [ 2580.208874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2580.212789] ? apic_timer_interrupt+0xf/0x20 [ 2580.220342] [ 2580.223668] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2580.233805] do_raw_spin_lock+0x177/0x190 [ 2580.243698] _raw_spin_lock+0x21/0x30 [ 2580.251738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2580.261857] ? do_raw_spin_unlock+0x75/0x190 [ 2580.272694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2580.278957] ? do_wait_intr+0xf0/0xf0 [ 2580.283248] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2580.290142] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2580.294176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2580.300671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2580.306230] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2580.310076] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2580.317436] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2580.319836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2580.322390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2580.324643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2580.329814] lmv_intent_open+0x338/0x1150 [lmv] [ 2580.335048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2580.337718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2580.342362] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2580.355614] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2580.367294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2580.375017] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2580.383822] ? do_raw_spin_unlock+0x75/0x190 [ 2580.391949] ? _raw_spin_unlock+0x12/0x30 [ 2580.397221] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2580.406379] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2580.415606] ll_atomic_open+0x36a/0xd70 [lustre] [ 2580.421193] lookup_open+0x356/0xb40 [ 2580.427187] do_last+0x313/0xfc0 [ 2580.428219] ? nd_jump_root+0xe5/0x160 [ 2580.432626] ? path_init+0x437/0x520 [ 2580.434168] path_openat+0xf7/0x500 [ 2580.436211] ? do_raw_spin_unlock+0x75/0x190 [ 2580.437758] do_filp_open+0x99/0x140 [ 2580.440300] ? getname_flags+0x6e/0x330 [ 2580.442834] ? __check_object_size+0xff/0x256 [ 2580.446946] ? do_raw_spin_unlock+0x75/0x190 [ 2580.448376] ? _raw_spin_unlock+0x12/0x30 [ 2580.449500] do_sys_openat2+0x2b4/0x410 [ 2580.452861] do_sys_open+0x73/0xa0 [ 2580.456394] __x64_sys_openat+0x24/0x30 [ 2580.458584] do_syscall_64+0xc1/0x440 [ 2580.461773] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2580.466172] RIP: 0033:0x7fa749552332 [ 2580.471721] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2580.496311] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2580.505456] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2580.509798] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2580.519328] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2580.524178] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2580.529915] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2583.525036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 666s! [ 2583.528232] Showing busy workqueues and worker pools: [ 2583.537783] workqueue events: flags=0x0 [ 2583.541730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2583.550251] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2583.571008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2583.573301] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2583.581342] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.586081] pending: kfree_rcu_monitor [ 2583.590804] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2583.592517] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2583.615316] workqueue events_long: flags=0x0 [ 2583.622301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.629164] pending: lu_objects_flush [obdclass] [ 2583.637012] workqueue events_unbound: flags=0x2 [ 2583.646329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2583.651249] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2583.664792] workqueue events_power_efficient: flags=0x80 [ 2583.671117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.676077] pending: check_lifetime [ 2583.677044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.687597] pending: neigh_periodic_work [ 2583.688774] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.700884] pending: neigh_periodic_work [ 2583.706862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.717984] pending: do_cache_clean [ 2583.719270] workqueue mm_percpu_wq: flags=0x8 [ 2583.723894] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.732433] pending: vmstat_update [ 2583.735212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.741359] pending: vmstat_update [ 2583.744203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2583.761773] pending: vmstat_update [ 2583.766076] workqueue writeback: flags=0x4a [ 2583.773460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2583.777067] pending: wb_workfn, wb_workfn [ 2583.788455] workqueue xprtiod: flags=0xa [ 2583.791362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2583.797989] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2583.806969] workqueue ipv6_addrconf: flags=0x40008 [ 2583.808230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2583.821526] pending: addrconf_verify_work [ 2583.825358] workqueue knbd0-recv: flags=0x1a [ 2583.830403] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2583.836901] in-flight: 651:recv_work [ 2583.840739] workqueue ptlrpc_pinger: flags=0x2 [ 2583.841940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2583.843483] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2583.855287] pool 8: cpus=0-3 flags=0x4 nice=0 hung=666s workers=4 idle: 37 31846 28897 [ 2583.860863] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2573s workers=2 idle: 653 [ 2588.061950] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2588.070200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2588.107182] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2588.119769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2588.126441] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2588.132304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2588.134671] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2588.151653] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2588.163872] RAX: 0000000000006f89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2588.170894] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2588.182682] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2588.199637] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2588.208424] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2588.214278] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2588.228343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2588.233002] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2588.245106] Call Trace: [ 2588.247867] [ 2588.250684] ? show_regs.cold.9+0x22/0x2f [ 2588.254669] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2588.258250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2588.261454] ? watchdog+0x40/0x40 [ 2588.263424] ? hrtimer_interrupt+0x128/0x2c0 [ 2588.266557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2588.268236] ? apic_timer_interrupt+0xf/0x20 [ 2588.271909] [ 2588.275457] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2588.279464] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2588.282091] do_raw_spin_lock+0x177/0x190 [ 2588.284380] _raw_spin_lock+0x21/0x30 [ 2588.286424] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2588.290187] process_one_work+0x2c8/0x700 [ 2588.292298] worker_thread+0x296/0x6e0 [ 2588.295618] ? rescuer_thread+0x570/0x570 [ 2588.299647] kthread+0x1d1/0x200 [ 2588.301114] ? set_kthread_struct+0x70/0x70 [ 2588.303219] ret_from_fork+0x1f/0x30 [ 2596.059077] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2596.062585] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2596.064645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.064720] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.064728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.064730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2596.064746] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2596.064749] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2596.064754] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2596.064756] RDX: 00000000000076f3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2596.064758] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2596.064761] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2596.064763] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2596.064766] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2596.064769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.064771] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2596.064779] Call Trace: [ 2596.064784] [ 2596.064789] ? show_regs.cold.9+0x22/0x2f [ 2596.064798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.064804] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.064808] ? watchdog+0x40/0x40 [ 2596.064812] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.064816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.064824] ? apic_timer_interrupt+0xf/0x20 [ 2596.064828] [ 2596.064832] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2596.064837] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2596.064841] do_raw_spin_lock+0x177/0x190 [ 2596.064846] _raw_spin_lock+0x21/0x30 [ 2596.064851] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2596.065135] obd_mod_free+0x46/0x140 [ptlrpc] [ 2596.071052] mdc_commit_open+0x91/0xb0 [mdc] [ 2596.071091] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2596.071439] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2596.072098] after_reply+0xe2c/0x1780 [ptlrpc] [ 2596.072370] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2596.072731] ? __next_timer_interrupt+0x160/0x160 [ 2596.072741] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2596.073152] ? do_wait_intr+0xf0/0xf0 [ 2596.073162] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2596.073522] kthread+0x1d1/0x200 [ 2596.073528] ? set_kthread_struct+0x70/0x70 [ 2596.073680] ret_from_fork+0x1f/0x30 [ 2596.433712] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.480389] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.491394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.512130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.526686] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2596.547669] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2596.555331] RAX: 000000000000148c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2596.566571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2596.580175] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2596.582553] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2596.592598] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2596.599148] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2596.607639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.615159] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2596.622467] Call Trace: [ 2596.623636] [ 2596.624922] ? show_regs.cold.9+0x22/0x2f [ 2596.631912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.637440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.644381] ? watchdog+0x40/0x40 [ 2596.650723] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.652483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.661446] ? apic_timer_interrupt+0xf/0x20 [ 2596.664108] [ 2596.665620] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.667389] do_raw_spin_lock+0x177/0x190 [ 2596.670875] _raw_spin_lock+0x21/0x30 [ 2596.674009] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2596.675331] ? do_raw_spin_unlock+0x75/0x190 [ 2596.676673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2596.678606] ? do_wait_intr+0xf0/0xf0 [ 2596.680201] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2596.682695] mdc_close+0x53b/0xd70 [mdc] [ 2596.684659] lmv_close+0x18d/0x490 [lmv] [ 2596.686534] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2596.695298] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2596.715965] ll_md_close+0x274/0x800 [lustre] [ 2596.718818] ll_file_release+0x128/0xc50 [lustre] [ 2596.722528] __fput+0x10d/0x440 [ 2596.723877] ____fput+0x12/0x20 [ 2596.725442] task_work_run+0xc6/0x110 [ 2596.730901] exit_to_usermode_loop+0x1dd/0x1f0 [ 2596.733452] do_syscall_64+0x3d6/0x440 [ 2596.735703] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2596.738128] RIP: 0033:0x7fa749552cd8 [ 2596.745467] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2596.761020] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2596.764906] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2596.769616] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2596.779877] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2596.784081] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2596.788079] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2608.054398] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2608.070210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2608.105334] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2608.111158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2608.115788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2608.118791] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2608.128358] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2608.131716] RAX: 000000000000105d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2608.135164] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2608.138433] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2608.141958] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2608.145354] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2608.148785] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2608.152329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2608.154962] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2608.158545] Call Trace: [ 2608.159675] [ 2608.160921] ? show_regs.cold.9+0x22/0x2f [ 2608.162947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2608.165222] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2608.167445] ? watchdog+0x40/0x40 [ 2608.168843] ? hrtimer_interrupt+0x128/0x2c0 [ 2608.170835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2608.173213] ? apic_timer_interrupt+0xf/0x20 [ 2608.175159] [ 2608.176198] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2608.178820] do_raw_spin_lock+0x177/0x190 [ 2608.180443] _raw_spin_lock+0x21/0x30 [ 2608.182429] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2608.185105] ? do_raw_spin_unlock+0x75/0x190 [ 2608.187164] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2608.189609] ? do_wait_intr+0xf0/0xf0 [ 2608.191314] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2608.193934] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2608.196148] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2608.199195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2608.201803] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2608.204788] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2608.207115] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2608.209327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2608.211837] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2608.214890] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2608.217554] lmv_intent_open+0x338/0x1150 [lmv] [ 2608.219564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2608.222185] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2608.224840] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2608.227088] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2608.228765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2608.230750] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2608.232857] ? do_raw_spin_unlock+0x75/0x190 [ 2608.235055] ? _raw_spin_unlock+0x12/0x30 [ 2608.237063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2608.239749] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2608.242124] ll_atomic_open+0x36a/0xd70 [lustre] [ 2608.244545] lookup_open+0x356/0xb40 [ 2608.246317] do_last+0x313/0xfc0 [ 2608.247680] ? nd_jump_root+0xe5/0x160 [ 2608.249131] ? path_init+0x437/0x520 [ 2608.250756] path_openat+0xf7/0x500 [ 2608.252414] ? do_raw_spin_unlock+0x75/0x190 [ 2608.254556] do_filp_open+0x99/0x140 [ 2608.256386] ? getname_flags+0x6e/0x330 [ 2608.258114] ? __check_object_size+0xff/0x256 [ 2608.260115] ? do_raw_spin_unlock+0x75/0x190 [ 2608.262394] ? _raw_spin_unlock+0x12/0x30 [ 2608.264376] do_sys_openat2+0x2b4/0x410 [ 2608.266201] do_sys_open+0x73/0xa0 [ 2608.267823] __x64_sys_openat+0x24/0x30 [ 2608.269800] do_syscall_64+0xc1/0x440 [ 2608.271652] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2608.273795] RIP: 0033:0x7fa749552332 [ 2608.275668] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2608.284154] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2608.289140] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2608.292898] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2608.295532] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2608.298345] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2608.302109] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2614.255352] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 697s! [ 2614.260848] Showing busy workqueues and worker pools: [ 2614.263941] workqueue events: flags=0x0 [ 2614.265079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2614.272737] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2614.282074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2614.292034] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2614.302935] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.305650] pending: kfree_rcu_monitor [ 2614.307281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2614.310342] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2614.313200] workqueue events_long: flags=0x0 [ 2614.315283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.323111] pending: lu_objects_flush [obdclass] [ 2614.326202] workqueue events_unbound: flags=0x2 [ 2614.328678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2614.332198] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2614.337262] workqueue events_power_efficient: flags=0x80 [ 2614.340865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.345124] pending: check_lifetime [ 2614.347213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.351101] pending: neigh_periodic_work [ 2614.353685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.357451] pending: neigh_periodic_work [ 2614.359544] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.362859] pending: do_cache_clean [ 2614.364865] workqueue mm_percpu_wq: flags=0x8 [ 2614.366946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.370708] pending: vmstat_update [ 2614.372854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.376497] pending: vmstat_update [ 2614.378574] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2614.381773] pending: vmstat_update [ 2614.383663] workqueue writeback: flags=0x4a [ 2614.386264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2614.389621] pending: wb_workfn, wb_workfn [ 2614.391776] workqueue xprtiod: flags=0xa [ 2614.393777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2614.397039] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2614.401276] workqueue ipv6_addrconf: flags=0x40008 [ 2614.403968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2614.407199] pending: addrconf_verify_work [ 2614.409207] workqueue knbd0-recv: flags=0x1a [ 2614.411450] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2614.414003] in-flight: 651:recv_work [ 2614.415626] workqueue ptlrpc_pinger: flags=0x2 [ 2614.417521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2614.420098] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2614.422719] pool 8: cpus=0-3 flags=0x4 nice=0 hung=697s workers=4 idle: 37 31846 28897 [ 2614.426056] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2604s workers=2 idle: 653 [ 2616.061397] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2616.078695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2616.166907] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2616.191917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2616.213647] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2616.221517] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2616.224507] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2616.238170] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2616.263029] RAX: 0000000000004c8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2616.270316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2616.279332] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2616.290368] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2616.308983] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2616.331349] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2616.341990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2616.345868] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2616.360912] Call Trace: [ 2616.369000] [ 2616.370629] ? show_regs.cold.9+0x22/0x2f [ 2616.378703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2616.388966] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2616.390558] ? watchdog+0x40/0x40 [ 2616.399393] ? hrtimer_interrupt+0x128/0x2c0 [ 2616.401652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2616.404441] ? apic_timer_interrupt+0xf/0x20 [ 2616.408500] [ 2616.410017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2616.413277] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2616.415911] do_raw_spin_lock+0x177/0x190 [ 2616.417656] _raw_spin_lock+0x21/0x30 [ 2616.425784] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2616.427458] process_one_work+0x2c8/0x700 [ 2616.432000] worker_thread+0x296/0x6e0 [ 2616.434395] ? rescuer_thread+0x570/0x570 [ 2616.437870] kthread+0x1d1/0x200 [ 2616.438963] ? set_kthread_struct+0x70/0x70 [ 2616.440179] ret_from_fork+0x1f/0x30 [ 2624.058289] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2624.060026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2624.064250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.064330] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.064337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.064340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2624.064354] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2624.064358] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2624.064363] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2624.064365] RDX: 0000000000005539 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2624.064367] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2624.064370] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2624.064372] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2624.064375] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2624.064378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.064380] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2624.064388] Call Trace: [ 2624.064393] [ 2624.064399] ? show_regs.cold.9+0x22/0x2f [ 2624.064408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.064414] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.064418] ? watchdog+0x40/0x40 [ 2624.064423] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.064427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.064435] ? apic_timer_interrupt+0xf/0x20 [ 2624.064440] [ 2624.064442] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2624.064447] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2624.064452] do_raw_spin_lock+0x177/0x190 [ 2624.064456] _raw_spin_lock+0x21/0x30 [ 2624.064462] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2624.065031] obd_mod_free+0x46/0x140 [ptlrpc] [ 2624.065284] mdc_commit_open+0x91/0xb0 [mdc] [ 2624.065318] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2624.065514] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2624.066032] after_reply+0xe2c/0x1780 [ptlrpc] [ 2624.066382] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2624.066926] ? __next_timer_interrupt+0x160/0x160 [ 2624.066935] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2624.067162] ? do_wait_intr+0xf0/0xf0 [ 2624.067170] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2624.067369] kthread+0x1d1/0x200 [ 2624.067375] ? set_kthread_struct+0x70/0x70 [ 2624.067379] ret_from_fork+0x1f/0x30 [ 2624.246372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.274006] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.282987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.289658] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.298030] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2624.314457] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2624.319645] RAX: 0000000000006748 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2624.329594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2624.334067] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2624.336814] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2624.339969] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2624.343029] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2624.346670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.349559] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2624.353137] Call Trace: [ 2624.354250] [ 2624.355311] ? show_regs.cold.9+0x22/0x2f [ 2624.357115] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.359126] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.361106] ? watchdog+0x40/0x40 [ 2624.362489] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.364181] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.366520] ? apic_timer_interrupt+0xf/0x20 [ 2624.368759] [ 2624.369533] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.372126] do_raw_spin_lock+0x177/0x190 [ 2624.374342] _raw_spin_lock+0x21/0x30 [ 2624.375970] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2624.379047] ? do_raw_spin_unlock+0x75/0x190 [ 2624.381044] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2624.383888] ? do_wait_intr+0xf0/0xf0 [ 2624.385602] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2624.388811] mdc_close+0x53b/0xd70 [mdc] [ 2624.391084] lmv_close+0x18d/0x490 [lmv] [ 2624.392848] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2624.395895] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2624.398543] ll_md_close+0x274/0x800 [lustre] [ 2624.401104] ll_file_release+0x128/0xc50 [lustre] [ 2624.403902] __fput+0x10d/0x440 [ 2624.405586] ____fput+0x12/0x20 [ 2624.407086] task_work_run+0xc6/0x110 [ 2624.409037] exit_to_usermode_loop+0x1dd/0x1f0 [ 2624.411316] do_syscall_64+0x3d6/0x440 [ 2624.413409] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2624.416002] RIP: 0033:0x7fa749552cd8 [ 2624.417572] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2624.427116] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2624.430959] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2624.434038] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2624.437559] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2624.441790] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2624.444668] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2640.054455] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2640.057338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2640.083166] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2640.088510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2640.093287] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2640.095710] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2640.106229] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2640.117495] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2640.132286] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2640.139911] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2640.142149] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2640.154058] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2640.157219] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2640.165599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2640.169401] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2640.172871] Call Trace: [ 2640.174499] [ 2640.175178] ? show_regs.cold.9+0x22/0x2f [ 2640.176556] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2640.178477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2640.179919] ? watchdog+0x40/0x40 [ 2640.180984] ? hrtimer_interrupt+0x128/0x2c0 [ 2640.183287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2640.185230] ? apic_timer_interrupt+0xf/0x20 [ 2640.188003] [ 2640.189176] ? native_safe_halt+0x2e/0x50 [ 2640.190945] kvm_wait+0xcd/0xe0 [ 2640.193114] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2640.195317] do_raw_spin_lock+0x177/0x190 [ 2640.197214] _raw_spin_lock+0x21/0x30 [ 2640.198559] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2640.202068] ? do_raw_spin_unlock+0x75/0x190 [ 2640.204629] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2640.208691] ? do_wait_intr+0xf0/0xf0 [ 2640.210097] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2640.215250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2640.218093] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2640.220092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2640.227432] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2640.230187] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2640.232751] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2640.236628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2640.241509] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2640.244943] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2640.247554] lmv_intent_open+0x338/0x1150 [lmv] [ 2640.249738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2640.251684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2640.254417] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2640.256731] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2640.258623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2640.262201] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2640.264378] ? do_raw_spin_unlock+0x75/0x190 [ 2640.266955] ? _raw_spin_unlock+0x12/0x30 [ 2640.269642] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2640.271803] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2640.273576] ll_atomic_open+0x36a/0xd70 [lustre] [ 2640.280153] lookup_open+0x356/0xb40 [ 2640.287056] do_last+0x313/0xfc0 [ 2640.288478] ? nd_jump_root+0xe5/0x160 [ 2640.290367] ? path_init+0x437/0x520 [ 2640.292534] path_openat+0xf7/0x500 [ 2640.294361] ? do_raw_spin_unlock+0x75/0x190 [ 2640.296898] do_filp_open+0x99/0x140 [ 2640.299178] ? getname_flags+0x6e/0x330 [ 2640.301479] ? __check_object_size+0xff/0x256 [ 2640.306761] ? do_raw_spin_unlock+0x75/0x190 [ 2640.310189] ? _raw_spin_unlock+0x12/0x30 [ 2640.312585] do_sys_openat2+0x2b4/0x410 [ 2640.316606] do_sys_open+0x73/0xa0 [ 2640.323282] __x64_sys_openat+0x24/0x30 [ 2640.325564] do_syscall_64+0xc1/0x440 [ 2640.330802] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2640.332678] RIP: 0033:0x7fa749552332 [ 2640.339020] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2640.351545] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2640.356055] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2640.360184] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2640.363465] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2640.370165] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2640.378011] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2644.057204] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2644.063954] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2644.109926] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2644.116036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2644.121008] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2644.126249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2644.133514] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2644.165162] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2644.187257] RAX: 0000000000004207 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2644.196054] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2644.200021] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2644.207049] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2644.213924] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2644.222320] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2644.226546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2644.230275] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2644.236404] Call Trace: [ 2644.238778] [ 2644.241584] ? show_regs.cold.9+0x22/0x2f [ 2644.245001] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2644.250125] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2644.253198] ? watchdog+0x40/0x40 [ 2644.255043] ? hrtimer_interrupt+0x128/0x2c0 [ 2644.257152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2644.262212] ? apic_timer_interrupt+0xf/0x20 [ 2644.266280] [ 2644.270118] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2644.272261] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2644.274742] do_raw_spin_lock+0x177/0x190 [ 2644.276595] _raw_spin_lock+0x21/0x30 [ 2644.278719] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2644.281540] process_one_work+0x2c8/0x700 [ 2644.283978] worker_thread+0x296/0x6e0 [ 2644.286497] ? rescuer_thread+0x570/0x570 [ 2644.288473] kthread+0x1d1/0x200 [ 2644.290753] ? set_kthread_struct+0x70/0x70 [ 2644.293069] ret_from_fork+0x1f/0x30 [ 2644.452032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 727s! [ 2644.459509] Showing busy workqueues and worker pools: [ 2644.461967] workqueue events: flags=0x0 [ 2644.464150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2644.469041] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2644.487914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2644.494939] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2644.504122] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.511963] pending: kfree_rcu_monitor [ 2644.516806] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2644.523591] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2644.527374] workqueue events_long: flags=0x0 [ 2644.532461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.541324] pending: lu_objects_flush [obdclass] [ 2644.545146] workqueue events_unbound: flags=0x2 [ 2644.547507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2644.550645] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2644.561629] workqueue events_power_efficient: flags=0x80 [ 2644.566085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.569519] pending: check_lifetime [ 2644.579178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.584952] pending: neigh_periodic_work [ 2644.589759] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.593485] pending: neigh_periodic_work [ 2644.595781] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.600274] pending: do_cache_clean [ 2644.602562] workqueue mm_percpu_wq: flags=0x8 [ 2644.605022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.608822] pending: vmstat_update [ 2644.610541] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.613327] pending: vmstat_update [ 2644.615638] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2644.618481] pending: vmstat_update [ 2644.620032] workqueue writeback: flags=0x4a [ 2644.622851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2644.625533] pending: wb_workfn, wb_workfn [ 2644.627036] workqueue xprtiod: flags=0xa [ 2644.629120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2644.631808] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2644.635165] workqueue ipv6_addrconf: flags=0x40008 [ 2644.637252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2644.641196] pending: addrconf_verify_work [ 2644.646737] workqueue knbd0-recv: flags=0x1a [ 2644.648692] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2644.654176] in-flight: 651:recv_work [ 2644.656010] workqueue ptlrpc_pinger: flags=0x2 [ 2644.662771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2644.672276] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2644.674488] pool 8: cpus=0-3 flags=0x4 nice=0 hung=727s workers=4 idle: 37 31846 28897 [ 2644.690250] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2634s workers=2 idle: 653 [ 2652.059006] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2652.059589] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2652.069328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.069408] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.069416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.069419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 2652.069434] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 2652.069438] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2652.069442] RAX: 00000000000041d1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2652.069445] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2652.069447] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2652.069449] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2652.069451] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2652.069454] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2652.069457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.069460] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2652.069469] Call Trace: [ 2652.069475] [ 2652.069481] ? show_regs.cold.9+0x22/0x2f [ 2652.069505] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.069512] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.069517] ? watchdog+0x40/0x40 [ 2652.069522] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.069526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.069533] ? apic_timer_interrupt+0xf/0x20 [ 2652.069537] [ 2652.069540] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 2652.069547] do_raw_spin_lock+0x177/0x190 [ 2652.069551] _raw_spin_lock+0x21/0x30 [ 2652.069556] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2652.069857] ? do_raw_spin_unlock+0x75/0x190 [ 2652.069863] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2652.070064] ? do_wait_intr+0xf0/0xf0 [ 2652.070071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2652.070264] mdc_close+0x53b/0xd70 [mdc] [ 2652.070303] lmv_close+0x18d/0x490 [lmv] [ 2652.070333] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2652.070443] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2652.075407] ll_md_close+0x274/0x800 [lustre] [ 2652.075800] ll_file_release+0x128/0xc50 [lustre] [ 2652.075882] __fput+0x10d/0x440 [ 2652.075892] ____fput+0x12/0x20 [ 2652.075896] task_work_run+0xc6/0x110 [ 2652.075904] exit_to_usermode_loop+0x1dd/0x1f0 [ 2652.075911] do_syscall_64+0x3d6/0x440 [ 2652.075915] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2652.075924] RIP: 0033:0x7fa749552cd8 [ 2652.075931] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2652.075935] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2652.075940] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2652.075943] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2652.075945] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2652.075947] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2652.075950] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2652.280533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.302172] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.307022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.310778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2652.313539] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2652.322536] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2652.327034] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2652.330473] RDX: 0000000000007b5a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2652.333410] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2652.336800] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2652.340458] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2652.344086] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2652.348536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.350790] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2652.354531] Call Trace: [ 2652.358208] [ 2652.362927] ? show_regs.cold.9+0x22/0x2f [ 2652.365435] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.368490] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.370066] ? watchdog+0x40/0x40 [ 2652.371304] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.374161] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.376381] ? apic_timer_interrupt+0xf/0x20 [ 2652.379330] [ 2652.380602] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2652.383461] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2652.388375] do_raw_spin_lock+0x177/0x190 [ 2652.391113] _raw_spin_lock+0x21/0x30 [ 2652.393014] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2652.396939] obd_mod_free+0x46/0x140 [ptlrpc] [ 2652.401035] mdc_commit_open+0x91/0xb0 [mdc] [ 2652.402578] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2652.404098] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2652.406393] after_reply+0xe2c/0x1780 [ptlrpc] [ 2652.411373] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2652.413953] ? __next_timer_interrupt+0x160/0x160 [ 2652.418836] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2652.421318] ? do_wait_intr+0xf0/0xf0 [ 2652.423501] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2652.426307] kthread+0x1d1/0x200 [ 2652.428800] ? set_kthread_struct+0x70/0x70 [ 2652.430631] ret_from_fork+0x1f/0x30 [ 2672.054034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2672.057106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2672.057768] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2672.094131] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2672.094162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2672.094166] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2672.094185] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2672.094190] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2672.094197] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2672.094199] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2672.094201] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2672.094203] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2672.094206] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2672.094209] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2672.094212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2672.094214] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2672.094223] Call Trace: [ 2672.094232] [ 2672.094241] ? show_regs.cold.9+0x22/0x2f [ 2672.094255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2672.094263] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2672.094267] ? watchdog+0x40/0x40 [ 2672.094271] ? hrtimer_interrupt+0x128/0x2c0 [ 2672.094275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2672.094282] ? apic_timer_interrupt+0xf/0x20 [ 2672.094287] [ 2672.094290] ? native_safe_halt+0x2e/0x50 [ 2672.094293] kvm_wait+0xcd/0xe0 [ 2672.094302] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2672.094315] do_raw_spin_lock+0x177/0x190 [ 2672.094319] _raw_spin_lock+0x21/0x30 [ 2672.094323] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2672.094641] ? do_raw_spin_unlock+0x75/0x190 [ 2672.094648] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2672.094841] ? do_wait_intr+0xf0/0xf0 [ 2672.094852] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2672.095048] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2672.108632] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2672.108826] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2672.108833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2672.108839] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2672.109158] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2672.109169] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2672.109173] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2672.109177] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2672.109179] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2672.109181] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2672.109183] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2672.109185] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2672.109188] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2672.109191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2672.109193] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2672.109202] Call Trace: [ 2672.109208] [ 2672.109213] ? show_regs.cold.9+0x22/0x2f [ 2672.109225] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2672.109231] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2672.109235] ? watchdog+0x40/0x40 [ 2672.109239] ? hrtimer_interrupt+0x128/0x2c0 [ 2672.109314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2672.109321] ? apic_timer_interrupt+0xf/0x20 [ 2672.109327] [ 2672.109330] ? native_safe_halt+0x2e/0x50 [ 2672.109334] kvm_wait+0xcd/0xe0 [ 2672.109342] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2672.109350] do_raw_spin_lock+0x177/0x190 [ 2672.109354] _raw_spin_lock+0x21/0x30 [ 2672.109359] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2672.109584] process_one_work+0x2c8/0x700 [ 2672.109590] worker_thread+0x296/0x6e0 [ 2672.109595] ? rescuer_thread+0x570/0x570 [ 2672.109598] kthread+0x1d1/0x200 [ 2672.109603] ? set_kthread_struct+0x70/0x70 [ 2672.109606] ret_from_fork+0x1f/0x30 [ 2672.530363] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2672.540575] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2672.549202] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2672.556036] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2672.561602] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2672.566031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2672.571111] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2672.583278] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2672.587037] lmv_intent_open+0x338/0x1150 [lmv] [ 2672.589579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2672.598475] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2672.601864] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2672.603517] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2672.610122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2672.620025] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2672.635355] ? do_raw_spin_unlock+0x75/0x190 [ 2672.644209] ? _raw_spin_unlock+0x12/0x30 [ 2672.651213] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2672.653093] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2672.664361] ll_atomic_open+0x36a/0xd70 [lustre] [ 2672.665684] lookup_open+0x356/0xb40 [ 2672.666897] do_last+0x313/0xfc0 [ 2672.676691] ? nd_jump_root+0xe5/0x160 [ 2672.677985] ? path_init+0x437/0x520 [ 2672.686176] path_openat+0xf7/0x500 [ 2672.690304] ? do_raw_spin_unlock+0x75/0x190 [ 2672.698866] do_filp_open+0x99/0x140 [ 2672.700102] ? getname_flags+0x6e/0x330 [ 2672.704304] ? __check_object_size+0xff/0x256 [ 2672.712101] ? do_raw_spin_unlock+0x75/0x190 [ 2672.720023] ? _raw_spin_unlock+0x12/0x30 [ 2672.724524] do_sys_openat2+0x2b4/0x410 [ 2672.731536] do_sys_open+0x73/0xa0 [ 2672.736976] __x64_sys_openat+0x24/0x30 [ 2672.741937] do_syscall_64+0xc1/0x440 [ 2672.743328] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2672.752221] RIP: 0033:0x7fa749552332 [ 2672.754248] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2672.783451] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2672.787616] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2672.794174] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2672.798508] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2672.802206] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2672.805150] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2675.175136] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 758s! [ 2675.179688] Showing busy workqueues and worker pools: [ 2675.182374] workqueue events: flags=0x0 [ 2675.184725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2675.188273] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2675.195550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2675.199904] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2675.208016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.212206] pending: kfree_rcu_monitor [ 2675.214762] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2675.218887] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2675.222736] workqueue events_long: flags=0x0 [ 2675.225202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.228759] pending: lu_objects_flush [obdclass] [ 2675.231773] workqueue events_unbound: flags=0x2 [ 2675.234367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2675.237973] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2675.243650] workqueue events_power_efficient: flags=0x80 [ 2675.246350] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.250010] pending: check_lifetime [ 2675.251094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.254262] pending: neigh_periodic_work [ 2675.255801] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.258553] pending: neigh_periodic_work [ 2675.259725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.265554] pending: do_cache_clean [ 2675.274757] workqueue mm_percpu_wq: flags=0x8 [ 2675.286164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.293350] pending: vmstat_update [ 2675.295276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.301751] pending: vmstat_update [ 2675.303737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2675.309765] pending: vmstat_update [ 2675.311053] workqueue writeback: flags=0x4a [ 2675.313247] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2675.317658] pending: wb_workfn, wb_workfn [ 2675.322341] workqueue xprtiod: flags=0xa [ 2675.330957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2675.332926] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2675.338301] workqueue ipv6_addrconf: flags=0x40008 [ 2675.340605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2675.345547] pending: addrconf_verify_work [ 2675.349318] workqueue knbd0-recv: flags=0x1a [ 2675.352958] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2675.357960] in-flight: 651:recv_work [ 2675.360316] workqueue ptlrpc_pinger: flags=0x2 [ 2675.363696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2675.366348] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2675.369683] pool 8: cpus=0-3 flags=0x4 nice=0 hung=758s workers=4 idle: 37 31846 28897 [ 2675.372523] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2665s workers=2 idle: 653 [ 2680.058412] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2680.059010] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2680.062429] Modules linked in: [ 2680.071315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.071400] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.071407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.071410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.071424] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2680.071428] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2680.071432] RAX: 0000000000002e83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2680.071435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2680.071438] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2680.071440] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2680.071442] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2680.071446] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2680.071448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.071450] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2680.071459] Call Trace: [ 2680.071465] [ 2680.071472] ? show_regs.cold.9+0x22/0x2f [ 2680.071481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.071488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.071492] ? watchdog+0x40/0x40 [ 2680.071497] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.071501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.071509] ? apic_timer_interrupt+0xf/0x20 [ 2680.071514] [ 2680.071517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.071525] do_raw_spin_lock+0x177/0x190 [ 2680.071530] _raw_spin_lock+0x21/0x30 [ 2680.071534] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2680.072351] ? do_raw_spin_unlock+0x75/0x190 [ 2680.072358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2680.072573] ? do_wait_intr+0xf0/0xf0 [ 2680.072581] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2680.078770] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.078857] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.078864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.078867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2680.078881] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2680.078886] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2680.078890] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2680.078893] RDX: 000000000000470d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2680.078895] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2680.078898] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2680.078901] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2680.078904] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2680.078907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.078909] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2680.078918] Call Trace: [ 2680.078923] [ 2680.078929] ? show_regs.cold.9+0x22/0x2f [ 2680.078938] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.078945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.078949] ? watchdog+0x40/0x40 [ 2680.078953] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.078958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.078967] ? apic_timer_interrupt+0xf/0x20 [ 2680.078971] [ 2680.078974] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2680.078979] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2680.078984] do_raw_spin_lock+0x177/0x190 [ 2680.078989] _raw_spin_lock+0x21/0x30 [ 2680.079006] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2680.079303] obd_mod_free+0x46/0x140 [ptlrpc] [ 2680.079508] mdc_commit_open+0x91/0xb0 [mdc] [ 2680.079554] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2680.079755] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2680.079964] after_reply+0xe2c/0x1780 [ptlrpc] [ 2680.080289] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2680.080562] ? __next_timer_interrupt+0x160/0x160 [ 2680.080572] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2680.080880] ? do_wait_intr+0xf0/0xf0 [ 2680.080890] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2680.081276] kthread+0x1d1/0x200 [ 2680.081282] ? set_kthread_struct+0x70/0x70 [ 2680.081285] ret_from_fork+0x1f/0x30 [ 2680.606934] mdc_close+0x53b/0xd70 [mdc] [ 2680.611291] lmv_close+0x18d/0x490 [lmv] [ 2680.616152] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2680.619901] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2680.622937] ll_md_close+0x274/0x800 [lustre] [ 2680.626947] ll_file_release+0x128/0xc50 [lustre] [ 2680.629530] __fput+0x10d/0x440 [ 2680.630915] ____fput+0x12/0x20 [ 2680.632393] task_work_run+0xc6/0x110 [ 2680.636200] exit_to_usermode_loop+0x1dd/0x1f0 [ 2680.638394] do_syscall_64+0x3d6/0x440 [ 2680.643671] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2680.645899] RIP: 0033:0x7fa749552cd8 [ 2680.649575] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2680.667429] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2680.675378] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2680.683119] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2680.690920] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2680.698008] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2680.709899] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2693.704037] rcu: INFO: rcu_sched self-detected stall on CPU [ 2693.706805] rcu: 2-...!: (434923 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 2693.710782] (t=780015 jiffies g=258701 q=1706) [ 2693.712811] rcu: rcu_sched kthread starved for 776810 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2693.716573] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2693.719800] rcu: RCU grace-period kthread stack dump: [ 2693.723901] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2693.729257] Call Trace: [ 2693.730338] __schedule+0x351/0xcb0 [ 2693.732449] schedule+0xc0/0x180 [ 2693.733911] schedule_timeout+0xb4/0x190 [ 2693.735594] ? do_raw_spin_unlock+0x75/0x190 [ 2693.738044] ? __next_timer_interrupt+0x160/0x160 [ 2693.740464] ? prepare_to_swait_event+0xf3/0x140 [ 2693.743018] rcu_gp_kthread+0x8dd/0x15b0 [ 2693.745079] ? force_qs_rnp+0x2c0/0x2c0 [ 2693.747533] kthread+0x1d1/0x200 [ 2693.749325] ? set_kthread_struct+0x70/0x70 [ 2693.751491] ret_from_fork+0x1f/0x30 [ 2693.753341] rcu: Stack dump where RCU GP kthread last ran: [ 2693.757162] NMI backtrace for cpu 2 [ 2693.758469] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2693.764084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2693.767294] Call Trace: [ 2693.768220] [ 2693.769088] ? dump_stack+0xbb/0x10e [ 2693.770816] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2693.773264] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2693.776190] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2693.778908] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2693.781903] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2693.784390] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2693.787192] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 2693.789637] ? update_process_times+0x6d/0xc0 [ 2693.791594] ? tick_sched_handle.isra.18+0x27/0x90 [ 2693.794367] ? tick_sched_timer+0x53/0xb0 [ 2693.796904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2693.799479] ? tick_sched_handle.isra.18+0x90/0x90 [ 2693.801995] ? hrtimer_interrupt+0x128/0x2c0 [ 2693.803581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2693.805348] ? apic_timer_interrupt+0xf/0x20 [ 2693.806344] [ 2693.807017] ? native_safe_halt+0x2e/0x50 [ 2693.808314] ? kvm_wait+0xcd/0xe0 [ 2693.809806] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2693.812451] ? do_raw_spin_lock+0x177/0x190 [ 2693.813857] ? _raw_spin_lock+0x21/0x30 [ 2693.815401] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2693.818037] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2693.820898] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2693.823674] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2693.826897] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2693.830169] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2693.833537] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2693.838153] ? __next_timer_interrupt+0x160/0x160 [ 2693.839607] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2693.841386] ? do_wait_intr+0xf0/0xf0 [ 2693.843303] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2693.846200] ? kthread+0x1d1/0x200 [ 2693.847721] ? set_kthread_struct+0x70/0x70 [ 2693.852861] ? ret_from_fork+0x1f/0x30 [ 2693.853697] NMI backtrace for cpu 2 [ 2693.855677] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2693.860852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2693.864745] Call Trace: [ 2693.865745] [ 2693.866732] ? dump_stack+0xbb/0x10e [ 2693.868192] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2693.870378] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2693.872623] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2693.875090] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2693.877209] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2693.880234] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2693.882208] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2693.884892] ? update_process_times+0x6d/0xc0 [ 2693.886516] ? tick_sched_handle.isra.18+0x27/0x90 [ 2693.889080] ? tick_sched_timer+0x53/0xb0 [ 2693.891226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2693.893316] ? tick_sched_handle.isra.18+0x90/0x90 [ 2693.895937] ? hrtimer_interrupt+0x128/0x2c0 [ 2693.897976] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2693.900017] ? apic_timer_interrupt+0xf/0x20 [ 2693.901790] [ 2693.902331] ? native_safe_halt+0x2e/0x50 [ 2693.903867] ? kvm_wait+0xcd/0xe0 [ 2693.905080] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2693.906856] ? do_raw_spin_lock+0x177/0x190 [ 2693.908692] ? _raw_spin_lock+0x21/0x30 [ 2693.910451] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2693.912842] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2693.915062] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2693.917178] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2693.921188] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2693.926140] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2693.929048] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2693.931309] ? __next_timer_interrupt+0x160/0x160 [ 2693.933230] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2693.937089] ? do_wait_intr+0xf0/0xf0 [ 2693.938695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2693.942818] ? kthread+0x1d1/0x200 [ 2693.944335] ? set_kthread_struct+0x70/0x70 [ 2693.946028] ? ret_from_fork+0x1f/0x30 [ 2700.054143] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2700.059468] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2700.061274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2700.061699] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2700.061706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2700.061709] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2700.061734] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2700.061738] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2700.061743] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2700.061745] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2700.061747] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2700.061750] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2700.061752] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2700.061755] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2700.061758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2700.061760] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2700.061768] Call Trace: [ 2700.061784] [ 2700.061791] ? show_regs.cold.9+0x22/0x2f [ 2700.061801] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2700.061807] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2700.061812] ? watchdog+0x40/0x40 [ 2700.061816] ? hrtimer_interrupt+0x128/0x2c0 [ 2700.061820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2700.061827] ? apic_timer_interrupt+0xf/0x20 [ 2700.061831] [ 2700.061833] ? native_safe_halt+0x2e/0x50 [ 2700.061837] kvm_wait+0xcd/0xe0 [ 2700.061845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2700.061853] do_raw_spin_lock+0x177/0x190 [ 2700.061857] _raw_spin_lock+0x21/0x30 [ 2700.061861] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2700.062462] ? do_raw_spin_unlock+0x75/0x190 [ 2700.062469] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2700.062820] ? do_wait_intr+0xf0/0xf0 [ 2700.062828] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2700.063205] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2700.063715] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2700.064147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2700.064273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2700.064309] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2700.064336] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2700.064435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2700.064594] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2700.064905] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2700.064938] lmv_intent_open+0x338/0x1150 [lmv] [ 2700.064967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2700.065201] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2700.065286] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2700.065310] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2700.065554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2700.065706] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2700.065789] ? do_raw_spin_unlock+0x75/0x190 [ 2700.065796] ? _raw_spin_unlock+0x12/0x30 [ 2700.065802] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2700.065826] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2700.065845] ll_atomic_open+0x36a/0xd70 [lustre] [ 2700.065929] lookup_open+0x356/0xb40 [ 2700.065939] do_last+0x313/0xfc0 [ 2700.065944] ? nd_jump_root+0xe5/0x160 [ 2700.065949] ? path_init+0x437/0x520 [ 2700.065953] path_openat+0xf7/0x500 [ 2700.065958] ? do_raw_spin_unlock+0x75/0x190 [ 2700.065962] do_filp_open+0x99/0x140 [ 2700.065968] ? getname_flags+0x6e/0x330 [ 2700.065972] ? __check_object_size+0xff/0x256 [ 2700.065979] ? do_raw_spin_unlock+0x75/0x190 [ 2700.065983] ? _raw_spin_unlock+0x12/0x30 [ 2700.065987] do_sys_openat2+0x2b4/0x410 [ 2700.065991] do_sys_open+0x73/0xa0 [ 2700.065995] __x64_sys_openat+0x24/0x30 [ 2700.065998] do_syscall_64+0xc1/0x440 [ 2700.066004] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2700.066011] RIP: 0033:0x7fa749552332 [ 2700.066343] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2700.066349] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2700.066702] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2700.066704] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2700.066706] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2700.066708] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2700.066711] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2700.325665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2700.348484] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2700.354432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2700.359165] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2700.362743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2700.365595] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2700.375850] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2700.379621] RAX: 0000000000006a89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2700.382958] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2700.386764] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2700.390244] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2700.394145] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2700.397292] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2700.401991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2700.404909] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2700.408245] Call Trace: [ 2700.409581] [ 2700.410554] ? show_regs.cold.9+0x22/0x2f [ 2700.412457] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2700.414862] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2700.417187] ? watchdog+0x40/0x40 [ 2700.418827] ? hrtimer_interrupt+0x128/0x2c0 [ 2700.420804] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2700.423489] ? apic_timer_interrupt+0xf/0x20 [ 2700.425821] [ 2700.426863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2700.429838] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2700.432827] do_raw_spin_lock+0x177/0x190 [ 2700.438863] _raw_spin_lock+0x21/0x30 [ 2700.440942] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2700.447978] process_one_work+0x2c8/0x700 [ 2700.450494] worker_thread+0x296/0x6e0 [ 2700.452691] ? rescuer_thread+0x570/0x570 [ 2700.455182] kthread+0x1d1/0x200 [ 2700.456923] ? set_kthread_struct+0x70/0x70 [ 2700.459867] ret_from_fork+0x1f/0x30 [ 2705.384975] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 788s! [ 2705.391873] Showing busy workqueues and worker pools: [ 2705.401919] workqueue events: flags=0x0 [ 2705.407877] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2705.413166] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2705.420562] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2705.433284] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2705.438970] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.441349] pending: kfree_rcu_monitor [ 2705.457769] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2705.466671] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2705.474388] workqueue events_long: flags=0x0 [ 2705.476774] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.485786] pending: lu_objects_flush [obdclass] [ 2705.491368] workqueue events_unbound: flags=0x2 [ 2705.494815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2705.498567] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2705.504034] workqueue events_power_efficient: flags=0x80 [ 2705.506432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.509810] pending: check_lifetime [ 2705.511656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.515321] pending: neigh_periodic_work [ 2705.519020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.527496] pending: neigh_periodic_work [ 2705.531373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.537787] pending: do_cache_clean [ 2705.542471] workqueue mm_percpu_wq: flags=0x8 [ 2705.547405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.550865] pending: vmstat_update [ 2705.554701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.559356] pending: vmstat_update [ 2705.563175] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2705.567814] pending: vmstat_update [ 2705.570040] workqueue writeback: flags=0x4a [ 2705.572389] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2705.575580] pending: wb_workfn, wb_workfn [ 2705.579835] workqueue xprtiod: flags=0xa [ 2705.583239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2705.585483] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2705.590339] workqueue ipv6_addrconf: flags=0x40008 [ 2705.592392] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2705.595472] pending: addrconf_verify_work [ 2705.601333] workqueue knbd0-recv: flags=0x1a [ 2705.606341] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2705.620421] in-flight: 651:recv_work [ 2705.626605] workqueue ptlrpc_pinger: flags=0x2 [ 2705.633619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2705.635842] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2705.649370] pool 8: cpus=0-3 flags=0x4 nice=0 hung=788s workers=4 idle: 37 31846 28897 [ 2705.656487] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2695s workers=2 idle: 653 [ 2708.063949] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2708.069943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2708.112810] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2708.124749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2708.130780] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2708.132913] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2708.154926] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2708.167981] RAX: 0000000000000b31 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2708.171911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2708.181044] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2708.185441] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2708.191357] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2708.199386] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2708.205922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2708.211907] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2708.218411] Call Trace: [ 2708.220879] [ 2708.222957] ? show_regs.cold.9+0x22/0x2f [ 2708.224374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2708.227697] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2708.230414] ? watchdog+0x40/0x40 [ 2708.233312] ? hrtimer_interrupt+0x128/0x2c0 [ 2708.237003] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2708.242805] ? apic_timer_interrupt+0xf/0x20 [ 2708.246763] [ 2708.251651] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2708.253345] do_raw_spin_lock+0x177/0x190 [ 2708.255946] _raw_spin_lock+0x21/0x30 [ 2708.257292] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2708.258963] ? do_raw_spin_unlock+0x75/0x190 [ 2708.266761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2708.271893] ? do_wait_intr+0xf0/0xf0 [ 2708.275455] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2708.280686] mdc_close+0x53b/0xd70 [mdc] [ 2708.282807] lmv_close+0x18d/0x490 [lmv] [ 2708.285908] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2708.292773] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2708.299140] ll_md_close+0x274/0x800 [lustre] [ 2708.302203] ll_file_release+0x128/0xc50 [lustre] [ 2708.308611] __fput+0x10d/0x440 [ 2708.310009] ____fput+0x12/0x20 [ 2708.312475] task_work_run+0xc6/0x110 [ 2708.316174] exit_to_usermode_loop+0x1dd/0x1f0 [ 2708.317990] do_syscall_64+0x3d6/0x440 [ 2708.319448] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2708.325088] RIP: 0033:0x7fa749552cd8 [ 2708.328704] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2708.343726] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2708.355129] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2708.369765] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2708.377403] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2708.383039] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2708.390829] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2720.058598] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2720.067198] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2720.104712] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2720.119012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2720.125732] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2720.129139] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2720.143704] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2720.147856] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2720.149715] RDX: 00000000000028be RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2720.153164] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2720.156429] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2720.160757] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2720.165762] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2720.170745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2720.174917] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2720.178964] Call Trace: [ 2720.180470] [ 2720.183151] ? show_regs.cold.9+0x22/0x2f [ 2720.185137] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2720.192201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2720.195122] ? watchdog+0x40/0x40 [ 2720.196258] ? hrtimer_interrupt+0x128/0x2c0 [ 2720.202980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2720.209487] ? apic_timer_interrupt+0xf/0x20 [ 2720.212678] [ 2720.214897] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2720.218202] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2720.222331] do_raw_spin_lock+0x177/0x190 [ 2720.224924] _raw_spin_lock+0x21/0x30 [ 2720.226176] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2720.237454] obd_mod_free+0x46/0x140 [ptlrpc] [ 2720.246265] mdc_commit_open+0x91/0xb0 [mdc] [ 2720.250988] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2720.257692] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2720.259753] after_reply+0xe2c/0x1780 [ptlrpc] [ 2720.266703] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2720.270764] ? __next_timer_interrupt+0x160/0x160 [ 2720.274752] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2720.276957] ? do_wait_intr+0xf0/0xf0 [ 2720.280617] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2720.283278] kthread+0x1d1/0x200 [ 2720.288709] ? set_kthread_struct+0x70/0x70 [ 2720.290642] ret_from_fork+0x1f/0x30 [ 2728.063443] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2728.067151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2728.106767] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2728.119327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2728.123296] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2728.128668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2728.131197] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2728.146016] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2728.149663] RAX: 000000000000378b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2728.159012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2728.166143] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2728.177547] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2728.184141] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2728.189562] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2728.202732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2728.205289] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2728.212366] Call Trace: [ 2728.213292] [ 2728.214024] ? show_regs.cold.9+0x22/0x2f [ 2728.225755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2728.230054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2728.232117] ? watchdog+0x40/0x40 [ 2728.237017] ? hrtimer_interrupt+0x128/0x2c0 [ 2728.238508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2728.242568] ? apic_timer_interrupt+0xf/0x20 [ 2728.244115] [ 2728.244867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2728.255818] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2728.258600] do_raw_spin_lock+0x177/0x190 [ 2728.260721] _raw_spin_lock+0x21/0x30 [ 2728.268091] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2728.277073] process_one_work+0x2c8/0x700 [ 2728.279670] worker_thread+0x296/0x6e0 [ 2728.282185] ? rescuer_thread+0x570/0x570 [ 2728.284341] kthread+0x1d1/0x200 [ 2728.285844] ? set_kthread_struct+0x70/0x70 [ 2728.289226] ret_from_fork+0x1f/0x30 [ 2732.054788] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2732.060763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2732.108298] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2732.123271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2732.129481] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2732.137333] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2732.158633] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2732.168052] RAX: 0000000000007b84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2732.172935] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2732.189881] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2732.202814] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2732.206621] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2732.211257] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2732.216162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2732.219759] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2732.229187] Call Trace: [ 2732.230459] [ 2732.231414] ? show_regs.cold.9+0x22/0x2f [ 2732.237709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2732.241059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2732.242624] ? watchdog+0x40/0x40 [ 2732.249668] ? hrtimer_interrupt+0x128/0x2c0 [ 2732.252750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2732.256408] ? apic_timer_interrupt+0xf/0x20 [ 2732.260539] [ 2732.261323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2732.263534] do_raw_spin_lock+0x177/0x190 [ 2732.270779] _raw_spin_lock+0x21/0x30 [ 2732.272620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2732.277826] ? do_raw_spin_unlock+0x75/0x190 [ 2732.280725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2732.283440] ? do_wait_intr+0xf0/0xf0 [ 2732.285064] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2732.287488] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2732.295144] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2732.301171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2732.306142] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2732.308891] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2732.311072] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2732.314407] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2732.319137] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2732.322542] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2732.326681] lmv_intent_open+0x338/0x1150 [lmv] [ 2732.331551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2732.334531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2732.337531] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2732.339410] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2732.344848] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2732.348902] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2732.352413] ? do_raw_spin_unlock+0x75/0x190 [ 2732.355059] ? _raw_spin_unlock+0x12/0x30 [ 2732.356878] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2732.363399] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2732.365763] ll_atomic_open+0x36a/0xd70 [lustre] [ 2732.373457] lookup_open+0x356/0xb40 [ 2732.376456] do_last+0x313/0xfc0 [ 2732.380946] ? nd_jump_root+0xe5/0x160 [ 2732.384301] ? path_init+0x437/0x520 [ 2732.386438] path_openat+0xf7/0x500 [ 2732.388458] ? do_raw_spin_unlock+0x75/0x190 [ 2732.392578] do_filp_open+0x99/0x140 [ 2732.398935] ? getname_flags+0x6e/0x330 [ 2732.402174] ? __check_object_size+0xff/0x256 [ 2732.405471] ? do_raw_spin_unlock+0x75/0x190 [ 2732.408060] ? _raw_spin_unlock+0x12/0x30 [ 2732.409769] do_sys_openat2+0x2b4/0x410 [ 2732.413508] do_sys_open+0x73/0xa0 [ 2732.417428] __x64_sys_openat+0x24/0x30 [ 2732.423484] do_syscall_64+0xc1/0x440 [ 2732.425183] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2732.426957] RIP: 0033:0x7fa749552332 [ 2732.430665] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2732.444565] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2732.451702] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2732.460740] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2732.469210] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2732.476528] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2732.481523] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2736.061214] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2736.066101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2736.097050] BUG: workqueue lockup - pool [ 2736.107576] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2736.107589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2736.107592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2736.107608] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2736.107613] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2736.107618] RAX: 0000000000007d07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2736.107620] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2736.107622] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2736.107624] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2736.107626] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2736.107629] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2736.107631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2736.107633] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2736.107641] Call Trace: [ 2736.107646] [ 2736.107652] ? show_regs.cold.9+0x22/0x2f [ 2736.107660] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2736.107666] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2736.107670] ? watchdog+0x40/0x40 [ 2736.107673] ? hrtimer_interrupt+0x128/0x2c0 [ 2736.107676] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2736.107684] ? apic_timer_interrupt+0xf/0x20 [ 2736.107689] [ 2736.107692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2736.107699] do_raw_spin_lock+0x177/0x190 [ 2736.109413] cpus=0-3 flags=0x4 nice=0 stuck for 818s! [ 2736.109430] Showing busy workqueues and worker pools: [ 2736.109432] workqueue events: flags=0x0 [ 2736.109435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2736.109780] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2736.110285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2736.110296] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2736.110721] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.110729] pending: kfree_rcu_monitor [ 2736.110740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2736.110747] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2736.110876] workqueue events_long: flags=0x0 [ 2736.110879] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.110885] pending: lu_objects_flush [obdclass] [ 2736.111162] workqueue events_unbound: flags=0x2 [ 2736.111165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2736.111173] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2736.111194] workqueue events_power_efficient: flags=0x80 [ 2736.111197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.111203] pending: check_lifetime [ 2736.111210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.111217] pending: neigh_periodic_work [ 2736.111224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.111231] pending: neigh_periodic_work [ 2736.111236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.111243] pending: do_cache_clean [ 2736.111254] workqueue mm_percpu_wq: flags=0x8 [ 2736.111256] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.111260] pending: vmstat_update [ 2736.111265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.111271] pending: vmstat_update [ 2736.111276] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2736.111282] pending: vmstat_update [ 2736.111289] workqueue writeback: flags=0x4a [ 2736.111291] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2736.111297] pending: wb_workfn, wb_workfn [ 2736.111312] workqueue xprtiod: flags=0xa [ 2736.111314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2736.111319] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2736.111337] workqueue ipv6_addrconf: flags=0x40008 [ 2736.111340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2736.111346] pending: addrconf_verify_work [ 2736.111357] workqueue knbd0-recv: flags=0x1a [ 2736.111359] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2736.111365] in-flight: 651:recv_work [ 2736.111374] workqueue ptlrpc_pinger: flags=0x2 [ 2736.111376] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2736.111381] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2736.111609] pool 8: cpus=0-3 flags=0x4 nice=0 hung=818s workers=4 idle: 37 31846 28897 [ 2736.111621] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2726s workers=2 idle: 653 [ 2736.481878] _raw_spin_lock+0x21/0x30 [ 2736.482932] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2736.487669] ? do_raw_spin_unlock+0x75/0x190 [ 2736.490687] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2736.496951] ? do_wait_intr+0xf0/0xf0 [ 2736.498109] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2736.499683] mdc_close+0x53b/0xd70 [mdc] [ 2736.501125] lmv_close+0x18d/0x490 [lmv] [ 2736.502226] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2736.507581] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2736.512422] ll_md_close+0x274/0x800 [lustre] [ 2736.518021] ll_file_release+0x128/0xc50 [lustre] [ 2736.521671] __fput+0x10d/0x440 [ 2736.522699] ____fput+0x12/0x20 [ 2736.524498] task_work_run+0xc6/0x110 [ 2736.530974] exit_to_usermode_loop+0x1dd/0x1f0 [ 2736.532341] do_syscall_64+0x3d6/0x440 [ 2736.536552] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2736.539471] RIP: 0033:0x7fa749552cd8 [ 2736.541710] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2736.548940] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2736.550779] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2736.553742] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2736.558815] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2736.562960] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2736.569860] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2748.058071] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2748.060832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2748.073408] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2748.076011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2748.077862] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2748.079469] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2748.083904] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2748.086160] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2748.088698] RDX: 0000000000004ad0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2748.090384] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2748.092561] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2748.094007] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2748.096069] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2748.098084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2748.099989] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2748.102233] Call Trace: [ 2748.103093] [ 2748.103510] ? show_regs.cold.9+0x22/0x2f [ 2748.104354] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2748.105461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2748.107009] ? watchdog+0x40/0x40 [ 2748.108576] ? hrtimer_interrupt+0x128/0x2c0 [ 2748.110959] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2748.112795] ? apic_timer_interrupt+0xf/0x20 [ 2748.113955] [ 2748.114404] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2748.115888] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2748.117328] do_raw_spin_lock+0x177/0x190 [ 2748.118423] _raw_spin_lock+0x21/0x30 [ 2748.119370] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2748.122591] obd_mod_free+0x46/0x140 [ptlrpc] [ 2748.123857] mdc_commit_open+0x91/0xb0 [mdc] [ 2748.125333] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2748.127022] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2748.129379] after_reply+0xe2c/0x1780 [ptlrpc] [ 2748.131219] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2748.133163] ? __next_timer_interrupt+0x160/0x160 [ 2748.134291] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2748.135801] ? do_wait_intr+0xf0/0xf0 [ 2748.137161] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2748.139928] kthread+0x1d1/0x200 [ 2748.141367] ? set_kthread_struct+0x70/0x70 [ 2748.143266] ret_from_fork+0x1f/0x30 [ 2756.057615] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2756.067836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2756.097044] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2756.102110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2756.106131] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2756.109197] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2756.112087] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2756.120803] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2756.124119] RAX: 0000000000004a8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2756.127148] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2756.135744] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2756.137827] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2756.145977] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2756.152121] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2756.156877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2756.159602] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2756.162595] Call Trace: [ 2756.163596] [ 2756.164218] ? show_regs.cold.9+0x22/0x2f [ 2756.165896] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2756.167955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2756.169868] ? watchdog+0x40/0x40 [ 2756.171136] ? hrtimer_interrupt+0x128/0x2c0 [ 2756.174884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2756.176942] ? apic_timer_interrupt+0xf/0x20 [ 2756.181715] [ 2756.182624] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2756.184862] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2756.187145] do_raw_spin_lock+0x177/0x190 [ 2756.193350] _raw_spin_lock+0x21/0x30 [ 2756.195041] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2756.202077] process_one_work+0x2c8/0x700 [ 2756.203838] worker_thread+0x296/0x6e0 [ 2756.205193] ? rescuer_thread+0x570/0x570 [ 2756.207174] kthread+0x1d1/0x200 [ 2756.208774] ? set_kthread_struct+0x70/0x70 [ 2756.214760] ret_from_fork+0x1f/0x30 [ 2764.055312] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2764.062361] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2764.071425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2764.071509] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2764.071517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2764.071520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.071545] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2764.071549] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2764.071554] RAX: 0000000000006d8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2764.071557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2764.071559] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2764.071561] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2764.071564] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2764.071567] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2764.071570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2764.071573] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2764.071581] Call Trace: [ 2764.071587] [ 2764.071593] ? show_regs.cold.9+0x22/0x2f [ 2764.071602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2764.071608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2764.071612] ? watchdog+0x40/0x40 [ 2764.071616] ? hrtimer_interrupt+0x128/0x2c0 [ 2764.071620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2764.071629] ? apic_timer_interrupt+0xf/0x20 [ 2764.080553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2764.080629] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2764.080637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2764.080640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.080655] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2764.080659] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2764.080664] RAX: 000000000000590a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2764.080667] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2764.080669] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2764.080671] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2764.080673] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2764.080676] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2764.080678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2764.080681] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2764.080689] Call Trace: [ 2764.080694] [ 2764.080700] ? show_regs.cold.9+0x22/0x2f [ 2764.080709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2764.080715] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2764.080719] ? watchdog+0x40/0x40 [ 2764.080722] ? hrtimer_interrupt+0x128/0x2c0 [ 2764.080726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2764.080735] ? apic_timer_interrupt+0xf/0x20 [ 2764.080739] [ 2764.080742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.080749] do_raw_spin_lock+0x177/0x190 [ 2764.080754] _raw_spin_lock+0x21/0x30 [ 2764.080759] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2764.081025] ? do_raw_spin_unlock+0x75/0x190 [ 2764.081031] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2764.081224] ? do_wait_intr+0xf0/0xf0 [ 2764.081233] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2764.081425] mdc_close+0x53b/0xd70 [mdc] [ 2764.081463] lmv_close+0x18d/0x490 [lmv] [ 2764.081491] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2764.081619] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2764.081701] ll_md_close+0x274/0x800 [lustre] [ 2764.081782] ll_file_release+0x128/0xc50 [lustre] [ 2764.081858] __fput+0x10d/0x440 [ 2764.081867] ____fput+0x12/0x20 [ 2764.081871] task_work_run+0xc6/0x110 [ 2764.081877] exit_to_usermode_loop+0x1dd/0x1f0 [ 2764.081883] do_syscall_64+0x3d6/0x440 [ 2764.081887] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2764.081893] RIP: 0033:0x7fa749552cd8 [ 2764.081901] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2764.081903] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2764.081907] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2764.081909] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2764.081911] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2764.081912] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2764.081915] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2764.458986] [ 2764.460184] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.464295] do_raw_spin_lock+0x177/0x190 [ 2764.466468] _raw_spin_lock+0x21/0x30 [ 2764.468636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2764.475464] ? do_raw_spin_unlock+0x75/0x190 [ 2764.480868] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2764.484170] ? do_wait_intr+0xf0/0xf0 [ 2764.487228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2764.491287] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2764.494701] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2764.498215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.501044] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2764.504410] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2764.507180] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2764.509404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.512067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2764.515199] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2764.520353] lmv_intent_open+0x338/0x1150 [lmv] [ 2764.521992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.526158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.533404] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2764.537508] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2764.541802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.545939] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2764.549208] ? do_raw_spin_unlock+0x75/0x190 [ 2764.554917] ? _raw_spin_unlock+0x12/0x30 [ 2764.564844] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2764.568334] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2764.578460] ll_atomic_open+0x36a/0xd70 [lustre] [ 2764.584954] lookup_open+0x356/0xb40 [ 2764.586634] do_last+0x313/0xfc0 [ 2764.587730] ? nd_jump_root+0xe5/0x160 [ 2764.592705] ? path_init+0x437/0x520 [ 2764.595179] path_openat+0xf7/0x500 [ 2764.597456] ? do_raw_spin_unlock+0x75/0x190 [ 2764.600015] do_filp_open+0x99/0x140 [ 2764.601929] ? getname_flags+0x6e/0x330 [ 2764.605491] ? __check_object_size+0xff/0x256 [ 2764.609415] ? do_raw_spin_unlock+0x75/0x190 [ 2764.611454] ? _raw_spin_unlock+0x12/0x30 [ 2764.619509] do_sys_openat2+0x2b4/0x410 [ 2764.622490] do_sys_open+0x73/0xa0 [ 2764.624532] __x64_sys_openat+0x24/0x30 [ 2764.627024] do_syscall_64+0xc1/0x440 [ 2764.631856] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2764.637471] RIP: 0033:0x7fa749552332 [ 2764.639349] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2764.655433] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2764.662563] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2764.666474] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2764.670458] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2764.675640] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2764.684379] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2766.306308] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 849s! [ 2766.310195] Showing busy workqueues and worker pools: [ 2766.317471] workqueue events: flags=0x0 [ 2766.321328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2766.325873] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2766.337587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2766.352027] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2766.357141] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.359306] pending: kfree_rcu_monitor [ 2766.366015] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2766.369356] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2766.373267] workqueue events_long: flags=0x0 [ 2766.375781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.378804] pending: lu_objects_flush [obdclass] [ 2766.381521] workqueue events_unbound: flags=0x2 [ 2766.384607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2766.387970] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2766.393019] workqueue events_power_efficient: flags=0x80 [ 2766.396576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.400856] pending: check_lifetime [ 2766.404341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.407803] pending: neigh_periodic_work [ 2766.410391] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.414033] pending: neigh_periodic_work [ 2766.417385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.425925] pending: do_cache_clean [ 2766.428861] workqueue mm_percpu_wq: flags=0x8 [ 2766.430884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.451738] pending: vmstat_update [ 2766.456456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.469143] pending: vmstat_update [ 2766.470541] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2766.483028] pending: vmstat_update [ 2766.489846] workqueue writeback: flags=0x4a [ 2766.492379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2766.498317] pending: wb_workfn, wb_workfn [ 2766.500288] workqueue xprtiod: flags=0xa [ 2766.502943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2766.508513] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2766.518572] workqueue ipv6_addrconf: flags=0x40008 [ 2766.524604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2766.535489] pending: addrconf_verify_work [ 2766.541359] workqueue knbd0-recv: flags=0x1a [ 2766.549026] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2766.558059] in-flight: 651:recv_work [ 2766.560599] workqueue ptlrpc_pinger: flags=0x2 [ 2766.574392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2766.583190] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2766.593400] pool 8: cpus=0-3 flags=0x4 nice=0 hung=849s workers=4 idle: 37 31846 28897 [ 2766.600419] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2756s workers=2 idle: 653 [ 2776.058027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2776.063011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2776.093769] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2776.105718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2776.111177] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2776.118130] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2776.128645] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2776.137672] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2776.143085] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2776.147126] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2776.150741] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2776.154600] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2776.158040] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2776.162300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2776.169952] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2776.175228] Call Trace: [ 2776.177231] [ 2776.178341] ? show_regs.cold.9+0x22/0x2f [ 2776.183214] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2776.185178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2776.188102] ? watchdog+0x40/0x40 [ 2776.189656] ? hrtimer_interrupt+0x128/0x2c0 [ 2776.192542] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2776.198248] ? apic_timer_interrupt+0xf/0x20 [ 2776.201768] [ 2776.205364] ? native_safe_halt+0x2e/0x50 [ 2776.207840] kvm_wait+0xcd/0xe0 [ 2776.209158] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2776.215437] do_raw_spin_lock+0x177/0x190 [ 2776.217474] _raw_spin_lock+0x21/0x30 [ 2776.223074] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2776.229490] obd_mod_free+0x46/0x140 [ptlrpc] [ 2776.234633] mdc_commit_open+0x91/0xb0 [mdc] [ 2776.239778] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2776.246925] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2776.255358] after_reply+0xe2c/0x1780 [ptlrpc] [ 2776.258157] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2776.262078] ? __next_timer_interrupt+0x160/0x160 [ 2776.264477] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2776.266994] ? do_wait_intr+0xf0/0xf0 [ 2776.268704] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2776.271273] kthread+0x1d1/0x200 [ 2776.272558] ? set_kthread_struct+0x70/0x70 [ 2776.274375] ret_from_fork+0x1f/0x30 [ 2784.057645] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2784.063733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2784.106293] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2784.112569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2784.120665] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2784.127716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2784.132073] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2784.144660] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2784.147809] RAX: 0000000000007f06 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2784.151591] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2784.156159] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2784.161681] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2784.169347] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2784.175088] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2784.181381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2784.183335] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2784.189339] Call Trace: [ 2784.190798] [ 2784.192465] ? show_regs.cold.9+0x22/0x2f [ 2784.196564] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2784.199912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2784.203858] ? watchdog+0x40/0x40 [ 2784.208657] ? hrtimer_interrupt+0x128/0x2c0 [ 2784.211556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2784.220495] ? apic_timer_interrupt+0xf/0x20 [ 2784.232663] [ 2784.234383] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2784.241426] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2784.247379] do_raw_spin_lock+0x177/0x190 [ 2784.254520] _raw_spin_lock+0x21/0x30 [ 2784.256190] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2784.268571] process_one_work+0x2c8/0x700 [ 2784.274549] worker_thread+0x296/0x6e0 [ 2784.279876] ? rescuer_thread+0x570/0x570 [ 2784.284020] kthread+0x1d1/0x200 [ 2784.288803] ? set_kthread_struct+0x70/0x70 [ 2784.296295] ret_from_fork+0x1f/0x30 [ 2792.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2792.057502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2792.059289] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 2792.099712] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2792.099724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2792.099728] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2792.099742] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2792.099746] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2792.099751] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2792.099753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2792.099755] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2792.099757] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2792.099760] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2792.099763] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2792.099766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2792.099768] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2792.099777] Call Trace: [ 2792.099782] [ 2792.099787] ? show_regs.cold.9+0x22/0x2f [ 2792.099796] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2792.099802] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2792.099807] ? watchdog+0x40/0x40 [ 2792.099810] ? hrtimer_interrupt+0x128/0x2c0 [ 2792.099814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2792.099820] ? apic_timer_interrupt+0xf/0x20 [ 2792.099825] [ 2792.099828] ? native_safe_halt+0x2e/0x50 [ 2792.099831] kvm_wait+0xcd/0xe0 [ 2792.099838] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2792.099849] do_raw_spin_lock+0x177/0x190 [ 2792.099854] _raw_spin_lock+0x21/0x30 [ 2792.099858] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2792.100157] ? do_raw_spin_unlock+0x75/0x190 [ 2792.100163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2792.100364] ? do_wait_intr+0xf0/0xf0 [ 2792.100373] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2792.100573] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2792.100796] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2792.101012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.101141] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2792.101176] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2792.101202] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2792.101226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.101309] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2792.101518] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2792.101545] lmv_intent_open+0x338/0x1150 [lmv] [ 2792.101573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.101657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.101740] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2792.101762] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2792.101853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.101936] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2792.102022] ? do_raw_spin_unlock+0x75/0x190 [ 2792.102028] ? _raw_spin_unlock+0x12/0x30 [ 2792.122295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2792.122368] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2792.122376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2792.122379] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2792.122392] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2792.122396] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2792.122406] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2792.122408] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2792.122410] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2792.122411] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2792.122414] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2792.122416] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2792.122419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2792.122421] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2792.122428] Call Trace: [ 2792.122434] [ 2792.122440] ? show_regs.cold.9+0x22/0x2f [ 2792.122450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2792.122455] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2792.122459] ? watchdog+0x40/0x40 [ 2792.122463] ? hrtimer_interrupt+0x128/0x2c0 [ 2792.122467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2792.122473] ? apic_timer_interrupt+0xf/0x20 [ 2792.122477] [ 2792.122479] ? native_safe_halt+0x2e/0x50 [ 2792.122483] kvm_wait+0xcd/0xe0 [ 2792.122490] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2792.122498] do_raw_spin_lock+0x177/0x190 [ 2792.122502] _raw_spin_lock+0x21/0x30 [ 2792.122506] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2792.122851] ? do_raw_spin_unlock+0x75/0x190 [ 2792.122858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2792.150484] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2792.150538] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2792.150565] ll_atomic_open+0x36a/0xd70 [lustre] [ 2792.150690] lookup_open+0x356/0xb40 [ 2792.150701] do_last+0x313/0xfc0 [ 2792.150706] ? nd_jump_root+0xe5/0x160 [ 2792.150711] ? path_init+0x437/0x520 [ 2792.150716] path_openat+0xf7/0x500 [ 2792.150721] ? do_raw_spin_unlock+0x75/0x190 [ 2792.150728] do_filp_open+0x99/0x140 [ 2792.150735] ? getname_flags+0x6e/0x330 [ 2792.150740] ? __check_object_size+0xff/0x256 [ 2792.150747] ? do_raw_spin_unlock+0x75/0x190 [ 2792.150751] ? _raw_spin_unlock+0x12/0x30 [ 2792.150758] do_sys_openat2+0x2b4/0x410 [ 2792.150762] do_sys_open+0x73/0xa0 [ 2792.150767] __x64_sys_openat+0x24/0x30 [ 2792.150771] do_syscall_64+0xc1/0x440 [ 2792.150777] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2792.150784] RIP: 0033:0x7fa749552332 [ 2792.150791] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2792.150795] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2792.150801] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2792.150803] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2792.150806] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2792.150808] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2792.150810] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2792.848066] ? do_wait_intr+0xf0/0xf0 [ 2792.852872] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2792.861090] mdc_close+0x53b/0xd70 [mdc] [ 2792.868391] lmv_close+0x18d/0x490 [lmv] [ 2792.872223] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2792.879004] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2792.883257] ll_md_close+0x274/0x800 [lustre] [ 2792.888478] ll_file_release+0x128/0xc50 [lustre] [ 2792.897844] __fput+0x10d/0x440 [ 2792.899507] ____fput+0x12/0x20 [ 2792.900520] task_work_run+0xc6/0x110 [ 2792.902446] exit_to_usermode_loop+0x1dd/0x1f0 [ 2792.906713] do_syscall_64+0x3d6/0x440 [ 2792.908461] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2792.911479] RIP: 0033:0x7fa749552cd8 [ 2792.916038] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2792.931430] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2792.944031] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2792.947475] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2792.953909] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2792.959564] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2792.963686] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2797.029136] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 879s! [ 2797.045414] Showing busy workqueues and worker pools: [ 2797.049464] workqueue events: flags=0x0 [ 2797.056348] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2797.067296] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2797.077462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2797.083215] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2797.090523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.096181] pending: kfree_rcu_monitor [ 2797.098287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2797.107892] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2797.114225] workqueue events_long: flags=0x0 [ 2797.117345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.120624] pending: lu_objects_flush [obdclass] [ 2797.123546] workqueue events_unbound: flags=0x2 [ 2797.125863] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2797.128651] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2797.132939] workqueue events_power_efficient: flags=0x80 [ 2797.135856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.138726] pending: check_lifetime [ 2797.140278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.143390] pending: neigh_periodic_work [ 2797.145364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.147500] pending: neigh_periodic_work [ 2797.150357] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.153420] pending: do_cache_clean [ 2797.154981] workqueue mm_percpu_wq: flags=0x8 [ 2797.156969] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.159797] pending: vmstat_update [ 2797.161198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.163809] pending: vmstat_update [ 2797.164822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2797.167093] pending: vmstat_update [ 2797.168216] workqueue writeback: flags=0x4a [ 2797.169388] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2797.171136] pending: wb_workfn, wb_workfn [ 2797.172508] workqueue xprtiod: flags=0xa [ 2797.173796] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2797.176114] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2797.179443] workqueue ipv6_addrconf: flags=0x40008 [ 2797.180956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2797.183301] pending: addrconf_verify_work [ 2797.184424] workqueue knbd0-recv: flags=0x1a [ 2797.186203] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2797.188079] in-flight: 651:recv_work [ 2797.189164] workqueue ptlrpc_pinger: flags=0x2 [ 2797.190446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2797.193051] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2797.194871] pool 8: cpus=0-3 flags=0x4 nice=0 hung=880s workers=4 idle: 37 31846 28897 [ 2797.196881] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2787s workers=2 idle: 653 [ 2804.061822] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2804.069479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2804.115822] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2804.124456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2804.129814] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2804.136036] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2804.151781] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2804.171734] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2804.176919] RDX: 0000000000007f84 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2804.188565] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2804.197236] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2804.199598] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2804.206018] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2804.213039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2804.217467] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2804.224926] Call Trace: [ 2804.226520] [ 2804.227417] ? show_regs.cold.9+0x22/0x2f [ 2804.230396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2804.239518] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2804.244301] ? watchdog+0x40/0x40 [ 2804.246039] ? hrtimer_interrupt+0x128/0x2c0 [ 2804.247612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2804.249316] ? apic_timer_interrupt+0xf/0x20 [ 2804.253331] [ 2804.254042] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2804.256604] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2804.261339] do_raw_spin_lock+0x177/0x190 [ 2804.265444] _raw_spin_lock+0x21/0x30 [ 2804.266841] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2804.274197] obd_mod_free+0x46/0x140 [ptlrpc] [ 2804.279867] mdc_commit_open+0x91/0xb0 [mdc] [ 2804.281418] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2804.288995] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2804.290996] after_reply+0xe2c/0x1780 [ptlrpc] [ 2804.294932] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2804.299459] ? __next_timer_interrupt+0x160/0x160 [ 2804.302359] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2804.310376] ? do_wait_intr+0xf0/0xf0 [ 2804.311441] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2804.319185] kthread+0x1d1/0x200 [ 2804.320335] ? set_kthread_struct+0x70/0x70 [ 2804.321793] ret_from_fork+0x1f/0x30 [ 2812.057028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2812.063555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2812.118646] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2812.144940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2812.164299] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2812.179946] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2812.181586] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2812.222052] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2812.227721] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2812.234604] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2812.241252] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2812.247601] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2812.256691] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2812.264737] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2812.278188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2812.291558] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2812.298114] Call Trace: [ 2812.299179] [ 2812.300321] ? show_regs.cold.9+0x22/0x2f [ 2812.305125] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2812.312913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2812.315863] ? watchdog+0x40/0x40 [ 2812.317646] ? hrtimer_interrupt+0x128/0x2c0 [ 2812.319352] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2812.322597] ? apic_timer_interrupt+0xf/0x20 [ 2812.327781] [ 2812.328735] ? native_safe_halt+0x2e/0x50 [ 2812.332815] kvm_wait+0xcd/0xe0 [ 2812.334167] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2812.337495] do_raw_spin_lock+0x177/0x190 [ 2812.339469] _raw_spin_lock+0x21/0x30 [ 2812.340765] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2812.346658] process_one_work+0x2c8/0x700 [ 2812.348848] worker_thread+0x296/0x6e0 [ 2812.350634] ? rescuer_thread+0x570/0x570 [ 2812.352249] kthread+0x1d1/0x200 [ 2812.356762] ? set_kthread_struct+0x70/0x70 [ 2812.359317] ret_from_fork+0x1f/0x30 [ 2820.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2820.062010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2820.083458] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2820.088873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2820.092411] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2820.094160] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2820.102638] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2820.106079] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2820.109134] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2820.112796] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2820.115933] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2820.119042] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2820.121901] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2820.128838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2820.132372] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2820.138536] Call Trace: [ 2820.140253] [ 2820.141037] ? show_regs.cold.9+0x22/0x2f [ 2820.142364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2820.158406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2820.163632] ? watchdog+0x40/0x40 [ 2820.168069] ? hrtimer_interrupt+0x128/0x2c0 [ 2820.170995] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2820.179251] ? apic_timer_interrupt+0xf/0x20 [ 2820.186392] [ 2820.190664] ? native_safe_halt+0x2e/0x50 [ 2820.193523] kvm_wait+0xcd/0xe0 [ 2820.195272] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2820.198162] do_raw_spin_lock+0x177/0x190 [ 2820.200356] _raw_spin_lock+0x21/0x30 [ 2820.202912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2820.206221] ? do_raw_spin_unlock+0x75/0x190 [ 2820.208920] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2820.213715] ? do_wait_intr+0xf0/0xf0 [ 2820.214906] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2820.222211] mdc_close+0x53b/0xd70 [mdc] [ 2820.224813] lmv_close+0x18d/0x490 [lmv] [ 2820.226907] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2820.230241] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2820.232906] ll_md_close+0x274/0x800 [lustre] [ 2820.235108] ll_file_release+0x128/0xc50 [lustre] [ 2820.237035] __fput+0x10d/0x440 [ 2820.238381] ____fput+0x12/0x20 [ 2820.240085] task_work_run+0xc6/0x110 [ 2820.241384] exit_to_usermode_loop+0x1dd/0x1f0 [ 2820.243726] do_syscall_64+0x3d6/0x440 [ 2820.245390] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2820.248328] RIP: 0033:0x7fa749552cd8 [ 2820.250169] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2820.268528] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2820.273918] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2820.281953] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2820.285832] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2820.294938] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2820.306657] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2824.054056] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2824.057469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2824.104288] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2824.116254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2824.125563] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2824.128589] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2824.137095] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2824.140759] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2824.142964] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2824.146323] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2824.149844] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2824.153201] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2824.156550] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2824.160980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2824.163885] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2824.166590] Call Trace: [ 2824.167630] [ 2824.168534] ? show_regs.cold.9+0x22/0x2f [ 2824.170305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2824.172285] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2824.174739] ? watchdog+0x40/0x40 [ 2824.176337] ? hrtimer_interrupt+0x128/0x2c0 [ 2824.178350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2824.180724] ? apic_timer_interrupt+0xf/0x20 [ 2824.182839] [ 2824.183831] ? native_safe_halt+0x2e/0x50 [ 2824.185602] kvm_wait+0xcd/0xe0 [ 2824.187108] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2824.189145] do_raw_spin_lock+0x177/0x190 [ 2824.190964] _raw_spin_lock+0x21/0x30 [ 2824.192639] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2824.195400] ? do_raw_spin_unlock+0x75/0x190 [ 2824.197446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2824.200296] ? do_wait_intr+0xf0/0xf0 [ 2824.201817] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2824.203705] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2824.205966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2824.208901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.211894] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2824.214448] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2824.216740] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2824.225066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.227272] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2824.233824] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2824.237133] lmv_intent_open+0x338/0x1150 [lmv] [ 2824.239930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.244093] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.252502] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2824.258105] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2824.263369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.269166] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2824.271743] ? do_raw_spin_unlock+0x75/0x190 [ 2824.276358] ? _raw_spin_unlock+0x12/0x30 [ 2824.280113] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2824.285360] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2824.289166] ll_atomic_open+0x36a/0xd70 [lustre] [ 2824.295034] lookup_open+0x356/0xb40 [ 2824.297081] do_last+0x313/0xfc0 [ 2824.301775] ? nd_jump_root+0xe5/0x160 [ 2824.305100] ? path_init+0x437/0x520 [ 2824.309641] path_openat+0xf7/0x500 [ 2824.316284] ? do_raw_spin_unlock+0x75/0x190 [ 2824.318154] do_filp_open+0x99/0x140 [ 2824.319619] ? getname_flags+0x6e/0x330 [ 2824.323879] ? __check_object_size+0xff/0x256 [ 2824.325399] ? do_raw_spin_unlock+0x75/0x190 [ 2824.333954] ? _raw_spin_unlock+0x12/0x30 [ 2824.336136] do_sys_openat2+0x2b4/0x410 [ 2824.341529] do_sys_open+0x73/0xa0 [ 2824.344723] __x64_sys_openat+0x24/0x30 [ 2824.346730] do_syscall_64+0xc1/0x440 [ 2824.348780] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2824.352998] RIP: 0033:0x7fa749552332 [ 2824.356397] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2824.377900] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2824.382611] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2824.385048] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2824.392044] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2824.397421] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2824.406502] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2827.232319] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 910s! [ 2827.240907] Showing busy workqueues and worker pools: [ 2827.243899] workqueue events: flags=0x0 [ 2827.245896] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2827.249379] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2827.257440] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2827.263881] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2827.271130] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.278893] pending: kfree_rcu_monitor [ 2827.283585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2827.290952] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2827.297454] workqueue events_long: flags=0x0 [ 2827.300941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.308672] pending: lu_objects_flush [obdclass] [ 2827.315303] workqueue events_unbound: flags=0x2 [ 2827.318102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2827.320360] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2827.332124] workqueue events_power_efficient: flags=0x80 [ 2827.336124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.344447] pending: check_lifetime [ 2827.346446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.353788] pending: neigh_periodic_work [ 2827.356589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.359952] pending: neigh_periodic_work [ 2827.361823] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.368930] pending: do_cache_clean [ 2827.376831] workqueue mm_percpu_wq: flags=0x8 [ 2827.382701] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.389134] pending: vmstat_update [ 2827.397065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.409872] pending: vmstat_update [ 2827.413659] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2827.417054] pending: vmstat_update [ 2827.421528] workqueue writeback: flags=0x4a [ 2827.425861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2827.428079] pending: wb_workfn, wb_workfn [ 2827.431824] workqueue xprtiod: flags=0xa [ 2827.433180] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2827.437888] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2827.443507] workqueue ipv6_addrconf: flags=0x40008 [ 2827.446705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2827.453897] pending: addrconf_verify_work [ 2827.456995] workqueue knbd0-recv: flags=0x1a [ 2827.460369] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2827.466373] in-flight: 651:recv_work [ 2827.468322] workqueue ptlrpc_pinger: flags=0x2 [ 2827.471269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2827.476892] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2827.487381] pool 8: cpus=0-3 flags=0x4 nice=0 hung=910s workers=4 idle: 37 31846 28897 [ 2827.490232] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2817s workers=2 idle: 653 [ 2832.060156] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2832.065349] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2832.088918] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2832.093758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2832.098130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2832.100605] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2832.108378] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2832.111318] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2832.114650] RDX: 0000000000000afb RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2832.119035] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2832.127855] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2832.143089] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2832.156760] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2832.173914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2832.177314] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2832.181932] Call Trace: [ 2832.183076] [ 2832.184213] ? show_regs.cold.9+0x22/0x2f [ 2832.186924] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2832.194085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2832.196222] ? watchdog+0x40/0x40 [ 2832.197783] ? hrtimer_interrupt+0x128/0x2c0 [ 2832.199769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2832.202389] ? apic_timer_interrupt+0xf/0x20 [ 2832.205446] [ 2832.209303] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2832.214322] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2832.220946] do_raw_spin_lock+0x177/0x190 [ 2832.226361] _raw_spin_lock+0x21/0x30 [ 2832.228478] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2832.236013] obd_mod_free+0x46/0x140 [ptlrpc] [ 2832.241772] mdc_commit_open+0x91/0xb0 [mdc] [ 2832.245369] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2832.252880] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2832.263297] after_reply+0xe2c/0x1780 [ptlrpc] [ 2832.266300] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2832.275193] ? __next_timer_interrupt+0x160/0x160 [ 2832.283547] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2832.287042] ? do_wait_intr+0xf0/0xf0 [ 2832.293211] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2832.300477] kthread+0x1d1/0x200 [ 2832.366004] ? set_kthread_struct+0x70/0x70 [ 2832.378811] ret_from_fork+0x1f/0x30 [ 2840.059467] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2840.063702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2840.115307] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2840.123259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2840.131780] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2840.139068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2840.144327] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2840.159381] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2840.164077] RAX: 0000000000004562 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2840.168773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2840.172088] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2840.175978] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2840.182904] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2840.186146] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2840.198641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2840.204863] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2840.214514] Call Trace: [ 2840.217989] [ 2840.219300] ? show_regs.cold.9+0x22/0x2f [ 2840.235111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2840.248351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2840.249989] ? watchdog+0x40/0x40 [ 2840.256899] ? hrtimer_interrupt+0x128/0x2c0 [ 2840.259032] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2840.262384] ? apic_timer_interrupt+0xf/0x20 [ 2840.265156] [ 2840.266638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2840.273953] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2840.278025] do_raw_spin_lock+0x177/0x190 [ 2840.280175] _raw_spin_lock+0x21/0x30 [ 2840.284486] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2840.288199] process_one_work+0x2c8/0x700 [ 2840.290780] worker_thread+0x296/0x6e0 [ 2840.293846] ? rescuer_thread+0x570/0x570 [ 2840.296360] kthread+0x1d1/0x200 [ 2840.303041] ? set_kthread_struct+0x70/0x70 [ 2840.306506] ret_from_fork+0x1f/0x30 [ 2848.061032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2848.069385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.145329] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.155971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.165209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.168871] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2848.183637] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2848.186616] RAX: 0000000000007f92 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2848.190011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2848.194205] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2848.201554] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2848.209707] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2848.215958] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2848.222636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.226698] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2848.235432] Call Trace: [ 2848.236564] [ 2848.237404] ? show_regs.cold.9+0x22/0x2f [ 2848.242118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.246996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.251270] ? watchdog+0x40/0x40 [ 2848.263372] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.266811] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.269655] ? apic_timer_interrupt+0xf/0x20 [ 2848.271949] [ 2848.272716] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.278391] do_raw_spin_lock+0x177/0x190 [ 2848.281988] _raw_spin_lock+0x21/0x30 [ 2848.283722] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2848.286920] ? do_raw_spin_unlock+0x75/0x190 [ 2848.289411] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2848.292402] ? do_wait_intr+0xf0/0xf0 [ 2848.295488] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2848.300007] mdc_close+0x53b/0xd70 [mdc] [ 2848.303202] lmv_close+0x18d/0x490 [lmv] [ 2848.308617] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2848.311589] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2848.319027] ll_md_close+0x274/0x800 [lustre] [ 2848.322771] ll_file_release+0x128/0xc50 [lustre] [ 2848.324312] __fput+0x10d/0x440 [ 2848.325243] ____fput+0x12/0x20 [ 2848.328134] task_work_run+0xc6/0x110 [ 2848.330133] exit_to_usermode_loop+0x1dd/0x1f0 [ 2848.332895] do_syscall_64+0x3d6/0x440 [ 2848.334438] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2848.337093] RIP: 0033:0x7fa749552cd8 [ 2848.338415] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2848.350014] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2848.353251] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2848.356550] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2848.363433] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2848.367064] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2848.370184] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2852.055857] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 2852.060387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2852.100348] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2852.114437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2852.123511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2852.130763] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2852.146733] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2852.156766] RAX: 0000000000002a8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2852.163452] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2852.171042] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2852.185549] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2852.200361] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2852.213336] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2852.229178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2852.231532] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2852.245098] Call Trace: [ 2852.253195] [ 2852.254324] ? show_regs.cold.9+0x22/0x2f [ 2852.260257] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2852.267772] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2852.269405] ? watchdog+0x40/0x40 [ 2852.278434] ? hrtimer_interrupt+0x128/0x2c0 [ 2852.284024] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2852.290356] ? apic_timer_interrupt+0xf/0x20 [ 2852.291912] [ 2852.308857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2852.319230] do_raw_spin_lock+0x177/0x190 [ 2852.320634] _raw_spin_lock+0x21/0x30 [ 2852.321974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2852.338582] ? do_raw_spin_unlock+0x75/0x190 [ 2852.352362] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2852.369445] ? do_wait_intr+0xf0/0xf0 [ 2852.376955] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2852.387705] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2852.398930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2852.409371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.417128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2852.423283] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2852.432670] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2852.437994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.443920] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2852.452421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2852.457350] lmv_intent_open+0x338/0x1150 [lmv] [ 2852.464965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.473304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.485445] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2852.490182] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2852.496942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.508839] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2852.511752] ? do_raw_spin_unlock+0x75/0x190 [ 2852.523051] ? _raw_spin_unlock+0x12/0x30 [ 2852.529939] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2852.534248] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2852.537916] ll_atomic_open+0x36a/0xd70 [lustre] [ 2852.541915] lookup_open+0x356/0xb40 [ 2852.545927] do_last+0x313/0xfc0 [ 2852.551047] ? nd_jump_root+0xe5/0x160 [ 2852.552141] ? path_init+0x437/0x520 [ 2852.553530] path_openat+0xf7/0x500 [ 2852.557502] ? do_raw_spin_unlock+0x75/0x190 [ 2852.570001] do_filp_open+0x99/0x140 [ 2852.574505] ? getname_flags+0x6e/0x330 [ 2852.579505] ? __check_object_size+0xff/0x256 [ 2852.584888] ? do_raw_spin_unlock+0x75/0x190 [ 2852.595266] ? _raw_spin_unlock+0x12/0x30 [ 2852.596570] do_sys_openat2+0x2b4/0x410 [ 2852.603242] do_sys_open+0x73/0xa0 [ 2852.612215] __x64_sys_openat+0x24/0x30 [ 2852.619427] do_syscall_64+0xc1/0x440 [ 2852.621254] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2852.626906] RIP: 0033:0x7fa749552332 [ 2852.628523] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2852.640858] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2852.642943] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2852.648375] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2852.652036] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2852.655297] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2852.658870] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2857.957484] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 940s! [ 2857.964250] Showing busy workqueues and worker pools: [ 2857.968704] workqueue events: flags=0x0 [ 2857.972154] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2857.976320] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2858.003214] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2858.008698] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2858.015771] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.024783] pending: kfree_rcu_monitor [ 2858.031457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2858.036108] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2858.040540] workqueue events_long: flags=0x0 [ 2858.044258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.050265] pending: lu_objects_flush [obdclass] [ 2858.056763] workqueue events_unbound: flags=0x2 [ 2858.059921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2858.064143] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2858.072264] workqueue events_power_efficient: flags=0x80 [ 2858.075231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.078273] pending: check_lifetime [ 2858.079587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.082529] pending: neigh_periodic_work [ 2858.084421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.087301] pending: neigh_periodic_work [ 2858.090754] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.095458] pending: do_cache_clean [ 2858.100442] workqueue mm_percpu_wq: flags=0x8 [ 2858.105044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.109440] pending: vmstat_update [ 2858.113317] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.124307] pending: vmstat_update [ 2858.127543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2858.136666] pending: vmstat_update [ 2858.138564] workqueue writeback: flags=0x4a [ 2858.144342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2858.148427] pending: wb_workfn, wb_workfn [ 2858.151534] workqueue xprtiod: flags=0xa [ 2858.154153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2858.157780] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2858.161610] workqueue ipv6_addrconf: flags=0x40008 [ 2858.164285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2858.167909] pending: addrconf_verify_work [ 2858.170155] workqueue knbd0-recv: flags=0x1a [ 2858.172184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2858.175210] in-flight: 651:recv_work [ 2858.177736] workqueue ptlrpc_pinger: flags=0x2 [ 2858.179131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2858.181638] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2858.184608] pool 8: cpus=0-3 flags=0x4 nice=0 hung=941s workers=4 idle: 37 31846 28897 [ 2858.189639] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2848s workers=2 idle: 653 [ 2860.062141] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2860.068459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2860.098135] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2860.101212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2860.104336] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2860.106276] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2860.116537] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2860.122298] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2860.134098] RDX: 00000000000060b2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2860.139024] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2860.145301] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2860.153482] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2860.158193] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2860.164225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2860.166479] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2860.169396] Call Trace: [ 2860.170372] [ 2860.171273] ? show_regs.cold.9+0x22/0x2f [ 2860.173166] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2860.175301] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2860.176503] ? watchdog+0x40/0x40 [ 2860.178085] ? hrtimer_interrupt+0x128/0x2c0 [ 2860.181145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2860.186661] ? apic_timer_interrupt+0xf/0x20 [ 2860.187739] [ 2860.190419] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2860.201078] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2860.203991] do_raw_spin_lock+0x177/0x190 [ 2860.205317] _raw_spin_lock+0x21/0x30 [ 2860.206502] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2860.209469] obd_mod_free+0x46/0x140 [ptlrpc] [ 2860.211323] mdc_commit_open+0x91/0xb0 [mdc] [ 2860.213524] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2860.217362] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2860.222182] after_reply+0xe2c/0x1780 [ptlrpc] [ 2860.224733] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2860.228399] ? __next_timer_interrupt+0x160/0x160 [ 2860.231502] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2860.232933] ? do_wait_intr+0xf0/0xf0 [ 2860.238018] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2860.246645] kthread+0x1d1/0x200 [ 2860.250391] ? set_kthread_struct+0x70/0x70 [ 2860.252397] ret_from_fork+0x1f/0x30 [ 2868.057155] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2868.064039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2868.096159] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2868.110093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2868.116519] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2868.123204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2868.129035] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2868.144831] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2868.154749] RAX: 0000000000006e01 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2868.162240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2868.171352] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2868.179240] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2868.190840] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2868.194215] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2868.204127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2868.211680] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2868.218776] Call Trace: [ 2868.225193] [ 2868.227113] ? show_regs.cold.9+0x22/0x2f [ 2868.234410] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2868.238708] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2868.244979] ? watchdog+0x40/0x40 [ 2868.246230] ? hrtimer_interrupt+0x128/0x2c0 [ 2868.256105] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2868.261460] ? apic_timer_interrupt+0xf/0x20 [ 2868.263655] [ 2868.264200] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2868.268265] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2868.273125] do_raw_spin_lock+0x177/0x190 [ 2868.280104] _raw_spin_lock+0x21/0x30 [ 2868.283587] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2868.290214] process_one_work+0x2c8/0x700 [ 2868.295588] worker_thread+0x296/0x6e0 [ 2868.304138] ? rescuer_thread+0x570/0x570 [ 2868.312113] kthread+0x1d1/0x200 [ 2868.320812] ? set_kthread_struct+0x70/0x70 [ 2868.325073] ret_from_fork+0x1f/0x30 [ 2873.707752] rcu: INFO: rcu_sched self-detected stall on CPU [ 2873.712180] rcu: 2-...!: (527950 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 2873.725991] (t=960018 jiffies g=258701 q=1757) [ 2873.732475] rcu: rcu_sched kthread starved for 956813 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 2873.753999] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2873.784264] rcu: RCU grace-period kthread stack dump: [ 2873.796667] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2873.807511] Call Trace: [ 2873.815760] __schedule+0x351/0xcb0 [ 2873.820020] schedule+0xc0/0x180 [ 2873.822319] schedule_timeout+0xb4/0x190 [ 2873.823765] ? do_raw_spin_unlock+0x75/0x190 [ 2873.825330] ? __next_timer_interrupt+0x160/0x160 [ 2873.841990] ? prepare_to_swait_event+0xf3/0x140 [ 2873.862436] rcu_gp_kthread+0x8dd/0x15b0 [ 2873.869879] ? force_qs_rnp+0x2c0/0x2c0 [ 2873.879311] kthread+0x1d1/0x200 [ 2873.895732] ? set_kthread_struct+0x70/0x70 [ 2873.897220] ret_from_fork+0x1f/0x30 [ 2873.898431] rcu: Stack dump where RCU GP kthread last ran: [ 2873.921919] Sending NMI from CPU 2 to CPUs 1: [ 2873.924854] NMI backtrace for cpu 1 [ 2873.924859] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2873.924861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2873.924863] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2873.924865] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2873.924866] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2873.924868] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 [ 2873.924870] RAX: 0000000000000f31 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2873.924871] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2873.924872] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2873.924874] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2873.924875] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2873.924876] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2873.924877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2873.924879] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2873.924880] Call Trace: [ 2873.924881] [ 2873.924881] ? show_regs.cold.9+0x22/0x2f [ 2873.924882] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2873.924883] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2873.924884] ? nmi_handle+0xbd/0x210 [ 2873.924885] ? default_do_nmi+0x4b/0x1b0 [ 2873.924886] ? do_nmi+0x280/0x380 [ 2873.924887] ? end_repeat_nmi+0x16/0x1f [ 2873.924888] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2873.924889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2873.924890] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2873.924891] [ 2873.924892] do_raw_spin_lock+0x177/0x190 [ 2873.924893] _raw_spin_lock+0x21/0x30 [ 2873.924894] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2873.924895] process_one_work+0x2c8/0x700 [ 2873.924896] worker_thread+0x296/0x6e0 [ 2873.924897] ? rescuer_thread+0x570/0x570 [ 2873.924898] kthread+0x1d1/0x200 [ 2873.924899] ? set_kthread_struct+0x70/0x70 [ 2873.924900] ret_from_fork+0x1f/0x30 [ 2873.928444] NMI backtrace for cpu 2 [ 2874.173405] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2874.184375] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2874.192431] Call Trace: [ 2874.195277] [ 2874.197400] ? dump_stack+0xbb/0x10e [ 2874.199310] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2874.209667] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2874.212375] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2874.216475] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2874.222384] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2874.230249] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2874.234954] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2874.241123] ? update_process_times+0x6d/0xc0 [ 2874.245000] ? tick_sched_handle.isra.18+0x27/0x90 [ 2874.248618] ? tick_sched_timer+0x53/0xb0 [ 2874.249973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2874.259197] ? tick_sched_handle.isra.18+0x90/0x90 [ 2874.267021] ? hrtimer_interrupt+0x128/0x2c0 [ 2874.273990] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2874.277388] ? apic_timer_interrupt+0xf/0x20 [ 2874.281408] [ 2874.283442] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2874.288324] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2874.294490] ? do_raw_spin_lock+0x177/0x190 [ 2874.299014] ? _raw_spin_lock+0x21/0x30 [ 2874.302669] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2874.311429] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2874.317236] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2874.318843] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2874.329360] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2874.338366] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2874.344931] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2874.351024] ? __next_timer_interrupt+0x160/0x160 [ 2874.354885] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2874.360375] ? do_wait_intr+0xf0/0xf0 [ 2874.364226] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2874.369223] ? kthread+0x1d1/0x200 [ 2874.371874] ? set_kthread_struct+0x70/0x70 [ 2874.380330] ? ret_from_fork+0x1f/0x30 [ 2876.063317] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2876.068962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2876.100938] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2876.107862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2876.116976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.121001] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2876.131417] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2876.136264] RAX: 00000000000060a2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2876.140335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2876.145265] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2876.149545] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2876.153880] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2876.158073] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2876.163066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2876.166779] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2876.175151] Call Trace: [ 2876.180049] [ 2876.189058] ? show_regs.cold.9+0x22/0x2f [ 2876.197098] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2876.205742] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2876.212709] ? watchdog+0x40/0x40 [ 2876.214850] ? hrtimer_interrupt+0x128/0x2c0 [ 2876.218938] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2876.220548] ? apic_timer_interrupt+0xf/0x20 [ 2876.228681] [ 2876.232769] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.237011] do_raw_spin_lock+0x177/0x190 [ 2876.240107] _raw_spin_lock+0x21/0x30 [ 2876.243694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2876.252786] ? do_raw_spin_unlock+0x75/0x190 [ 2876.254034] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2876.260853] ? do_wait_intr+0xf0/0xf0 [ 2876.266544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2876.279922] mdc_close+0x53b/0xd70 [mdc] [ 2876.286673] lmv_close+0x18d/0x490 [lmv] [ 2876.292634] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2876.299367] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2876.307101] ll_md_close+0x274/0x800 [lustre] [ 2876.317708] ll_file_release+0x128/0xc50 [lustre] [ 2876.331808] __fput+0x10d/0x440 [ 2876.335938] ____fput+0x12/0x20 [ 2876.337149] task_work_run+0xc6/0x110 [ 2876.346472] exit_to_usermode_loop+0x1dd/0x1f0 [ 2876.353557] do_syscall_64+0x3d6/0x440 [ 2876.364544] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2876.366367] RIP: 0033:0x7fa749552cd8 [ 2876.377611] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2876.405042] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2876.418934] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2876.425635] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2876.427513] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2876.438658] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2876.446013] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2884.056129] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2884.061174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2884.091501] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2884.100633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2884.107359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2884.110055] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2884.124196] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2884.127232] RAX: 000000000000568a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2884.129863] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2884.131856] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2884.134538] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2884.138203] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2884.140860] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2884.144084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2884.146535] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2884.149349] Call Trace: [ 2884.150242] [ 2884.150908] ? show_regs.cold.9+0x22/0x2f [ 2884.152475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2884.153696] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2884.154846] ? watchdog+0x40/0x40 [ 2884.155667] ? hrtimer_interrupt+0x128/0x2c0 [ 2884.156816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2884.158070] ? apic_timer_interrupt+0xf/0x20 [ 2884.159211] [ 2884.159770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2884.161226] do_raw_spin_lock+0x177/0x190 [ 2884.162542] _raw_spin_lock+0x21/0x30 [ 2884.163831] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2884.166032] ? do_raw_spin_unlock+0x75/0x190 [ 2884.167616] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2884.169471] ? do_wait_intr+0xf0/0xf0 [ 2884.170654] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2884.172599] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2884.174664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2884.176789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2884.178682] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2884.180470] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2884.181877] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2884.183439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2884.185435] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2884.187699] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2884.189586] lmv_intent_open+0x338/0x1150 [lmv] [ 2884.191238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2884.193031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2884.194887] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2884.196313] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2884.197780] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2884.199558] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2884.201078] ? do_raw_spin_unlock+0x75/0x190 [ 2884.202596] ? _raw_spin_unlock+0x12/0x30 [ 2884.204049] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2884.205732] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2884.207824] ll_atomic_open+0x36a/0xd70 [lustre] [ 2884.209517] lookup_open+0x356/0xb40 [ 2884.210668] do_last+0x313/0xfc0 [ 2884.211724] ? nd_jump_root+0xe5/0x160 [ 2884.213440] ? path_init+0x437/0x520 [ 2884.214645] path_openat+0xf7/0x500 [ 2884.215591] ? do_raw_spin_unlock+0x75/0x190 [ 2884.216822] do_filp_open+0x99/0x140 [ 2884.217851] ? getname_flags+0x6e/0x330 [ 2884.218910] ? __check_object_size+0xff/0x256 [ 2884.220323] ? do_raw_spin_unlock+0x75/0x190 [ 2884.221813] ? _raw_spin_unlock+0x12/0x30 [ 2884.222930] do_sys_openat2+0x2b4/0x410 [ 2884.224221] do_sys_open+0x73/0xa0 [ 2884.225247] __x64_sys_openat+0x24/0x30 [ 2884.226340] do_syscall_64+0xc1/0x440 [ 2884.227346] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2884.228942] RIP: 0033:0x7fa749552332 [ 2884.230194] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2884.235504] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2884.237667] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2884.240047] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2884.242031] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2884.244096] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2884.246237] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2888.679488] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 971s! [ 2888.688833] Showing busy workqueues and worker pools: [ 2888.693871] workqueue events: flags=0x0 [ 2888.703000] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2888.709274] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2888.725802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2888.732872] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2888.740561] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.746919] pending: kfree_rcu_monitor [ 2888.756134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2888.763500] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2888.770850] workqueue events_long: flags=0x0 [ 2888.772396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.783356] pending: lu_objects_flush [obdclass] [ 2888.789093] workqueue events_unbound: flags=0x2 [ 2888.793840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2888.799437] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2888.808446] workqueue events_power_efficient: flags=0x80 [ 2888.813780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.818753] pending: check_lifetime [ 2888.821137] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.824380] pending: neigh_periodic_work [ 2888.826064] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.835023] pending: neigh_periodic_work [ 2888.838035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.844035] pending: do_cache_clean [ 2888.848046] workqueue mm_percpu_wq: flags=0x8 [ 2888.850428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.856758] pending: vmstat_update [ 2888.858860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.862954] pending: vmstat_update [ 2888.864901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2888.869807] pending: vmstat_update [ 2888.874904] workqueue writeback: flags=0x4a [ 2888.878973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2888.884853] pending: wb_workfn, wb_workfn [ 2888.888212] workqueue xprtiod: flags=0xa [ 2888.890142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2888.894133] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2888.904282] workqueue ipv6_addrconf: flags=0x40008 [ 2888.915775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2888.922126] pending: addrconf_verify_work [ 2888.928171] workqueue knbd0-recv: flags=0x1a [ 2888.934477] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2888.938568] in-flight: 651:recv_work [ 2888.939936] workqueue ptlrpc_pinger: flags=0x2 [ 2888.949014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2888.953868] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2888.961552] pool 8: cpus=0-3 flags=0x4 nice=0 hung=971s workers=4 idle: 37 31846 28897 [ 2888.968612] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2878s workers=2 idle: 653 [ 2900.057027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2900.058754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2900.059304] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2900.073094] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2900.073109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2900.073116] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2900.073410] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2900.073420] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2900.073424] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2900.073429] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2900.073431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2900.073433] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2900.073435] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2900.073438] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2900.073441] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2900.073443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2900.073445] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2900.073455] Call Trace: [ 2900.073460] [ 2900.073464] ? show_regs.cold.9+0x22/0x2f [ 2900.073474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2900.073480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2900.073485] ? watchdog+0x40/0x40 [ 2900.073490] ? hrtimer_interrupt+0x128/0x2c0 [ 2900.073494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2900.073502] ? apic_timer_interrupt+0xf/0x20 [ 2900.073507] [ 2900.073510] ? native_safe_halt+0x2e/0x50 [ 2900.073514] kvm_wait+0xcd/0xe0 [ 2900.073522] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2900.073532] do_raw_spin_lock+0x177/0x190 [ 2900.073536] _raw_spin_lock+0x21/0x30 [ 2900.073541] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2900.073760] process_one_work+0x2c8/0x700 [ 2900.073766] worker_thread+0x296/0x6e0 [ 2900.073771] ? rescuer_thread+0x570/0x570 [ 2900.073775] kthread+0x1d1/0x200 [ 2900.073779] ? set_kthread_struct+0x70/0x70 [ 2900.073783] ret_from_fork+0x1f/0x30 [ 2900.289471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2900.320272] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2900.331291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2900.340496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2900.343951] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2900.357622] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2900.365659] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2900.369622] RDX: 0000000000000a6c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2900.384448] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2900.393314] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2900.403858] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2900.416919] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2900.429996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2900.444490] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2900.452378] Call Trace: [ 2900.455418] [ 2900.458352] ? show_regs.cold.9+0x22/0x2f [ 2900.465831] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2900.473347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2900.476939] ? watchdog+0x40/0x40 [ 2900.482048] ? hrtimer_interrupt+0x128/0x2c0 [ 2900.484950] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2900.490700] ? apic_timer_interrupt+0xf/0x20 [ 2900.499809] [ 2900.503084] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2900.511621] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2900.515775] do_raw_spin_lock+0x177/0x190 [ 2900.518689] _raw_spin_lock+0x21/0x30 [ 2900.521696] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2900.528376] obd_mod_free+0x46/0x140 [ptlrpc] [ 2900.531088] mdc_commit_open+0x91/0xb0 [mdc] [ 2900.534105] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2900.537047] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2900.540366] after_reply+0xe2c/0x1780 [ptlrpc] [ 2900.543316] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2900.548307] ? __next_timer_interrupt+0x160/0x160 [ 2900.550659] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2900.555726] ? do_wait_intr+0xf0/0xf0 [ 2900.559005] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2900.562625] kthread+0x1d1/0x200 [ 2900.564659] ? set_kthread_struct+0x70/0x70 [ 2900.569003] ret_from_fork+0x1f/0x30 [ 2904.061692] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2904.075250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2904.122430] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2904.140378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2904.157279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2904.159156] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2904.192762] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2904.210219] RAX: 0000000000006994 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2904.222246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2904.231013] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2904.243943] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2904.259183] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2904.273517] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2904.288052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2904.303183] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2904.323758] Call Trace: [ 2904.324595] [ 2904.325336] ? show_regs.cold.9+0x22/0x2f [ 2904.326740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2904.336294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2904.337792] ? watchdog+0x40/0x40 [ 2904.349030] ? hrtimer_interrupt+0x128/0x2c0 [ 2904.362330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2904.363952] ? apic_timer_interrupt+0xf/0x20 [ 2904.365360] [ 2904.366045] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2904.375144] do_raw_spin_lock+0x177/0x190 [ 2904.384946] _raw_spin_lock+0x21/0x30 [ 2904.386168] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2904.389643] ? do_raw_spin_unlock+0x75/0x190 [ 2904.391155] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2904.395025] ? do_wait_intr+0xf0/0xf0 [ 2904.398635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2904.405175] mdc_close+0x53b/0xd70 [mdc] [ 2904.408274] lmv_close+0x18d/0x490 [lmv] [ 2904.410282] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2904.413544] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2904.419204] ll_md_close+0x274/0x800 [lustre] [ 2904.421261] ll_file_release+0x128/0xc50 [lustre] [ 2904.427240] __fput+0x10d/0x440 [ 2904.429387] ____fput+0x12/0x20 [ 2904.431521] task_work_run+0xc6/0x110 [ 2904.432803] exit_to_usermode_loop+0x1dd/0x1f0 [ 2904.438397] do_syscall_64+0x3d6/0x440 [ 2904.440245] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2904.443219] RIP: 0033:0x7fa749552cd8 [ 2904.445596] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2904.457580] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2904.461843] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2904.465318] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2904.469945] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2904.473909] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2904.478214] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2916.055759] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2916.062860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2916.112145] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2916.122167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2916.125462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2916.128563] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2916.139659] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2916.143403] RAX: 0000000000006c07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2916.146778] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2916.149537] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2916.155890] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2916.164037] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2916.171047] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2916.176795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2916.181858] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2916.186870] Call Trace: [ 2916.188720] [ 2916.190470] ? show_regs.cold.9+0x22/0x2f [ 2916.194626] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2916.199080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2916.203715] ? watchdog+0x40/0x40 [ 2916.206912] ? hrtimer_interrupt+0x128/0x2c0 [ 2916.211798] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2916.215922] ? apic_timer_interrupt+0xf/0x20 [ 2916.223157] [ 2916.226605] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2916.235744] do_raw_spin_lock+0x177/0x190 [ 2916.240078] _raw_spin_lock+0x21/0x30 [ 2916.244784] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2916.249034] ? do_raw_spin_unlock+0x75/0x190 [ 2916.254286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2916.264593] ? do_wait_intr+0xf0/0xf0 [ 2916.267473] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2916.274255] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2916.285560] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2916.295928] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2916.298977] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2916.304923] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2916.306602] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2916.313392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2916.321510] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2916.329018] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2916.330913] lmv_intent_open+0x338/0x1150 [lmv] [ 2916.336874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2916.340987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2916.348414] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2916.353476] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2916.355947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2916.358563] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2916.367601] ? do_raw_spin_unlock+0x75/0x190 [ 2916.373041] ? _raw_spin_unlock+0x12/0x30 [ 2916.376517] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2916.382656] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2916.384828] ll_atomic_open+0x36a/0xd70 [lustre] [ 2916.391193] lookup_open+0x356/0xb40 [ 2916.394507] do_last+0x313/0xfc0 [ 2916.397919] ? nd_jump_root+0xe5/0x160 [ 2916.399100] ? path_init+0x437/0x520 [ 2916.400331] path_openat+0xf7/0x500 [ 2916.405878] ? do_raw_spin_unlock+0x75/0x190 [ 2916.408953] do_filp_open+0x99/0x140 [ 2916.413008] ? getname_flags+0x6e/0x330 [ 2916.417013] ? __check_object_size+0xff/0x256 [ 2916.420816] ? do_raw_spin_unlock+0x75/0x190 [ 2916.426291] ? _raw_spin_unlock+0x12/0x30 [ 2916.428801] do_sys_openat2+0x2b4/0x410 [ 2916.432426] do_sys_open+0x73/0xa0 [ 2916.436129] __x64_sys_openat+0x24/0x30 [ 2916.440215] do_syscall_64+0xc1/0x440 [ 2916.443039] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2916.448007] RIP: 0033:0x7fa749552332 [ 2916.451936] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2916.465997] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2916.474378] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2916.480014] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2916.485386] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2916.491274] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2916.497236] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2919.393062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1002s! [ 2919.396304] Showing busy workqueues and worker pools: [ 2919.398384] workqueue events: flags=0x0 [ 2919.401354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2919.405672] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2919.412415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2919.416908] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2919.424822] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.428842] pending: kfree_rcu_monitor [ 2919.433338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2919.436469] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2919.440941] workqueue events_long: flags=0x0 [ 2919.443990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.452307] pending: lu_objects_flush [obdclass] [ 2919.455456] workqueue events_unbound: flags=0x2 [ 2919.459907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2919.464685] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2919.474048] workqueue events_power_efficient: flags=0x80 [ 2919.476346] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.484275] pending: check_lifetime [ 2919.485695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.496845] pending: neigh_periodic_work [ 2919.499593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.506405] pending: neigh_periodic_work [ 2919.512384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.519452] pending: do_cache_clean [ 2919.523357] workqueue mm_percpu_wq: flags=0x8 [ 2919.526428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.531726] pending: vmstat_update [ 2919.534493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.543629] pending: vmstat_update [ 2919.549202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2919.560204] pending: vmstat_update [ 2919.564921] workqueue writeback: flags=0x4a [ 2919.571743] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2919.579440] pending: wb_workfn, wb_workfn [ 2919.586172] workqueue xprtiod: flags=0xa [ 2919.592914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2919.599623] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2919.614373] workqueue ipv6_addrconf: flags=0x40008 [ 2919.624309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2919.637465] pending: addrconf_verify_work [ 2919.649673] workqueue knbd0-recv: flags=0x1a [ 2919.651197] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2919.686816] in-flight: 651:recv_work [ 2919.701112] workqueue ptlrpc_pinger: flags=0x2 [ 2919.707195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2919.715589] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2919.736886] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1002s workers=4 idle: 37 31846 28897 [ 2919.757879] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2909s workers=2 idle: 653 [ 2928.057102] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 2928.060163] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2928.069823] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 2928.078522] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2928.078535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2928.078541] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2928.078916] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2928.078926] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2928.078931] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2928.078935] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2928.078938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2928.078940] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2928.078942] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2928.078944] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2928.078947] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2928.078950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2928.078952] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2928.078961] Call Trace: [ 2928.078966] [ 2928.078971] ? show_regs.cold.9+0x22/0x2f [ 2928.078980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2928.078987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2928.078991] ? watchdog+0x40/0x40 [ 2928.078995] ? hrtimer_interrupt+0x128/0x2c0 [ 2928.078999] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2928.079006] ? apic_timer_interrupt+0xf/0x20 [ 2928.079010] [ 2928.079013] ? native_safe_halt+0x2e/0x50 [ 2928.079017] kvm_wait+0xcd/0xe0 [ 2928.079026] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2928.079034] do_raw_spin_lock+0x177/0x190 [ 2928.079039] _raw_spin_lock+0x21/0x30 [ 2928.079043] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2928.079261] process_one_work+0x2c8/0x700 [ 2928.079269] worker_thread+0x296/0x6e0 [ 2928.079274] ? rescuer_thread+0x570/0x570 [ 2928.079277] kthread+0x1d1/0x200 [ 2928.079282] ? set_kthread_struct+0x70/0x70 [ 2928.079286] ret_from_fork+0x1f/0x30 [ 2928.186805] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2928.205842] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2928.210440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2928.214172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2928.216576] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2928.224509] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2928.228062] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2928.230631] RDX: 0000000000006351 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2928.233625] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2928.236503] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2928.239610] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2928.242538] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2928.246133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2928.248529] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2928.251920] Call Trace: [ 2928.253044] [ 2928.253928] ? show_regs.cold.9+0x22/0x2f [ 2928.255491] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2928.257459] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2928.259458] ? watchdog+0x40/0x40 [ 2928.274446] ? hrtimer_interrupt+0x128/0x2c0 [ 2928.276718] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2928.279269] ? apic_timer_interrupt+0xf/0x20 [ 2928.281685] [ 2928.282811] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2928.285468] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2928.288142] do_raw_spin_lock+0x177/0x190 [ 2928.290004] _raw_spin_lock+0x21/0x30 [ 2928.291662] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2928.294810] obd_mod_free+0x46/0x140 [ptlrpc] [ 2928.297422] mdc_commit_open+0x91/0xb0 [mdc] [ 2928.299718] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2928.320682] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2928.323541] after_reply+0xe2c/0x1780 [ptlrpc] [ 2928.327595] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2928.333183] ? __next_timer_interrupt+0x160/0x160 [ 2928.337905] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2928.342851] ? do_wait_intr+0xf0/0xf0 [ 2928.346959] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2928.348452] kthread+0x1d1/0x200 [ 2928.350606] ? set_kthread_struct+0x70/0x70 [ 2928.352675] ret_from_fork+0x1f/0x30 [ 2932.059918] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2932.063243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2932.093012] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2932.100723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2932.106074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2932.119284] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2932.132012] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2932.136157] RAX: 0000000000000833 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2932.140216] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2932.149571] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2932.153511] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2932.158680] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2932.165742] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2932.174309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2932.178219] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2932.185933] Call Trace: [ 2932.187027] [ 2932.190905] ? show_regs.cold.9+0x22/0x2f [ 2932.194011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2932.198909] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2932.203666] ? watchdog+0x40/0x40 [ 2932.205911] ? hrtimer_interrupt+0x128/0x2c0 [ 2932.208700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2932.215831] ? apic_timer_interrupt+0xf/0x20 [ 2932.219651] [ 2932.222198] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2932.225248] do_raw_spin_lock+0x177/0x190 [ 2932.231882] _raw_spin_lock+0x21/0x30 [ 2932.234240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2932.240536] ? do_raw_spin_unlock+0x75/0x190 [ 2932.242768] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2932.251213] ? do_wait_intr+0xf0/0xf0 [ 2932.255499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2932.257986] mdc_close+0x53b/0xd70 [mdc] [ 2932.265992] lmv_close+0x18d/0x490 [lmv] [ 2932.272406] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2932.285551] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2932.294407] ll_md_close+0x274/0x800 [lustre] [ 2932.304685] ll_file_release+0x128/0xc50 [lustre] [ 2932.309174] __fput+0x10d/0x440 [ 2932.313351] ____fput+0x12/0x20 [ 2932.317077] task_work_run+0xc6/0x110 [ 2932.321823] exit_to_usermode_loop+0x1dd/0x1f0 [ 2932.327148] do_syscall_64+0x3d6/0x440 [ 2932.329304] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2932.332914] RIP: 0033:0x7fa749552cd8 [ 2932.334379] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2932.344492] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2932.348325] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2932.351216] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2932.354718] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2932.358106] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2932.361185] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2944.055703] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2944.065631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2944.114696] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2944.125693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2944.134643] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2944.137201] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2944.156031] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2944.165938] RAX: 0000000000002614 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2944.171825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2944.176692] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2944.184557] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2944.191658] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2944.199262] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2944.207757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2944.213539] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2944.221449] Call Trace: [ 2944.222379] [ 2944.223076] ? show_regs.cold.9+0x22/0x2f [ 2944.228578] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2944.238045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2944.239567] ? watchdog+0x40/0x40 [ 2944.240544] ? hrtimer_interrupt+0x128/0x2c0 [ 2944.245079] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2944.251348] ? apic_timer_interrupt+0xf/0x20 [ 2944.256316] [ 2944.258784] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2944.264649] do_raw_spin_lock+0x177/0x190 [ 2944.269603] _raw_spin_lock+0x21/0x30 [ 2944.274024] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2944.278730] ? do_raw_spin_unlock+0x75/0x190 [ 2944.284344] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2944.293983] ? do_wait_intr+0xf0/0xf0 [ 2944.298793] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2944.305898] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2944.311335] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2944.318608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2944.323908] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2944.327591] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2944.330074] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2944.334026] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2944.336794] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2944.342440] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2944.349822] lmv_intent_open+0x338/0x1150 [lmv] [ 2944.352343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2944.354741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2944.358199] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2944.359814] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2944.361461] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2944.364335] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2944.365571] ? do_raw_spin_unlock+0x75/0x190 [ 2944.367352] ? _raw_spin_unlock+0x12/0x30 [ 2944.368836] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2944.370717] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2944.375664] ll_atomic_open+0x36a/0xd70 [lustre] [ 2944.377787] lookup_open+0x356/0xb40 [ 2944.379750] do_last+0x313/0xfc0 [ 2944.381359] ? nd_jump_root+0xe5/0x160 [ 2944.382788] ? path_init+0x437/0x520 [ 2944.384686] path_openat+0xf7/0x500 [ 2944.385966] ? do_raw_spin_unlock+0x75/0x190 [ 2944.387803] do_filp_open+0x99/0x140 [ 2944.389004] ? getname_flags+0x6e/0x330 [ 2944.390306] ? __check_object_size+0xff/0x256 [ 2944.394035] ? do_raw_spin_unlock+0x75/0x190 [ 2944.396400] ? _raw_spin_unlock+0x12/0x30 [ 2944.398648] do_sys_openat2+0x2b4/0x410 [ 2944.404193] do_sys_open+0x73/0xa0 [ 2944.406759] __x64_sys_openat+0x24/0x30 [ 2944.411087] do_syscall_64+0xc1/0x440 [ 2944.415968] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2944.419026] RIP: 0033:0x7fa749552332 [ 2944.422516] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2944.437792] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2944.445785] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2944.449919] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2944.455950] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2944.461391] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2944.465547] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2950.113088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1032s! [ 2950.122418] Showing busy workqueues and worker pools: [ 2950.129165] workqueue events: flags=0x0 [ 2950.136949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2950.141384] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2950.162212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2950.176989] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2950.195450] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.197989] pending: kfree_rcu_monitor [ 2950.209044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2950.218847] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2950.227795] workqueue events_long: flags=0x0 [ 2950.235065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.249700] pending: lu_objects_flush [obdclass] [ 2950.257507] workqueue events_unbound: flags=0x2 [ 2950.276708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2950.280754] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2950.288200] workqueue events_power_efficient: flags=0x80 [ 2950.293707] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.302746] pending: check_lifetime [ 2950.304120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.312558] pending: neigh_periodic_work [ 2950.315204] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.320213] pending: neigh_periodic_work [ 2950.322807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.325205] pending: do_cache_clean [ 2950.332659] workqueue mm_percpu_wq: flags=0x8 [ 2950.336649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.340603] pending: vmstat_update [ 2950.342649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.350912] pending: vmstat_update [ 2950.353223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2950.357258] pending: vmstat_update [ 2950.359189] workqueue writeback: flags=0x4a [ 2950.361879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2950.365288] pending: wb_workfn, wb_workfn [ 2950.369362] workqueue xprtiod: flags=0xa [ 2950.372965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2950.381782] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2950.388749] workqueue ipv6_addrconf: flags=0x40008 [ 2950.394073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2950.400751] pending: addrconf_verify_work [ 2950.403131] workqueue knbd0-recv: flags=0x1a [ 2950.408212] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2950.412962] in-flight: 651:recv_work [ 2950.415330] workqueue ptlrpc_pinger: flags=0x2 [ 2950.417656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2950.421418] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2950.425550] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1033s workers=4 idle: 37 31846 28897 [ 2950.436672] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2940s workers=2 idle: 653 [ 2956.058543] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2956.058932] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2956.060977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2956.061221] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2956.061228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2956.061231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2956.061244] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2956.061248] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2956.061252] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2956.061255] RDX: 00000000000046ff RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2956.061257] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2956.061259] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2956.061261] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2956.061264] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2956.061266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2956.061268] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2956.061278] Call Trace: [ 2956.061283] [ 2956.061288] ? show_regs.cold.9+0x22/0x2f [ 2956.061297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2956.061304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2956.061308] ? watchdog+0x40/0x40 [ 2956.061312] ? hrtimer_interrupt+0x128/0x2c0 [ 2956.061316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2956.061324] ? apic_timer_interrupt+0xf/0x20 [ 2956.061328] [ 2956.061331] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2956.061336] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2956.061340] do_raw_spin_lock+0x177/0x190 [ 2956.061345] _raw_spin_lock+0x21/0x30 [ 2956.061351] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2956.061631] obd_mod_free+0x46/0x140 [ptlrpc] [ 2956.061827] mdc_commit_open+0x91/0xb0 [mdc] [ 2956.061863] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2956.065713] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2956.065980] after_reply+0xe2c/0x1780 [ptlrpc] [ 2956.066189] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2956.066384] ? __next_timer_interrupt+0x160/0x160 [ 2956.066394] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2956.066598] ? do_wait_intr+0xf0/0xf0 [ 2956.066606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2956.066797] kthread+0x1d1/0x200 [ 2956.066802] ? set_kthread_struct+0x70/0x70 [ 2956.066807] ret_from_fork+0x1f/0x30 [ 2956.249844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2956.282544] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2956.290704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2956.297462] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2956.301785] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2956.304423] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2956.318804] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2956.324052] RAX: 0000000000004485 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2956.328475] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2956.335605] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2956.340128] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2956.344030] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2956.347661] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2956.352052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2956.355618] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2956.360121] Call Trace: [ 2956.361671] [ 2956.364528] ? show_regs.cold.9+0x22/0x2f [ 2956.367972] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2956.370029] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2956.373930] ? watchdog+0x40/0x40 [ 2956.376076] ? hrtimer_interrupt+0x128/0x2c0 [ 2956.379445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2956.384646] ? apic_timer_interrupt+0xf/0x20 [ 2956.386740] [ 2956.388700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2956.393227] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2956.396751] do_raw_spin_lock+0x177/0x190 [ 2956.400076] _raw_spin_lock+0x21/0x30 [ 2956.401932] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2956.404958] process_one_work+0x2c8/0x700 [ 2956.406805] worker_thread+0x296/0x6e0 [ 2956.408808] ? rescuer_thread+0x570/0x570 [ 2956.409884] kthread+0x1d1/0x200 [ 2956.410993] ? set_kthread_struct+0x70/0x70 [ 2956.413485] ret_from_fork+0x1f/0x30 [ 2960.059359] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2960.062525] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2960.086528] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2960.096350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2960.102352] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2960.104988] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2960.111317] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2960.113104] RAX: 0000000000007f89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2960.115123] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2960.117661] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2960.120653] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2960.123641] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2960.126963] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2960.130460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2960.132453] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2960.134454] Call Trace: [ 2960.135060] [ 2960.135883] ? show_regs.cold.9+0x22/0x2f [ 2960.137552] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2960.139566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2960.141597] ? watchdog+0x40/0x40 [ 2960.142930] ? hrtimer_interrupt+0x128/0x2c0 [ 2960.145091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2960.147518] ? apic_timer_interrupt+0xf/0x20 [ 2960.150760] [ 2960.155549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2960.160570] do_raw_spin_lock+0x177/0x190 [ 2960.165733] _raw_spin_lock+0x21/0x30 [ 2960.166745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2960.170734] ? do_raw_spin_unlock+0x75/0x190 [ 2960.174288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2960.177045] ? do_wait_intr+0xf0/0xf0 [ 2960.179283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2960.182078] mdc_close+0x53b/0xd70 [mdc] [ 2960.184425] lmv_close+0x18d/0x490 [lmv] [ 2960.186501] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2960.193701] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2960.199543] ll_md_close+0x274/0x800 [lustre] [ 2960.201792] ll_file_release+0x128/0xc50 [lustre] [ 2960.205116] __fput+0x10d/0x440 [ 2960.207420] ____fput+0x12/0x20 [ 2960.209417] task_work_run+0xc6/0x110 [ 2960.210958] exit_to_usermode_loop+0x1dd/0x1f0 [ 2960.212725] do_syscall_64+0x3d6/0x440 [ 2960.215542] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2960.218048] RIP: 0033:0x7fa749552cd8 [ 2960.223524] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2960.241081] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2960.249510] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2960.255448] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2960.261452] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2960.269643] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2960.277029] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2976.059658] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 2976.066301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2976.098859] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2976.104147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2976.115049] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2976.118137] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2976.133809] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2976.138060] RAX: 0000000000003b85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2976.142734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 2976.146857] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 2976.152999] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 2976.157157] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 2976.159923] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 2976.164463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2976.174652] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 2976.179746] Call Trace: [ 2976.180488] [ 2976.183787] ? show_regs.cold.9+0x22/0x2f [ 2976.186287] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2976.190010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2976.191494] ? watchdog+0x40/0x40 [ 2976.195388] ? hrtimer_interrupt+0x128/0x2c0 [ 2976.210027] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2976.212080] ? apic_timer_interrupt+0xf/0x20 [ 2976.213669] [ 2976.214276] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2976.215779] do_raw_spin_lock+0x177/0x190 [ 2976.224167] _raw_spin_lock+0x21/0x30 [ 2976.225361] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2976.231898] ? do_raw_spin_unlock+0x75/0x190 [ 2976.236072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2976.241361] ? do_wait_intr+0xf0/0xf0 [ 2976.248774] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2976.251328] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2976.255929] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2976.260333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2976.264016] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2976.266084] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2976.276946] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2976.279275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2976.282365] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2976.285394] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2976.287338] lmv_intent_open+0x338/0x1150 [lmv] [ 2976.289079] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2976.291150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2976.293257] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2976.295880] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2976.300334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2976.307103] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2976.308546] ? do_raw_spin_unlock+0x75/0x190 [ 2976.310923] ? _raw_spin_unlock+0x12/0x30 [ 2976.312252] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2976.315041] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2976.319154] ll_atomic_open+0x36a/0xd70 [lustre] [ 2976.321128] lookup_open+0x356/0xb40 [ 2976.322258] do_last+0x313/0xfc0 [ 2976.325464] ? nd_jump_root+0xe5/0x160 [ 2976.327064] ? path_init+0x437/0x520 [ 2976.328719] path_openat+0xf7/0x500 [ 2976.330273] ? do_raw_spin_unlock+0x75/0x190 [ 2976.331866] do_filp_open+0x99/0x140 [ 2976.333069] ? getname_flags+0x6e/0x330 [ 2976.334412] ? __check_object_size+0xff/0x256 [ 2976.339230] ? do_raw_spin_unlock+0x75/0x190 [ 2976.343927] ? _raw_spin_unlock+0x12/0x30 [ 2976.345148] do_sys_openat2+0x2b4/0x410 [ 2976.346319] do_sys_open+0x73/0xa0 [ 2976.348170] __x64_sys_openat+0x24/0x30 [ 2976.349825] do_syscall_64+0xc1/0x440 [ 2976.351099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2976.353354] RIP: 0033:0x7fa749552332 [ 2976.354820] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2976.371155] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2976.388977] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 2976.396474] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 2976.399325] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 2976.402397] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 2976.412302] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 2980.832712] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1063s! [ 2980.842341] Showing busy workqueues and worker pools: [ 2980.846692] workqueue events: flags=0x0 [ 2980.852359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2980.860719] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 2980.873792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2980.876235] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2980.891398] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2980.921436] pending: kfree_rcu_monitor [ 2980.926090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2980.933791] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 2980.938519] workqueue events_long: flags=0x0 [ 2980.944986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2980.953353] pending: lu_objects_flush [obdclass] [ 2980.957404] workqueue events_unbound: flags=0x2 [ 2980.961604] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2980.965200] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2980.970263] workqueue events_power_efficient: flags=0x80 [ 2980.979527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2980.990757] pending: check_lifetime [ 2980.992423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2981.007149] pending: neigh_periodic_work [ 2981.012293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2981.017054] pending: neigh_periodic_work [ 2981.021749] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2981.030064] pending: do_cache_clean [ 2981.032924] workqueue mm_percpu_wq: flags=0x8 [ 2981.041344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2981.051791] pending: vmstat_update [ 2981.055234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2981.061414] pending: vmstat_update [ 2981.064671] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2981.072917] pending: vmstat_update [ 2981.074352] workqueue writeback: flags=0x4a [ 2981.078854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2981.085095] pending: wb_workfn, wb_workfn [ 2981.090179] workqueue xprtiod: flags=0xa [ 2981.094033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2981.099922] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2981.105125] workqueue ipv6_addrconf: flags=0x40008 [ 2981.107693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2981.110243] pending: addrconf_verify_work [ 2981.112135] workqueue knbd0-recv: flags=0x1a [ 2981.113544] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2981.116144] in-flight: 651:recv_work [ 2981.117853] workqueue ptlrpc_pinger: flags=0x2 [ 2981.119850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2981.122050] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2981.124351] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1063s workers=4 idle: 37 31846 28897 [ 2981.127637] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2971s workers=2 idle: 653 [ 2984.057239] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 2984.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 2984.067367] Modules linked in: [ 2984.070527] Modules linked in: [ 2984.070529] lustre(O) osp(O) [ 2984.071932] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2984.072002] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2984.072009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2984.072012] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2984.072025] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2984.072029] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2984.072035] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 2984.072038] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 2984.072040] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 2984.072041] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 2984.072044] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2984.072047] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 2984.072049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2984.072051] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 2984.072060] Call Trace: [ 2984.072065] [ 2984.072069] ? show_regs.cold.9+0x22/0x2f [ 2984.072079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2984.072085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2984.072089] ? watchdog+0x40/0x40 [ 2984.072093] ? hrtimer_interrupt+0x128/0x2c0 [ 2984.072098] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2984.072105] ? apic_timer_interrupt+0xf/0x20 [ 2984.072109] [ 2984.072112] ? native_safe_halt+0x2e/0x50 [ 2984.072116] kvm_wait+0xcd/0xe0 [ 2984.072124] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2984.072131] do_raw_spin_lock+0x177/0x190 [ 2984.072135] _raw_spin_lock+0x21/0x30 [ 2984.072140] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2984.072438] obd_mod_free+0x46/0x140 [ptlrpc] [ 2984.072640] mdc_commit_open+0x91/0xb0 [mdc] [ 2984.072684] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2984.074878] ofd(O) [ 2984.076032] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2984.132994] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2984.133074] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2984.133081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2984.133088] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2984.137753] after_reply+0xe2c/0x1780 [ptlrpc] [ 2984.138071] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2984.138301] ? __next_timer_interrupt+0x160/0x160 [ 2984.138313] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2984.138538] ? do_wait_intr+0xf0/0xf0 [ 2984.138548] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2984.138777] kthread+0x1d1/0x200 [ 2984.138782] ? set_kthread_struct+0x70/0x70 [ 2984.138786] ret_from_fork+0x1f/0x30 [ 2984.329371] [ 2984.330035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2984.332483] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2984.355651] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2984.360096] RAX: 000000000000720e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2984.365522] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 2984.373339] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2984.381860] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 2984.384816] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 2984.390688] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 2984.397299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2984.401347] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 2984.408539] Call Trace: [ 2984.410944] [ 2984.413869] ? show_regs.cold.9+0x22/0x2f [ 2984.417975] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2984.419884] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2984.425638] ? watchdog+0x40/0x40 [ 2984.430801] ? hrtimer_interrupt+0x128/0x2c0 [ 2984.435263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2984.440226] ? apic_timer_interrupt+0xf/0x20 [ 2984.442716] [ 2984.443665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2984.446911] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2984.449943] do_raw_spin_lock+0x177/0x190 [ 2984.452656] _raw_spin_lock+0x21/0x30 [ 2984.458258] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2984.462504] process_one_work+0x2c8/0x700 [ 2984.464145] worker_thread+0x296/0x6e0 [ 2984.465879] ? rescuer_thread+0x570/0x570 [ 2984.468346] kthread+0x1d1/0x200 [ 2984.469839] ? set_kthread_struct+0x70/0x70 [ 2984.473178] ret_from_fork+0x1f/0x30 [ 2988.069962] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 2988.080023] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2988.100794] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2988.106173] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2988.110740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2988.113109] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2988.121688] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2988.129146] RAX: 000000000000701a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 2988.138645] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 2988.145641] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2988.151979] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 2988.158632] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 2988.165527] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 2988.173077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2988.179880] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 2988.187130] Call Trace: [ 2988.188492] [ 2988.189764] ? show_regs.cold.9+0x22/0x2f [ 2988.194628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2988.197261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2988.202650] ? watchdog+0x40/0x40 [ 2988.205293] ? hrtimer_interrupt+0x128/0x2c0 [ 2988.210200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2988.216076] ? apic_timer_interrupt+0xf/0x20 [ 2988.218745] [ 2988.219734] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2988.223256] do_raw_spin_lock+0x177/0x190 [ 2988.229128] _raw_spin_lock+0x21/0x30 [ 2988.231074] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2988.237207] ? do_raw_spin_unlock+0x75/0x190 [ 2988.240045] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2988.248156] ? do_wait_intr+0xf0/0xf0 [ 2988.254015] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2988.256706] mdc_close+0x53b/0xd70 [mdc] [ 2988.261254] lmv_close+0x18d/0x490 [lmv] [ 2988.263150] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2988.269587] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2988.274100] ll_md_close+0x274/0x800 [lustre] [ 2988.278572] ll_file_release+0x128/0xc50 [lustre] [ 2988.282416] __fput+0x10d/0x440 [ 2988.286718] ____fput+0x12/0x20 [ 2988.288155] task_work_run+0xc6/0x110 [ 2988.292892] exit_to_usermode_loop+0x1dd/0x1f0 [ 2988.294960] do_syscall_64+0x3d6/0x440 [ 2988.299190] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2988.302413] RIP: 0033:0x7fa749552cd8 [ 2988.307453] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2988.323904] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2988.327162] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 2988.332723] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 2988.336015] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 2988.339158] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 2988.344278] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3008.054288] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3008.069922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3008.159947] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3008.179825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3008.192314] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3008.197005] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3008.238719] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3008.248724] RAX: 000000000000208c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3008.254069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3008.261049] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3008.263514] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3008.267113] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3008.273465] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3008.281951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3008.283869] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3008.287756] Call Trace: [ 3008.292658] [ 3008.293671] ? show_regs.cold.9+0x22/0x2f [ 3008.295507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3008.301580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3008.307608] ? watchdog+0x40/0x40 [ 3008.317546] ? hrtimer_interrupt+0x128/0x2c0 [ 3008.319842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3008.323945] ? apic_timer_interrupt+0xf/0x20 [ 3008.329764] [ 3008.333855] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3008.344448] do_raw_spin_lock+0x177/0x190 [ 3008.349185] _raw_spin_lock+0x21/0x30 [ 3008.357474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3008.364440] ? do_raw_spin_unlock+0x75/0x190 [ 3008.365870] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3008.371271] ? do_wait_intr+0xf0/0xf0 [ 3008.379938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3008.382427] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3008.386286] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3008.396515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3008.407048] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3008.413975] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3008.421792] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3008.424044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3008.431028] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3008.438280] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3008.441958] lmv_intent_open+0x338/0x1150 [lmv] [ 3008.444093] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3008.449210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3008.455546] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3008.457264] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3008.460396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3008.464300] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3008.466636] ? do_raw_spin_unlock+0x75/0x190 [ 3008.473637] ? _raw_spin_unlock+0x12/0x30 [ 3008.479151] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3008.483098] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3008.485868] ll_atomic_open+0x36a/0xd70 [lustre] [ 3008.488561] lookup_open+0x356/0xb40 [ 3008.490449] do_last+0x313/0xfc0 [ 3008.495842] ? nd_jump_root+0xe5/0x160 [ 3008.502138] ? path_init+0x437/0x520 [ 3008.503308] path_openat+0xf7/0x500 [ 3008.509445] ? do_raw_spin_unlock+0x75/0x190 [ 3008.512473] do_filp_open+0x99/0x140 [ 3008.515142] ? getname_flags+0x6e/0x330 [ 3008.517213] ? __check_object_size+0xff/0x256 [ 3008.519897] ? do_raw_spin_unlock+0x75/0x190 [ 3008.522045] ? _raw_spin_unlock+0x12/0x30 [ 3008.525274] do_sys_openat2+0x2b4/0x410 [ 3008.529767] do_sys_open+0x73/0xa0 [ 3008.531640] __x64_sys_openat+0x24/0x30 [ 3008.541468] do_syscall_64+0xc1/0x440 [ 3008.544199] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3008.547576] RIP: 0033:0x7fa749552332 [ 3008.550414] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3008.567191] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3008.571460] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3008.575302] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3008.583993] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3008.590585] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3008.594724] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3011.554917] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1094s! [ 3011.557487] Showing busy workqueues and worker pools: [ 3011.559764] workqueue events: flags=0x0 [ 3011.572076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3011.584806] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3011.602189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3011.617104] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3011.629919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.636037] pending: kfree_rcu_monitor [ 3011.638321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3011.645154] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3011.652046] workqueue events_long: flags=0x0 [ 3011.657112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.662273] pending: lu_objects_flush [obdclass] [ 3011.668993] workqueue events_unbound: flags=0x2 [ 3011.676313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3011.686099] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3011.697645] workqueue events_power_efficient: flags=0x80 [ 3011.703385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.713226] pending: check_lifetime [ 3011.718318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.726258] pending: neigh_periodic_work [ 3011.730956] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.739802] pending: neigh_periodic_work [ 3011.746636] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.757621] pending: do_cache_clean [ 3011.760711] workqueue mm_percpu_wq: flags=0x8 [ 3011.765344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.775854] pending: vmstat_update [ 3011.777058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.783171] pending: vmstat_update [ 3011.784545] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3011.791752] pending: vmstat_update [ 3011.797107] workqueue writeback: flags=0x4a [ 3011.802951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3011.806326] pending: wb_workfn, wb_workfn [ 3011.812765] workqueue xprtiod: flags=0xa [ 3011.814762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3011.820435] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3011.829353] workqueue ipv6_addrconf: flags=0x40008 [ 3011.830703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3011.837980] pending: addrconf_verify_work [ 3011.842677] workqueue knbd0-recv: flags=0x1a [ 3011.844544] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3011.848041] in-flight: 651:recv_work [ 3011.852071] workqueue ptlrpc_pinger: flags=0x2 [ 3011.853967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3011.863213] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3011.871914] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1094s workers=4 idle: 37 31846 28897 [ 3011.880855] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3001s workers=2 idle: 653 [ 3012.060381] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3012.062528] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3012.068409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3012.068633] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3012.068641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3012.068644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3012.068658] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3012.068662] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3012.068666] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3012.068668] RDX: 00000000000043d4 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3012.068670] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3012.068673] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3012.068675] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3012.068679] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3012.068681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3012.068684] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3012.068692] Call Trace: [ 3012.068697] [ 3012.068703] ? show_regs.cold.9+0x22/0x2f [ 3012.068712] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3012.068718] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3012.068722] ? watchdog+0x40/0x40 [ 3012.068726] ? hrtimer_interrupt+0x128/0x2c0 [ 3012.068797] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3012.068837] ? apic_timer_interrupt+0xf/0x20 [ 3012.068841] [ 3012.068843] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3012.068848] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3012.068852] do_raw_spin_lock+0x177/0x190 [ 3012.068856] _raw_spin_lock+0x21/0x30 [ 3012.068862] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3012.069550] obd_mod_free+0x46/0x140 [ptlrpc] [ 3012.069824] mdc_commit_open+0x91/0xb0 [mdc] [ 3012.069858] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3012.070154] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3012.070440] after_reply+0xe2c/0x1780 [ptlrpc] [ 3012.070670] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3012.071146] ? __next_timer_interrupt+0x160/0x160 [ 3012.071161] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3012.071411] ? do_wait_intr+0xf0/0xf0 [ 3012.071420] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3012.071994] kthread+0x1d1/0x200 [ 3012.072001] ? set_kthread_struct+0x70/0x70 [ 3012.072009] ret_from_fork+0x1f/0x30 [ 3012.496515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3012.555686] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3012.569732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3012.579962] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3012.585173] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3012.587027] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3012.616852] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3012.621612] RAX: 000000000000468b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3012.630646] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3012.639595] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3012.649535] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3012.662243] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3012.665115] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3012.672972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3012.679777] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3012.695079] Call Trace: [ 3012.695996] [ 3012.701577] ? show_regs.cold.9+0x22/0x2f [ 3012.709697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3012.718047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3012.721552] ? watchdog+0x40/0x40 [ 3012.725464] ? hrtimer_interrupt+0x128/0x2c0 [ 3012.729646] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3012.736311] ? apic_timer_interrupt+0xf/0x20 [ 3012.744357] [ 3012.745856] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3012.754942] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3012.759726] do_raw_spin_lock+0x177/0x190 [ 3012.761126] _raw_spin_lock+0x21/0x30 [ 3012.774096] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3012.782006] process_one_work+0x2c8/0x700 [ 3012.787509] worker_thread+0x296/0x6e0 [ 3012.792190] ? rescuer_thread+0x570/0x570 [ 3012.800512] kthread+0x1d1/0x200 [ 3012.806750] ? set_kthread_struct+0x70/0x70 [ 3012.816723] ret_from_fork+0x1f/0x30 [ 3016.059022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3016.063907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3016.098497] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3016.102747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3016.113778] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3016.116135] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3016.126211] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3016.135093] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3016.139510] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3016.141726] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3016.149551] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3016.152936] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3016.156298] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3016.168298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3016.177000] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3016.183438] Call Trace: [ 3016.185576] [ 3016.186648] ? show_regs.cold.9+0x22/0x2f [ 3016.188723] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3016.193032] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3016.194872] ? watchdog+0x40/0x40 [ 3016.195996] ? hrtimer_interrupt+0x128/0x2c0 [ 3016.197795] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3016.200675] ? apic_timer_interrupt+0xf/0x20 [ 3016.202596] [ 3016.204556] ? native_safe_halt+0x2e/0x50 [ 3016.206575] kvm_wait+0xcd/0xe0 [ 3016.208425] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3016.211384] do_raw_spin_lock+0x177/0x190 [ 3016.213385] _raw_spin_lock+0x21/0x30 [ 3016.214900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3016.220862] ? do_raw_spin_unlock+0x75/0x190 [ 3016.224338] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3016.226691] ? do_wait_intr+0xf0/0xf0 [ 3016.228627] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3016.231666] mdc_close+0x53b/0xd70 [mdc] [ 3016.233707] lmv_close+0x18d/0x490 [lmv] [ 3016.236438] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3016.241665] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3016.253576] ll_md_close+0x274/0x800 [lustre] [ 3016.259182] ll_file_release+0x128/0xc50 [lustre] [ 3016.269347] __fput+0x10d/0x440 [ 3016.271494] ____fput+0x12/0x20 [ 3016.274544] task_work_run+0xc6/0x110 [ 3016.276883] exit_to_usermode_loop+0x1dd/0x1f0 [ 3016.278705] do_syscall_64+0x3d6/0x440 [ 3016.280390] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3016.282580] RIP: 0033:0x7fa749552cd8 [ 3016.284282] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3016.292049] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3016.295379] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3016.298332] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3016.301195] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3016.304422] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3016.307595] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3036.064019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3036.075543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3036.165644] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3036.195095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3036.211108] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3036.227293] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3036.291340] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3036.312381] RAX: 0000000000003c06 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3036.329190] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3036.334542] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3036.344263] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3036.349660] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3036.355417] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3036.365407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3036.378042] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3036.387079] Call Trace: [ 3036.390435] [ 3036.393338] ? show_regs.cold.9+0x22/0x2f [ 3036.395686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3036.403018] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3036.408063] ? watchdog+0x40/0x40 [ 3036.413153] ? hrtimer_interrupt+0x128/0x2c0 [ 3036.422720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3036.429661] ? apic_timer_interrupt+0xf/0x20 [ 3036.435164] [ 3036.441636] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3036.453448] do_raw_spin_lock+0x177/0x190 [ 3036.454793] _raw_spin_lock+0x21/0x30 [ 3036.471205] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3036.473127] ? do_raw_spin_unlock+0x75/0x190 [ 3036.474554] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3036.504065] ? do_wait_intr+0xf0/0xf0 [ 3036.515833] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3036.525206] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3036.536690] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3036.548393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3036.557725] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3036.561421] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3036.563338] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3036.569446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3036.576429] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3036.586058] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3036.594121] lmv_intent_open+0x338/0x1150 [lmv] [ 3036.598088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3036.603587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3036.606456] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3036.612252] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3036.615725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3036.620574] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3036.624131] ? do_raw_spin_unlock+0x75/0x190 [ 3036.627142] ? _raw_spin_unlock+0x12/0x30 [ 3036.629139] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3036.632162] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3036.634567] ll_atomic_open+0x36a/0xd70 [lustre] [ 3036.637322] lookup_open+0x356/0xb40 [ 3036.639969] do_last+0x313/0xfc0 [ 3036.641739] ? nd_jump_root+0xe5/0x160 [ 3036.652696] ? path_init+0x437/0x520 [ 3036.659937] path_openat+0xf7/0x500 [ 3036.660964] ? do_raw_spin_unlock+0x75/0x190 [ 3036.674192] do_filp_open+0x99/0x140 [ 3036.675404] ? getname_flags+0x6e/0x330 [ 3036.679245] ? __check_object_size+0xff/0x256 [ 3036.682222] ? do_raw_spin_unlock+0x75/0x190 [ 3036.693419] ? _raw_spin_unlock+0x12/0x30 [ 3036.694606] do_sys_openat2+0x2b4/0x410 [ 3036.700217] do_sys_open+0x73/0xa0 [ 3036.702159] __x64_sys_openat+0x24/0x30 [ 3036.705363] do_syscall_64+0xc1/0x440 [ 3036.710118] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3036.713431] RIP: 0033:0x7fa749552332 [ 3036.715090] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3036.735982] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3036.741932] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3036.746207] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3036.756072] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3036.770700] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3036.782795] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3040.060882] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3040.066074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [ 3040.066284] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3040.066299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3040.100225] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3040.100255] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3040.100265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3040.100273] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3040.100839] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3040.100853] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3040.100916] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3040.100921] RAX: 0000000000004b1a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3040.100924] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3040.100926] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3040.100928] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3040.100930] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3040.100934] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3040.100936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3040.100938] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3040.100946] Call Trace: [ 3040.100953] [ 3040.100959] ? show_regs.cold.9+0x22/0x2f [ 3040.101220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3040.101228] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3040.101232] ? watchdog+0x40/0x40 [ 3040.101236] ? hrtimer_interrupt+0x128/0x2c0 [ 3040.101240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3040.101250] ? apic_timer_interrupt+0xf/0x20 [ 3040.101254] [ 3040.101256] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3040.101261] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3040.101266] do_raw_spin_lock+0x177/0x190 [ 3040.101271] _raw_spin_lock+0x21/0x30 [ 3040.101275] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3040.102189] process_one_work+0x2c8/0x700 [ 3040.102197] worker_thread+0x296/0x6e0 [ 3040.102201] ? rescuer_thread+0x570/0x570 [ 3040.102204] kthread+0x1d1/0x200 [ 3040.102209] ? set_kthread_struct+0x70/0x70 [ 3040.102212] ret_from_fork+0x1f/0x30 [ 3040.583421] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3040.608203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3040.621105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3040.641103] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3040.677038] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3040.685897] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3040.687521] RDX: 00000000000063ee RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3040.714918] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3040.740271] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3040.754246] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3040.766721] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3040.791025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3040.802003] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3040.810268] Call Trace: [ 3040.815059] [ 3040.816022] ? show_regs.cold.9+0x22/0x2f [ 3040.818803] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3040.820507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3040.823229] ? watchdog+0x40/0x40 [ 3040.824978] ? hrtimer_interrupt+0x128/0x2c0 [ 3040.837252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3040.838771] ? apic_timer_interrupt+0xf/0x20 [ 3040.846616] [ 3040.847594] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3040.861401] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3040.863220] do_raw_spin_lock+0x177/0x190 [ 3040.864581] _raw_spin_lock+0x21/0x30 [ 3040.877130] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3040.894052] obd_mod_free+0x46/0x140 [ptlrpc] [ 3040.895825] mdc_commit_open+0x91/0xb0 [mdc] [ 3040.922475] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3040.940220] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3040.942269] after_reply+0xe2c/0x1780 [ptlrpc] [ 3040.964034] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3040.966045] ? __next_timer_interrupt+0x160/0x160 [ 3040.983454] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3040.988202] ? do_wait_intr+0xf0/0xf0 [ 3040.990599] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3040.996387] kthread+0x1d1/0x200 [ 3040.999507] ? set_kthread_struct+0x70/0x70 [ 3041.008977] ret_from_fork+0x1f/0x30 [ 3042.274088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1125s! [ 3042.290897] Showing busy workqueues and worker pools: [ 3042.297224] workqueue events: flags=0x0 [ 3042.307041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3042.316846] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3042.331429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3042.335195] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3042.351786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.372834] pending: kfree_rcu_monitor [ 3042.379873] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3042.385985] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3042.397599] workqueue events_long: flags=0x0 [ 3042.399113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.418578] pending: lu_objects_flush [obdclass] [ 3042.432459] workqueue events_unbound: flags=0x2 [ 3042.442111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3042.449769] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3042.454239] workqueue events_power_efficient: flags=0x80 [ 3042.459181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.464860] pending: check_lifetime [ 3042.468191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.483036] pending: neigh_periodic_work [ 3042.488952] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.497489] pending: neigh_periodic_work [ 3042.502503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.521165] pending: do_cache_clean [ 3042.522492] workqueue mm_percpu_wq: flags=0x8 [ 3042.538026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.540301] pending: vmstat_update [ 3042.558367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.560530] pending: vmstat_update [ 3042.572171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3042.582358] pending: vmstat_update [ 3042.583649] workqueue writeback: flags=0x4a [ 3042.594944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3042.597221] pending: wb_workfn, wb_workfn [ 3042.603132] workqueue xprtiod: flags=0xa [ 3042.604538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3042.618933] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3042.644026] workqueue ipv6_addrconf: flags=0x40008 [ 3042.660056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3042.677851] pending: addrconf_verify_work [ 3042.679384] workqueue knbd0-recv: flags=0x1a [ 3042.689559] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3042.702022] in-flight: 651:recv_work [ 3042.704450] workqueue ptlrpc_pinger: flags=0x2 [ 3042.716045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3042.732793] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3042.741481] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1125s workers=4 idle: 37 31846 28897 [ 3042.757870] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3032s workers=2 idle: 653 [ 3044.066078] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3044.073121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3044.135842] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3044.145742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3044.153102] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3044.164139] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3044.189956] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3044.205904] RAX: 0000000000006c70 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3044.214189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3044.222407] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3044.232869] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3044.248582] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3044.262666] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3044.272131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3044.280967] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3044.287429] Call Trace: [ 3044.288294] [ 3044.288958] ? show_regs.cold.9+0x22/0x2f [ 3044.303167] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3044.309564] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3044.320162] ? watchdog+0x40/0x40 [ 3044.324648] ? hrtimer_interrupt+0x128/0x2c0 [ 3044.326335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3044.327601] ? apic_timer_interrupt+0xf/0x20 [ 3044.334756] [ 3044.335442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3044.346252] do_raw_spin_lock+0x177/0x190 [ 3044.351663] _raw_spin_lock+0x21/0x30 [ 3044.356555] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3044.360442] ? do_raw_spin_unlock+0x75/0x190 [ 3044.367562] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3044.375983] ? do_wait_intr+0xf0/0xf0 [ 3044.384605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3044.398515] mdc_close+0x53b/0xd70 [mdc] [ 3044.405516] lmv_close+0x18d/0x490 [lmv] [ 3044.408084] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3044.414845] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3044.422014] ll_md_close+0x274/0x800 [lustre] [ 3044.435628] ll_file_release+0x128/0xc50 [lustre] [ 3044.449745] __fput+0x10d/0x440 [ 3044.451359] ____fput+0x12/0x20 [ 3044.453352] task_work_run+0xc6/0x110 [ 3044.455275] exit_to_usermode_loop+0x1dd/0x1f0 [ 3044.458885] do_syscall_64+0x3d6/0x440 [ 3044.460086] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3044.465785] RIP: 0033:0x7fa749552cd8 [ 3044.467787] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3044.489460] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3044.501829] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3044.510153] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3044.523707] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3044.534541] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3044.542021] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3053.711635] rcu: INFO: rcu_sched self-detected stall on CPU [ 3053.713725] rcu: 2-...!: (612387 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 3053.719876] (t=1140022 jiffies g=258701 q=1842) [ 3053.721469] rcu: rcu_sched kthread starved for 1136817 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3053.729944] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3053.732660] rcu: RCU grace-period kthread stack dump: [ 3053.736672] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3053.744964] Call Trace: [ 3053.745726] __schedule+0x351/0xcb0 [ 3053.747420] schedule+0xc0/0x180 [ 3053.748979] schedule_timeout+0xb4/0x190 [ 3053.750715] ? do_raw_spin_unlock+0x75/0x190 [ 3053.752030] ? __next_timer_interrupt+0x160/0x160 [ 3053.754351] ? prepare_to_swait_event+0xf3/0x140 [ 3053.758231] rcu_gp_kthread+0x8dd/0x15b0 [ 3053.759763] ? force_qs_rnp+0x2c0/0x2c0 [ 3053.761504] kthread+0x1d1/0x200 [ 3053.763993] ? set_kthread_struct+0x70/0x70 [ 3053.767988] ret_from_fork+0x1f/0x30 [ 3053.770772] rcu: Stack dump where RCU GP kthread last ran: [ 3053.774674] Sending NMI from CPU 2 to CPUs 3: [ 3053.778401] NMI backtrace for cpu 3 [ 3053.778407] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3053.778408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3053.778410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3053.778411] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3053.778414] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 [ 3053.778416] RAX: 0000000000007b88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3053.778417] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3053.778418] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3053.778420] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3053.778421] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3053.778422] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3053.778423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3053.778424] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3053.778425] Call Trace: [ 3053.778426] [ 3053.778427] ? show_regs.cold.9+0x22/0x2f [ 3053.778428] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3053.778429] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3053.778430] ? nmi_handle+0xbd/0x210 [ 3053.778430] ? default_do_nmi+0x4b/0x1b0 [ 3053.778431] ? do_nmi+0x280/0x380 [ 3053.778432] ? end_repeat_nmi+0x16/0x1f [ 3053.778433] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3053.778434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3053.778435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3053.778436] [ 3053.778437] do_raw_spin_lock+0x177/0x190 [ 3053.778438] _raw_spin_lock+0x21/0x30 [ 3053.778439] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3053.778440] ? do_raw_spin_unlock+0x75/0x190 [ 3053.778441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3053.778442] ? do_wait_intr+0xf0/0xf0 [ 3053.778443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3053.778444] mdc_close+0x53b/0xd70 [mdc] [ 3053.778445] lmv_close+0x18d/0x490 [lmv] [ 3053.778446] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3053.778447] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3053.778449] ll_md_close+0x274/0x800 [lustre] [ 3053.778450] ll_file_release+0x128/0xc50 [lustre] [ 3053.778451] __fput+0x10d/0x440 [ 3053.778452] ____fput+0x12/0x20 [ 3053.778453] task_work_run+0xc6/0x110 [ 3053.778463] exit_to_usermode_loop+0x1dd/0x1f0 [ 3053.778464] do_syscall_64+0x3d6/0x440 [ 3053.778465] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3053.778466] RIP: 0033:0x7fa749552cd8 [ 3053.778467] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3053.778469] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3053.778472] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3053.778473] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3053.778474] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3053.778476] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3053.778477] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3053.781104] NMI backtrace for cpu 2 [ 3054.016152] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3054.021383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3054.024440] Call Trace: [ 3054.025140] [ 3054.025909] ? dump_stack+0xbb/0x10e [ 3054.027596] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3054.029887] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3054.032730] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3054.034991] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3054.036913] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3054.039107] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3054.044468] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3054.045866] ? update_process_times+0x6d/0xc0 [ 3054.047062] ? tick_sched_handle.isra.18+0x27/0x90 [ 3054.049820] ? tick_sched_timer+0x53/0xb0 [ 3054.051837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3054.053053] ? tick_sched_handle.isra.18+0x90/0x90 [ 3054.054773] ? hrtimer_interrupt+0x128/0x2c0 [ 3054.056107] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3054.057831] ? apic_timer_interrupt+0xf/0x20 [ 3054.059526] [ 3054.060299] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3054.062372] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3054.064369] ? do_raw_spin_lock+0x177/0x190 [ 3054.066099] ? _raw_spin_lock+0x21/0x30 [ 3054.068204] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3054.070840] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3054.075735] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3054.078669] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3054.080927] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3054.082993] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3054.085285] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3054.087701] ? __next_timer_interrupt+0x160/0x160 [ 3054.092135] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3054.094305] ? do_wait_intr+0xf0/0xf0 [ 3054.096315] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3054.099436] ? kthread+0x1d1/0x200 [ 3054.100573] ? set_kthread_struct+0x70/0x70 [ 3054.103368] ? ret_from_fork+0x1f/0x30 [ 3068.055668] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3068.061614] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3068.064902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3068.064982] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3068.065116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3068.065119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3068.065133] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3068.065138] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3068.065147] RAX: 0000000000005bd9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3068.065149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3068.065151] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3068.065153] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3068.065156] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3068.065159] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3068.065162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3068.065165] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3068.065173] Call Trace: [ 3068.065177] [ 3068.065183] ? show_regs.cold.9+0x22/0x2f [ 3068.065198] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3068.065289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3068.065294] ? watchdog+0x40/0x40 [ 3068.065298] ? hrtimer_interrupt+0x128/0x2c0 [ 3068.065307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3068.065315] ? apic_timer_interrupt+0xf/0x20 [ 3068.065320] [ 3068.065323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3068.065331] do_raw_spin_lock+0x177/0x190 [ 3068.065336] _raw_spin_lock+0x21/0x30 [ 3068.065341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3068.074852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3068.074929] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3068.074936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3068.074942] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3068.142306] ? do_raw_spin_unlock+0x75/0x190 [ 3068.149841] [ 3068.149850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3068.149866] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3068.149870] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3068.149875] RAX: 0000000000007eaf RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3068.149877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3068.149879] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3068.149881] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3068.149884] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3068.149887] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3068.149891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3068.149893] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3068.149902] Call Trace: [ 3068.149906] [ 3068.149912] ? show_regs.cold.9+0x22/0x2f [ 3068.149935] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3068.149941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3068.149945] ? watchdog+0x40/0x40 [ 3068.149950] ? hrtimer_interrupt+0x128/0x2c0 [ 3068.149953] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3068.149960] ? apic_timer_interrupt+0xf/0x20 [ 3068.149965] [ 3068.149967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3068.149972] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3068.149979] do_raw_spin_lock+0x177/0x190 [ 3068.149983] _raw_spin_lock+0x21/0x30 [ 3068.149989] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3068.150281] process_one_work+0x2c8/0x700 [ 3068.150288] worker_thread+0x296/0x6e0 [ 3068.150292] ? rescuer_thread+0x570/0x570 [ 3068.150296] kthread+0x1d1/0x200 [ 3068.150300] ? set_kthread_struct+0x70/0x70 [ 3068.150304] ret_from_fork+0x1f/0x30 [ 3068.588440] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3068.595350] ? do_wait_intr+0xf0/0xf0 [ 3068.597516] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3068.600443] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3068.603363] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3068.606957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3068.608579] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3068.610287] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3068.612387] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3068.614327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3068.618568] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3068.623064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3068.629105] lmv_intent_open+0x338/0x1150 [lmv] [ 3068.633043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3068.638269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3068.642776] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3068.646017] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3068.651564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3068.657344] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3068.660822] ? do_raw_spin_unlock+0x75/0x190 [ 3068.663498] ? _raw_spin_unlock+0x12/0x30 [ 3068.668580] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3068.675632] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3068.681560] ll_atomic_open+0x36a/0xd70 [lustre] [ 3068.684871] lookup_open+0x356/0xb40 [ 3068.690686] do_last+0x313/0xfc0 [ 3068.694864] ? nd_jump_root+0xe5/0x160 [ 3068.696511] ? path_init+0x437/0x520 [ 3068.697776] path_openat+0xf7/0x500 [ 3068.700546] ? do_raw_spin_unlock+0x75/0x190 [ 3068.702462] do_filp_open+0x99/0x140 [ 3068.705533] ? getname_flags+0x6e/0x330 [ 3068.708107] ? __check_object_size+0xff/0x256 [ 3068.712034] ? do_raw_spin_unlock+0x75/0x190 [ 3068.715521] ? _raw_spin_unlock+0x12/0x30 [ 3068.719338] do_sys_openat2+0x2b4/0x410 [ 3068.722686] do_sys_open+0x73/0xa0 [ 3068.726122] __x64_sys_openat+0x24/0x30 [ 3068.728421] do_syscall_64+0xc1/0x440 [ 3068.730135] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3068.732621] RIP: 0033:0x7fa749552332 [ 3068.734454] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3068.749790] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3068.756505] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3068.762777] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3068.770803] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3068.777689] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3068.785062] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3072.996507] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1155s! [ 3073.006236] Showing busy workqueues and worker pools: [ 3073.011255] workqueue events: flags=0x0 [ 3073.016374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3073.022688] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3073.039116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3073.051492] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3073.061127] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.071378] pending: kfree_rcu_monitor [ 3073.078379] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3073.088568] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3073.095078] workqueue events_long: flags=0x0 [ 3073.100110] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.101923] pending: lu_objects_flush [obdclass] [ 3073.110397] workqueue events_unbound: flags=0x2 [ 3073.117730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3073.123689] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3073.132681] workqueue events_power_efficient: flags=0x80 [ 3073.139232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.150232] pending: check_lifetime [ 3073.154139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.159695] pending: neigh_periodic_work [ 3073.166366] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.171361] pending: neigh_periodic_work [ 3073.179183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.188901] pending: do_cache_clean [ 3073.193642] workqueue mm_percpu_wq: flags=0x8 [ 3073.200426] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.212434] pending: vmstat_update [ 3073.214298] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.224034] pending: vmstat_update [ 3073.228036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3073.238589] pending: vmstat_update [ 3073.240382] workqueue writeback: flags=0x4a [ 3073.250515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3073.262665] pending: wb_workfn, wb_workfn [ 3073.267707] workqueue xprtiod: flags=0xa [ 3073.274051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3073.278727] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3073.289669] workqueue ipv6_addrconf: flags=0x40008 [ 3073.294594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3073.305172] pending: addrconf_verify_work [ 3073.312523] workqueue knbd0-recv: flags=0x1a [ 3073.313634] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3073.319638] in-flight: 651:recv_work [ 3073.324309] workqueue ptlrpc_pinger: flags=0x2 [ 3073.329012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3073.334596] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3073.340599] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1156s workers=4 idle: 37 31846 28897 [ 3073.346939] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3063s workers=2 idle: 653 [ 3080.058054] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3080.060090] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3080.061424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3080.061496] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3080.061502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3080.061504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3080.061517] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3080.061522] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3080.061525] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3080.061528] RDX: 0000000000005d74 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3080.061530] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3080.061532] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3080.061534] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3080.061537] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3080.061540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3080.061542] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3080.061550] Call Trace: [ 3080.061555] [ 3080.061561] ? show_regs.cold.9+0x22/0x2f [ 3080.061571] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3080.061577] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3080.061581] ? watchdog+0x40/0x40 [ 3080.061585] ? hrtimer_interrupt+0x128/0x2c0 [ 3080.061589] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3080.061597] ? apic_timer_interrupt+0xf/0x20 [ 3080.061601] [ 3080.061603] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3080.061608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3080.061612] do_raw_spin_lock+0x177/0x190 [ 3080.061617] _raw_spin_lock+0x21/0x30 [ 3080.061623] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3080.061874] obd_mod_free+0x46/0x140 [ptlrpc] [ 3080.062083] mdc_commit_open+0x91/0xb0 [mdc] [ 3080.062117] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3080.062304] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3080.062500] after_reply+0xe2c/0x1780 [ptlrpc] [ 3080.062705] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3080.062910] ? __next_timer_interrupt+0x160/0x160 [ 3080.062919] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3080.073980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3080.074812] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3080.074819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3080.074822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3080.074832] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3080.074835] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3080.074838] RAX: 0000000000007c84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3080.074840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3080.074842] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3080.074843] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3080.074845] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3080.074847] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3080.074849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3080.074850] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3080.074858] Call Trace: [ 3080.074862] [ 3080.074866] ? show_regs.cold.9+0x22/0x2f [ 3080.074872] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3080.074877] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3080.074879] ? watchdog+0x40/0x40 [ 3080.074882] ? hrtimer_interrupt+0x128/0x2c0 [ 3080.074884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3080.074891] ? apic_timer_interrupt+0xf/0x20 [ 3080.074894] [ 3080.074896] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3080.074900] do_raw_spin_lock+0x177/0x190 [ 3080.074903] _raw_spin_lock+0x21/0x30 [ 3080.074906] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3080.075403] ? do_raw_spin_unlock+0x75/0x190 [ 3080.075410] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3080.103399] ? do_wait_intr+0xf0/0xf0 [ 3080.103419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3080.103944] kthread+0x1d1/0x200 [ 3080.103951] ? set_kthread_struct+0x70/0x70 [ 3080.103955] ret_from_fork+0x1f/0x30 [ 3080.567114] ? do_wait_intr+0xf0/0xf0 [ 3080.574729] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3080.582548] mdc_close+0x53b/0xd70 [mdc] [ 3080.586084] lmv_close+0x18d/0x490 [lmv] [ 3080.591270] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3080.602916] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3080.606091] ll_md_close+0x274/0x800 [lustre] [ 3080.612465] ll_file_release+0x128/0xc50 [lustre] [ 3080.616115] __fput+0x10d/0x440 [ 3080.618032] ____fput+0x12/0x20 [ 3080.628353] task_work_run+0xc6/0x110 [ 3080.632339] exit_to_usermode_loop+0x1dd/0x1f0 [ 3080.639327] do_syscall_64+0x3d6/0x440 [ 3080.641551] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3080.651417] RIP: 0033:0x7fa749552cd8 [ 3080.655449] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3080.671260] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3080.679111] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3080.685994] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3080.693324] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3080.700506] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3080.706935] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3096.058942] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3096.078941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3096.164358] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3096.178798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3096.192210] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3096.199093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3096.202970] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3096.231172] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3096.239510] RAX: 0000000000005511 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3096.250801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3096.257119] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3096.271131] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3096.278906] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3096.284826] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3096.298345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3096.302101] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3096.306779] Call Trace: [ 3096.309286] [ 3096.313163] ? show_regs.cold.9+0x22/0x2f [ 3096.317195] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3096.321261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3096.322398] ? watchdog+0x40/0x40 [ 3096.323573] ? hrtimer_interrupt+0x128/0x2c0 [ 3096.326648] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3096.329356] ? apic_timer_interrupt+0xf/0x20 [ 3096.331800] [ 3096.333179] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3096.336784] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3096.339833] do_raw_spin_lock+0x177/0x190 [ 3096.341787] _raw_spin_lock+0x21/0x30 [ 3096.344267] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3096.351283] process_one_work+0x2c8/0x700 [ 3096.354321] worker_thread+0x296/0x6e0 [ 3096.359235] ? rescuer_thread+0x570/0x570 [ 3096.362187] kthread+0x1d1/0x200 [ 3096.363240] ? set_kthread_struct+0x70/0x70 [ 3096.365262] ret_from_fork+0x1f/0x30 [ 3100.057799] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3100.068144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3100.125164] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3100.135686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3100.149295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3100.160789] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3100.203492] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3100.214195] RAX: 0000000000007b06 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3100.221020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3100.235832] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3100.243771] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3100.249264] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3100.255045] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3100.270809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3100.276067] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3100.289940] Call Trace: [ 3100.294765] [ 3100.304735] ? show_regs.cold.9+0x22/0x2f [ 3100.310261] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3100.324566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3100.329005] ? watchdog+0x40/0x40 [ 3100.332512] ? hrtimer_interrupt+0x128/0x2c0 [ 3100.340041] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3100.342915] ? apic_timer_interrupt+0xf/0x20 [ 3100.350439] [ 3100.353385] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3100.362268] do_raw_spin_lock+0x177/0x190 [ 3100.363721] _raw_spin_lock+0x21/0x30 [ 3100.370435] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3100.374575] ? do_raw_spin_unlock+0x75/0x190 [ 3100.379826] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3100.381737] ? do_wait_intr+0xf0/0xf0 [ 3100.390616] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3100.394079] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3100.402336] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3100.410589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.415695] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3100.423291] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3100.426327] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3100.433529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.441043] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3100.447144] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3100.455088] lmv_intent_open+0x338/0x1150 [lmv] [ 3100.458051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.468218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.471955] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3100.474832] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3100.477074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.491249] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3100.498573] ? do_raw_spin_unlock+0x75/0x190 [ 3100.501596] ? _raw_spin_unlock+0x12/0x30 [ 3100.508238] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3100.515261] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3100.521073] ll_atomic_open+0x36a/0xd70 [lustre] [ 3100.530430] lookup_open+0x356/0xb40 [ 3100.534980] do_last+0x313/0xfc0 [ 3100.537009] ? nd_jump_root+0xe5/0x160 [ 3100.544510] ? path_init+0x437/0x520 [ 3100.545861] path_openat+0xf7/0x500 [ 3100.547110] ? do_raw_spin_unlock+0x75/0x190 [ 3100.567191] do_filp_open+0x99/0x140 [ 3100.576772] ? getname_flags+0x6e/0x330 [ 3100.579008] ? __check_object_size+0xff/0x256 [ 3100.585610] ? do_raw_spin_unlock+0x75/0x190 [ 3100.591934] ? _raw_spin_unlock+0x12/0x30 [ 3100.595012] do_sys_openat2+0x2b4/0x410 [ 3100.604560] do_sys_open+0x73/0xa0 [ 3100.608287] __x64_sys_openat+0x24/0x30 [ 3100.623914] do_syscall_64+0xc1/0x440 [ 3100.628253] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3100.643029] RIP: 0033:0x7fa749552332 [ 3100.655344] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3100.686300] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3100.696169] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3100.708557] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3100.719064] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3100.729728] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3100.739928] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3103.719652] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1186s! [ 3103.735517] Showing busy workqueues and worker pools: [ 3103.744461] workqueue events: flags=0x0 [ 3103.748115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3103.755748] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3103.770057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3103.774935] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3103.786540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.803473] pending: kfree_rcu_monitor [ 3103.812826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3103.817642] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3103.821872] workqueue events_long: flags=0x0 [ 3103.834561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.843405] pending: lu_objects_flush [obdclass] [ 3103.850845] workqueue events_unbound: flags=0x2 [ 3103.855705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3103.860817] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3103.865303] workqueue events_power_efficient: flags=0x80 [ 3103.869552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.873015] pending: check_lifetime [ 3103.875838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.880970] pending: neigh_periodic_work [ 3103.887080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.894830] pending: neigh_periodic_work [ 3103.900519] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.908940] pending: do_cache_clean [ 3103.916378] workqueue mm_percpu_wq: flags=0x8 [ 3103.921115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.930839] pending: vmstat_update [ 3103.936270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.942138] pending: vmstat_update [ 3103.944078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3103.947818] pending: vmstat_update [ 3103.951941] workqueue writeback: flags=0x4a [ 3103.953382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3103.966764] pending: wb_workfn, wb_workfn [ 3103.968376] workqueue xprtiod: flags=0xa [ 3103.972417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3103.979982] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3103.988594] workqueue ipv6_addrconf: flags=0x40008 [ 3103.992855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3104.002534] pending: addrconf_verify_work [ 3104.008778] workqueue knbd0-recv: flags=0x1a [ 3104.013015] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3104.022047] in-flight: 651:recv_work [ 3104.026116] workqueue ptlrpc_pinger: flags=0x2 [ 3104.029902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3104.036108] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3104.040097] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1186s workers=4 idle: 37 31846 28897 [ 3104.050930] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3093s workers=2 idle: 653 [ 3108.059308] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3108.062589] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3108.062717] Modules linked in: [ 3108.074029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 3108.076355] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3108.076428] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3108.076434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3108.076437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3108.076449] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3108.076453] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3108.076540] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3108.076544] RDX: 0000000000001776 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3108.077073] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3108.077078] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3108.077080] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3108.077083] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3108.077085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3108.077087] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3108.077095] Call Trace: [ 3108.077100] [ 3108.077106] ? show_regs.cold.9+0x22/0x2f [ 3108.077116] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3108.077122] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3108.077126] ? watchdog+0x40/0x40 [ 3108.077130] ? hrtimer_interrupt+0x128/0x2c0 [ 3108.077133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3108.077140] ? apic_timer_interrupt+0xf/0x20 [ 3108.077144] [ 3108.077146] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3108.077151] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3108.077155] do_raw_spin_lock+0x177/0x190 [ 3108.077159] _raw_spin_lock+0x21/0x30 [ 3108.077165] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3108.077446] obd_mod_free+0x46/0x140 [ptlrpc] [ 3108.077748] mdc_commit_open+0x91/0xb0 [mdc] [ 3108.077785] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3108.089178] ptlrpc(O) [ 3108.106653] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3108.115579] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3108.115627] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3108.115635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3108.115638] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3108.115652] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3108.115655] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3108.115660] RAX: 0000000000007f0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3108.115663] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3108.115664] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3108.115666] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3108.115669] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3108.115672] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3108.115674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3108.115676] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3108.115684] Call Trace: [ 3108.115690] [ 3108.115696] ? show_regs.cold.9+0x22/0x2f [ 3108.115705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3108.115711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3108.115715] ? watchdog+0x40/0x40 [ 3108.115719] ? hrtimer_interrupt+0x128/0x2c0 [ 3108.115723] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3108.115731] ? apic_timer_interrupt+0xf/0x20 [ 3108.115735] [ 3108.115738] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3108.115745] do_raw_spin_lock+0x177/0x190 [ 3108.115749] _raw_spin_lock+0x21/0x30 [ 3108.115753] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3108.119951] ? do_raw_spin_unlock+0x75/0x190 [ 3108.120604] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3108.134796] after_reply+0xe2c/0x1780 [ptlrpc] [ 3108.139438] ? do_wait_intr+0xf0/0xf0 [ 3108.139462] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3108.139750] mdc_close+0x53b/0xd70 [mdc] [ 3108.139792] lmv_close+0x18d/0x490 [lmv] [ 3108.139826] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3108.139933] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3108.140018] ll_md_close+0x274/0x800 [lustre] [ 3108.140101] ll_file_release+0x128/0xc50 [lustre] [ 3108.140188] __fput+0x10d/0x440 [ 3108.140198] ____fput+0x12/0x20 [ 3108.140203] task_work_run+0xc6/0x110 [ 3108.140210] exit_to_usermode_loop+0x1dd/0x1f0 [ 3108.140218] do_syscall_64+0x3d6/0x440 [ 3108.140223] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3108.140232] RIP: 0033:0x7fa749552cd8 [ 3108.140239] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3108.140244] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3108.140250] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3108.140252] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3108.140254] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3108.140256] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3108.140258] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3108.756222] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3108.765822] ? __next_timer_interrupt+0x160/0x160 [ 3108.767353] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3108.771522] ? do_wait_intr+0xf0/0xf0 [ 3108.773179] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3108.777488] kthread+0x1d1/0x200 [ 3108.782236] ? set_kthread_struct+0x70/0x70 [ 3108.786134] ret_from_fork+0x1f/0x30 [ 3124.057054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3124.059576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3124.080753] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3124.085906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3124.089568] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3124.092085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.095081] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3124.103490] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3124.107123] RAX: 0000000000003a91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3124.110629] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3124.115010] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3124.118022] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3124.121674] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3124.124802] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3124.128798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3124.131411] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3124.134821] Call Trace: [ 3124.136089] [ 3124.136976] ? show_regs.cold.9+0x22/0x2f [ 3124.138888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3124.140924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3124.143448] ? watchdog+0x40/0x40 [ 3124.144901] ? hrtimer_interrupt+0x128/0x2c0 [ 3124.146835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3124.149015] ? apic_timer_interrupt+0xf/0x20 [ 3124.151364] [ 3124.152490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.155157] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3124.157832] do_raw_spin_lock+0x177/0x190 [ 3124.159683] _raw_spin_lock+0x21/0x30 [ 3124.161079] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3124.163890] process_one_work+0x2c8/0x700 [ 3124.165899] worker_thread+0x296/0x6e0 [ 3124.167764] ? rescuer_thread+0x570/0x570 [ 3124.169680] kthread+0x1d1/0x200 [ 3124.171147] ? set_kthread_struct+0x70/0x70 [ 3124.173137] ret_from_fork+0x1f/0x30 [ 3128.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3128.062117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3128.122692] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3128.134419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3128.146445] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3128.154872] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3128.175197] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3128.177838] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3128.180516] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3128.183301] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3128.185949] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3128.188690] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3128.191447] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3128.194442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3128.196387] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3128.198929] Call Trace: [ 3128.199710] [ 3128.200444] ? show_regs.cold.9+0x22/0x2f [ 3128.202290] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3128.204027] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3128.205978] ? watchdog+0x40/0x40 [ 3128.207520] ? hrtimer_interrupt+0x128/0x2c0 [ 3128.209335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3128.211420] ? apic_timer_interrupt+0xf/0x20 [ 3128.212791] [ 3128.213572] ? native_safe_halt+0x2e/0x50 [ 3128.215147] kvm_wait+0xcd/0xe0 [ 3128.216232] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3128.218230] do_raw_spin_lock+0x177/0x190 [ 3128.219862] _raw_spin_lock+0x21/0x30 [ 3128.221369] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3128.223535] ? do_raw_spin_unlock+0x75/0x190 [ 3128.225332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3128.227589] ? do_wait_intr+0xf0/0xf0 [ 3128.228785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3128.231191] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3128.233371] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3128.235893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.237924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3128.240178] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3128.241675] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3128.243587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.245533] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3128.248098] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3128.250307] lmv_intent_open+0x338/0x1150 [lmv] [ 3128.252306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.254259] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.256487] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3128.258489] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3128.260487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.262790] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3128.264607] ? do_raw_spin_unlock+0x75/0x190 [ 3128.266472] ? _raw_spin_unlock+0x12/0x30 [ 3128.267772] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3128.269543] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3128.271311] ll_atomic_open+0x36a/0xd70 [lustre] [ 3128.273438] lookup_open+0x356/0xb40 [ 3128.274721] do_last+0x313/0xfc0 [ 3128.275670] ? nd_jump_root+0xe5/0x160 [ 3128.276686] ? path_init+0x437/0x520 [ 3128.277726] path_openat+0xf7/0x500 [ 3128.279271] ? do_raw_spin_unlock+0x75/0x190 [ 3128.281720] do_filp_open+0x99/0x140 [ 3128.283468] ? getname_flags+0x6e/0x330 [ 3128.284817] ? __check_object_size+0xff/0x256 [ 3128.286792] ? do_raw_spin_unlock+0x75/0x190 [ 3128.288755] ? _raw_spin_unlock+0x12/0x30 [ 3128.290739] do_sys_openat2+0x2b4/0x410 [ 3128.292623] do_sys_open+0x73/0xa0 [ 3128.293748] __x64_sys_openat+0x24/0x30 [ 3128.295221] do_syscall_64+0xc1/0x440 [ 3128.296591] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3128.298842] RIP: 0033:0x7fa749552332 [ 3128.300352] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3128.308749] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3128.311779] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3128.314456] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3128.316859] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3128.320237] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3128.323060] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3134.432420] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1217s! [ 3134.435475] Showing busy workqueues and worker pools: [ 3134.444275] workqueue events: flags=0x0 [ 3134.446918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3134.456269] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3134.461682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3134.466521] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3134.478501] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.484165] pending: kfree_rcu_monitor [ 3134.487407] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3134.495478] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3134.500386] workqueue events_long: flags=0x0 [ 3134.506300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.508744] pending: lu_objects_flush [obdclass] [ 3134.511250] workqueue events_unbound: flags=0x2 [ 3134.517080] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3134.523667] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3134.529287] workqueue events_power_efficient: flags=0x80 [ 3134.536349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.539284] pending: check_lifetime [ 3134.540500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.543365] pending: neigh_periodic_work [ 3134.546845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.552150] pending: neigh_periodic_work [ 3134.556051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.558451] pending: do_cache_clean [ 3134.562376] workqueue mm_percpu_wq: flags=0x8 [ 3134.566634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.577530] pending: vmstat_update [ 3134.581955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.585339] pending: vmstat_update [ 3134.586853] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3134.589329] pending: vmstat_update [ 3134.590725] workqueue writeback: flags=0x4a [ 3134.592391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3134.597735] pending: wb_workfn, wb_workfn [ 3134.605079] workqueue xprtiod: flags=0xa [ 3134.609259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3134.619847] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3134.627484] workqueue ipv6_addrconf: flags=0x40008 [ 3134.639033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3134.646315] pending: addrconf_verify_work [ 3134.649176] workqueue knbd0-recv: flags=0x1a [ 3134.651874] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3134.661325] in-flight: 651:recv_work [ 3134.665476] workqueue ptlrpc_pinger: flags=0x2 [ 3134.666967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3134.674264] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3134.677072] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1217s workers=4 idle: 37 31846 28897 [ 3134.680580] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3124s workers=2 idle: 653 [ 3136.058028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3136.059105] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3136.062120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3136.062193] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3136.062201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3136.062204] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3136.062217] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3136.062221] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3136.062225] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3136.062227] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3136.062230] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3136.062232] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3136.062234] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3136.062237] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3136.062240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3136.062242] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3136.062251] Call Trace: [ 3136.062257] [ 3136.062262] ? show_regs.cold.9+0x22/0x2f [ 3136.078326] Modules linked in: lustre(O) [ 3136.118398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3136.118416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3136.138804] osp(O) [ 3136.143324] ? watchdog+0x40/0x40 [ 3136.143344] ? hrtimer_interrupt+0x128/0x2c0 [ 3136.143350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3136.143359] ? apic_timer_interrupt+0xf/0x20 [ 3136.143363] [ 3136.143368] ? native_safe_halt+0x2e/0x50 [ 3136.143373] kvm_wait+0xcd/0xe0 [ 3136.143381] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3136.143388] do_raw_spin_lock+0x177/0x190 [ 3136.143393] _raw_spin_lock+0x21/0x30 [ 3136.143399] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3136.143840] obd_mod_free+0x46/0x140 [ptlrpc] [ 3136.144700] mdc_commit_open+0x91/0xb0 [mdc] [ 3136.144901] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3136.145288] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3136.145574] after_reply+0xe2c/0x1780 [ptlrpc] [ 3136.145832] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3136.146036] ? __next_timer_interrupt+0x160/0x160 [ 3136.146046] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3136.146368] ? do_wait_intr+0xf0/0xf0 [ 3136.146379] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3136.146810] kthread+0x1d1/0x200 [ 3136.146817] ? set_kthread_struct+0x70/0x70 [ 3136.146821] ret_from_fork+0x1f/0x30 [ 3136.308685] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3136.353409] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3136.363508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3136.374345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3136.379956] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3136.400568] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3136.405647] RAX: 0000000000002acd RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3136.408990] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3136.415069] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3136.428633] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3136.443425] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3136.451080] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3136.458997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3136.463445] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3136.467321] Call Trace: [ 3136.468626] [ 3136.469430] ? show_regs.cold.9+0x22/0x2f [ 3136.471512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3136.473916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3136.476958] ? watchdog+0x40/0x40 [ 3136.480264] ? hrtimer_interrupt+0x128/0x2c0 [ 3136.484762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3136.491495] ? apic_timer_interrupt+0xf/0x20 [ 3136.493671] [ 3136.494665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3136.497195] do_raw_spin_lock+0x177/0x190 [ 3136.498903] _raw_spin_lock+0x21/0x30 [ 3136.509408] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3136.512892] ? do_raw_spin_unlock+0x75/0x190 [ 3136.519882] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3136.523311] ? do_wait_intr+0xf0/0xf0 [ 3136.524944] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3136.528358] mdc_close+0x53b/0xd70 [mdc] [ 3136.530163] lmv_close+0x18d/0x490 [lmv] [ 3136.532251] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3136.542387] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3136.544483] ll_md_close+0x274/0x800 [lustre] [ 3136.551051] ll_file_release+0x128/0xc50 [lustre] [ 3136.553865] __fput+0x10d/0x440 [ 3136.556647] ____fput+0x12/0x20 [ 3136.559714] task_work_run+0xc6/0x110 [ 3136.563103] exit_to_usermode_loop+0x1dd/0x1f0 [ 3136.567135] do_syscall_64+0x3d6/0x440 [ 3136.568451] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3136.574956] RIP: 0033:0x7fa749552cd8 [ 3136.580342] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3136.598075] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3136.610330] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3136.617162] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3136.622550] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3136.626687] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3136.637552] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3152.060556] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3152.065638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3152.101258] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3152.107055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3152.111854] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3152.114786] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3152.117181] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3152.126088] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3152.129982] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3152.133816] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3152.137347] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3152.150300] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3152.154879] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3152.161062] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3152.165893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3152.169676] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3152.174101] Call Trace: [ 3152.175252] [ 3152.176648] ? show_regs.cold.9+0x22/0x2f [ 3152.178416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3152.181278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3152.184845] ? watchdog+0x40/0x40 [ 3152.189001] ? hrtimer_interrupt+0x128/0x2c0 [ 3152.192084] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3152.194333] ? apic_timer_interrupt+0xf/0x20 [ 3152.197995] [ 3152.199131] ? native_safe_halt+0x2e/0x50 [ 3152.201282] kvm_wait+0xcd/0xe0 [ 3152.203061] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3152.205079] do_raw_spin_lock+0x177/0x190 [ 3152.206762] _raw_spin_lock+0x21/0x30 [ 3152.209134] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3152.212114] process_one_work+0x2c8/0x700 [ 3152.214123] worker_thread+0x296/0x6e0 [ 3152.216029] ? rescuer_thread+0x570/0x570 [ 3152.219099] kthread+0x1d1/0x200 [ 3152.222327] ? set_kthread_struct+0x70/0x70 [ 3152.224917] ret_from_fork+0x1f/0x30 [ 3160.054018] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3160.057747] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3160.086916] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3160.092040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3160.095988] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3160.099620] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3160.106379] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3160.110288] RAX: 0000000000000e85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3160.113555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3160.116189] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3160.121550] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3160.124864] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3160.130117] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3160.132334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3160.134263] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3160.137491] Call Trace: [ 3160.138628] [ 3160.139441] ? show_regs.cold.9+0x22/0x2f [ 3160.141004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3160.142604] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3160.144400] ? watchdog+0x40/0x40 [ 3160.145462] ? hrtimer_interrupt+0x128/0x2c0 [ 3160.146856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3160.148424] ? apic_timer_interrupt+0xf/0x20 [ 3160.150950] [ 3160.152033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3160.154416] do_raw_spin_lock+0x177/0x190 [ 3160.155879] _raw_spin_lock+0x21/0x30 [ 3160.157239] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3160.159822] ? do_raw_spin_unlock+0x75/0x190 [ 3160.161993] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3160.164845] ? do_wait_intr+0xf0/0xf0 [ 3160.166430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3160.168967] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3160.171659] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3160.174382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3160.176227] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3160.178161] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3160.179907] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3160.181529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3160.183982] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3160.187054] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3160.189571] lmv_intent_open+0x338/0x1150 [lmv] [ 3160.191530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3160.193568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3160.195909] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3160.197498] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3160.198973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3160.202589] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3160.204692] ? do_raw_spin_unlock+0x75/0x190 [ 3160.206572] ? _raw_spin_unlock+0x12/0x30 [ 3160.208399] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3160.210297] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3160.212315] ll_atomic_open+0x36a/0xd70 [lustre] [ 3160.214223] lookup_open+0x356/0xb40 [ 3160.215950] do_last+0x313/0xfc0 [ 3160.217283] ? nd_jump_root+0xe5/0x160 [ 3160.218637] ? path_init+0x437/0x520 [ 3160.220102] path_openat+0xf7/0x500 [ 3160.221493] ? do_raw_spin_unlock+0x75/0x190 [ 3160.223291] do_filp_open+0x99/0x140 [ 3160.224805] ? getname_flags+0x6e/0x330 [ 3160.226536] ? __check_object_size+0xff/0x256 [ 3160.228473] ? do_raw_spin_unlock+0x75/0x190 [ 3160.229801] ? _raw_spin_unlock+0x12/0x30 [ 3160.231259] do_sys_openat2+0x2b4/0x410 [ 3160.232778] do_sys_open+0x73/0xa0 [ 3160.234070] __x64_sys_openat+0x24/0x30 [ 3160.235251] do_syscall_64+0xc1/0x440 [ 3160.236726] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3160.239514] RIP: 0033:0x7fa749552332 [ 3160.240870] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3160.248428] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3160.251396] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3160.254710] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3160.259241] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3160.264273] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3160.267803] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3164.059029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3164.062138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3164.063708] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3164.098936] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3164.098949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3164.098952] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3164.098967] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3164.098971] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3164.098976] RAX: 000000000000330c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3164.098978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3164.098980] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3164.098982] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3164.098985] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3164.098988] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3164.098991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3164.098993] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3164.099001] Call Trace: [ 3164.099006] [ 3164.099011] ? show_regs.cold.9+0x22/0x2f [ 3164.099021] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3164.099027] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3164.099031] ? watchdog+0x40/0x40 [ 3164.099036] ? hrtimer_interrupt+0x128/0x2c0 [ 3164.099040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3164.099049] ? apic_timer_interrupt+0xf/0x20 [ 3164.099053] [ 3164.099056] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3164.099067] do_raw_spin_lock+0x177/0x190 [ 3164.099071] _raw_spin_lock+0x21/0x30 [ 3164.099075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3164.099346] ? do_raw_spin_unlock+0x75/0x190 [ 3164.099352] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3164.099654] ? do_wait_intr+0xf0/0xf0 [ 3164.099661] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3164.109157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3164.109240] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3164.110360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3164.110363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3164.110376] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3164.110381] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3164.110387] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3164.110389] RDX: 0000000000006cf9 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3164.110392] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3164.110394] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3164.110396] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3164.110453] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3164.110457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3164.110460] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3164.110467] Call Trace: [ 3164.110473] [ 3164.110479] ? show_regs.cold.9+0x22/0x2f [ 3164.110493] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3164.110501] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3164.110505] ? watchdog+0x40/0x40 [ 3164.110509] ? hrtimer_interrupt+0x128/0x2c0 [ 3164.110513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3164.110522] ? apic_timer_interrupt+0xf/0x20 [ 3164.110536] [ 3164.110539] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3164.110544] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3164.110549] do_raw_spin_lock+0x177/0x190 [ 3164.110553] _raw_spin_lock+0x21/0x30 [ 3164.110560] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3164.110886] obd_mod_free+0x46/0x140 [ptlrpc] [ 3164.111095] mdc_commit_open+0x91/0xb0 [mdc] [ 3164.111131] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3164.111999] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3164.114424] after_reply+0xe2c/0x1780 [ptlrpc] [ 3164.123319] mdc_close+0x53b/0xd70 [mdc] [ 3164.123440] lmv_close+0x18d/0x490 [lmv] [ 3164.123474] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3164.123603] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3164.123685] ll_md_close+0x274/0x800 [lustre] [ 3164.123900] ll_file_release+0x128/0xc50 [lustre] [ 3164.123982] __fput+0x10d/0x440 [ 3164.123992] ____fput+0x12/0x20 [ 3164.123997] task_work_run+0xc6/0x110 [ 3164.124005] exit_to_usermode_loop+0x1dd/0x1f0 [ 3164.124011] do_syscall_64+0x3d6/0x440 [ 3164.124016] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3164.124024] RIP: 0033:0x7fa749552cd8 [ 3164.124031] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3164.124035] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3164.124040] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3164.124043] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3164.124045] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3164.124047] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3164.124050] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3164.503401] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3164.506058] ? __next_timer_interrupt+0x160/0x160 [ 3164.508180] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3164.510823] ? do_wait_intr+0xf0/0xf0 [ 3164.518566] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3164.520423] kthread+0x1d1/0x200 [ 3164.522011] ? set_kthread_struct+0x70/0x70 [ 3164.524725] ret_from_fork+0x1f/0x30 [ 3165.154099] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1248s! [ 3165.156412] Showing busy workqueues and worker pools: [ 3165.158840] workqueue events: flags=0x0 [ 3165.160513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3165.162730] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3165.172676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3165.176235] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3165.180816] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.205098] pending: kfree_rcu_monitor [ 3165.209302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3165.212686] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3165.218443] workqueue events_long: flags=0x0 [ 3165.222736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.225804] pending: lu_objects_flush [obdclass] [ 3165.230128] workqueue events_unbound: flags=0x2 [ 3165.235807] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3165.241141] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3165.252382] workqueue events_power_efficient: flags=0x80 [ 3165.259445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.262924] pending: check_lifetime [ 3165.265263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.270278] pending: neigh_periodic_work [ 3165.272187] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.275853] pending: neigh_periodic_work [ 3165.281189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.286151] pending: do_cache_clean [ 3165.288350] workqueue mm_percpu_wq: flags=0x8 [ 3165.290996] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.294870] pending: vmstat_update [ 3165.296781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.300434] pending: vmstat_update [ 3165.302123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3165.310421] pending: vmstat_update [ 3165.313278] workqueue writeback: flags=0x4a [ 3165.315775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3165.319321] pending: wb_workfn, wb_workfn [ 3165.321934] workqueue xprtiod: flags=0xa [ 3165.325758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3165.332526] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3165.343115] workqueue ipv6_addrconf: flags=0x40008 [ 3165.346927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3165.351118] pending: addrconf_verify_work [ 3165.353407] workqueue knbd0-recv: flags=0x1a [ 3165.355792] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3165.358910] in-flight: 651:recv_work [ 3165.361028] workqueue ptlrpc_pinger: flags=0x2 [ 3165.364253] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3165.367477] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3165.373364] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1248s workers=4 idle: 37 31846 28897 [ 3165.380651] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3155s workers=2 idle: 653 [ 3180.061540] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3180.068893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3180.112036] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3180.115922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3180.119321] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3180.121750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.123829] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3180.130279] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3180.133334] RAX: 0000000000002c8c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3180.135838] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3180.138416] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3180.141090] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3180.143741] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3180.146262] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3180.149537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3180.151684] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3180.154347] Call Trace: [ 3180.155343] [ 3180.156183] ? show_regs.cold.9+0x22/0x2f [ 3180.157662] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3180.160085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3180.163929] ? watchdog+0x40/0x40 [ 3180.166768] ? hrtimer_interrupt+0x128/0x2c0 [ 3180.171103] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3180.174349] ? apic_timer_interrupt+0xf/0x20 [ 3180.175766] [ 3180.177861] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.184747] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3180.188415] do_raw_spin_lock+0x177/0x190 [ 3180.189764] _raw_spin_lock+0x21/0x30 [ 3180.194161] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3180.205166] process_one_work+0x2c8/0x700 [ 3180.209597] worker_thread+0x296/0x6e0 [ 3180.210767] ? rescuer_thread+0x570/0x570 [ 3180.218069] kthread+0x1d1/0x200 [ 3180.220616] ? set_kthread_struct+0x70/0x70 [ 3180.222936] ret_from_fork+0x1f/0x30 [ 3192.054179] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3192.058063] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3192.059067] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3192.059075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3192.059261] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3192.059268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3192.059271] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3192.059284] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3192.059288] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3192.059292] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3192.059295] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3192.059297] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3192.059300] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3192.059303] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3192.059306] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3192.059309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3192.059311] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3192.059319] Call Trace: [ 3192.059323] [ 3192.059328] ? show_regs.cold.9+0x22/0x2f [ 3192.059337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3192.059343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3192.059347] ? watchdog+0x40/0x40 [ 3192.059351] ? hrtimer_interrupt+0x128/0x2c0 [ 3192.059355] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3192.059361] ? apic_timer_interrupt+0xf/0x20 [ 3192.059367] [ 3192.059369] ? native_safe_halt+0x2e/0x50 [ 3192.059373] kvm_wait+0xcd/0xe0 [ 3192.059380] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3192.059388] do_raw_spin_lock+0x177/0x190 [ 3192.059393] _raw_spin_lock+0x21/0x30 [ 3192.059397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3192.059665] ? do_raw_spin_unlock+0x75/0x190 [ 3192.059671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3192.059866] ? do_wait_intr+0xf0/0xf0 [ 3192.059873] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3192.060137] mdc_close+0x53b/0xd70 [mdc] [ 3192.060165] lmv_close+0x18d/0x490 [lmv] [ 3192.060191] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3192.060393] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3192.060445] ll_md_close+0x274/0x800 [lustre] [ 3192.060499] ll_file_release+0x128/0xc50 [lustre] [ 3192.060552] __fput+0x10d/0x440 [ 3192.060559] ____fput+0x12/0x20 [ 3192.060562] task_work_run+0xc6/0x110 [ 3192.060569] exit_to_usermode_loop+0x1dd/0x1f0 [ 3192.060578] do_syscall_64+0x3d6/0x440 [ 3192.060583] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3192.060590] RIP: 0033:0x7fa749552cd8 [ 3192.060616] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3192.060620] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3192.060624] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3192.060632] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3192.060634] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3192.060636] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3192.060638] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3192.064358] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3192.064435] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3192.064453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3192.064457] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3192.064471] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3192.064475] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3192.064479] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3192.064481] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3192.064483] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3192.064486] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3192.064488] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3192.064491] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3192.064493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3192.064496] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3192.064504] Call Trace: [ 3192.064509] [ 3192.064514] ? show_regs.cold.9+0x22/0x2f [ 3192.064524] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3192.064530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3192.064534] ? watchdog+0x40/0x40 [ 3192.064538] ? hrtimer_interrupt+0x128/0x2c0 [ 3192.064541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3192.064548] ? apic_timer_interrupt+0xf/0x20 [ 3192.064552] [ 3192.064554] ? native_safe_halt+0x2e/0x50 [ 3192.064557] kvm_wait+0xcd/0xe0 [ 3192.064564] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3192.064572] do_raw_spin_lock+0x177/0x190 [ 3192.064576] _raw_spin_lock+0x21/0x30 [ 3192.064580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3192.064849] ? do_raw_spin_unlock+0x75/0x190 [ 3192.064855] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3192.065072] ? do_wait_intr+0xf0/0xf0 [ 3192.065082] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3192.065275] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3192.065491] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3192.073133] Modules linked in: [ 3192.114313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.114437] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3192.114472] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3192.119119] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3192.119190] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3192.119197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3192.119200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3192.119214] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3192.119218] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3192.119222] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3192.119224] RDX: 0000000000004919 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3192.119226] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3192.119228] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3192.119230] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3192.119233] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3192.119235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3192.119237] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3192.119245] Call Trace: [ 3192.119251] [ 3192.119257] ? show_regs.cold.9+0x22/0x2f [ 3192.119265] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3192.119271] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3192.119275] ? watchdog+0x40/0x40 [ 3192.119280] ? hrtimer_interrupt+0x128/0x2c0 [ 3192.119284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3192.119292] ? apic_timer_interrupt+0xf/0x20 [ 3192.119297] [ 3192.119300] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3192.119305] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3192.119310] do_raw_spin_lock+0x177/0x190 [ 3192.119314] _raw_spin_lock+0x21/0x30 [ 3192.119320] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3192.119587] obd_mod_free+0x46/0x140 [ptlrpc] [ 3192.119855] mdc_commit_open+0x91/0xb0 [mdc] [ 3192.119893] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3192.120112] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3192.120314] after_reply+0xe2c/0x1780 [ptlrpc] [ 3192.120521] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3192.124435] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3192.124493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.124610] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3192.124914] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3192.124946] lmv_intent_open+0x338/0x1150 [lmv] [ 3192.124978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.125075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.125156] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3192.125179] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3192.125261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.125340] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3192.125418] ? do_raw_spin_unlock+0x75/0x190 [ 3192.125426] ? _raw_spin_unlock+0x12/0x30 [ 3192.125433] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3192.125461] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3192.125487] ll_atomic_open+0x36a/0xd70 [lustre] [ 3192.125569] lookup_open+0x356/0xb40 [ 3192.125578] do_last+0x313/0xfc0 [ 3192.125583] ? nd_jump_root+0xe5/0x160 [ 3192.125587] ? path_init+0x437/0x520 [ 3192.125592] path_openat+0xf7/0x500 [ 3192.125597] ? do_raw_spin_unlock+0x75/0x190 [ 3192.125602] do_filp_open+0x99/0x140 [ 3192.125608] ? getname_flags+0x6e/0x330 [ 3192.125613] ? __check_object_size+0xff/0x256 [ 3192.125619] ? do_raw_spin_unlock+0x75/0x190 [ 3192.125624] ? _raw_spin_unlock+0x12/0x30 [ 3192.125628] do_sys_openat2+0x2b4/0x410 [ 3192.125632] do_sys_open+0x73/0xa0 [ 3192.125635] __x64_sys_openat+0x24/0x30 [ 3192.125640] do_syscall_64+0xc1/0x440 [ 3192.125647] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3192.125654] RIP: 0033:0x7fa749552332 [ 3192.125700] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3192.125706] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3192.125710] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3192.125713] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3192.125715] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3192.125717] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3192.125719] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3192.707075] ? __next_timer_interrupt+0x160/0x160 [ 3192.713480] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3192.717520] ? do_wait_intr+0xf0/0xf0 [ 3192.719679] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3192.723419] kthread+0x1d1/0x200 [ 3192.724699] ? set_kthread_struct+0x70/0x70 [ 3192.726637] ret_from_fork+0x1f/0x30 [ 3195.875854] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1278s! [ 3195.879360] Showing busy workqueues and worker pools: [ 3195.880918] workqueue events: flags=0x0 [ 3195.881872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3195.884045] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3195.888549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3195.893643] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3195.898851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.902642] pending: kfree_rcu_monitor [ 3195.905233] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3195.909191] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3195.914144] workqueue events_long: flags=0x0 [ 3195.917859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.920150] pending: lu_objects_flush [obdclass] [ 3195.921537] workqueue events_unbound: flags=0x2 [ 3195.927450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3195.931342] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3195.935754] workqueue events_power_efficient: flags=0x80 [ 3195.937189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.940095] pending: check_lifetime [ 3195.941583] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.945349] pending: neigh_periodic_work [ 3195.947131] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.949872] pending: neigh_periodic_work [ 3195.951391] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.958454] pending: do_cache_clean [ 3195.959605] workqueue mm_percpu_wq: flags=0x8 [ 3195.961922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.965159] pending: vmstat_update [ 3195.967328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.971037] pending: vmstat_update [ 3195.972238] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3195.974274] pending: vmstat_update [ 3195.975638] workqueue writeback: flags=0x4a [ 3195.980725] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3195.988203] pending: wb_workfn, wb_workfn [ 3195.991332] workqueue xprtiod: flags=0xa [ 3195.994589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3196.000603] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3196.003684] workqueue ipv6_addrconf: flags=0x40008 [ 3196.005849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3196.008495] pending: addrconf_verify_work [ 3196.015974] workqueue knbd0-recv: flags=0x1a [ 3196.017490] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3196.021184] in-flight: 651:recv_work [ 3196.023666] workqueue ptlrpc_pinger: flags=0x2 [ 3196.026305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3196.029060] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3196.035401] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1278s workers=4 idle: 37 31846 28897 [ 3196.038263] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3185s workers=2 idle: 653 [ 3208.057080] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3208.059433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3208.097862] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3208.111144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3208.118833] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3208.125643] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3208.130194] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3208.145603] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3208.150305] RAX: 00000000000055b9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3208.156821] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3208.162087] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3208.168945] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3208.175006] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3208.185760] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3208.190004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3208.196126] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3208.199639] Call Trace: [ 3208.200977] [ 3208.201896] ? show_regs.cold.9+0x22/0x2f [ 3208.203999] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3208.208015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3208.209420] ? watchdog+0x40/0x40 [ 3208.210345] ? hrtimer_interrupt+0x128/0x2c0 [ 3208.213292] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3208.220228] ? apic_timer_interrupt+0xf/0x20 [ 3208.222130] [ 3208.224301] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3208.226994] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3208.229759] do_raw_spin_lock+0x177/0x190 [ 3208.231438] _raw_spin_lock+0x21/0x30 [ 3208.233412] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3208.237701] process_one_work+0x2c8/0x700 [ 3208.242069] worker_thread+0x296/0x6e0 [ 3208.243956] ? rescuer_thread+0x570/0x570 [ 3208.246674] kthread+0x1d1/0x200 [ 3208.251287] ? set_kthread_struct+0x70/0x70 [ 3208.255719] ret_from_fork+0x1f/0x30 [ 3220.056285] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3220.059468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3220.063106] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3220.066009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3220.066025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3220.067003] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3220.067011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3220.067014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3220.067032] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3220.067037] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3220.067041] RAX: 0000000000007b86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3220.067044] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3220.067046] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3220.067049] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3220.067051] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3220.067054] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3220.067057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3220.067060] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3220.067068] Call Trace: [ 3220.067076] [ 3220.067083] ? show_regs.cold.9+0x22/0x2f [ 3220.067179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3220.067186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3220.067190] ? watchdog+0x40/0x40 [ 3220.067195] ? hrtimer_interrupt+0x128/0x2c0 [ 3220.067199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3220.067208] ? apic_timer_interrupt+0xf/0x20 [ 3220.067213] [ 3220.067216] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3220.067225] do_raw_spin_lock+0x177/0x190 [ 3220.067230] _raw_spin_lock+0x21/0x30 [ 3220.067236] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3220.069289] ? do_raw_spin_unlock+0x75/0x190 [ 3220.069300] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3220.069917] ? do_wait_intr+0xf0/0xf0 [ 3220.069927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3220.070206] mdc_close+0x53b/0xd70 [mdc] [ 3220.070246] lmv_close+0x18d/0x490 [lmv] [ 3220.070314] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3220.070429] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3220.071605] ll_md_close+0x274/0x800 [lustre] [ 3220.071861] ll_file_release+0x128/0xc50 [lustre] [ 3220.071953] __fput+0x10d/0x440 [ 3220.071964] ____fput+0x12/0x20 [ 3220.071969] task_work_run+0xc6/0x110 [ 3220.071977] exit_to_usermode_loop+0x1dd/0x1f0 [ 3220.071985] do_syscall_64+0x3d6/0x440 [ 3220.071990] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3220.072000] RIP: 0033:0x7fa749552cd8 [ 3220.072009] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3220.072013] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3220.072019] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3220.072021] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3220.072023] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3220.072026] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3220.072028] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3220.093860] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3220.093873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3220.093876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3220.093890] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3220.093894] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3220.093900] RAX: 0000000000007b38 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3220.093903] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3220.093905] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3220.093908] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3220.093910] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3220.093913] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3220.093916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3220.093918] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3220.093925] Call Trace: [ 3220.093930] [ 3220.093935] ? show_regs.cold.9+0x22/0x2f [ 3220.093944] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3220.093950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3220.093955] ? watchdog+0x40/0x40 [ 3220.093959] ? hrtimer_interrupt+0x128/0x2c0 [ 3220.093963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3220.093971] ? apic_timer_interrupt+0xf/0x20 [ 3220.093976] [ 3220.093979] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3220.093987] do_raw_spin_lock+0x177/0x190 [ 3220.093992] _raw_spin_lock+0x21/0x30 [ 3220.093997] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3220.111158] Modules linked in: [ 3220.114514] ? do_raw_spin_unlock+0x75/0x190 [ 3220.114538] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3220.115015] ? do_wait_intr+0xf0/0xf0 [ 3220.115029] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3220.115341] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3220.115626] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3220.116165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.116333] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3220.116393] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3220.116420] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3220.116446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.116552] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3220.116879] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3220.116906] lmv_intent_open+0x338/0x1150 [lmv] [ 3220.116936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.117036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.117122] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3220.117145] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3220.117312] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.117411] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3220.117515] ? do_raw_spin_unlock+0x75/0x190 [ 3220.117522] ? _raw_spin_unlock+0x12/0x30 [ 3220.117529] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3220.117572] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3220.117594] ll_atomic_open+0x36a/0xd70 [lustre] [ 3220.117827] lookup_open+0x356/0xb40 [ 3220.117841] do_last+0x313/0xfc0 [ 3220.117850] ? nd_jump_root+0xe5/0x160 [ 3220.117855] ? path_init+0x437/0x520 [ 3220.117860] path_openat+0xf7/0x500 [ 3220.117866] ? do_raw_spin_unlock+0x75/0x190 [ 3220.117871] do_filp_open+0x99/0x140 [ 3220.117877] ? getname_flags+0x6e/0x330 [ 3220.117882] ? __check_object_size+0xff/0x256 [ 3220.117890] ? do_raw_spin_unlock+0x75/0x190 [ 3220.117893] ? _raw_spin_unlock+0x12/0x30 [ 3220.117898] do_sys_openat2+0x2b4/0x410 [ 3220.117903] do_sys_open+0x73/0xa0 [ 3220.117908] __x64_sys_openat+0x24/0x30 [ 3220.117912] do_syscall_64+0xc1/0x440 [ 3220.117919] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3220.117926] RIP: 0033:0x7fa749552332 [ 3220.117934] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3220.117939] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3220.117945] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3220.117949] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3220.159452] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3220.159550] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3220.159557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3220.159560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3220.159574] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3220.159578] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3220.159583] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3220.159585] RDX: 0000000000004bd6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3220.159588] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3220.159591] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3220.159593] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3220.159597] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3220.159600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3220.159602] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3220.159610] Call Trace: [ 3220.159616] [ 3220.159623] ? show_regs.cold.9+0x22/0x2f [ 3220.159633] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3220.159640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3220.159644] ? watchdog+0x40/0x40 [ 3220.159649] ? hrtimer_interrupt+0x128/0x2c0 [ 3220.159653] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3220.159662] ? apic_timer_interrupt+0xf/0x20 [ 3220.159666] [ 3220.159670] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3220.159675] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3220.159679] do_raw_spin_lock+0x177/0x190 [ 3220.159683] _raw_spin_lock+0x21/0x30 [ 3220.159689] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3220.159985] obd_mod_free+0x46/0x140 [ptlrpc] [ 3220.160176] mdc_commit_open+0x91/0xb0 [mdc] [ 3220.160213] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3220.160408] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3220.160776] after_reply+0xe2c/0x1780 [ptlrpc] [ 3220.161027] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3220.165828] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3220.165840] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3220.165843] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3220.576594] ? __next_timer_interrupt+0x160/0x160 [ 3220.577944] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3220.579452] ? do_wait_intr+0xf0/0xf0 [ 3220.580898] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3220.583086] kthread+0x1d1/0x200 [ 3220.584506] ? set_kthread_struct+0x70/0x70 [ 3220.585928] ret_from_fork+0x1f/0x30 [ 3226.084627] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1308s! [ 3226.089731] Showing busy workqueues and worker pools: [ 3226.092183] workqueue events: flags=0x0 [ 3226.093557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3226.098078] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3226.119091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3226.126637] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3226.138309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.145581] pending: kfree_rcu_monitor [ 3226.147875] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3226.155798] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3226.162084] workqueue events_long: flags=0x0 [ 3226.164662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.167988] pending: lu_objects_flush [obdclass] [ 3226.170876] workqueue events_unbound: flags=0x2 [ 3226.175700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3226.179908] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3226.194268] workqueue events_power_efficient: flags=0x80 [ 3226.200192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.205982] pending: check_lifetime [ 3226.210235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.214513] pending: neigh_periodic_work [ 3226.219421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.225065] pending: neigh_periodic_work [ 3226.229907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.235955] pending: do_cache_clean [ 3226.238066] workqueue mm_percpu_wq: flags=0x8 [ 3226.240287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.246865] pending: vmstat_update [ 3226.248498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.253526] pending: vmstat_update [ 3226.256078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3226.259596] pending: vmstat_update [ 3226.261923] workqueue writeback: flags=0x4a [ 3226.264868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3226.270060] pending: wb_workfn, wb_workfn [ 3226.272393] workqueue xprtiod: flags=0xa [ 3226.274465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3226.276932] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3226.280425] workqueue ipv6_addrconf: flags=0x40008 [ 3226.282452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3226.290275] pending: addrconf_verify_work [ 3226.292134] workqueue knbd0-recv: flags=0x1a [ 3226.298244] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3226.302932] in-flight: 651:recv_work [ 3226.305013] workqueue ptlrpc_pinger: flags=0x2 [ 3226.307802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3226.314960] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3226.319905] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1309s workers=4 idle: 37 31846 28897 [ 3226.324348] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3216s workers=2 idle: 653 [ 3233.718607] rcu: INFO: rcu_sched self-detected stall on CPU [ 3233.720873] rcu: 2-...!: (706442 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 3233.725837] (t=1320029 jiffies g=258701 q=1928) [ 3233.727987] rcu: rcu_sched kthread starved for 1316824 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3233.732812] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3233.739256] rcu: RCU grace-period kthread stack dump: [ 3233.742951] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3233.749535] Call Trace: [ 3233.751540] __schedule+0x351/0xcb0 [ 3233.753776] schedule+0xc0/0x180 [ 3233.755819] schedule_timeout+0xb4/0x190 [ 3233.759464] ? do_raw_spin_unlock+0x75/0x190 [ 3233.762774] ? __next_timer_interrupt+0x160/0x160 [ 3233.764696] ? prepare_to_swait_event+0xf3/0x140 [ 3233.766923] rcu_gp_kthread+0x8dd/0x15b0 [ 3233.768638] ? force_qs_rnp+0x2c0/0x2c0 [ 3233.770373] kthread+0x1d1/0x200 [ 3233.771938] ? set_kthread_struct+0x70/0x70 [ 3233.773698] ret_from_fork+0x1f/0x30 [ 3233.775511] rcu: Stack dump where RCU GP kthread last ran: [ 3233.778224] Sending NMI from CPU 2 to CPUs 3: [ 3233.781662] NMI backtrace for cpu 3 [ 3233.781668] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3233.781670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3233.781671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3233.781673] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3233.781675] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 [ 3233.781677] RAX: 0000000000006c29 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3233.781683] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3233.781684] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3233.781685] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3233.781686] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3233.781687] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3233.781689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3233.781690] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3233.781691] Call Trace: [ 3233.781692] [ 3233.781692] ? show_regs.cold.9+0x22/0x2f [ 3233.781693] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3233.781694] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3233.781695] ? nmi_handle+0xbd/0x210 [ 3233.781696] ? default_do_nmi+0x4b/0x1b0 [ 3233.781697] ? do_nmi+0x280/0x380 [ 3233.781698] ? end_repeat_nmi+0x16/0x1f [ 3233.781699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3233.781700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3233.781701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3233.781702] [ 3233.781703] do_raw_spin_lock+0x177/0x190 [ 3233.781704] _raw_spin_lock+0x21/0x30 [ 3233.781705] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3233.781705] ? do_raw_spin_unlock+0x75/0x190 [ 3233.781706] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3233.781710] ? do_wait_intr+0xf0/0xf0 [ 3233.781711] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3233.781712] mdc_close+0x53b/0xd70 [mdc] [ 3233.781713] lmv_close+0x18d/0x490 [lmv] [ 3233.781714] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3233.781715] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3233.781716] ll_md_close+0x274/0x800 [lustre] [ 3233.781717] ll_file_release+0x128/0xc50 [lustre] [ 3233.781718] __fput+0x10d/0x440 [ 3233.781718] ____fput+0x12/0x20 [ 3233.781719] task_work_run+0xc6/0x110 [ 3233.781720] exit_to_usermode_loop+0x1dd/0x1f0 [ 3233.781721] do_syscall_64+0x3d6/0x440 [ 3233.781722] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3233.781723] RIP: 0033:0x7fa749552cd8 [ 3233.781724] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3233.781726] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3233.781728] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3233.781729] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3233.781730] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3233.781731] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3233.781732] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3233.783762] NMI backtrace for cpu 2 [ 3233.924426] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3233.928430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3233.931585] Call Trace: [ 3233.932691] [ 3233.933637] ? dump_stack+0xbb/0x10e [ 3233.934958] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3233.936898] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3233.938618] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3233.940488] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3233.942276] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3233.943783] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3233.945319] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3233.947258] ? update_process_times+0x6d/0xc0 [ 3233.948778] ? tick_sched_handle.isra.18+0x27/0x90 [ 3233.950820] ? tick_sched_timer+0x53/0xb0 [ 3233.952559] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3233.954681] ? tick_sched_handle.isra.18+0x90/0x90 [ 3233.956645] ? hrtimer_interrupt+0x128/0x2c0 [ 3233.958203] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3233.959850] ? apic_timer_interrupt+0xf/0x20 [ 3233.961506] [ 3233.962398] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3233.964376] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3233.966435] ? do_raw_spin_lock+0x177/0x190 [ 3233.967702] ? _raw_spin_lock+0x21/0x30 [ 3233.968763] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3233.970965] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3233.973229] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3233.975116] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3233.980213] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3233.984550] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3233.987541] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3233.989547] ? __next_timer_interrupt+0x160/0x160 [ 3233.991533] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3233.993860] ? do_wait_intr+0xf0/0xf0 [ 3233.995430] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3233.998504] ? kthread+0x1d1/0x200 [ 3234.000247] ? set_kthread_struct+0x70/0x70 [ 3234.002121] ? ret_from_fork+0x1f/0x30 [ 3236.059413] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3236.064634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3236.098711] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3236.104613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3236.108926] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3236.112323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3236.114732] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3236.122935] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3236.126294] RAX: 0000000000004986 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3236.129640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3236.132997] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3236.137301] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3236.140889] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3236.144124] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3236.147747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3236.150304] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3236.153315] Call Trace: [ 3236.154232] [ 3236.155296] ? show_regs.cold.9+0x22/0x2f [ 3236.157135] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3236.159665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3236.161614] ? watchdog+0x40/0x40 [ 3236.163405] ? hrtimer_interrupt+0x128/0x2c0 [ 3236.165367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3236.167182] ? apic_timer_interrupt+0xf/0x20 [ 3236.168669] [ 3236.169494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3236.171455] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3236.173829] do_raw_spin_lock+0x177/0x190 [ 3236.175583] _raw_spin_lock+0x21/0x30 [ 3236.177846] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3236.180515] process_one_work+0x2c8/0x700 [ 3236.182042] worker_thread+0x296/0x6e0 [ 3236.183990] ? rescuer_thread+0x570/0x570 [ 3236.185914] kthread+0x1d1/0x200 [ 3236.187328] ? set_kthread_struct+0x70/0x70 [ 3236.189369] ret_from_fork+0x1f/0x30 [ 3252.056340] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3252.062838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3252.098207] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3252.104548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3252.109219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3252.111373] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3252.124081] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3252.129874] RAX: 0000000000007e87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3252.133015] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3252.136420] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3252.139393] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3252.142906] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3252.146231] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3252.150015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3252.152198] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3252.154773] Call Trace: [ 3252.155542] [ 3252.156554] ? show_regs.cold.9+0x22/0x2f [ 3252.158381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3252.160468] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3252.162830] ? watchdog+0x40/0x40 [ 3252.164286] ? hrtimer_interrupt+0x128/0x2c0 [ 3252.166434] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3252.169092] ? apic_timer_interrupt+0xf/0x20 [ 3252.171278] [ 3252.172350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3252.174897] do_raw_spin_lock+0x177/0x190 [ 3252.176914] _raw_spin_lock+0x21/0x30 [ 3252.178666] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3252.181257] ? do_raw_spin_unlock+0x75/0x190 [ 3252.183332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3252.186312] ? do_wait_intr+0xf0/0xf0 [ 3252.187852] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3252.190277] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3252.192352] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3252.194663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3252.197191] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3252.200191] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3252.202266] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3252.204267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3252.207237] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3252.210201] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3252.212843] lmv_intent_open+0x338/0x1150 [lmv] [ 3252.214868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3252.217360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3252.219865] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3252.221869] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3252.224754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3252.230015] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3252.231792] ? do_raw_spin_unlock+0x75/0x190 [ 3252.237292] ? _raw_spin_unlock+0x12/0x30 [ 3252.238332] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3252.240034] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3252.242295] ll_atomic_open+0x36a/0xd70 [lustre] [ 3252.244294] lookup_open+0x356/0xb40 [ 3252.245587] do_last+0x313/0xfc0 [ 3252.247095] ? nd_jump_root+0xe5/0x160 [ 3252.248526] ? path_init+0x437/0x520 [ 3252.250375] path_openat+0xf7/0x500 [ 3252.252189] ? do_raw_spin_unlock+0x75/0x190 [ 3252.253566] do_filp_open+0x99/0x140 [ 3252.254834] ? getname_flags+0x6e/0x330 [ 3252.259684] ? __check_object_size+0xff/0x256 [ 3252.262261] ? do_raw_spin_unlock+0x75/0x190 [ 3252.264155] ? _raw_spin_unlock+0x12/0x30 [ 3252.265566] do_sys_openat2+0x2b4/0x410 [ 3252.266877] do_sys_open+0x73/0xa0 [ 3252.268182] __x64_sys_openat+0x24/0x30 [ 3252.269624] do_syscall_64+0xc1/0x440 [ 3252.271230] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3252.273359] RIP: 0033:0x7fa749552332 [ 3252.274602] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3252.282891] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3252.286139] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3252.289831] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3252.293036] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3252.299592] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3252.303611] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3256.800505] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1339s! [ 3256.819033] Showing busy workqueues and worker pools: [ 3256.822052] workqueue events: flags=0x0 [ 3256.825169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3256.833383] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3256.846770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3256.852962] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3256.861838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.871282] pending: kfree_rcu_monitor [ 3256.876618] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3256.880777] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3256.883091] workqueue events_long: flags=0x0 [ 3256.889951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.898049] pending: lu_objects_flush [obdclass] [ 3256.900881] workqueue events_unbound: flags=0x2 [ 3256.904404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3256.909777] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3256.914951] workqueue events_power_efficient: flags=0x80 [ 3256.919884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.926824] pending: check_lifetime [ 3256.928351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.932268] pending: neigh_periodic_work [ 3256.933952] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.937862] pending: neigh_periodic_work [ 3256.940011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.945886] pending: do_cache_clean [ 3256.948486] workqueue mm_percpu_wq: flags=0x8 [ 3256.950022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.952976] pending: vmstat_update [ 3256.954351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.957357] pending: vmstat_update [ 3256.959773] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3256.962693] pending: vmstat_update [ 3256.963961] workqueue writeback: flags=0x4a [ 3256.965786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3256.967942] pending: wb_workfn, wb_workfn [ 3256.969424] workqueue xprtiod: flags=0xa [ 3256.971329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3256.975655] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3256.982078] workqueue ipv6_addrconf: flags=0x40008 [ 3256.986700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3256.992430] pending: addrconf_verify_work [ 3256.994839] workqueue knbd0-recv: flags=0x1a [ 3256.997308] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3257.003135] in-flight: 651:recv_work [ 3257.008279] workqueue ptlrpc_pinger: flags=0x2 [ 3257.017033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3257.026045] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3257.032163] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1339s workers=4 idle: 37 31846 28897 [ 3257.043853] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3246s workers=2 idle: 653 [ 3260.058017] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3260.060112] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3260.061718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3260.061788] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3260.061795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3260.061799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3260.061812] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3260.061815] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3260.061819] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3260.061821] RDX: 0000000000001cbd RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3260.061824] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3260.061825] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3260.061827] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3260.061830] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3260.061833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3260.061835] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3260.061843] Call Trace: [ 3260.064975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3260.065197] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3260.065204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3260.065207] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3260.065220] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3260.065224] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3260.065228] RAX: 0000000000005676 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3260.065230] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3260.065232] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3260.065234] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3260.065236] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3260.065239] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3260.065242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3260.065243] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3260.065251] Call Trace: [ 3260.065257] [ 3260.065263] ? show_regs.cold.9+0x22/0x2f [ 3260.065272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3260.065277] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3260.065281] ? watchdog+0x40/0x40 [ 3260.065285] ? hrtimer_interrupt+0x128/0x2c0 [ 3260.065289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3260.065298] ? apic_timer_interrupt+0xf/0x20 [ 3260.065302] [ 3260.065305] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3260.065311] do_raw_spin_lock+0x177/0x190 [ 3260.065315] _raw_spin_lock+0x21/0x30 [ 3260.065319] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3260.088402] [ 3260.088414] ? show_regs.cold.9+0x22/0x2f [ 3260.088428] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3260.088436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3260.088440] ? watchdog+0x40/0x40 [ 3260.088445] ? hrtimer_interrupt+0x128/0x2c0 [ 3260.088449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3260.088457] ? apic_timer_interrupt+0xf/0x20 [ 3260.088461] [ 3260.088464] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3260.088470] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3260.088475] do_raw_spin_lock+0x177/0x190 [ 3260.088479] _raw_spin_lock+0x21/0x30 [ 3260.088488] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3260.088782] obd_mod_free+0x46/0x140 [ptlrpc] [ 3260.089197] mdc_commit_open+0x91/0xb0 [mdc] [ 3260.089236] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3260.089454] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3260.089730] after_reply+0xe2c/0x1780 [ptlrpc] [ 3260.089954] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3260.090294] ? __next_timer_interrupt+0x160/0x160 [ 3260.090304] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3260.090528] ? do_wait_intr+0xf0/0xf0 [ 3260.090537] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3260.090951] kthread+0x1d1/0x200 [ 3260.090956] ? set_kthread_struct+0x70/0x70 [ 3260.090961] ret_from_fork+0x1f/0x30 [ 3260.310518] ? do_raw_spin_unlock+0x75/0x190 [ 3260.312948] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3260.316791] ? do_wait_intr+0xf0/0xf0 [ 3260.318626] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3260.321011] mdc_close+0x53b/0xd70 [mdc] [ 3260.323296] lmv_close+0x18d/0x490 [lmv] [ 3260.325341] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3260.327678] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3260.330516] ll_md_close+0x274/0x800 [lustre] [ 3260.333006] ll_file_release+0x128/0xc50 [lustre] [ 3260.335618] __fput+0x10d/0x440 [ 3260.337290] ____fput+0x12/0x20 [ 3260.338919] task_work_run+0xc6/0x110 [ 3260.340934] exit_to_usermode_loop+0x1dd/0x1f0 [ 3260.343426] do_syscall_64+0x3d6/0x440 [ 3260.345369] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3260.347977] RIP: 0033:0x7fa749552cd8 [ 3260.350351] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3260.360095] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3260.364379] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3260.367833] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3260.371417] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3260.374784] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3260.378446] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3264.057092] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3264.064938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3264.089552] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3264.096028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3264.100691] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3264.104566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3264.107330] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3264.120205] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3264.123687] RAX: 0000000000001043 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3264.127025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3264.130131] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3264.133191] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3264.136190] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3264.139104] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3264.145126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3264.148287] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3264.152325] Call Trace: [ 3264.153447] [ 3264.154685] ? show_regs.cold.9+0x22/0x2f [ 3264.158762] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3264.162218] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3264.164143] ? watchdog+0x40/0x40 [ 3264.165287] ? hrtimer_interrupt+0x128/0x2c0 [ 3264.167226] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3264.169223] ? apic_timer_interrupt+0xf/0x20 [ 3264.171448] [ 3264.172440] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3264.174415] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3264.182248] do_raw_spin_lock+0x177/0x190 [ 3264.184101] _raw_spin_lock+0x21/0x30 [ 3264.185451] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3264.188736] process_one_work+0x2c8/0x700 [ 3264.190404] worker_thread+0x296/0x6e0 [ 3264.192092] ? rescuer_thread+0x570/0x570 [ 3264.194124] kthread+0x1d1/0x200 [ 3264.196464] ? set_kthread_struct+0x70/0x70 [ 3264.199204] ret_from_fork+0x1f/0x30 [ 3284.055824] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3284.062343] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3284.100886] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3284.114593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3284.123080] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3284.128944] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3284.137794] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3284.140559] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3284.143850] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3284.146858] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3284.149482] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3284.152068] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3284.154613] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3284.159117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3284.161632] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3284.164806] Call Trace: [ 3284.165971] [ 3284.166740] ? show_regs.cold.9+0x22/0x2f [ 3284.168406] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3284.170752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3284.173759] ? watchdog+0x40/0x40 [ 3284.175669] ? hrtimer_interrupt+0x128/0x2c0 [ 3284.178366] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3284.180553] ? apic_timer_interrupt+0xf/0x20 [ 3284.182258] [ 3284.183810] ? native_safe_halt+0x2e/0x50 [ 3284.186523] kvm_wait+0xcd/0xe0 [ 3284.189824] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3284.194824] do_raw_spin_lock+0x177/0x190 [ 3284.198696] _raw_spin_lock+0x21/0x30 [ 3284.200646] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3284.205498] ? do_raw_spin_unlock+0x75/0x190 [ 3284.209137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3284.211050] ? do_wait_intr+0xf0/0xf0 [ 3284.218442] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3284.222618] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3284.225280] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3284.229538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.235594] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3284.243595] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3284.249597] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3284.255657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.262734] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3284.269501] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3284.274577] lmv_intent_open+0x338/0x1150 [lmv] [ 3284.278648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.283010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.286007] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3284.290867] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3284.296785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.303026] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3284.306855] ? do_raw_spin_unlock+0x75/0x190 [ 3284.311279] ? _raw_spin_unlock+0x12/0x30 [ 3284.314626] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3284.323248] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3284.326519] ll_atomic_open+0x36a/0xd70 [lustre] [ 3284.330684] lookup_open+0x356/0xb40 [ 3284.334486] do_last+0x313/0xfc0 [ 3284.339125] ? nd_jump_root+0xe5/0x160 [ 3284.341104] ? path_init+0x437/0x520 [ 3284.343550] path_openat+0xf7/0x500 [ 3284.345735] ? do_raw_spin_unlock+0x75/0x190 [ 3284.348247] do_filp_open+0x99/0x140 [ 3284.350508] ? getname_flags+0x6e/0x330 [ 3284.353188] ? __check_object_size+0xff/0x256 [ 3284.359522] ? do_raw_spin_unlock+0x75/0x190 [ 3284.362970] ? _raw_spin_unlock+0x12/0x30 [ 3284.365872] do_sys_openat2+0x2b4/0x410 [ 3284.369874] do_sys_open+0x73/0xa0 [ 3284.373525] __x64_sys_openat+0x24/0x30 [ 3284.376371] do_syscall_64+0xc1/0x440 [ 3284.381398] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3284.391115] RIP: 0033:0x7fa749552332 [ 3284.393872] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3284.414354] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3284.421371] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3284.425654] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3284.428718] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3284.435239] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3284.442551] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3287.520659] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1370s! [ 3287.525388] Showing busy workqueues and worker pools: [ 3287.531190] workqueue events: flags=0x0 [ 3287.534066] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3287.538133] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3287.545272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3287.550976] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3287.555873] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.564096] pending: kfree_rcu_monitor [ 3287.567312] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3287.571396] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3287.574935] workqueue events_long: flags=0x0 [ 3287.579123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.584989] pending: lu_objects_flush [obdclass] [ 3287.589351] workqueue events_unbound: flags=0x2 [ 3287.591469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3287.594412] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3287.609467] workqueue events_power_efficient: flags=0x80 [ 3287.612630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.620132] pending: check_lifetime [ 3287.622831] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.627262] pending: neigh_periodic_work [ 3287.629645] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.632716] pending: neigh_periodic_work [ 3287.635188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.643517] pending: do_cache_clean [ 3287.645794] workqueue mm_percpu_wq: flags=0x8 [ 3287.647740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.649954] pending: vmstat_update [ 3287.653122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.658934] pending: vmstat_update [ 3287.664162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3287.668507] pending: vmstat_update [ 3287.671202] workqueue writeback: flags=0x4a [ 3287.673702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3287.679992] pending: wb_workfn, wb_workfn [ 3287.682892] workqueue xprtiod: flags=0xa [ 3287.685433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3287.689196] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3287.696478] workqueue ipv6_addrconf: flags=0x40008 [ 3287.700142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3287.703304] pending: addrconf_verify_work [ 3287.706313] workqueue knbd0-recv: flags=0x1a [ 3287.708306] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3287.713920] in-flight: 651:recv_work [ 3287.718197] workqueue ptlrpc_pinger: flags=0x2 [ 3287.721337] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3287.725614] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3287.729312] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1370s workers=4 idle: 37 31846 28897 [ 3287.736684] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3277s workers=2 idle: 653 [ 3288.058131] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3288.061209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3288.061384] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3288.081608] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3288.081624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3288.081627] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3288.081641] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3288.081645] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3288.081651] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3288.081653] RDX: 000000000000377d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3288.081655] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3288.081657] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3288.081659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3288.081662] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3288.081665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3288.081668] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3288.081677] Call Trace: [ 3288.081682] [ 3288.081687] ? show_regs.cold.9+0x22/0x2f [ 3288.081698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3288.081705] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3288.081709] ? watchdog+0x40/0x40 [ 3288.081714] ? hrtimer_interrupt+0x128/0x2c0 [ 3288.081719] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3288.081727] ? apic_timer_interrupt+0xf/0x20 [ 3288.081732] [ 3288.081735] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3288.081740] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3288.081745] do_raw_spin_lock+0x177/0x190 [ 3288.081750] _raw_spin_lock+0x21/0x30 [ 3288.081759] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3288.082241] obd_mod_free+0x46/0x140 [ptlrpc] [ 3288.082470] mdc_commit_open+0x91/0xb0 [mdc] [ 3288.082508] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3288.082720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3288.083194] after_reply+0xe2c/0x1780 [ptlrpc] [ 3288.083416] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3288.083608] ? __next_timer_interrupt+0x160/0x160 [ 3288.083617] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3288.083816] ? do_wait_intr+0xf0/0xf0 [ 3288.083826] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3288.084227] kthread+0x1d1/0x200 [ 3288.084234] ? set_kthread_struct+0x70/0x70 [ 3288.084238] ret_from_fork+0x1f/0x30 [ 3288.192505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3288.211257] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3288.215607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3288.219413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3288.221842] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3288.229299] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3288.232269] RAX: 00000000000072b3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3288.235030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3288.237812] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3288.240787] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3288.243765] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3288.246675] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3288.249854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3288.252245] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3288.255278] Call Trace: [ 3288.256322] [ 3288.257182] ? show_regs.cold.9+0x22/0x2f [ 3288.258739] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3288.260922] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3288.263016] ? watchdog+0x40/0x40 [ 3288.264566] ? hrtimer_interrupt+0x128/0x2c0 [ 3288.266334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3288.267787] ? apic_timer_interrupt+0xf/0x20 [ 3288.269599] [ 3288.270687] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3288.272761] do_raw_spin_lock+0x177/0x190 [ 3288.274019] _raw_spin_lock+0x21/0x30 [ 3288.275738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3288.277796] ? do_raw_spin_unlock+0x75/0x190 [ 3288.279512] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3288.281767] ? do_wait_intr+0xf0/0xf0 [ 3288.283331] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3288.285100] mdc_close+0x53b/0xd70 [mdc] [ 3288.286867] lmv_close+0x18d/0x490 [lmv] [ 3288.288437] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3288.290791] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3288.292934] ll_md_close+0x274/0x800 [lustre] [ 3288.294819] ll_file_release+0x128/0xc50 [lustre] [ 3288.297055] __fput+0x10d/0x440 [ 3288.298514] ____fput+0x12/0x20 [ 3288.299548] task_work_run+0xc6/0x110 [ 3288.301005] exit_to_usermode_loop+0x1dd/0x1f0 [ 3288.302747] do_syscall_64+0x3d6/0x440 [ 3288.304551] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3288.306390] RIP: 0033:0x7fa749552cd8 [ 3288.307795] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3288.315010] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3288.318033] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3288.320784] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3288.323602] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3288.326690] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3288.329708] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3292.057580] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3292.064749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3292.116865] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3292.132907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3292.139547] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3292.150967] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3292.154844] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3292.164129] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3292.167915] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3292.170808] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3292.173642] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3292.176510] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3292.179563] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3292.184944] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3292.188552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3292.191125] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3292.198880] Call Trace: [ 3292.200666] [ 3292.201694] ? show_regs.cold.9+0x22/0x2f [ 3292.203815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3292.206353] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3292.209057] ? watchdog+0x40/0x40 [ 3292.210743] ? hrtimer_interrupt+0x128/0x2c0 [ 3292.213587] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3292.218733] ? apic_timer_interrupt+0xf/0x20 [ 3292.222190] [ 3292.224209] ? native_safe_halt+0x2e/0x50 [ 3292.231604] kvm_wait+0xcd/0xe0 [ 3292.234648] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3292.244981] do_raw_spin_lock+0x177/0x190 [ 3292.247067] _raw_spin_lock+0x21/0x30 [ 3292.251560] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3292.257893] process_one_work+0x2c8/0x700 [ 3292.260849] worker_thread+0x296/0x6e0 [ 3292.262711] ? rescuer_thread+0x570/0x570 [ 3292.266337] kthread+0x1d1/0x200 [ 3292.268117] ? set_kthread_struct+0x70/0x70 [ 3292.271558] ret_from_fork+0x1f/0x30 [ 3312.054009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3312.056949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3312.081554] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3312.093218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3312.096717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3312.100062] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3312.106784] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3312.109115] RAX: 0000000000004f83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3312.113807] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3312.116016] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3312.118962] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3312.121526] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3312.124369] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3312.126509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3312.130222] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3312.134014] Call Trace: [ 3312.135827] [ 3312.136785] ? show_regs.cold.9+0x22/0x2f [ 3312.138631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3312.140395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3312.141898] ? watchdog+0x40/0x40 [ 3312.143376] ? hrtimer_interrupt+0x128/0x2c0 [ 3312.147088] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3312.148902] ? apic_timer_interrupt+0xf/0x20 [ 3312.150779] [ 3312.151373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3312.153497] do_raw_spin_lock+0x177/0x190 [ 3312.158034] _raw_spin_lock+0x21/0x30 [ 3312.160025] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3312.162388] ? do_raw_spin_unlock+0x75/0x190 [ 3312.164226] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3312.169771] ? do_wait_intr+0xf0/0xf0 [ 3312.170854] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3312.172409] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3312.176416] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3312.179392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.181801] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3312.183708] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3312.185527] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3312.187186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.188825] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3312.193436] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3312.195322] lmv_intent_open+0x338/0x1150 [lmv] [ 3312.196997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.199422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.202309] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3312.205151] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3312.207122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.208865] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3312.211067] ? do_raw_spin_unlock+0x75/0x190 [ 3312.212840] ? _raw_spin_unlock+0x12/0x30 [ 3312.214294] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3312.219397] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3312.223454] ll_atomic_open+0x36a/0xd70 [lustre] [ 3312.225913] lookup_open+0x356/0xb40 [ 3312.227395] do_last+0x313/0xfc0 [ 3312.229827] ? nd_jump_root+0xe5/0x160 [ 3312.232049] ? path_init+0x437/0x520 [ 3312.234997] path_openat+0xf7/0x500 [ 3312.236714] ? do_raw_spin_unlock+0x75/0x190 [ 3312.238729] do_filp_open+0x99/0x140 [ 3312.240360] ? getname_flags+0x6e/0x330 [ 3312.242549] ? __check_object_size+0xff/0x256 [ 3312.243955] ? do_raw_spin_unlock+0x75/0x190 [ 3312.246248] ? _raw_spin_unlock+0x12/0x30 [ 3312.249345] do_sys_openat2+0x2b4/0x410 [ 3312.253128] do_sys_open+0x73/0xa0 [ 3312.257052] __x64_sys_openat+0x24/0x30 [ 3312.260409] do_syscall_64+0xc1/0x440 [ 3312.264304] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3312.266831] RIP: 0033:0x7fa749552332 [ 3312.271240] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3312.280321] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3312.286350] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3312.290327] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3312.295926] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3312.299542] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3312.301591] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3316.058039] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3316.061027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3316.061324] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3316.087830] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3316.087843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3316.087846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3316.087875] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3316.087880] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3316.087885] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3316.087888] RDX: 0000000000004742 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3316.087890] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3316.087892] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3316.087895] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3316.087898] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3316.087900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3316.087906] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3316.087913] Call Trace: [ 3316.087919] [ 3316.087924] ? show_regs.cold.9+0x22/0x2f [ 3316.087933] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3316.087943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3316.087947] ? watchdog+0x40/0x40 [ 3316.087951] ? hrtimer_interrupt+0x128/0x2c0 [ 3316.087955] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3316.087963] ? apic_timer_interrupt+0xf/0x20 [ 3316.087967] [ 3316.087972] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3316.087979] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3316.087984] do_raw_spin_lock+0x177/0x190 [ 3316.087988] _raw_spin_lock+0x21/0x30 [ 3316.087994] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3316.088338] obd_mod_free+0x46/0x140 [ptlrpc] [ 3316.088570] mdc_commit_open+0x91/0xb0 [mdc] [ 3316.088719] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3316.088936] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3316.089165] after_reply+0xe2c/0x1780 [ptlrpc] [ 3316.089386] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3316.089738] ? __next_timer_interrupt+0x160/0x160 [ 3316.089747] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3316.089976] ? do_wait_intr+0xf0/0xf0 [ 3316.089985] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3316.090232] kthread+0x1d1/0x200 [ 3316.090238] ? set_kthread_struct+0x70/0x70 [ 3316.090241] ret_from_fork+0x1f/0x30 [ 3316.250729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3316.283849] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3316.293961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3316.303667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3316.309323] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3316.321826] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3316.328373] RAX: 0000000000007b76 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3316.332132] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3316.335290] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3316.341337] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3316.343918] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3316.349640] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3316.358471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3316.361670] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3316.369279] Call Trace: [ 3316.373132] [ 3316.374032] ? show_regs.cold.9+0x22/0x2f [ 3316.377890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3316.380313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3316.382190] ? watchdog+0x40/0x40 [ 3316.384720] ? hrtimer_interrupt+0x128/0x2c0 [ 3316.386065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3316.387273] ? apic_timer_interrupt+0xf/0x20 [ 3316.393675] [ 3316.395314] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3316.402835] do_raw_spin_lock+0x177/0x190 [ 3316.409160] _raw_spin_lock+0x21/0x30 [ 3316.413438] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3316.420201] ? do_raw_spin_unlock+0x75/0x190 [ 3316.422347] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3316.429810] ? do_wait_intr+0xf0/0xf0 [ 3316.432009] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3316.435223] mdc_close+0x53b/0xd70 [mdc] [ 3316.438933] lmv_close+0x18d/0x490 [lmv] [ 3316.441305] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3316.446149] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3316.449315] ll_md_close+0x274/0x800 [lustre] [ 3316.450548] ll_file_release+0x128/0xc50 [lustre] [ 3316.457091] __fput+0x10d/0x440 [ 3316.458264] ____fput+0x12/0x20 [ 3316.459823] task_work_run+0xc6/0x110 [ 3316.463057] exit_to_usermode_loop+0x1dd/0x1f0 [ 3316.464885] do_syscall_64+0x3d6/0x440 [ 3316.466342] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3316.470074] RIP: 0033:0x7fa749552cd8 [ 3316.472394] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3316.482384] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3316.488332] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3316.491814] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3316.504151] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3316.516203] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3316.527224] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3318.244446] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1401s! [ 3318.249345] Showing busy workqueues and worker pools: [ 3318.252739] workqueue events: flags=0x0 [ 3318.257040] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3318.265102] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3318.273183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3318.278130] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3318.283917] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.287529] pending: kfree_rcu_monitor [ 3318.290056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3318.293609] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3318.296226] workqueue events_long: flags=0x0 [ 3318.298969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.307702] pending: lu_objects_flush [obdclass] [ 3318.311054] workqueue events_unbound: flags=0x2 [ 3318.314209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3318.322739] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3318.332866] workqueue events_power_efficient: flags=0x80 [ 3318.334566] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.342051] pending: check_lifetime [ 3318.348585] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.357819] pending: neigh_periodic_work [ 3318.359471] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.364592] pending: neigh_periodic_work [ 3318.366685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.369866] pending: do_cache_clean [ 3318.374857] workqueue mm_percpu_wq: flags=0x8 [ 3318.379479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.384979] pending: vmstat_update [ 3318.388638] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.393952] pending: vmstat_update [ 3318.395841] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3318.397648] pending: vmstat_update [ 3318.400446] workqueue writeback: flags=0x4a [ 3318.402581] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3318.405018] pending: wb_workfn, wb_workfn [ 3318.408665] workqueue xprtiod: flags=0xa [ 3318.412102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3318.414249] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3318.419832] workqueue ipv6_addrconf: flags=0x40008 [ 3318.428686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3318.435602] pending: addrconf_verify_work [ 3318.437246] workqueue knbd0-recv: flags=0x1a [ 3318.444914] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3318.448926] in-flight: 651:recv_work [ 3318.453035] workqueue ptlrpc_pinger: flags=0x2 [ 3318.457354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3318.466125] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3318.475028] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1401s workers=4 idle: 37 31846 28897 [ 3318.483107] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3308s workers=2 idle: 653 [ 3320.059072] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3320.066337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3320.103512] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3320.115581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3320.123812] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3320.133693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3320.139119] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3320.161091] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3320.163307] RAX: 0000000000004c08 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3320.167889] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3320.171290] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3320.174737] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3320.182450] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3320.188621] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3320.192195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3320.194404] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3320.199875] Call Trace: [ 3320.200599] [ 3320.201108] ? show_regs.cold.9+0x22/0x2f [ 3320.205857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3320.209629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3320.214113] ? watchdog+0x40/0x40 [ 3320.217110] ? hrtimer_interrupt+0x128/0x2c0 [ 3320.221324] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3320.223965] ? apic_timer_interrupt+0xf/0x20 [ 3320.230344] [ 3320.234534] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3320.240990] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3320.255773] do_raw_spin_lock+0x177/0x190 [ 3320.258039] _raw_spin_lock+0x21/0x30 [ 3320.262167] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3320.265107] process_one_work+0x2c8/0x700 [ 3320.268721] worker_thread+0x296/0x6e0 [ 3320.274450] ? rescuer_thread+0x570/0x570 [ 3320.279088] kthread+0x1d1/0x200 [ 3320.280247] ? set_kthread_struct+0x70/0x70 [ 3320.288827] ret_from_fork+0x1f/0x30 [ 3344.054064] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3344.056654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3344.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3344.059785] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3344.059794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3344.059865] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3344.059870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3344.059873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3344.059887] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3344.059891] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3344.059895] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3344.059898] RDX: 0000000000005121 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3344.059900] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3344.059903] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3344.059905] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3344.059908] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3344.059911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3344.059913] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3344.059921] Call Trace: [ 3344.059926] [ 3344.059934] ? show_regs.cold.9+0x22/0x2f [ 3344.059944] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3344.059951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3344.059955] ? watchdog+0x40/0x40 [ 3344.059960] ? hrtimer_interrupt+0x128/0x2c0 [ 3344.059964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3344.059973] ? apic_timer_interrupt+0xf/0x20 [ 3344.059977] [ 3344.059980] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3344.059985] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3344.059990] do_raw_spin_lock+0x177/0x190 [ 3344.059995] _raw_spin_lock+0x21/0x30 [ 3344.060001] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3344.060347] obd_mod_free+0x46/0x140 [ptlrpc] [ 3344.060559] mdc_commit_open+0x91/0xb0 [mdc] [ 3344.060598] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3344.060868] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3344.061066] after_reply+0xe2c/0x1780 [ptlrpc] [ 3344.063516] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3344.063872] ? __next_timer_interrupt+0x160/0x160 [ 3344.063882] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3344.064088] ? do_wait_intr+0xf0/0xf0 [ 3344.064099] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3344.064510] kthread+0x1d1/0x200 [ 3344.064516] ? set_kthread_struct+0x70/0x70 [ 3344.064520] ret_from_fork+0x1f/0x30 [ 3344.078871] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3344.078884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3344.078887] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3344.078900] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3344.078904] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3344.078911] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3344.078913] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3344.078915] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3344.078917] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3344.078919] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3344.078923] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3344.078925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3344.078927] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3344.078935] Call Trace: [ 3344.078941] [ 3344.078947] ? show_regs.cold.9+0x22/0x2f [ 3344.078956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3344.078962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3344.078966] ? watchdog+0x40/0x40 [ 3344.078970] ? hrtimer_interrupt+0x128/0x2c0 [ 3344.078973] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3344.078980] ? apic_timer_interrupt+0xf/0x20 [ 3344.078984] [ 3344.078987] ? native_safe_halt+0x2e/0x50 [ 3344.078991] kvm_wait+0xcd/0xe0 [ 3344.078997] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3344.079008] do_raw_spin_lock+0x177/0x190 [ 3344.079013] _raw_spin_lock+0x21/0x30 [ 3344.079017] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3344.079286] ? do_raw_spin_unlock+0x75/0x190 [ 3344.079292] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3344.079612] ? do_wait_intr+0xf0/0xf0 [ 3344.079622] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3344.079821] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3344.080047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3344.080274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.080705] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3344.080742] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3344.080772] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3344.080797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.080883] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3344.081136] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3344.081165] lmv_intent_open+0x338/0x1150 [lmv] [ 3344.081200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.081287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.081498] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3344.081521] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3344.081611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.081744] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3344.081828] ? do_raw_spin_unlock+0x75/0x190 [ 3344.081836] ? _raw_spin_unlock+0x12/0x30 [ 3344.081842] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3344.081870] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3344.081889] ll_atomic_open+0x36a/0xd70 [lustre] [ 3344.081984] lookup_open+0x356/0xb40 [ 3344.081993] do_last+0x313/0xfc0 [ 3344.081998] ? nd_jump_root+0xe5/0x160 [ 3344.082003] ? path_init+0x437/0x520 [ 3344.082008] path_openat+0xf7/0x500 [ 3344.082013] ? do_raw_spin_unlock+0x75/0x190 [ 3344.082017] do_filp_open+0x99/0x140 [ 3344.082025] ? getname_flags+0x6e/0x330 [ 3344.082030] ? __check_object_size+0xff/0x256 [ 3344.082039] ? do_raw_spin_unlock+0x75/0x190 [ 3344.082043] ? _raw_spin_unlock+0x12/0x30 [ 3344.082047] do_sys_openat2+0x2b4/0x410 [ 3344.082052] do_sys_open+0x73/0xa0 [ 3344.082056] __x64_sys_openat+0x24/0x30 [ 3344.082059] do_syscall_64+0xc1/0x440 [ 3344.082067] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3344.082073] RIP: 0033:0x7fa749552332 [ 3344.082081] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3344.082085] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3344.082091] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3344.082094] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3344.082096] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3344.082098] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3344.082101] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3344.572344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3344.594512] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3344.598916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3344.603952] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3344.605898] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3344.613174] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3344.618820] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3344.623502] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3344.629902] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3344.632349] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3344.634270] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3344.640019] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3344.645998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3344.649442] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3344.654575] Call Trace: [ 3344.658179] [ 3344.661186] ? show_regs.cold.9+0x22/0x2f [ 3344.664300] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3344.666779] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3344.668235] ? watchdog+0x40/0x40 [ 3344.669960] ? hrtimer_interrupt+0x128/0x2c0 [ 3344.672331] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3344.674693] ? apic_timer_interrupt+0xf/0x20 [ 3344.679264] [ 3344.680828] ? native_safe_halt+0x2e/0x50 [ 3344.682232] kvm_wait+0xcd/0xe0 [ 3344.683819] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3344.687228] do_raw_spin_lock+0x177/0x190 [ 3344.690326] _raw_spin_lock+0x21/0x30 [ 3344.693658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3344.696788] ? do_raw_spin_unlock+0x75/0x190 [ 3344.699553] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3344.701754] ? do_wait_intr+0xf0/0xf0 [ 3344.703259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3344.706858] mdc_close+0x53b/0xd70 [mdc] [ 3344.709572] lmv_close+0x18d/0x490 [lmv] [ 3344.713309] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3344.715774] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3344.719914] ll_md_close+0x274/0x800 [lustre] [ 3344.724046] ll_file_release+0x128/0xc50 [lustre] [ 3344.729248] __fput+0x10d/0x440 [ 3344.731644] ____fput+0x12/0x20 [ 3344.733559] task_work_run+0xc6/0x110 [ 3344.735152] exit_to_usermode_loop+0x1dd/0x1f0 [ 3344.738811] do_syscall_64+0x3d6/0x440 [ 3344.740228] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3344.742660] RIP: 0033:0x7fa749552cd8 [ 3344.743875] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3344.750221] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3344.752682] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3344.755048] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3344.757331] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3344.759977] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3344.762437] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3348.058954] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3348.064179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3348.117126] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3348.129132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3348.134658] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3348.143559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3348.150758] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3348.163625] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3348.170227] RAX: 0000000000007d10 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3348.180183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3348.186468] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3348.196366] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3348.201678] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3348.208405] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3348.219165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3348.222764] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3348.231143] Call Trace: [ 3348.233521] [ 3348.237354] ? show_regs.cold.9+0x22/0x2f [ 3348.240331] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3348.244653] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3348.246598] ? watchdog+0x40/0x40 [ 3348.248197] ? hrtimer_interrupt+0x128/0x2c0 [ 3348.251578] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3348.256818] ? apic_timer_interrupt+0xf/0x20 [ 3348.260500] [ 3348.263600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3348.266163] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3348.268793] do_raw_spin_lock+0x177/0x190 [ 3348.270388] _raw_spin_lock+0x21/0x30 [ 3348.272118] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3348.280159] process_one_work+0x2c8/0x700 [ 3348.284699] worker_thread+0x296/0x6e0 [ 3348.290188] ? rescuer_thread+0x570/0x570 [ 3348.293793] kthread+0x1d1/0x200 [ 3348.296361] ? set_kthread_struct+0x70/0x70 [ 3348.298747] ret_from_fork+0x1f/0x30 [ 3348.963027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1431s! [ 3348.965642] Showing busy workqueues and worker pools: [ 3348.969516] workqueue events: flags=0x0 [ 3348.972156] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3348.976324] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3348.982394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3348.985795] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3348.991295] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3348.996331] pending: kfree_rcu_monitor [ 3348.997508] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3349.004796] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3349.010591] workqueue events_long: flags=0x0 [ 3349.014552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3349.021511] pending: lu_objects_flush [obdclass] [ 3349.025740] workqueue events_unbound: flags=0x2 [ 3349.029390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3349.037828] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3349.050325] workqueue events_power_efficient: flags=0x80 [ 3349.057092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3349.062145] pending: check_lifetime [ 3349.064167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3349.068309] pending: neigh_periodic_work [ 3349.070616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3349.075799] pending: neigh_periodic_work [ 3349.079652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3349.085315] pending: do_cache_clean [ 3349.090560] workqueue mm_percpu_wq: flags=0x8 [ 3349.098084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3349.104981] pending: vmstat_update [ 3349.107656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3349.113704] pending: vmstat_update [ 3349.121221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3349.124764] pending: vmstat_update [ 3349.127339] workqueue writeback: flags=0x4a [ 3349.129264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3349.138937] pending: wb_workfn, wb_workfn [ 3349.140671] workqueue xprtiod: flags=0xa [ 3349.142136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3349.150457] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3349.154210] workqueue ipv6_addrconf: flags=0x40008 [ 3349.156339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3349.159127] pending: addrconf_verify_work [ 3349.160683] workqueue knbd0-recv: flags=0x1a [ 3349.162600] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3349.165704] in-flight: 651:recv_work [ 3349.168115] workqueue ptlrpc_pinger: flags=0x2 [ 3349.170937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3349.176390] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3349.182944] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1432s workers=4 idle: 37 31846 28897 [ 3349.188131] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3339s workers=2 idle: 653 [ 3372.060679] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3372.065433] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3372.066279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3372.066351] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3372.066358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3372.066361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3372.066375] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3372.066380] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3372.066384] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3372.066386] RDX: 0000000000005e38 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3372.066389] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3372.066391] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3372.066394] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3372.066397] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3372.066400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3372.066402] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3372.066411] Call Trace: [ 3372.066417] [ 3372.066423] ? show_regs.cold.9+0x22/0x2f [ 3372.066433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3372.066441] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3372.066445] ? watchdog+0x40/0x40 [ 3372.066450] ? hrtimer_interrupt+0x128/0x2c0 [ 3372.066454] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3372.066464] ? apic_timer_interrupt+0xf/0x20 [ 3372.066469] [ 3372.066472] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3372.066477] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3372.066482] do_raw_spin_lock+0x177/0x190 [ 3372.066487] _raw_spin_lock+0x21/0x30 [ 3372.066493] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3372.074335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3372.074412] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3372.074420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3372.074424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3372.074438] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3372.074442] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3372.074447] RAX: 0000000000007383 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3372.074449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3372.074451] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3372.074453] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3372.074455] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3372.074457] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3372.074460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3372.074463] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3372.074472] Call Trace: [ 3372.074477] [ 3372.074482] ? show_regs.cold.9+0x22/0x2f [ 3372.074507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3372.074514] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3372.074518] ? watchdog+0x40/0x40 [ 3372.074522] ? hrtimer_interrupt+0x128/0x2c0 [ 3372.074526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3372.074533] ? apic_timer_interrupt+0xf/0x20 [ 3372.074538] [ 3372.074541] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3372.074548] do_raw_spin_lock+0x177/0x190 [ 3372.074553] _raw_spin_lock+0x21/0x30 [ 3372.074558] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3372.074846] ? do_raw_spin_unlock+0x75/0x190 [ 3372.074852] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3372.075056] ? do_wait_intr+0xf0/0xf0 [ 3372.075064] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3372.075272] mdc_close+0x53b/0xd70 [mdc] [ 3372.075319] lmv_close+0x18d/0x490 [lmv] [ 3372.075351] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3372.116504] obd_mod_free+0x46/0x140 [ptlrpc] [ 3372.116820] mdc_commit_open+0x91/0xb0 [mdc] [ 3372.116861] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3372.117080] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3372.117280] after_reply+0xe2c/0x1780 [ptlrpc] [ 3372.117478] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3372.117674] ? __next_timer_interrupt+0x160/0x160 [ 3372.117686] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3372.117911] ? do_wait_intr+0xf0/0xf0 [ 3372.117921] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3372.118131] kthread+0x1d1/0x200 [ 3372.118136] ? set_kthread_struct+0x70/0x70 [ 3372.118139] ret_from_fork+0x1f/0x30 [ 3372.409308] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3372.411421] ll_md_close+0x274/0x800 [lustre] [ 3372.416540] ll_file_release+0x128/0xc50 [lustre] [ 3372.419391] __fput+0x10d/0x440 [ 3372.421155] ____fput+0x12/0x20 [ 3372.422950] task_work_run+0xc6/0x110 [ 3372.424587] exit_to_usermode_loop+0x1dd/0x1f0 [ 3372.426428] do_syscall_64+0x3d6/0x440 [ 3372.428181] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3372.431011] RIP: 0033:0x7fa749552cd8 [ 3372.432306] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3372.448721] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3372.453086] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3372.459413] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3372.468301] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3372.484088] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3372.495263] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3376.054008] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3376.058716] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3376.064767] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3376.064861] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3376.064870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3376.064873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3376.064888] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 3376.064893] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3376.064897] RAX: 0000000000003adb RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3376.064900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3376.064902] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3376.064904] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3376.064907] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3376.064910] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3376.064913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3376.064915] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3376.064923] Call Trace: [ 3376.064929] [ 3376.064935] ? show_regs.cold.9+0x22/0x2f [ 3376.064944] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3376.064951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3376.064956] ? watchdog+0x40/0x40 [ 3376.064960] ? hrtimer_interrupt+0x128/0x2c0 [ 3376.064964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3376.064972] ? apic_timer_interrupt+0xf/0x20 [ 3376.064976] [ 3376.064979] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3376.064985] do_raw_spin_lock+0x177/0x190 [ 3376.064990] _raw_spin_lock+0x21/0x30 [ 3376.064995] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3376.065276] ? do_raw_spin_unlock+0x75/0x190 [ 3376.065281] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3376.065474] ? do_wait_intr+0xf0/0xf0 [ 3376.065483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3376.065901] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3376.074505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3376.074597] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3376.074606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3376.074612] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3376.081384] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3376.081399] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3376.081404] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3376.081408] RAX: 0000000000007a11 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3376.081411] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3376.081413] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3376.081415] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3376.081417] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3376.081420] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3376.081423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3376.081425] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3376.081434] Call Trace: [ 3376.081440] [ 3376.081446] ? show_regs.cold.9+0x22/0x2f [ 3376.081456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3376.081462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3376.081466] ? watchdog+0x40/0x40 [ 3376.081470] ? hrtimer_interrupt+0x128/0x2c0 [ 3376.081474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3376.081482] ? apic_timer_interrupt+0xf/0x20 [ 3376.081486] [ 3376.081489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3376.081493] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3376.081500] do_raw_spin_lock+0x177/0x190 [ 3376.081505] _raw_spin_lock+0x21/0x30 [ 3376.081510] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3376.081775] process_one_work+0x2c8/0x700 [ 3376.081782] worker_thread+0x296/0x6e0 [ 3376.081786] ? rescuer_thread+0x570/0x570 [ 3376.081790] kthread+0x1d1/0x200 [ 3376.081794] ? set_kthread_struct+0x70/0x70 [ 3376.081797] ret_from_fork+0x1f/0x30 [ 3376.710270] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3376.720336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3376.728427] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3376.734968] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3376.737487] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3376.740506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3376.753912] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3376.760914] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3376.764537] lmv_intent_open+0x338/0x1150 [lmv] [ 3376.769452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3376.774358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3376.778584] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3376.786469] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3376.788046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3376.797087] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3376.801477] ? do_raw_spin_unlock+0x75/0x190 [ 3376.804858] ? _raw_spin_unlock+0x12/0x30 [ 3376.811277] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3376.821214] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3376.826865] ll_atomic_open+0x36a/0xd70 [lustre] [ 3376.830863] lookup_open+0x356/0xb40 [ 3376.834185] do_last+0x313/0xfc0 [ 3376.835756] ? nd_jump_root+0xe5/0x160 [ 3376.838213] ? path_init+0x437/0x520 [ 3376.840661] path_openat+0xf7/0x500 [ 3376.845044] ? do_raw_spin_unlock+0x75/0x190 [ 3376.850104] do_filp_open+0x99/0x140 [ 3376.854251] ? getname_flags+0x6e/0x330 [ 3376.859242] ? __check_object_size+0xff/0x256 [ 3376.867341] ? do_raw_spin_unlock+0x75/0x190 [ 3376.868730] ? _raw_spin_unlock+0x12/0x30 [ 3376.873263] do_sys_openat2+0x2b4/0x410 [ 3376.879493] do_sys_open+0x73/0xa0 [ 3376.884508] __x64_sys_openat+0x24/0x30 [ 3376.893743] do_syscall_64+0xc1/0x440 [ 3376.902248] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3376.915367] RIP: 0033:0x7fa749552332 [ 3376.919435] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3376.944085] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3376.953106] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3376.965039] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3376.967606] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3376.978444] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3376.987566] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3379.681359] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1462s! [ 3379.685038] Showing busy workqueues and worker pools: [ 3379.688672] workqueue events: flags=0x0 [ 3379.690788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3379.694027] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3379.704766] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3379.709063] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3379.717823] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.723344] pending: kfree_rcu_monitor [ 3379.726875] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3379.733490] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3379.739293] workqueue events_long: flags=0x0 [ 3379.744666] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.751919] pending: lu_objects_flush [obdclass] [ 3379.755922] workqueue events_unbound: flags=0x2 [ 3379.763076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3379.768663] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3379.774277] workqueue events_power_efficient: flags=0x80 [ 3379.781617] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.789712] pending: check_lifetime [ 3379.792660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.799612] pending: neigh_periodic_work [ 3379.807431] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.815545] pending: neigh_periodic_work [ 3379.823891] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.834737] pending: do_cache_clean [ 3379.837008] workqueue mm_percpu_wq: flags=0x8 [ 3379.841637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.853630] pending: vmstat_update [ 3379.855957] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.864156] pending: vmstat_update [ 3379.870349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3379.875012] pending: vmstat_update [ 3379.879900] workqueue writeback: flags=0x4a [ 3379.883531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3379.891807] pending: wb_workfn, wb_workfn [ 3379.894984] workqueue xprtiod: flags=0xa [ 3379.904516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3379.911494] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3379.926391] workqueue ipv6_addrconf: flags=0x40008 [ 3379.931797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3379.937134] pending: addrconf_verify_work [ 3379.943632] workqueue knbd0-recv: flags=0x1a [ 3379.945783] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3379.950427] in-flight: 651:recv_work [ 3379.954966] workqueue ptlrpc_pinger: flags=0x2 [ 3379.957053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3379.963465] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3379.969573] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1462s workers=4 idle: 37 31846 28897 [ 3379.979518] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3369s workers=2 idle: 653 [ 3400.059359] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3400.061410] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3400.061761] Modules linked in: lustre(O) [ 3400.069683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3400.069859] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3400.069867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3400.069870] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3400.069894] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3400.069898] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3400.069902] RAX: 0000000000006e5a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3400.069904] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3400.069906] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3400.071570] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3400.071652] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3400.071659] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3400.071663] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3400.071677] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3400.071681] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3400.071686] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3400.071689] RDX: 0000000000002f58 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3400.071691] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3400.071694] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3400.071696] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3400.071700] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3400.071703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3400.071705] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3400.071714] Call Trace: [ 3400.071752] [ 3400.071758] ? show_regs.cold.9+0x22/0x2f [ 3400.071768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3400.071774] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3400.071778] ? watchdog+0x40/0x40 [ 3400.071782] ? hrtimer_interrupt+0x128/0x2c0 [ 3400.071785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3400.071792] ? apic_timer_interrupt+0xf/0x20 [ 3400.071796] [ 3400.071799] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3400.071804] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3400.071808] do_raw_spin_lock+0x177/0x190 [ 3400.071813] _raw_spin_lock+0x21/0x30 [ 3400.071819] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3400.072176] obd_mod_free+0x46/0x140 [ptlrpc] [ 3400.072405] mdc_commit_open+0x91/0xb0 [mdc] [ 3400.072446] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3400.072641] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3400.072866] after_reply+0xe2c/0x1780 [ptlrpc] [ 3400.115446] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3400.120449] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3400.128626] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3400.131041] ? __next_timer_interrupt+0x160/0x160 [ 3400.131062] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3400.131353] ? do_wait_intr+0xf0/0xf0 [ 3400.131363] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3400.131574] kthread+0x1d1/0x200 [ 3400.131580] ? set_kthread_struct+0x70/0x70 [ 3400.131583] ret_from_fork+0x1f/0x30 [ 3400.306682] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3400.311045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3400.314638] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3400.319579] Call Trace: [ 3400.322547] [ 3400.324229] ? show_regs.cold.9+0x22/0x2f [ 3400.325726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3400.328303] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3400.330538] ? watchdog+0x40/0x40 [ 3400.332668] ? hrtimer_interrupt+0x128/0x2c0 [ 3400.335842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3400.339912] ? apic_timer_interrupt+0xf/0x20 [ 3400.343791] [ 3400.345752] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3400.348993] do_raw_spin_lock+0x177/0x190 [ 3400.351026] _raw_spin_lock+0x21/0x30 [ 3400.353454] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3400.357539] ? do_raw_spin_unlock+0x75/0x190 [ 3400.360494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3400.363027] ? do_wait_intr+0xf0/0xf0 [ 3400.364717] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3400.367709] mdc_close+0x53b/0xd70 [mdc] [ 3400.369064] lmv_close+0x18d/0x490 [lmv] [ 3400.370163] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3400.373762] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3400.376623] ll_md_close+0x274/0x800 [lustre] [ 3400.379093] ll_file_release+0x128/0xc50 [lustre] [ 3400.383829] __fput+0x10d/0x440 [ 3400.385803] ____fput+0x12/0x20 [ 3400.387765] task_work_run+0xc6/0x110 [ 3400.390731] exit_to_usermode_loop+0x1dd/0x1f0 [ 3400.392505] do_syscall_64+0x3d6/0x440 [ 3400.394024] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3400.396022] RIP: 0033:0x7fa749552cd8 [ 3400.397546] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3400.407355] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3400.412317] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3400.418526] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3400.422552] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3400.425910] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3400.429731] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3404.056282] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3404.058018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3404.058773] Modules linked in: lustre(O) [ 3404.065861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3404.065947] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3404.065954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3404.065961] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3404.066653] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3404.066664] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3404.066668] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3404.066672] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3404.066675] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3404.066677] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3404.066679] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3404.066682] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3404.066685] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3404.066688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3404.066690] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3404.066698] Call Trace: [ 3404.066703] [ 3404.066857] ? show_regs.cold.9+0x22/0x2f [ 3404.066868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3404.066875] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3404.066880] ? watchdog+0x40/0x40 [ 3404.066884] ? hrtimer_interrupt+0x128/0x2c0 [ 3404.066889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3404.066906] ? apic_timer_interrupt+0xf/0x20 [ 3404.066910] [ 3404.066913] ? native_safe_halt+0x2e/0x50 [ 3404.066916] kvm_wait+0xcd/0xe0 [ 3404.066924] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3404.066931] do_raw_spin_lock+0x177/0x190 [ 3404.066935] _raw_spin_lock+0x21/0x30 [ 3404.066939] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3404.068959] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3404.069029] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3404.069037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3404.069040] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3404.069052] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3404.069055] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3404.069059] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3404.069062] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3404.069063] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3404.069065] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3404.069067] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3404.069071] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3404.069074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3404.069076] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3404.069084] Call Trace: [ 3404.069089] [ 3404.069093] ? show_regs.cold.9+0x22/0x2f [ 3404.069103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3404.069110] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3404.069113] ? watchdog+0x40/0x40 [ 3404.069118] ? hrtimer_interrupt+0x128/0x2c0 [ 3404.069121] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3404.069128] ? apic_timer_interrupt+0xf/0x20 [ 3404.069132] [ 3404.069135] ? native_safe_halt+0x2e/0x50 [ 3404.069138] kvm_wait+0xcd/0xe0 [ 3404.069146] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3404.069154] do_raw_spin_lock+0x177/0x190 [ 3404.069158] _raw_spin_lock+0x21/0x30 [ 3404.069162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3404.121552] process_one_work+0x2c8/0x700 [ 3404.121568] worker_thread+0x296/0x6e0 [ 3404.121571] ? rescuer_thread+0x570/0x570 [ 3404.121574] kthread+0x1d1/0x200 [ 3404.121577] ? set_kthread_struct+0x70/0x70 [ 3404.121579] ret_from_fork+0x1f/0x30 [ 3404.379492] ? do_raw_spin_unlock+0x75/0x190 [ 3404.381223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3404.386528] ? do_wait_intr+0xf0/0xf0 [ 3404.388742] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3404.392114] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3404.393988] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3404.396160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3404.397955] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3404.399641] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3404.401163] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3404.406257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3404.408467] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3404.415692] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3404.420021] lmv_intent_open+0x338/0x1150 [lmv] [ 3404.421636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3404.428721] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3404.432753] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3404.441707] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3404.445534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3404.454663] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3404.458964] ? do_raw_spin_unlock+0x75/0x190 [ 3404.460882] ? _raw_spin_unlock+0x12/0x30 [ 3404.463565] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3404.467000] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3404.473470] ll_atomic_open+0x36a/0xd70 [lustre] [ 3404.482967] lookup_open+0x356/0xb40 [ 3404.486641] do_last+0x313/0xfc0 [ 3404.488251] ? nd_jump_root+0xe5/0x160 [ 3404.489966] ? path_init+0x437/0x520 [ 3404.495627] path_openat+0xf7/0x500 [ 3404.497600] ? do_raw_spin_unlock+0x75/0x190 [ 3404.500559] do_filp_open+0x99/0x140 [ 3404.502730] ? getname_flags+0x6e/0x330 [ 3404.504493] ? __check_object_size+0xff/0x256 [ 3404.515816] ? do_raw_spin_unlock+0x75/0x190 [ 3404.519511] ? _raw_spin_unlock+0x12/0x30 [ 3404.522763] do_sys_openat2+0x2b4/0x410 [ 3404.524436] do_sys_open+0x73/0xa0 [ 3404.526866] __x64_sys_openat+0x24/0x30 [ 3404.530807] do_syscall_64+0xc1/0x440 [ 3404.533777] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3404.539284] RIP: 0033:0x7fa749552332 [ 3404.540670] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3404.583621] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3404.585953] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3404.603283] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3404.610307] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3404.617419] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3404.619940] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3410.400166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1493s! [ 3410.410412] Showing busy workqueues and worker pools: [ 3410.425710] workqueue events: flags=0x0 [ 3410.431737] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3410.447189] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3410.468361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3410.482787] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3410.501325] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.514622] pending: kfree_rcu_monitor [ 3410.516127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3410.526792] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3410.540636] workqueue events_long: flags=0x0 [ 3410.542220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.550352] pending: lu_objects_flush [obdclass] [ 3410.562495] workqueue events_unbound: flags=0x2 [ 3410.564208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3410.577366] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3410.592629] workqueue events_power_efficient: flags=0x80 [ 3410.594603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.609010] pending: check_lifetime [ 3410.615096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.621704] pending: neigh_periodic_work [ 3410.626173] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.634435] pending: neigh_periodic_work [ 3410.646663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.664074] pending: do_cache_clean [ 3410.665515] workqueue mm_percpu_wq: flags=0x8 [ 3410.675875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.689231] pending: vmstat_update [ 3410.697303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.703436] pending: vmstat_update [ 3410.706384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3410.710656] pending: vmstat_update [ 3410.713735] workqueue writeback: flags=0x4a [ 3410.715979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3410.720273] pending: wb_workfn, wb_workfn [ 3410.721924] workqueue xprtiod: flags=0xa [ 3410.723191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3410.728628] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3410.735153] workqueue ipv6_addrconf: flags=0x40008 [ 3410.739619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3410.744428] pending: addrconf_verify_work [ 3410.747176] workqueue knbd0-recv: flags=0x1a [ 3410.749294] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3410.754805] in-flight: 651:recv_work [ 3410.756296] workqueue ptlrpc_pinger: flags=0x2 [ 3410.759725] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3410.767135] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3410.771359] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1493s workers=4 idle: 37 31846 28897 [ 3410.775336] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3400s workers=2 idle: 653 [ 3413.725856] rcu: INFO: rcu_sched self-detected stall on CPU [ 3413.735820] rcu: 2-...!: (809715 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 3413.747078] (t=1500036 jiffies g=258701 q=1961) [ 3413.753066] rcu: rcu_sched kthread starved for 1496831 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 3413.762256] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3413.771584] rcu: RCU grace-period kthread stack dump: [ 3413.781533] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3413.792973] Call Trace: [ 3413.794013] __schedule+0x351/0xcb0 [ 3413.797741] schedule+0xc0/0x180 [ 3413.800349] schedule_timeout+0xb4/0x190 [ 3413.802447] ? do_raw_spin_unlock+0x75/0x190 [ 3413.805036] ? __next_timer_interrupt+0x160/0x160 [ 3413.809007] ? prepare_to_swait_event+0xf3/0x140 [ 3413.812227] rcu_gp_kthread+0x8dd/0x15b0 [ 3413.815103] ? force_qs_rnp+0x2c0/0x2c0 [ 3413.817520] kthread+0x1d1/0x200 [ 3413.819275] ? set_kthread_struct+0x70/0x70 [ 3413.821487] ret_from_fork+0x1f/0x30 [ 3413.828333] rcu: Stack dump where RCU GP kthread last ran: [ 3413.831769] Sending NMI from CPU 2 to CPUs 1: [ 3413.833790] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 3413.835047] NMI backtrace for cpu 2 [ 3413.841347] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3413.850577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3413.854397] Call Trace: [ 3413.855801] [ 3413.856934] ? dump_stack+0xbb/0x10e [ 3413.858967] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3413.861555] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3413.865280] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3413.868532] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3413.871415] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3413.873783] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3413.875851] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3413.878069] ? update_process_times+0x6d/0xc0 [ 3413.880149] ? tick_sched_handle.isra.18+0x27/0x90 [ 3413.882401] ? tick_sched_timer+0x53/0xb0 [ 3413.884225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3413.886757] ? tick_sched_handle.isra.18+0x90/0x90 [ 3413.888822] ? hrtimer_interrupt+0x128/0x2c0 [ 3413.890733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3413.893049] ? apic_timer_interrupt+0xf/0x20 [ 3413.895426] [ 3413.896487] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3413.899227] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3413.902034] ? do_raw_spin_lock+0x177/0x190 [ 3413.904031] ? _raw_spin_lock+0x21/0x30 [ 3413.906534] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3413.909424] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3413.912043] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3413.914373] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3413.917515] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3413.919916] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3413.922513] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3413.925438] ? __next_timer_interrupt+0x160/0x160 [ 3413.927691] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3413.929727] ? do_wait_intr+0xf0/0xf0 [ 3413.930992] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3413.933434] ? kthread+0x1d1/0x200 [ 3413.934783] ? set_kthread_struct+0x70/0x70 [ 3413.937366] ? ret_from_fork+0x1f/0x30 [ 3428.059094] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3428.061682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3428.080286] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3428.086277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3428.091726] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3428.094024] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3428.107272] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3428.113237] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3428.119162] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3428.122808] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3428.126175] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3428.129766] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3428.135382] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3428.145750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3428.149254] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3428.152970] Call Trace: [ 3428.154247] [ 3428.156056] ? show_regs.cold.9+0x22/0x2f [ 3428.159386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3428.163361] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3428.167591] ? watchdog+0x40/0x40 [ 3428.170982] ? hrtimer_interrupt+0x128/0x2c0 [ 3428.176884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3428.178675] ? apic_timer_interrupt+0xf/0x20 [ 3428.187936] [ 3428.188922] ? native_safe_halt+0x2e/0x50 [ 3428.195239] kvm_wait+0xcd/0xe0 [ 3428.197877] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3428.203201] do_raw_spin_lock+0x177/0x190 [ 3428.205920] _raw_spin_lock+0x21/0x30 [ 3428.210916] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3428.215215] ? do_raw_spin_unlock+0x75/0x190 [ 3428.220546] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3428.224319] ? do_wait_intr+0xf0/0xf0 [ 3428.227678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3428.229982] mdc_close+0x53b/0xd70 [mdc] [ 3428.231148] lmv_close+0x18d/0x490 [lmv] [ 3428.232715] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3428.235341] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3428.237610] ll_md_close+0x274/0x800 [lustre] [ 3428.239722] ll_file_release+0x128/0xc50 [lustre] [ 3428.241822] __fput+0x10d/0x440 [ 3428.243054] ____fput+0x12/0x20 [ 3428.244215] task_work_run+0xc6/0x110 [ 3428.245406] exit_to_usermode_loop+0x1dd/0x1f0 [ 3428.246892] do_syscall_64+0x3d6/0x440 [ 3428.248219] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3428.249853] RIP: 0033:0x7fa749552cd8 [ 3428.251203] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3428.257285] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3428.259845] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3428.262402] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3428.264856] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3428.267202] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3428.269855] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3436.054032] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3436.059799] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3436.097623] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3436.102672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3436.106142] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3436.109904] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3436.129661] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3436.132549] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3436.139586] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3436.144751] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3436.149411] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3436.156172] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3436.168569] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3436.172736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3436.176456] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3436.183432] Call Trace: [ 3436.184435] [ 3436.187563] ? show_regs.cold.9+0x22/0x2f [ 3436.189288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3436.194980] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3436.197451] ? watchdog+0x40/0x40 [ 3436.201232] ? hrtimer_interrupt+0x128/0x2c0 [ 3436.203817] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3436.207911] ? apic_timer_interrupt+0xf/0x20 [ 3436.210891] [ 3436.212643] ? native_safe_halt+0x2e/0x50 [ 3436.220682] kvm_wait+0xcd/0xe0 [ 3436.224266] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3436.228830] do_raw_spin_lock+0x177/0x190 [ 3436.234354] _raw_spin_lock+0x21/0x30 [ 3436.241703] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3436.249522] ? do_raw_spin_unlock+0x75/0x190 [ 3436.253583] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3436.256324] ? do_wait_intr+0xf0/0xf0 [ 3436.263984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3436.268350] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3436.271869] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3436.275569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.281882] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3436.286612] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3436.289133] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3436.293879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.301580] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3436.308740] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3436.311851] lmv_intent_open+0x338/0x1150 [lmv] [ 3436.314691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.317739] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.321841] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3436.324556] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3436.327111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.332576] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3436.335371] ? do_raw_spin_unlock+0x75/0x190 [ 3436.338266] ? _raw_spin_unlock+0x12/0x30 [ 3436.339517] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3436.343262] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3436.347131] ll_atomic_open+0x36a/0xd70 [lustre] [ 3436.350909] lookup_open+0x356/0xb40 [ 3436.353399] do_last+0x313/0xfc0 [ 3436.359934] ? nd_jump_root+0xe5/0x160 [ 3436.362312] ? path_init+0x437/0x520 [ 3436.364569] path_openat+0xf7/0x500 [ 3436.368300] ? do_raw_spin_unlock+0x75/0x190 [ 3436.372786] do_filp_open+0x99/0x140 [ 3436.377348] ? getname_flags+0x6e/0x330 [ 3436.381438] ? __check_object_size+0xff/0x256 [ 3436.384880] ? do_raw_spin_unlock+0x75/0x190 [ 3436.386724] ? _raw_spin_unlock+0x12/0x30 [ 3436.389614] do_sys_openat2+0x2b4/0x410 [ 3436.390629] do_sys_open+0x73/0xa0 [ 3436.393399] __x64_sys_openat+0x24/0x30 [ 3436.399048] do_syscall_64+0xc1/0x440 [ 3436.401423] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3436.402898] RIP: 0033:0x7fa749552332 [ 3436.406481] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3436.422538] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3436.426416] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3436.435799] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3436.442454] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3436.448824] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3436.455227] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3440.057959] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3440.065038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.065500] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3440.131937] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.131950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.131956] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3440.135613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix [ 3440.140749] [ 3440.147935] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.147959] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.147966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.147969] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3440.147983] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3440.147987] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3440.147991] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3440.147994] RDX: 0000000000007e06 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3440.147996] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3440.147999] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3440.148000] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3440.148003] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3440.148006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.148009] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3440.148019] Call Trace: [ 3440.148025] [ 3440.148031] ? show_regs.cold.9+0x22/0x2f [ 3440.148040] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.148046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.148052] ? watchdog+0x40/0x40 [ 3440.148056] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.148060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.148068] ? apic_timer_interrupt+0xf/0x20 [ 3440.148302] [ 3440.148364] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3440.148370] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3440.148375] do_raw_spin_lock+0x177/0x190 [ 3440.148381] _raw_spin_lock+0x21/0x30 [ 3440.148388] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3440.148768] obd_mod_free+0x46/0x140 [ptlrpc] [ 3440.148963] mdc_commit_open+0x91/0xb0 [mdc] [ 3440.149006] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3440.149547] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3440.149756] after_reply+0xe2c/0x1780 [ptlrpc] [ 3440.149953] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3440.150382] ? __next_timer_interrupt+0x160/0x160 [ 3440.150395] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3440.150735] ? do_wait_intr+0xf0/0xf0 [ 3440.150745] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3440.150951] kthread+0x1d1/0x200 [ 3440.150957] ? set_kthread_struct+0x70/0x70 [ 3440.150961] ret_from_fork+0x1f/0x30 [ 3440.456083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.458839] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3440.469320] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3440.471695] RAX: 000000000000543c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3440.476128] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3440.482195] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3440.495787] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3440.501442] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3440.508098] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3440.512311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.519970] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3440.523811] Call Trace: [ 3440.526313] [ 3440.527547] ? show_regs.cold.9+0x22/0x2f [ 3440.531622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.539325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.541380] ? watchdog+0x40/0x40 [ 3440.545152] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.547937] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.549699] ? apic_timer_interrupt+0xf/0x20 [ 3440.552372] [ 3440.552980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.555894] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3440.557714] do_raw_spin_lock+0x177/0x190 [ 3440.560858] _raw_spin_lock+0x21/0x30 [ 3440.563076] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3440.568379] process_one_work+0x2c8/0x700 [ 3440.569503] worker_thread+0x296/0x6e0 [ 3440.575894] ? rescuer_thread+0x570/0x570 [ 3440.580202] kthread+0x1d1/0x200 [ 3440.584306] ? set_kthread_struct+0x70/0x70 [ 3440.589166] ret_from_fork+0x1f/0x30 [ 3441.120615] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1523s! [ 3441.124396] Showing busy workqueues and worker pools: [ 3441.127251] workqueue events: flags=0x0 [ 3441.130571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3441.134090] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3441.139106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3441.141451] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3441.149309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.151969] pending: kfree_rcu_monitor [ 3441.153836] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3441.156532] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3441.159862] workqueue events_long: flags=0x0 [ 3441.162545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.166526] pending: lu_objects_flush [obdclass] [ 3441.169567] workqueue events_unbound: flags=0x2 [ 3441.172516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3441.176323] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3441.181312] workqueue events_power_efficient: flags=0x80 [ 3441.183197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.185702] pending: check_lifetime [ 3441.186856] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.188928] pending: neigh_periodic_work [ 3441.190605] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.197176] pending: neigh_periodic_work [ 3441.199686] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.205431] pending: do_cache_clean [ 3441.207681] workqueue mm_percpu_wq: flags=0x8 [ 3441.209568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.213452] pending: vmstat_update [ 3441.215220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.218089] pending: vmstat_update [ 3441.221614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3441.225030] pending: vmstat_update [ 3441.227089] workqueue writeback: flags=0x4a [ 3441.228702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3441.230989] pending: wb_workfn, wb_workfn [ 3441.233462] workqueue xprtiod: flags=0xa [ 3441.238141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3441.240619] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3441.246016] workqueue ipv6_addrconf: flags=0x40008 [ 3441.251428] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3441.256295] pending: addrconf_verify_work [ 3441.260153] workqueue knbd0-recv: flags=0x1a [ 3441.262748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3441.268956] in-flight: 651:recv_work [ 3441.271209] workqueue ptlrpc_pinger: flags=0x2 [ 3441.273053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3441.276283] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3441.285645] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1524s workers=4 idle: 37 31846 28897 [ 3441.292700] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3431s workers=2 idle: 653 [ 3456.061431] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3456.071847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3456.126770] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3456.132872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3456.141788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3456.149075] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3456.160519] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3456.163574] RAX: 0000000000001115 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3456.173787] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3456.182603] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3456.192395] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3456.202624] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3456.210179] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3456.219080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3456.225265] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3456.233374] Call Trace: [ 3456.235949] [ 3456.237146] ? show_regs.cold.9+0x22/0x2f [ 3456.239190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3456.247602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3456.249806] ? watchdog+0x40/0x40 [ 3456.251013] ? hrtimer_interrupt+0x128/0x2c0 [ 3456.254431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3456.257617] ? apic_timer_interrupt+0xf/0x20 [ 3456.265550] [ 3456.267636] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3456.271731] do_raw_spin_lock+0x177/0x190 [ 3456.275952] _raw_spin_lock+0x21/0x30 [ 3456.281107] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3456.290102] ? do_raw_spin_unlock+0x75/0x190 [ 3456.303690] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3456.317321] ? do_wait_intr+0xf0/0xf0 [ 3456.325446] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3456.334638] mdc_close+0x53b/0xd70 [mdc] [ 3456.336780] lmv_close+0x18d/0x490 [lmv] [ 3456.343481] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3456.348717] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3456.356237] ll_md_close+0x274/0x800 [lustre] [ 3456.359476] ll_file_release+0x128/0xc50 [lustre] [ 3456.361983] __fput+0x10d/0x440 [ 3456.363615] ____fput+0x12/0x20 [ 3456.365563] task_work_run+0xc6/0x110 [ 3456.367387] exit_to_usermode_loop+0x1dd/0x1f0 [ 3456.369710] do_syscall_64+0x3d6/0x440 [ 3456.371595] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3456.374231] RIP: 0033:0x7fa749552cd8 [ 3456.375914] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3456.383238] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3456.386066] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3456.388992] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3456.391947] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3456.394463] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3456.397444] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3468.057010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3468.057994] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3468.058661] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3468.058669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.058738] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.058743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.058746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3468.058761] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3468.058765] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3468.058770] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3468.058772] RDX: 0000000000003105 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3468.058774] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3468.058776] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3468.058779] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3468.058781] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3468.058785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.058787] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3468.058796] Call Trace: [ 3468.058802] [ 3468.058807] ? show_regs.cold.9+0x22/0x2f [ 3468.058819] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.058826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.058830] ? watchdog+0x40/0x40 [ 3468.058835] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.058839] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.058847] ? apic_timer_interrupt+0xf/0x20 [ 3468.058852] [ 3468.058855] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3468.058859] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3468.058864] do_raw_spin_lock+0x177/0x190 [ 3468.058868] _raw_spin_lock+0x21/0x30 [ 3468.058874] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3468.059300] obd_mod_free+0x46/0x140 [ptlrpc] [ 3468.059635] mdc_commit_open+0x91/0xb0 [mdc] [ 3468.059736] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 3468.059670] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3468.059768] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.059802] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.059806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.059953] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3468.060736] after_reply+0xe2c/0x1780 [ptlrpc] [ 3468.059811] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3468.061184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.061195] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3468.061198] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3468.061213] RAX: 000000000000200a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3468.061216] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3468.061226] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3468.061228] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3468.061231] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3468.061233] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3468.061236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.061238] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3468.061245] Call Trace: [ 3468.061248] [ 3468.061251] ? show_regs.cold.9+0x22/0x2f [ 3468.061260] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.061266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.061270] ? watchdog+0x40/0x40 [ 3468.061274] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.061277] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.061284] ? apic_timer_interrupt+0xf/0x20 [ 3468.061289] [ 3468.061292] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.061297] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3468.061302] do_raw_spin_lock+0x177/0x190 [ 3468.061308] _raw_spin_lock+0x21/0x30 [ 3468.061313] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3468.061602] process_one_work+0x2c8/0x700 [ 3468.061608] worker_thread+0x296/0x6e0 [ 3468.061613] ? rescuer_thread+0x570/0x570 [ 3468.061616] kthread+0x1d1/0x200 [ 3468.061620] ? set_kthread_struct+0x70/0x70 [ 3468.061623] ret_from_fork+0x1f/0x30 [ 3468.062230] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3468.062619] ? __next_timer_interrupt+0x160/0x160 [ 3468.062629] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3468.062834] ? do_wait_intr+0xf0/0xf0 [ 3468.062843] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3468.071375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.071565] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.071572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.071575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.071589] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3468.071594] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3468.071598] RAX: 0000000000001b4b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3468.071601] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3468.071603] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3468.071605] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3468.071608] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3468.071611] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3468.071614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.071616] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3468.071626] Call Trace: [ 3468.071632] [ 3468.071637] ? show_regs.cold.9+0x22/0x2f [ 3468.071647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.071653] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.071657] ? watchdog+0x40/0x40 [ 3468.071661] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.071665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.071674] ? apic_timer_interrupt+0xf/0x20 [ 3468.071679] [ 3468.071682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.077293] kthread+0x1d1/0x200 [ 3468.077313] ? set_kthread_struct+0x70/0x70 [ 3468.077318] ret_from_fork+0x1f/0x30 [ 3468.616219] do_raw_spin_lock+0x177/0x190 [ 3468.623069] _raw_spin_lock+0x21/0x30 [ 3468.624870] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3468.632081] ? do_raw_spin_unlock+0x75/0x190 [ 3468.637828] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3468.640614] ? do_wait_intr+0xf0/0xf0 [ 3468.642918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3468.645984] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3468.650159] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3468.653929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.658754] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3468.664785] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3468.667847] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3468.670784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.679908] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3468.687120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3468.693484] lmv_intent_open+0x338/0x1150 [lmv] [ 3468.696270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.698803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.702071] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3468.704687] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3468.706647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.709067] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3468.712683] ? do_raw_spin_unlock+0x75/0x190 [ 3468.715004] ? _raw_spin_unlock+0x12/0x30 [ 3468.717062] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3468.720014] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3468.722732] ll_atomic_open+0x36a/0xd70 [lustre] [ 3468.727609] lookup_open+0x356/0xb40 [ 3468.730497] do_last+0x313/0xfc0 [ 3468.732007] ? nd_jump_root+0xe5/0x160 [ 3468.734544] ? path_init+0x437/0x520 [ 3468.736482] path_openat+0xf7/0x500 [ 3468.738554] ? do_raw_spin_unlock+0x75/0x190 [ 3468.744640] do_filp_open+0x99/0x140 [ 3468.751004] ? getname_flags+0x6e/0x330 [ 3468.753924] ? __check_object_size+0xff/0x256 [ 3468.758095] ? do_raw_spin_unlock+0x75/0x190 [ 3468.763549] ? _raw_spin_unlock+0x12/0x30 [ 3468.765904] do_sys_openat2+0x2b4/0x410 [ 3468.767794] do_sys_open+0x73/0xa0 [ 3468.769575] __x64_sys_openat+0x24/0x30 [ 3468.771678] do_syscall_64+0xc1/0x440 [ 3468.786478] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3468.790241] RIP: 0033:0x7fa749552332 [ 3468.797683] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3468.813902] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3468.823468] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3468.826864] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3468.830572] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3468.835732] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3468.838997] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3471.329262] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1554s! [ 3471.334735] Showing busy workqueues and worker pools: [ 3471.337750] workqueue events: flags=0x0 [ 3471.339738] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3471.343365] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3471.350081] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3471.354851] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3471.360523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.363038] pending: kfree_rcu_monitor [ 3471.365852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3471.370269] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3471.374851] workqueue events_long: flags=0x0 [ 3471.378132] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.383318] pending: lu_objects_flush [obdclass] [ 3471.388063] workqueue events_unbound: flags=0x2 [ 3471.398121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3471.407295] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3471.415556] workqueue events_power_efficient: flags=0x80 [ 3471.418308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.423303] pending: check_lifetime [ 3471.426631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.431234] pending: neigh_periodic_work [ 3471.436129] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.442014] pending: neigh_periodic_work [ 3471.445086] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.452893] pending: do_cache_clean [ 3471.456759] workqueue mm_percpu_wq: flags=0x8 [ 3471.462460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.471105] pending: vmstat_update [ 3471.474441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.479726] pending: vmstat_update [ 3471.486236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3471.493224] pending: vmstat_update [ 3471.497202] workqueue writeback: flags=0x4a [ 3471.502711] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3471.509958] pending: wb_workfn, wb_workfn [ 3471.514054] workqueue xprtiod: flags=0xa [ 3471.517057] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3471.519777] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3471.526982] workqueue ipv6_addrconf: flags=0x40008 [ 3471.531788] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3471.539176] pending: addrconf_verify_work [ 3471.541636] workqueue knbd0-recv: flags=0x1a [ 3471.544267] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3471.547657] in-flight: 651:recv_work [ 3471.549781] workqueue ptlrpc_pinger: flags=0x2 [ 3471.551507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3471.554347] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3471.557245] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1554s workers=4 idle: 37 31846 28897 [ 3471.561068] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3461s workers=2 idle: 653 [ 3484.059529] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3484.067895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3484.101459] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3484.118422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3484.129888] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3484.136430] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3484.152000] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3484.155734] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3484.159520] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3484.165329] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3484.169959] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3484.171881] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3484.178701] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3484.182849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3484.196316] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3484.198821] Call Trace: [ 3484.204760] [ 3484.206832] ? show_regs.cold.9+0x22/0x2f [ 3484.215896] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3484.217293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3484.218804] ? watchdog+0x40/0x40 [ 3484.225899] ? hrtimer_interrupt+0x128/0x2c0 [ 3484.229357] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3484.231521] ? apic_timer_interrupt+0xf/0x20 [ 3484.234418] [ 3484.238413] ? native_safe_halt+0x2e/0x50 [ 3484.241261] kvm_wait+0xcd/0xe0 [ 3484.247528] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3484.252827] do_raw_spin_lock+0x177/0x190 [ 3484.258332] _raw_spin_lock+0x21/0x30 [ 3484.260442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3484.264430] ? do_raw_spin_unlock+0x75/0x190 [ 3484.265898] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3484.271037] ? do_wait_intr+0xf0/0xf0 [ 3484.273416] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3484.281402] mdc_close+0x53b/0xd70 [mdc] [ 3484.282689] lmv_close+0x18d/0x490 [lmv] [ 3484.283959] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3484.301488] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3484.307404] ll_md_close+0x274/0x800 [lustre] [ 3484.308733] ll_file_release+0x128/0xc50 [lustre] [ 3484.319270] __fput+0x10d/0x440 [ 3484.325711] ____fput+0x12/0x20 [ 3484.328481] task_work_run+0xc6/0x110 [ 3484.335852] exit_to_usermode_loop+0x1dd/0x1f0 [ 3484.341687] do_syscall_64+0x3d6/0x440 [ 3484.344885] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3484.360357] RIP: 0033:0x7fa749552cd8 [ 3484.362073] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3484.392400] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3484.396297] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3484.404085] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3484.412998] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3484.419130] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3484.423804] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3496.054647] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3496.057174] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3496.058442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.058520] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.058527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.058530] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.058545] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3496.058550] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3496.058555] RAX: 000000000000041d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3496.058557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3496.058560] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3496.058562] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3496.058565] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3496.058569] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3496.058572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.058575] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3496.058584] Call Trace: [ 3496.058590] [ 3496.058596] ? show_regs.cold.9+0x22/0x2f [ 3496.058606] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.058612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.058616] ? watchdog+0x40/0x40 [ 3496.058621] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.058625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.058634] ? apic_timer_interrupt+0xf/0x20 [ 3496.058639] [ 3496.058642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.058649] do_raw_spin_lock+0x177/0x190 [ 3496.058655] _raw_spin_lock+0x21/0x30 [ 3496.058660] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3496.059160] ? do_raw_spin_unlock+0x75/0x190 [ 3496.059166] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3496.059376] ? do_wait_intr+0xf0/0xf0 [ 3496.059384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3496.060352] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3496.060611] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3496.060859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.060981] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3496.061020] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3496.061048] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3496.061076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.061167] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3496.061394] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3496.061432] lmv_intent_open+0x338/0x1150 [lmv] [ 3496.061462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.061542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.061623] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3496.061646] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3496.061761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.061872] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3496.061879] Modules linked in: lustre(O) [ 3496.061844] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3496.061885] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 3496.061931] ? do_raw_spin_unlock+0x75/0x190 [ 3496.061938] nfsv4 dns_resolver intel_rapl_msr [ 3496.061944] ? _raw_spin_unlock+0x12/0x30 [ 3496.061950] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix [ 3496.061957] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3496.061978] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.061994] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3496.062000] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.062009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.062011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3496.062024] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3496.062028] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3496.062033] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3496.062036] RDX: 00000000000017b6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3496.062038] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3496.062041] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3496.062043] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3496.062051] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3496.062054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.062057] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3496.062020] ll_atomic_open+0x36a/0xd70 [lustre] [ 3496.062065] Call Trace: [ 3496.062070] [ 3496.062073] ? show_regs.cold.9+0x22/0x2f [ 3496.062082] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.062088] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.062093] ? watchdog+0x40/0x40 [ 3496.062097] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.062102] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.062110] lookup_open+0x356/0xb40 [ 3496.062113] ? apic_timer_interrupt+0xf/0x20 [ 3496.062126] do_last+0x313/0xfc0 [ 3496.062129] [ 3496.062135] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3496.062137] ? nd_jump_root+0xe5/0x160 [ 3496.062147] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3496.062149] ? path_init+0x437/0x520 [ 3496.062163] do_raw_spin_lock+0x177/0x190 [ 3496.062164] path_openat+0xf7/0x500 [ 3496.062178] _raw_spin_lock+0x21/0x30 [ 3496.062179] ? do_raw_spin_unlock+0x75/0x190 [ 3496.062190] do_filp_open+0x99/0x140 [ 3496.062198] ? getname_flags+0x6e/0x330 [ 3496.062203] ? __check_object_size+0xff/0x256 [ 3496.062211] ? do_raw_spin_unlock+0x75/0x190 [ 3496.062215] ? _raw_spin_unlock+0x12/0x30 [ 3496.062220] do_sys_openat2+0x2b4/0x410 [ 3496.062225] do_sys_open+0x73/0xa0 [ 3496.062230] __x64_sys_openat+0x24/0x30 [ 3496.062234] do_syscall_64+0xc1/0x440 [ 3496.062241] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3496.062247] RIP: 0033:0x7fa749552332 [ 3496.062253] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3496.062257] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3496.062261] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3496.062264] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3496.062267] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3496.062269] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3496.062272] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3496.062189] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3496.062449] obd_mod_free+0x46/0x140 [ptlrpc] [ 3496.069229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.069310] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.069318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.069324] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3496.070071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.070084] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3496.070089] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3496.070094] RAX: 000000000000199f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3496.070096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3496.070099] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3496.070101] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3496.070104] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3496.070107] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3496.070110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.070112] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3496.070120] Call Trace: [ 3496.070125] [ 3496.070130] ? show_regs.cold.9+0x22/0x2f [ 3496.070138] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.070144] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.070148] ? watchdog+0x40/0x40 [ 3496.070153] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.070156] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.070165] ? apic_timer_interrupt+0xf/0x20 [ 3496.070169] [ 3496.070172] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.070176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3496.070181] do_raw_spin_lock+0x177/0x190 [ 3496.070186] _raw_spin_lock+0x21/0x30 [ 3496.070191] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3496.070935] process_one_work+0x2c8/0x700 [ 3496.070949] worker_thread+0x296/0x6e0 [ 3496.070954] ? rescuer_thread+0x570/0x570 [ 3496.070957] kthread+0x1d1/0x200 [ 3496.070962] ? set_kthread_struct+0x70/0x70 [ 3496.070965] ret_from_fork+0x1f/0x30 [ 3496.751697] mdc_commit_open+0x91/0xb0 [mdc] [ 3496.753201] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3496.755292] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3496.758091] after_reply+0xe2c/0x1780 [ptlrpc] [ 3496.759812] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3496.766806] ? __next_timer_interrupt+0x160/0x160 [ 3496.769320] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3496.774722] ? do_wait_intr+0xf0/0xf0 [ 3496.779145] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3496.786360] kthread+0x1d1/0x200 [ 3496.788319] ? set_kthread_struct+0x70/0x70 [ 3496.790142] ret_from_fork+0x1f/0x30 [ 3502.057637] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1584s! [ 3502.065224] Showing busy workqueues and worker pools: [ 3502.068262] workqueue events: flags=0x0 [ 3502.077209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3502.079377] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3502.087260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3502.095189] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3502.101584] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.107877] pending: kfree_rcu_monitor [ 3502.111557] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3502.116457] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3502.122191] workqueue events_long: flags=0x0 [ 3502.123751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.127652] pending: lu_objects_flush [obdclass] [ 3502.129984] workqueue events_unbound: flags=0x2 [ 3502.132176] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3502.134926] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3502.138463] workqueue events_power_efficient: flags=0x80 [ 3502.140277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.142957] pending: check_lifetime [ 3502.144405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.146898] pending: neigh_periodic_work [ 3502.148576] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.151285] pending: neigh_periodic_work [ 3502.153732] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.156893] pending: do_cache_clean [ 3502.158384] workqueue mm_percpu_wq: flags=0x8 [ 3502.159796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.161928] pending: vmstat_update [ 3502.163052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.165269] pending: vmstat_update [ 3502.167083] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3502.170029] pending: vmstat_update [ 3502.171870] workqueue writeback: flags=0x4a [ 3502.173807] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3502.176503] pending: wb_workfn, wb_workfn [ 3502.179302] workqueue xprtiod: flags=0xa [ 3502.183214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3502.188844] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3502.193222] workqueue ipv6_addrconf: flags=0x40008 [ 3502.194969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3502.197287] pending: addrconf_verify_work [ 3502.199113] workqueue knbd0-recv: flags=0x1a [ 3502.200867] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3502.203446] in-flight: 651:recv_work [ 3502.205302] workqueue ptlrpc_pinger: flags=0x2 [ 3502.207277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3502.210076] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3502.212784] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1585s workers=4 idle: 37 31846 28897 [ 3502.216290] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3492s workers=2 idle: 653 [ 3512.059192] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3512.067265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3512.098498] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3512.117102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3512.120122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3512.123855] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3512.132671] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3512.136291] RAX: 00000000000011e8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3512.140460] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3512.145116] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3512.148705] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3512.152648] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3512.155069] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3512.157881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3512.160317] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3512.173247] Call Trace: [ 3512.177639] [ 3512.178133] ? show_regs.cold.9+0x22/0x2f [ 3512.189494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3512.191413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3512.193179] ? watchdog+0x40/0x40 [ 3512.194445] ? hrtimer_interrupt+0x128/0x2c0 [ 3512.196546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3512.199483] ? apic_timer_interrupt+0xf/0x20 [ 3512.200901] [ 3512.203728] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3512.207781] do_raw_spin_lock+0x177/0x190 [ 3512.209714] _raw_spin_lock+0x21/0x30 [ 3512.211733] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3512.214591] ? do_raw_spin_unlock+0x75/0x190 [ 3512.216745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3512.220709] ? do_wait_intr+0xf0/0xf0 [ 3512.223343] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3512.225552] mdc_close+0x53b/0xd70 [mdc] [ 3512.226868] lmv_close+0x18d/0x490 [lmv] [ 3512.230536] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3512.233125] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3512.235558] ll_md_close+0x274/0x800 [lustre] [ 3512.237902] ll_file_release+0x128/0xc50 [lustre] [ 3512.239935] __fput+0x10d/0x440 [ 3512.241466] ____fput+0x12/0x20 [ 3512.242749] task_work_run+0xc6/0x110 [ 3512.244060] exit_to_usermode_loop+0x1dd/0x1f0 [ 3512.246042] do_syscall_64+0x3d6/0x440 [ 3512.247213] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3512.248662] RIP: 0033:0x7fa749552cd8 [ 3512.249663] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3512.255208] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3512.257872] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3512.260590] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3512.262632] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3512.264477] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3512.269027] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3524.058758] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3524.059855] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3524.063384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.063666] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.063672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.063678] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3524.064100] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3524.064127] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3524.064131] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3524.064135] RAX: 000000000000778c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3524.064138] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3524.064140] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3524.064143] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3524.064145] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3524.064148] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3524.064150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.064153] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3524.064162] Call Trace: [ 3524.064178] [ 3524.064184] ? show_regs.cold.9+0x22/0x2f [ 3524.064196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.064202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.064207] ? watchdog+0x40/0x40 [ 3524.064211] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.064215] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.064239] ? apic_timer_interrupt+0xf/0x20 [ 3524.064244] [ 3524.064246] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3524.064251] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3524.064257] do_raw_spin_lock+0x177/0x190 [ 3524.064262] _raw_spin_lock+0x21/0x30 [ 3524.064267] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3524.064669] process_one_work+0x2c8/0x700 [ 3524.064677] worker_thread+0x296/0x6e0 [ 3524.064682] ? rescuer_thread+0x570/0x570 [ 3524.064685] kthread+0x1d1/0x200 [ 3524.064689] ? set_kthread_struct+0x70/0x70 [ 3524.064693] ret_from_fork+0x1f/0x30 [ 3524.293135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.346329] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.354874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.359536] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3524.362478] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3524.370281] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3524.373844] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3524.376861] RDX: 0000000000005176 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3524.380158] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3524.383304] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3524.386048] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3524.388690] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3524.399379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.403307] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3524.405875] Call Trace: [ 3524.410503] [ 3524.412317] ? show_regs.cold.9+0x22/0x2f [ 3524.415428] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.419425] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.422151] ? watchdog+0x40/0x40 [ 3524.425880] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.428161] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.430051] ? apic_timer_interrupt+0xf/0x20 [ 3524.431208] [ 3524.431869] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3524.433882] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3524.437142] do_raw_spin_lock+0x177/0x190 [ 3524.444134] _raw_spin_lock+0x21/0x30 [ 3524.447268] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3524.448849] obd_mod_free+0x46/0x140 [ptlrpc] [ 3524.457785] mdc_commit_open+0x91/0xb0 [mdc] [ 3524.462181] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3524.466923] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3524.472879] after_reply+0xe2c/0x1780 [ptlrpc] [ 3524.476212] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3524.480802] ? __next_timer_interrupt+0x160/0x160 [ 3524.483852] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3524.490320] ? do_wait_intr+0xf0/0xf0 [ 3524.491776] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3524.497690] kthread+0x1d1/0x200 [ 3524.499379] ? set_kthread_struct+0x70/0x70 [ 3524.501053] ret_from_fork+0x1f/0x30 [ 3528.056589] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3528.058746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3528.090653] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3528.096108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3528.101853] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3528.104935] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3528.112626] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3528.115646] RAX: 00000000000016fa RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3528.118050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3528.120386] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3528.122774] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3528.125276] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3528.128302] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3528.131001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3528.132853] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3528.134980] Call Trace: [ 3528.135817] [ 3528.136359] ? show_regs.cold.9+0x22/0x2f [ 3528.137915] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3528.139470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3528.146126] ? watchdog+0x40/0x40 [ 3528.147400] ? hrtimer_interrupt+0x128/0x2c0 [ 3528.151407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3528.158878] ? apic_timer_interrupt+0xf/0x20 [ 3528.162285] [ 3528.164386] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3528.167401] do_raw_spin_lock+0x177/0x190 [ 3528.169346] _raw_spin_lock+0x21/0x30 [ 3528.176424] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3528.184712] ? do_raw_spin_unlock+0x75/0x190 [ 3528.189807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3528.199017] ? do_wait_intr+0xf0/0xf0 [ 3528.201890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3528.206498] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3528.209020] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3528.215065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3528.217945] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3528.223130] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3528.225107] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3528.230893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3528.235930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3528.238373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3528.242714] lmv_intent_open+0x338/0x1150 [lmv] [ 3528.244681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3528.250773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3528.254649] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3528.256361] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3528.258180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3528.265002] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3528.269743] ? do_raw_spin_unlock+0x75/0x190 [ 3528.271912] ? _raw_spin_unlock+0x12/0x30 [ 3528.273215] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3528.278829] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3528.283245] ll_atomic_open+0x36a/0xd70 [lustre] [ 3528.285730] lookup_open+0x356/0xb40 [ 3528.286910] do_last+0x313/0xfc0 [ 3528.290141] ? nd_jump_root+0xe5/0x160 [ 3528.292006] ? path_init+0x437/0x520 [ 3528.293288] path_openat+0xf7/0x500 [ 3528.294517] ? do_raw_spin_unlock+0x75/0x190 [ 3528.295880] do_filp_open+0x99/0x140 [ 3528.297177] ? getname_flags+0x6e/0x330 [ 3528.299766] ? __check_object_size+0xff/0x256 [ 3528.301547] ? do_raw_spin_unlock+0x75/0x190 [ 3528.303117] ? _raw_spin_unlock+0x12/0x30 [ 3528.304947] do_sys_openat2+0x2b4/0x410 [ 3528.315076] do_sys_open+0x73/0xa0 [ 3528.316724] __x64_sys_openat+0x24/0x30 [ 3528.319416] do_syscall_64+0xc1/0x440 [ 3528.326258] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3528.334016] RIP: 0033:0x7fa749552332 [ 3528.335834] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3528.370362] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3528.392975] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3528.405135] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3528.419427] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3528.428714] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3528.431377] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3532.256093] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1615s! [ 3532.259439] Showing busy workqueues and worker pools: [ 3532.262741] workqueue events: flags=0x0 [ 3532.264857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3532.268442] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3532.276204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3532.279177] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3532.285646] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.287432] pending: kfree_rcu_monitor [ 3532.288802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3532.290651] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3532.293009] workqueue events_long: flags=0x0 [ 3532.294505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.296763] pending: lu_objects_flush [obdclass] [ 3532.298719] workqueue events_unbound: flags=0x2 [ 3532.300515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3532.302568] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3532.305537] workqueue events_power_efficient: flags=0x80 [ 3532.307220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.309440] pending: check_lifetime [ 3532.310696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.312472] pending: neigh_periodic_work [ 3532.313776] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.316218] pending: neigh_periodic_work [ 3532.317344] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.319236] pending: do_cache_clean [ 3532.320412] workqueue mm_percpu_wq: flags=0x8 [ 3532.321847] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.324202] pending: vmstat_update [ 3532.325542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.328231] pending: vmstat_update [ 3532.329655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3532.331919] pending: vmstat_update [ 3532.333066] workqueue writeback: flags=0x4a [ 3532.334476] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3532.336769] pending: wb_workfn, wb_workfn [ 3532.338217] workqueue xprtiod: flags=0xa [ 3532.339695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3532.342115] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3532.345111] workqueue ipv6_addrconf: flags=0x40008 [ 3532.346470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3532.349122] pending: addrconf_verify_work [ 3532.350776] workqueue knbd0-recv: flags=0x1a [ 3532.352463] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3532.354987] in-flight: 651:recv_work [ 3532.356488] workqueue ptlrpc_pinger: flags=0x2 [ 3532.358205] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3532.360439] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3532.362740] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1615s workers=4 idle: 37 31846 28897 [ 3532.365618] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3522s workers=2 idle: 653 [ 3540.064502] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3540.075208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3540.109197] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3540.118972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3540.124867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3540.130419] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3540.148446] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3540.155117] RAX: 0000000000005c86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3540.160823] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3540.174262] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3540.179499] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3540.181674] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3540.193426] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3540.205231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3540.210741] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3540.218183] Call Trace: [ 3540.219054] [ 3540.219701] ? show_regs.cold.9+0x22/0x2f [ 3540.221856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3540.224384] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3540.227554] ? watchdog+0x40/0x40 [ 3540.230009] ? hrtimer_interrupt+0x128/0x2c0 [ 3540.233506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3540.237433] ? apic_timer_interrupt+0xf/0x20 [ 3540.240029] [ 3540.241081] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3540.242890] do_raw_spin_lock+0x177/0x190 [ 3540.250208] _raw_spin_lock+0x21/0x30 [ 3540.253403] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3540.256548] ? do_raw_spin_unlock+0x75/0x190 [ 3540.258839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3540.261542] ? do_wait_intr+0xf0/0xf0 [ 3540.263772] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3540.267024] mdc_close+0x53b/0xd70 [mdc] [ 3540.269155] lmv_close+0x18d/0x490 [lmv] [ 3540.270861] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3540.277849] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3540.282033] ll_md_close+0x274/0x800 [lustre] [ 3540.284358] ll_file_release+0x128/0xc50 [lustre] [ 3540.289982] __fput+0x10d/0x440 [ 3540.294489] ____fput+0x12/0x20 [ 3540.295733] task_work_run+0xc6/0x110 [ 3540.296659] exit_to_usermode_loop+0x1dd/0x1f0 [ 3540.299278] do_syscall_64+0x3d6/0x440 [ 3540.303212] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3540.306046] RIP: 0033:0x7fa749552cd8 [ 3540.308410] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3540.324580] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3540.332883] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3540.337287] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3540.342627] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3540.349054] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3540.351386] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3552.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3552.059924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.063445] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3552.079194] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.079207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.079214] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3552.079494] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3552.079503] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3552.079507] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.079512] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3552.079514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3552.079516] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3552.079518] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3552.079520] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3552.079523] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3552.079526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.079528] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3552.079537] Call Trace: [ 3552.079542] [ 3552.079547] ? show_regs.cold.9+0x22/0x2f [ 3552.079557] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.079564] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.079568] ? watchdog+0x40/0x40 [ 3552.079572] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.079576] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.079582] ? apic_timer_interrupt+0xf/0x20 [ 3552.079587] [ 3552.079589] ? native_safe_halt+0x2e/0x50 [ 3552.079593] kvm_wait+0xcd/0xe0 [ 3552.079601] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3552.079609] do_raw_spin_lock+0x177/0x190 [ 3552.079613] _raw_spin_lock+0x21/0x30 [ 3552.079616] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3552.079849] process_one_work+0x2c8/0x700 [ 3552.079856] worker_thread+0x296/0x6e0 [ 3552.079861] ? rescuer_thread+0x570/0x570 [ 3552.079864] kthread+0x1d1/0x200 [ 3552.079869] ? set_kthread_struct+0x70/0x70 [ 3552.079872] ret_from_fork+0x1f/0x30 [ 3552.156646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.174294] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.178410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.182507] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3552.185165] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3552.193246] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.200526] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3552.204876] RDX: 0000000000003f1b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3552.209045] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3552.214501] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3552.217287] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3552.220196] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3552.223431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.225934] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3552.228647] Call Trace: [ 3552.229682] [ 3552.230295] ? show_regs.cold.9+0x22/0x2f [ 3552.231705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.233365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.235001] ? watchdog+0x40/0x40 [ 3552.236102] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.237767] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.239482] ? apic_timer_interrupt+0xf/0x20 [ 3552.241003] [ 3552.241600] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3552.243698] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3552.245624] do_raw_spin_lock+0x177/0x190 [ 3552.246921] _raw_spin_lock+0x21/0x30 [ 3552.248184] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3552.250255] obd_mod_free+0x46/0x140 [ptlrpc] [ 3552.252217] mdc_commit_open+0x91/0xb0 [mdc] [ 3552.253919] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3552.255924] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3552.258133] after_reply+0xe2c/0x1780 [ptlrpc] [ 3552.259815] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3552.261902] ? __next_timer_interrupt+0x160/0x160 [ 3552.263980] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3552.265684] ? do_wait_intr+0xf0/0xf0 [ 3552.267340] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3552.269424] kthread+0x1d1/0x200 [ 3552.270793] ? set_kthread_struct+0x70/0x70 [ 3552.272316] ret_from_fork+0x1f/0x30 [ 3560.063005] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3560.076859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3560.116144] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3560.122270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3560.127457] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3560.135515] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3560.155565] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3560.159882] RAX: 000000000000498a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3560.165466] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3560.174267] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3560.181367] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3560.184524] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3560.191610] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3560.199899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3560.207118] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3560.209587] Call Trace: [ 3560.214987] [ 3560.218716] ? show_regs.cold.9+0x22/0x2f [ 3560.228026] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3560.229477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3560.242040] ? watchdog+0x40/0x40 [ 3560.243298] ? hrtimer_interrupt+0x128/0x2c0 [ 3560.247416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3560.252193] ? apic_timer_interrupt+0xf/0x20 [ 3560.257549] [ 3560.259438] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3560.262780] do_raw_spin_lock+0x177/0x190 [ 3560.267455] _raw_spin_lock+0x21/0x30 [ 3560.268724] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3560.270286] ? do_raw_spin_unlock+0x75/0x190 [ 3560.277017] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3560.279879] ? do_wait_intr+0xf0/0xf0 [ 3560.282769] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3560.289883] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3560.295091] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3560.300137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3560.307466] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3560.311474] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3560.313833] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3560.315427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3560.323469] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3560.331779] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3560.336404] lmv_intent_open+0x338/0x1150 [lmv] [ 3560.339632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3560.348264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3560.351887] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3560.353730] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3560.359400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3560.364187] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3560.369113] ? do_raw_spin_unlock+0x75/0x190 [ 3560.372696] ? _raw_spin_unlock+0x12/0x30 [ 3560.379537] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3560.384161] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3560.387448] ll_atomic_open+0x36a/0xd70 [lustre] [ 3560.393025] lookup_open+0x356/0xb40 [ 3560.395472] do_last+0x313/0xfc0 [ 3560.398468] ? nd_jump_root+0xe5/0x160 [ 3560.404636] ? path_init+0x437/0x520 [ 3560.408898] path_openat+0xf7/0x500 [ 3560.413802] ? do_raw_spin_unlock+0x75/0x190 [ 3560.419788] do_filp_open+0x99/0x140 [ 3560.422891] ? getname_flags+0x6e/0x330 [ 3560.424990] ? __check_object_size+0xff/0x256 [ 3560.436684] ? do_raw_spin_unlock+0x75/0x190 [ 3560.438181] ? _raw_spin_unlock+0x12/0x30 [ 3560.439275] do_sys_openat2+0x2b4/0x410 [ 3560.444949] do_sys_open+0x73/0xa0 [ 3560.453602] __x64_sys_openat+0x24/0x30 [ 3560.456875] do_syscall_64+0xc1/0x440 [ 3560.461454] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3560.464464] RIP: 0033:0x7fa749552332 [ 3560.467531] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3560.491782] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3560.497985] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3560.506171] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3560.511815] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3560.518443] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3560.522430] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3562.471251] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1645s! [ 3562.477694] Showing busy workqueues and worker pools: [ 3562.496426] workqueue events: flags=0x0 [ 3562.501672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3562.510433] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3562.527938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3562.534333] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3562.543778] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.548066] pending: kfree_rcu_monitor [ 3562.550596] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3562.554502] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3562.569455] workqueue events_long: flags=0x0 [ 3562.572050] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.586995] pending: lu_objects_flush [obdclass] [ 3562.592350] workqueue events_unbound: flags=0x2 [ 3562.597487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3562.605398] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3562.613457] workqueue events_power_efficient: flags=0x80 [ 3562.618650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.622421] pending: check_lifetime [ 3562.627490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.636802] pending: neigh_periodic_work [ 3562.642838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.648911] pending: neigh_periodic_work [ 3562.655492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.665375] pending: do_cache_clean [ 3562.669982] workqueue mm_percpu_wq: flags=0x8 [ 3562.676064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.683396] pending: vmstat_update [ 3562.691945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.700533] pending: vmstat_update [ 3562.705108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3562.709955] pending: vmstat_update [ 3562.714584] workqueue writeback: flags=0x4a [ 3562.720786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3562.728594] pending: wb_workfn, wb_workfn [ 3562.730230] workqueue xprtiod: flags=0xa [ 3562.737028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3562.746492] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3562.752719] workqueue ipv6_addrconf: flags=0x40008 [ 3562.755484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3562.761534] pending: addrconf_verify_work [ 3562.763326] workqueue knbd0-recv: flags=0x1a [ 3562.770420] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3562.778836] in-flight: 651:recv_work [ 3562.782987] workqueue ptlrpc_pinger: flags=0x2 [ 3562.787409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3562.792446] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3562.796579] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1645s workers=4 idle: 37 31846 28897 [ 3562.803473] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3552s workers=2 idle: 653 [ 3568.059011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3568.063665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3568.087056] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3568.091140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3568.094368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3568.097254] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3568.105952] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3568.109516] RAX: 0000000000006f25 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3568.112807] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3568.116006] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3568.118975] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3568.122029] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3568.124898] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3568.126972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3568.129423] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3568.132297] Call Trace: [ 3568.133753] [ 3568.134595] ? show_regs.cold.9+0x22/0x2f [ 3568.136518] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3568.138844] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3568.140754] ? watchdog+0x40/0x40 [ 3568.142100] ? hrtimer_interrupt+0x128/0x2c0 [ 3568.143936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3568.146329] ? apic_timer_interrupt+0xf/0x20 [ 3568.147989] [ 3568.148777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3568.150934] do_raw_spin_lock+0x177/0x190 [ 3568.152588] _raw_spin_lock+0x21/0x30 [ 3568.153661] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3568.155800] ? do_raw_spin_unlock+0x75/0x190 [ 3568.157767] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3568.160258] ? do_wait_intr+0xf0/0xf0 [ 3568.161784] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3568.164530] mdc_close+0x53b/0xd70 [mdc] [ 3568.166330] lmv_close+0x18d/0x490 [lmv] [ 3568.168056] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3568.170684] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3568.173019] ll_md_close+0x274/0x800 [lustre] [ 3568.174974] ll_file_release+0x128/0xc50 [lustre] [ 3568.177363] __fput+0x10d/0x440 [ 3568.178515] ____fput+0x12/0x20 [ 3568.180163] task_work_run+0xc6/0x110 [ 3568.181826] exit_to_usermode_loop+0x1dd/0x1f0 [ 3568.183848] do_syscall_64+0x3d6/0x440 [ 3568.185444] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3568.187824] RIP: 0033:0x7fa749552cd8 [ 3568.189401] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3568.197744] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3568.201482] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3568.204378] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3568.208150] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3568.211118] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3568.214220] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3580.059091] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3580.060324] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3580.061791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.061858] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.061864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.061866] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3580.061879] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3580.061883] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.061887] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3580.061890] RDX: 000000000000565c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3580.061892] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3580.061894] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3580.061897] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3580.061899] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3580.061902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.061905] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3580.061912] Call Trace: [ 3580.061916] [ 3580.061921] ? show_regs.cold.9+0x22/0x2f [ 3580.061929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.061935] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.061939] ? watchdog+0x40/0x40 [ 3580.061943] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.061947] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.061954] ? apic_timer_interrupt+0xf/0x20 [ 3580.064026] [ 3580.064030] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3580.064034] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3580.064037] do_raw_spin_lock+0x177/0x190 [ 3580.064041] _raw_spin_lock+0x21/0x30 [ 3580.064046] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3580.064281] obd_mod_free+0x46/0x140 [ptlrpc] [ 3580.064424] mdc_commit_open+0x91/0xb0 [mdc] [ 3580.064460] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3580.064604] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3580.064730] after_reply+0xe2c/0x1780 [ptlrpc] [ 3580.064878] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3580.069914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.069996] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.070003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.070009] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3580.070304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.070315] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.070320] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.070325] RAX: 0000000000007788 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3580.070327] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3580.070329] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.070331] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3580.070334] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3580.070336] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3580.070339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.070341] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3580.070350] Call Trace: [ 3580.070354] [ 3580.070360] ? show_regs.cold.9+0x22/0x2f [ 3580.070370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.070376] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.070381] ? watchdog+0x40/0x40 [ 3580.070385] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.070389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.070397] ? apic_timer_interrupt+0xf/0x20 [ 3580.070401] [ 3580.070404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.070409] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3580.070413] do_raw_spin_lock+0x177/0x190 [ 3580.070419] _raw_spin_lock+0x21/0x30 [ 3580.070423] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3580.098788] ? __next_timer_interrupt+0x160/0x160 [ 3580.098809] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3580.099089] ? do_wait_intr+0xf0/0xf0 [ 3580.099096] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3580.099238] kthread+0x1d1/0x200 [ 3580.099252] ? set_kthread_struct+0x70/0x70 [ 3580.099257] ret_from_fork+0x1f/0x30 [ 3580.354349] process_one_work+0x2c8/0x700 [ 3580.355771] worker_thread+0x296/0x6e0 [ 3580.357231] ? rescuer_thread+0x570/0x570 [ 3580.358761] kthread+0x1d1/0x200 [ 3580.360052] ? set_kthread_struct+0x70/0x70 [ 3580.361755] ret_from_fork+0x1f/0x30 [ 3588.054280] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3588.057475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3588.088112] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3588.095477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3588.100724] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3588.104176] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3588.116728] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3588.120975] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3588.126881] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3588.131703] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3588.136170] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3588.139763] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3588.143646] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3588.148085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3588.153203] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3588.157634] Call Trace: [ 3588.159511] [ 3588.161064] ? show_regs.cold.9+0x22/0x2f [ 3588.163867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3588.166945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3588.169967] ? watchdog+0x40/0x40 [ 3588.171708] ? hrtimer_interrupt+0x128/0x2c0 [ 3588.175357] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3588.179288] ? apic_timer_interrupt+0xf/0x20 [ 3588.182089] [ 3588.182763] ? native_safe_halt+0x2e/0x50 [ 3588.191561] kvm_wait+0xcd/0xe0 [ 3588.196033] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3588.200579] do_raw_spin_lock+0x177/0x190 [ 3588.203048] _raw_spin_lock+0x21/0x30 [ 3588.209433] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3588.212751] ? do_raw_spin_unlock+0x75/0x190 [ 3588.219974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3588.224552] ? do_wait_intr+0xf0/0xf0 [ 3588.227493] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3588.233123] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3588.237448] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3588.244680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3588.250296] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3588.256449] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3588.260040] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3588.262765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3588.268437] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3588.276566] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3588.284900] lmv_intent_open+0x338/0x1150 [lmv] [ 3588.293871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3588.303353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3588.306965] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3588.310729] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3588.313313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3588.319795] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3588.322366] ? do_raw_spin_unlock+0x75/0x190 [ 3588.328427] ? _raw_spin_unlock+0x12/0x30 [ 3588.331468] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3588.336188] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3588.344395] ll_atomic_open+0x36a/0xd70 [lustre] [ 3588.347737] lookup_open+0x356/0xb40 [ 3588.349803] do_last+0x313/0xfc0 [ 3588.352485] ? nd_jump_root+0xe5/0x160 [ 3588.359455] ? path_init+0x437/0x520 [ 3588.368726] path_openat+0xf7/0x500 [ 3588.375005] ? do_raw_spin_unlock+0x75/0x190 [ 3588.379963] do_filp_open+0x99/0x140 [ 3588.386412] ? getname_flags+0x6e/0x330 [ 3588.392887] ? __check_object_size+0xff/0x256 [ 3588.399421] ? do_raw_spin_unlock+0x75/0x190 [ 3588.403758] ? _raw_spin_unlock+0x12/0x30 [ 3588.405439] do_sys_openat2+0x2b4/0x410 [ 3588.412173] do_sys_open+0x73/0xa0 [ 3588.418988] __x64_sys_openat+0x24/0x30 [ 3588.424676] do_syscall_64+0xc1/0x440 [ 3588.426493] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3588.428115] RIP: 0033:0x7fa749552332 [ 3588.429370] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3588.452749] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3588.454681] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3588.464185] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3588.471759] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3588.478563] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3588.485677] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3593.193828] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1676s! [ 3593.206362] Showing busy workqueues and worker pools: [ 3593.212029] workqueue events: flags=0x0 [ 3593.215954] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3593.221376] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3593.231166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3593.236550] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3593.246219] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.250265] pending: kfree_rcu_monitor [ 3593.257223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3593.260841] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3593.263244] workqueue events_long: flags=0x0 [ 3593.265328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.269682] pending: lu_objects_flush [obdclass] [ 3593.272375] workqueue events_unbound: flags=0x2 [ 3593.274204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3593.283289] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3593.293255] workqueue events_power_efficient: flags=0x80 [ 3593.295342] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.296759] pending: check_lifetime [ 3593.301717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.306216] pending: neigh_periodic_work [ 3593.308440] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.313298] pending: neigh_periodic_work [ 3593.315961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.320352] pending: do_cache_clean [ 3593.323213] workqueue mm_percpu_wq: flags=0x8 [ 3593.325716] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.330576] pending: vmstat_update [ 3593.332686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.336506] pending: vmstat_update [ 3593.339309] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3593.343370] pending: vmstat_update [ 3593.345908] workqueue writeback: flags=0x4a [ 3593.348660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3593.350818] pending: wb_workfn, wb_workfn [ 3593.352660] workqueue xprtiod: flags=0xa [ 3593.353691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3593.359999] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3593.363875] workqueue ipv6_addrconf: flags=0x40008 [ 3593.369484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3593.376336] pending: addrconf_verify_work [ 3593.380098] workqueue knbd0-recv: flags=0x1a [ 3593.382923] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3593.385568] in-flight: 651:recv_work [ 3593.387280] workqueue ptlrpc_pinger: flags=0x2 [ 3593.389186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3593.391592] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3593.394167] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1676s workers=4 idle: 37 31846 28897 [ 3593.397641] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3583s workers=2 idle: 653 [ 3593.728031] rcu: INFO: rcu_sched self-detected stall on CPU [ 3593.730215] rcu: 2-...!: (912005 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 3593.737267] (t=1680039 jiffies g=258701 q=2012) [ 3593.743307] rcu: rcu_sched kthread starved for 1676834 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3593.747727] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3593.751212] rcu: RCU grace-period kthread stack dump: [ 3593.752880] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3593.760065] Call Trace: [ 3593.764110] __schedule+0x351/0xcb0 [ 3593.766272] schedule+0xc0/0x180 [ 3593.769978] schedule_timeout+0xb4/0x190 [ 3593.773360] ? do_raw_spin_unlock+0x75/0x190 [ 3593.776278] ? __next_timer_interrupt+0x160/0x160 [ 3593.778547] ? prepare_to_swait_event+0xf3/0x140 [ 3593.785177] rcu_gp_kthread+0x8dd/0x15b0 [ 3593.788911] ? force_qs_rnp+0x2c0/0x2c0 [ 3593.791198] kthread+0x1d1/0x200 [ 3593.792782] ? set_kthread_struct+0x70/0x70 [ 3593.799523] ret_from_fork+0x1f/0x30 [ 3593.800752] rcu: Stack dump where RCU GP kthread last ran: [ 3593.805897] NMI backtrace for cpu 2 [ 3593.813167] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3593.830667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3593.838975] Call Trace: [ 3593.840594] [ 3593.842563] ? dump_stack+0xbb/0x10e [ 3593.851134] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3593.861773] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3593.863242] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3593.869627] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3593.872037] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3593.873936] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3593.879639] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 3593.881665] ? update_process_times+0x6d/0xc0 [ 3593.887457] ? tick_sched_handle.isra.18+0x27/0x90 [ 3593.889105] ? tick_sched_timer+0x53/0xb0 [ 3593.892583] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3593.897878] ? tick_sched_handle.isra.18+0x90/0x90 [ 3593.904903] ? hrtimer_interrupt+0x128/0x2c0 [ 3593.908519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3593.912018] ? apic_timer_interrupt+0xf/0x20 [ 3593.913966] [ 3593.915733] ? native_safe_halt+0x2e/0x50 [ 3593.921363] ? kvm_wait+0xcd/0xe0 [ 3593.922621] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3593.935052] ? do_raw_spin_lock+0x177/0x190 [ 3593.940019] ? _raw_spin_lock+0x21/0x30 [ 3593.941958] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3593.945232] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3593.955692] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3593.958408] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3593.960629] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3593.962698] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3593.965603] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3593.972063] ? __next_timer_interrupt+0x160/0x160 [ 3593.976081] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3593.980127] ? do_wait_intr+0xf0/0xf0 [ 3593.982504] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3593.985490] ? kthread+0x1d1/0x200 [ 3593.986682] ? set_kthread_struct+0x70/0x70 [ 3593.989429] ? ret_from_fork+0x1f/0x30 [ 3593.990926] NMI backtrace for cpu 2 [ 3593.992506] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3593.999012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3594.006665] Call Trace: [ 3594.007504] [ 3594.008059] ? dump_stack+0xbb/0x10e [ 3594.017089] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3594.021895] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3594.029540] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3594.033655] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3594.037175] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3594.040141] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3594.041870] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3594.043453] ? update_process_times+0x6d/0xc0 [ 3594.044613] ? tick_sched_handle.isra.18+0x27/0x90 [ 3594.050136] ? tick_sched_timer+0x53/0xb0 [ 3594.056041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3594.057797] ? tick_sched_handle.isra.18+0x90/0x90 [ 3594.059915] ? hrtimer_interrupt+0x128/0x2c0 [ 3594.061645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3594.063709] ? apic_timer_interrupt+0xf/0x20 [ 3594.065333] [ 3594.066103] ? native_safe_halt+0x2e/0x50 [ 3594.067873] ? kvm_wait+0xcd/0xe0 [ 3594.071680] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3594.075417] ? do_raw_spin_lock+0x177/0x190 [ 3594.077205] ? _raw_spin_lock+0x21/0x30 [ 3594.079956] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3594.082078] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3594.084156] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3594.085485] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3594.088869] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3594.091909] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3594.094018] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3594.096297] ? __next_timer_interrupt+0x160/0x160 [ 3594.097817] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3594.099405] ? do_wait_intr+0xf0/0xf0 [ 3594.100893] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3594.104276] ? kthread+0x1d1/0x200 [ 3594.106382] ? set_kthread_struct+0x70/0x70 [ 3594.109286] ? ret_from_fork+0x1f/0x30 [ 3596.060637] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3596.064280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3596.105030] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3596.112339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3596.119592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3596.124461] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3596.137670] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3596.141460] RAX: 0000000000007e0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3596.147611] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3596.151921] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3596.160508] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3596.164336] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3596.171173] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3596.177088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3596.179621] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3596.187500] Call Trace: [ 3596.189147] [ 3596.190263] ? show_regs.cold.9+0x22/0x2f [ 3596.191976] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3596.198440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3596.200956] ? watchdog+0x40/0x40 [ 3596.202629] ? hrtimer_interrupt+0x128/0x2c0 [ 3596.204238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3596.210485] ? apic_timer_interrupt+0xf/0x20 [ 3596.214355] [ 3596.215393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3596.221441] do_raw_spin_lock+0x177/0x190 [ 3596.223922] _raw_spin_lock+0x21/0x30 [ 3596.227494] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3596.230643] ? do_raw_spin_unlock+0x75/0x190 [ 3596.236560] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3596.243638] ? do_wait_intr+0xf0/0xf0 [ 3596.245266] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3596.248654] mdc_close+0x53b/0xd70 [mdc] [ 3596.253982] lmv_close+0x18d/0x490 [lmv] [ 3596.257329] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3596.261054] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3596.268024] ll_md_close+0x274/0x800 [lustre] [ 3596.272434] ll_file_release+0x128/0xc50 [lustre] [ 3596.275081] __fput+0x10d/0x440 [ 3596.277202] ____fput+0x12/0x20 [ 3596.280063] task_work_run+0xc6/0x110 [ 3596.281582] exit_to_usermode_loop+0x1dd/0x1f0 [ 3596.287993] do_syscall_64+0x3d6/0x440 [ 3596.290120] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3596.292201] RIP: 0033:0x7fa749552cd8 [ 3596.293880] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3596.311137] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3596.318867] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3596.323220] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3596.326347] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3596.329805] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3596.336249] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3608.065728] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3608.068842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.116779] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.134544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3608.145707] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3608.155634] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3608.161068] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3608.185781] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3608.191533] RAX: 000000000000088a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3608.195797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3608.199932] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3608.209361] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3608.215620] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3608.223731] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3608.232811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.238733] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3608.241869] Call Trace: [ 3608.243224] [ 3608.244038] ? show_regs.cold.9+0x22/0x2f [ 3608.246106] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.248157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.250157] ? watchdog+0x40/0x40 [ 3608.251800] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.253905] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.255618] ? apic_timer_interrupt+0xf/0x20 [ 3608.257282] [ 3608.258052] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3608.260390] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3608.262872] do_raw_spin_lock+0x177/0x190 [ 3608.264425] _raw_spin_lock+0x21/0x30 [ 3608.266063] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3608.268358] process_one_work+0x2c8/0x700 [ 3608.270007] worker_thread+0x296/0x6e0 [ 3608.271295] ? rescuer_thread+0x570/0x570 [ 3608.272912] kthread+0x1d1/0x200 [ 3608.274051] ? set_kthread_struct+0x70/0x70 [ 3608.275457] ret_from_fork+0x1f/0x30 [ 3620.058073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3620.058479] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3620.062042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3620.062134] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3620.062142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3620.062144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3620.062157] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3620.062162] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3620.062166] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3620.062341] RDX: 0000000000006e6f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3620.062346] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3620.062348] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3620.062351] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3620.062355] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3620.062358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3620.062360] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3620.062369] Call Trace: [ 3620.062376] [ 3620.062381] ? show_regs.cold.9+0x22/0x2f [ 3620.062391] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3620.062398] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3620.062401] ? watchdog+0x40/0x40 [ 3620.062406] ? hrtimer_interrupt+0x128/0x2c0 [ 3620.062410] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3620.062418] ? apic_timer_interrupt+0xf/0x20 [ 3620.062423] [ 3620.062426] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3620.062431] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3620.062436] do_raw_spin_lock+0x177/0x190 [ 3620.062440] _raw_spin_lock+0x21/0x30 [ 3620.062446] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3620.062821] obd_mod_free+0x46/0x140 [ptlrpc] [ 3620.063022] mdc_commit_open+0x91/0xb0 [mdc] [ 3620.063058] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3620.063523] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3620.063787] after_reply+0xe2c/0x1780 [ptlrpc] [ 3620.063997] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3620.064459] ? __next_timer_interrupt+0x160/0x160 [ 3620.064472] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3620.064780] ? do_wait_intr+0xf0/0xf0 [ 3620.064790] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3620.065012] kthread+0x1d1/0x200 [ 3620.065018] ? set_kthread_struct+0x70/0x70 [ 3620.065021] ret_from_fork+0x1f/0x30 [ 3620.258996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3620.294607] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3620.302114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3620.306244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3620.313633] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3620.336477] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3620.341264] RAX: 0000000000006992 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3620.344984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3620.351627] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3620.360020] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3620.366483] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3620.370265] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3620.377583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3620.381828] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3620.385702] Call Trace: [ 3620.387599] [ 3620.388337] ? show_regs.cold.9+0x22/0x2f [ 3620.391103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3620.394762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3620.399957] ? watchdog+0x40/0x40 [ 3620.404506] ? hrtimer_interrupt+0x128/0x2c0 [ 3620.406082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3620.416127] ? apic_timer_interrupt+0xf/0x20 [ 3620.421635] [ 3620.423185] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3620.424987] do_raw_spin_lock+0x177/0x190 [ 3620.426483] _raw_spin_lock+0x21/0x30 [ 3620.430153] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3620.433015] ? do_raw_spin_unlock+0x75/0x190 [ 3620.436158] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3620.442590] ? do_wait_intr+0xf0/0xf0 [ 3620.443863] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3620.456203] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3620.460547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3620.465199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.477423] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3620.483897] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3620.486844] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3620.491602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.493278] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3620.499323] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3620.501111] lmv_intent_open+0x338/0x1150 [lmv] [ 3620.505747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.507491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.518680] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3620.522656] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3620.527774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.536740] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3620.540104] ? do_raw_spin_unlock+0x75/0x190 [ 3620.544055] ? _raw_spin_unlock+0x12/0x30 [ 3620.548453] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3620.551731] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3620.554791] ll_atomic_open+0x36a/0xd70 [lustre] [ 3620.557826] lookup_open+0x356/0xb40 [ 3620.559709] do_last+0x313/0xfc0 [ 3620.561645] ? nd_jump_root+0xe5/0x160 [ 3620.566142] ? path_init+0x437/0x520 [ 3620.571234] path_openat+0xf7/0x500 [ 3620.572970] ? do_raw_spin_unlock+0x75/0x190 [ 3620.579194] do_filp_open+0x99/0x140 [ 3620.584789] ? getname_flags+0x6e/0x330 [ 3620.586045] ? __check_object_size+0xff/0x256 [ 3620.591090] ? do_raw_spin_unlock+0x75/0x190 [ 3620.594666] ? _raw_spin_unlock+0x12/0x30 [ 3620.604021] do_sys_openat2+0x2b4/0x410 [ 3620.605283] do_sys_open+0x73/0xa0 [ 3620.608217] __x64_sys_openat+0x24/0x30 [ 3620.611031] do_syscall_64+0xc1/0x440 [ 3620.612667] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3620.622390] RIP: 0033:0x7fa749552332 [ 3620.623617] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3620.651955] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3620.661758] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3620.666435] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3620.675507] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3620.687389] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3620.695608] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3623.909324] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1706s! [ 3623.916797] Showing busy workqueues and worker pools: [ 3623.923144] workqueue events: flags=0x0 [ 3623.929137] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3623.937003] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3623.947310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3623.952847] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3623.961967] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3623.966344] pending: kfree_rcu_monitor [ 3623.969129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3623.975957] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3623.982913] workqueue events_long: flags=0x0 [ 3623.985794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3623.993009] pending: lu_objects_flush [obdclass] [ 3623.999591] workqueue events_unbound: flags=0x2 [ 3624.003964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3624.008805] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3624.017385] workqueue events_power_efficient: flags=0x80 [ 3624.021306] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3624.027779] pending: check_lifetime [ 3624.034819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3624.041451] pending: neigh_periodic_work [ 3624.047180] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3624.056802] pending: neigh_periodic_work [ 3624.059108] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3624.061597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3624.061617] pending: do_cache_clean [ 3624.061632] workqueue mm_percpu_wq: flags=0x8 [ 3624.061635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3624.061642] pending: vmstat_update [ 3624.061649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3624.061656] pending: vmstat_update [ 3624.061662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3624.061669] pending: vmstat_update [ 3624.061675] workqueue writeback: flags=0x4a [ 3624.061677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3624.061684] pending: wb_workfn, wb_workfn [ 3624.062030] workqueue xprtiod: flags=0xa [ 3624.062035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3624.062044] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3624.062064] workqueue ipv6_addrconf: flags=0x40008 [ 3624.062067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3624.062073] pending: addrconf_verify_work [ 3624.062418] workqueue knbd0-recv: flags=0x1a [ 3624.062421] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3624.062428] in-flight: 651:recv_work [ 3624.062437] workqueue ptlrpc_pinger: flags=0x2 [ 3624.062439] pwq 8: [ 3624.070582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3624.070663] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3624.070672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3624.070675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3624.070689] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3624.070693] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3624.070698] RAX: 0000000000005630 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3624.070700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3624.070703] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3624.070705] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3624.070707] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3624.070872] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3624.070875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3624.070878] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3624.070891] Call Trace: [ 3624.070896] [ 3624.070902] ? show_regs.cold.9+0x22/0x2f [ 3624.070912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3624.070918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3624.070923] ? watchdog+0x40/0x40 [ 3624.070928] ? hrtimer_interrupt+0x128/0x2c0 [ 3624.070932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3624.070940] ? apic_timer_interrupt+0xf/0x20 [ 3624.070945] [ 3624.070948] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3624.070955] do_raw_spin_lock+0x177/0x190 [ 3624.070960] _raw_spin_lock+0x21/0x30 [ 3624.070965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3624.071339] ? do_raw_spin_unlock+0x75/0x190 [ 3624.071345] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3624.071567] ? do_wait_intr+0xf0/0xf0 [ 3624.071574] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3624.072321] mdc_close+0x53b/0xd70 [mdc] [ 3624.072363] lmv_close+0x18d/0x490 [lmv] [ 3624.072391] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3624.072521] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3624.072603] ll_md_close+0x274/0x800 [lustre] [ 3624.072689] ll_file_release+0x128/0xc50 [lustre] [ 3624.072782] __fput+0x10d/0x440 [ 3624.072792] ____fput+0x12/0x20 [ 3624.072797] task_work_run+0xc6/0x110 [ 3624.072808] exit_to_usermode_loop+0x1dd/0x1f0 [ 3624.072815] do_syscall_64+0x3d6/0x440 [ 3624.072820] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3624.072828] RIP: 0033:0x7fa749552cd8 [ 3624.072835] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3624.072842] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3624.072848] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3624.072850] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3624.072852] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3624.072855] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3624.072857] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3624.430839] cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3624.433403] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3624.439379] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1707s workers=4 idle: 37 31846 28897 [ 3624.445420] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3614s workers=2 idle: 653 [ 3636.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3636.060595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3636.086907] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3636.091785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3636.095756] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3636.098776] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3636.100777] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3636.123789] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3636.135853] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3636.150881] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3636.153149] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3636.166971] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3636.178233] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3636.184531] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3636.190700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3636.197746] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3636.212402] Call Trace: [ 3636.213910] [ 3636.215158] ? show_regs.cold.9+0x22/0x2f [ 3636.220118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3636.223909] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3636.226484] ? watchdog+0x40/0x40 [ 3636.228168] ? hrtimer_interrupt+0x128/0x2c0 [ 3636.230561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3636.232563] ? apic_timer_interrupt+0xf/0x20 [ 3636.236729] [ 3636.237490] ? native_safe_halt+0x2e/0x50 [ 3636.239724] kvm_wait+0xcd/0xe0 [ 3636.240956] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3636.247382] do_raw_spin_lock+0x177/0x190 [ 3636.251362] _raw_spin_lock+0x21/0x30 [ 3636.255040] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3636.259014] process_one_work+0x2c8/0x700 [ 3636.260670] worker_thread+0x296/0x6e0 [ 3636.261855] ? rescuer_thread+0x570/0x570 [ 3636.263159] kthread+0x1d1/0x200 [ 3636.264159] ? set_kthread_struct+0x70/0x70 [ 3636.269390] ret_from_fork+0x1f/0x30 [ 3648.058561] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3648.065269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3648.107125] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3648.129869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3648.140150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3648.152332] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3648.174371] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3648.188733] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3648.202537] RDX: 0000000000006b99 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3648.205881] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3648.213911] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3648.231161] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3648.239927] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3648.246825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3648.251083] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3648.263006] Call Trace: [ 3648.265186] [ 3648.269193] ? show_regs.cold.9+0x22/0x2f [ 3648.273573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3648.280964] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3648.283460] ? watchdog+0x40/0x40 [ 3648.286665] ? hrtimer_interrupt+0x128/0x2c0 [ 3648.290017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3648.292803] ? apic_timer_interrupt+0xf/0x20 [ 3648.294868] [ 3648.298664] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3648.302470] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3648.308611] do_raw_spin_lock+0x177/0x190 [ 3648.313014] _raw_spin_lock+0x21/0x30 [ 3648.316814] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3648.324914] obd_mod_free+0x46/0x140 [ptlrpc] [ 3648.331045] mdc_commit_open+0x91/0xb0 [mdc] [ 3648.332441] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3648.343750] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3648.345633] after_reply+0xe2c/0x1780 [ptlrpc] [ 3648.354854] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3648.356765] ? __next_timer_interrupt+0x160/0x160 [ 3648.365123] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3648.367198] ? do_wait_intr+0xf0/0xf0 [ 3648.370143] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3648.381668] kthread+0x1d1/0x200 [ 3648.383525] ? set_kthread_struct+0x70/0x70 [ 3648.385905] ret_from_fork+0x1f/0x30 [ 3652.055855] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3652.062153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3652.073962] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3652.098616] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3652.098632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3652.098636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3652.098652] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3652.098657] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3652.098662] RAX: 000000000000555f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3652.098665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3652.098667] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3652.098669] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3652.098671] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3652.098675] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3652.098677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3652.098679] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3652.098688] Call Trace: [ 3652.098693] [ 3652.098709] ? show_regs.cold.9+0x22/0x2f [ 3652.098719] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3652.098726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3652.098730] ? watchdog+0x40/0x40 [ 3652.098734] ? hrtimer_interrupt+0x128/0x2c0 [ 3652.098739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3652.098746] ? apic_timer_interrupt+0xf/0x20 [ 3652.098750] [ 3652.098753] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3652.098761] do_raw_spin_lock+0x177/0x190 [ 3652.098766] _raw_spin_lock+0x21/0x30 [ 3652.098771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3652.099199] ? do_raw_spin_unlock+0x75/0x190 [ 3652.099274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3652.099622] ? do_wait_intr+0xf0/0xf0 [ 3652.099630] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3652.100122] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3652.100552] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3652.100826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3652.101021] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3652.101060] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3652.101090] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3652.101117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3652.101198] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3652.107145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3652.107218] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3652.107226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3652.107229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3652.107244] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3652.107248] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3652.107253] RAX: 0000000000007c52 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3652.107255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3652.107258] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3652.107260] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3652.107262] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3652.107265] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3652.107268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3652.107270] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3652.107279] Call Trace: [ 3652.107284] [ 3652.107290] ? show_regs.cold.9+0x22/0x2f [ 3652.107300] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3652.107306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3652.107311] ? watchdog+0x40/0x40 [ 3652.107595] ? hrtimer_interrupt+0x128/0x2c0 [ 3652.107600] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3652.107609] ? apic_timer_interrupt+0xf/0x20 [ 3652.107613] [ 3652.107617] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3652.107624] do_raw_spin_lock+0x177/0x190 [ 3652.107629] _raw_spin_lock+0x21/0x30 [ 3652.107634] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3652.108071] ? do_raw_spin_unlock+0x75/0x190 [ 3652.108078] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3652.108417] ? do_wait_intr+0xf0/0xf0 [ 3652.108425] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3652.108776] mdc_close+0x53b/0xd70 [mdc] [ 3652.108814] lmv_close+0x18d/0x490 [lmv] [ 3652.108842] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3652.109028] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3652.109115] ll_md_close+0x274/0x800 [lustre] [ 3652.109198] ll_file_release+0x128/0xc50 [lustre] [ 3652.109421] __fput+0x10d/0x440 [ 3652.109430] ____fput+0x12/0x20 [ 3652.109435] task_work_run+0xc6/0x110 [ 3652.109443] exit_to_usermode_loop+0x1dd/0x1f0 [ 3652.109450] do_syscall_64+0x3d6/0x440 [ 3652.109455] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3652.109462] RIP: 0033:0x7fa749552cd8 [ 3652.109469] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3652.109473] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3652.109550] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3652.109553] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3652.109556] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3652.109558] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3652.109560] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3652.391260] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3652.393089] lmv_intent_open+0x338/0x1150 [lmv] [ 3652.394751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3652.397178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3652.399561] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3652.401521] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3652.403525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3652.405867] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3652.407936] ? do_raw_spin_unlock+0x75/0x190 [ 3652.409648] ? _raw_spin_unlock+0x12/0x30 [ 3652.411351] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3652.413000] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3652.414656] ll_atomic_open+0x36a/0xd70 [lustre] [ 3652.416545] lookup_open+0x356/0xb40 [ 3652.418078] do_last+0x313/0xfc0 [ 3652.419501] ? nd_jump_root+0xe5/0x160 [ 3652.421058] ? path_init+0x437/0x520 [ 3652.422583] path_openat+0xf7/0x500 [ 3652.424076] ? do_raw_spin_unlock+0x75/0x190 [ 3652.425793] do_filp_open+0x99/0x140 [ 3652.426772] ? getname_flags+0x6e/0x330 [ 3652.427828] ? __check_object_size+0xff/0x256 [ 3652.428999] ? do_raw_spin_unlock+0x75/0x190 [ 3652.430867] ? _raw_spin_unlock+0x12/0x30 [ 3652.432551] do_sys_openat2+0x2b4/0x410 [ 3652.434192] do_sys_open+0x73/0xa0 [ 3652.435686] __x64_sys_openat+0x24/0x30 [ 3652.437351] do_syscall_64+0xc1/0x440 [ 3652.438631] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3652.440159] RIP: 0033:0x7fa749552332 [ 3652.441339] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3652.449076] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3652.452331] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3652.454701] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3652.458992] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3652.463435] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3652.468175] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3654.627286] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1737s! [ 3654.632370] Showing busy workqueues and worker pools: [ 3654.634511] workqueue events: flags=0x0 [ 3654.636337] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3654.638694] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3654.643676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3654.645913] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3654.649342] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.652324] pending: kfree_rcu_monitor [ 3654.654096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3654.657292] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 3654.659570] workqueue events_long: flags=0x0 [ 3654.661393] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.664095] pending: lu_objects_flush [obdclass] [ 3654.666129] workqueue events_unbound: flags=0x2 [ 3654.668183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3654.670428] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3654.676600] workqueue events_power_efficient: flags=0x80 [ 3654.679511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.683368] pending: check_lifetime [ 3654.684927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.689254] pending: neigh_periodic_work [ 3654.691861] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.695388] pending: neigh_periodic_work [ 3654.698205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.701490] pending: do_cache_clean [ 3654.703484] workqueue mm_percpu_wq: flags=0x8 [ 3654.706146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.709815] pending: vmstat_update [ 3654.711726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.715224] pending: vmstat_update [ 3654.717248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3654.721219] pending: vmstat_update [ 3654.723258] workqueue writeback: flags=0x4a [ 3654.725376] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3654.728466] pending: wb_workfn, wb_workfn [ 3654.731314] workqueue xprtiod: flags=0xa [ 3654.732710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3654.734959] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3654.738593] workqueue ipv6_addrconf: flags=0x40008 [ 3654.741254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3654.747330] pending: addrconf_verify_work [ 3654.749404] workqueue knbd0-recv: flags=0x1a [ 3654.751518] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3654.754473] in-flight: 651:recv_work [ 3654.756988] workqueue ptlrpc_pinger: flags=0x2 [ 3654.761287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3654.765005] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3654.770321] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1737s workers=4 idle: 37 31846 28897 [ 3654.775622] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3644s workers=2 idle: 653 [ 3664.061045] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3664.068638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3664.110696] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3664.120365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3664.139189] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3664.141954] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3664.144688] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3664.159966] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3664.166225] RAX: 0000000000007908 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3664.173213] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3664.180593] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3664.188579] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3664.199845] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3664.207064] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3664.213961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3664.217973] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3664.225949] Call Trace: [ 3664.227267] [ 3664.228808] ? show_regs.cold.9+0x22/0x2f [ 3664.231199] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3664.238105] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3664.241102] ? watchdog+0x40/0x40 [ 3664.246619] ? hrtimer_interrupt+0x128/0x2c0 [ 3664.249973] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3664.255575] ? apic_timer_interrupt+0xf/0x20 [ 3664.257916] [ 3664.259638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3664.276278] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3664.280621] do_raw_spin_lock+0x177/0x190 [ 3664.285057] _raw_spin_lock+0x21/0x30 [ 3664.286351] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3664.292007] process_one_work+0x2c8/0x700 [ 3664.293237] worker_thread+0x296/0x6e0 [ 3664.300278] ? rescuer_thread+0x570/0x570 [ 3664.302103] kthread+0x1d1/0x200 [ 3664.309145] ? set_kthread_struct+0x70/0x70 [ 3664.312713] ret_from_fork+0x1f/0x30 [ 3676.063809] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3676.078535] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3676.147848] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3676.167264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3676.181092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3676.194207] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3676.219423] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3676.234732] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3676.251741] RDX: 00000000000016e1 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3676.266737] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3676.276108] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3676.286802] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3676.292473] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3676.296645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3676.301648] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3676.307239] Call Trace: [ 3676.311018] [ 3676.311766] ? show_regs.cold.9+0x22/0x2f [ 3676.320986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3676.322686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3676.325428] ? watchdog+0x40/0x40 [ 3676.326722] ? hrtimer_interrupt+0x128/0x2c0 [ 3676.332062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3676.334400] ? apic_timer_interrupt+0xf/0x20 [ 3676.336222] [ 3676.337238] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3676.339709] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3676.342173] do_raw_spin_lock+0x177/0x190 [ 3676.344339] _raw_spin_lock+0x21/0x30 [ 3676.345781] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3676.348724] obd_mod_free+0x46/0x140 [ptlrpc] [ 3676.351529] mdc_commit_open+0x91/0xb0 [mdc] [ 3676.353445] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3676.355812] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3676.358224] after_reply+0xe2c/0x1780 [ptlrpc] [ 3676.360491] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3676.365718] ? __next_timer_interrupt+0x160/0x160 [ 3676.368532] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3676.373671] ? do_wait_intr+0xf0/0xf0 [ 3676.375807] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3676.378342] kthread+0x1d1/0x200 [ 3676.379740] ? set_kthread_struct+0x70/0x70 [ 3676.381224] ret_from_fork+0x1f/0x30 [ 3680.054867] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3680.059456] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3680.066193] Modules linked in: lustre(O) [ 3680.070114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 3680.077640] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3680.077714] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3680.077721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3680.077723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3680.077737] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3680.077741] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3680.077745] RAX: 0000000000007e1b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3680.077748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3680.077750] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3680.077752] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3680.077754] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3680.077757] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3680.077760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3680.077762] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3680.077773] Call Trace: [ 3680.077778] [ 3680.077783] ? show_regs.cold.9+0x22/0x2f [ 3680.077792] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3680.077798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3680.077802] ? watchdog+0x40/0x40 [ 3680.077806] ? hrtimer_interrupt+0x128/0x2c0 [ 3680.077809] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3680.077817] ? apic_timer_interrupt+0xf/0x20 [ 3680.077821] [ 3680.077824] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3680.077830] do_raw_spin_lock+0x177/0x190 [ 3680.077835] _raw_spin_lock+0x21/0x30 [ 3680.077839] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3680.121830] sha512_generic [ 3680.121842] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3680.121849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3680.121852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3680.121866] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3680.121870] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3680.121874] RAX: 000000000000228f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3680.121876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3680.121879] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3680.121881] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3680.121883] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3680.121886] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3680.121889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3680.121892] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3680.121900] Call Trace: [ 3680.121905] [ 3680.121911] ? show_regs.cold.9+0x22/0x2f [ 3680.121920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3680.121926] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3680.121931] ? watchdog+0x40/0x40 [ 3680.121935] ? hrtimer_interrupt+0x128/0x2c0 [ 3680.121939] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3680.121946] ? apic_timer_interrupt+0xf/0x20 [ 3680.121950] [ 3680.121954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3680.121961] do_raw_spin_lock+0x177/0x190 [ 3680.121967] _raw_spin_lock+0x21/0x30 [ 3680.121972] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3680.122258] ? do_raw_spin_unlock+0x75/0x190 [ 3680.122265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3680.122474] ? do_wait_intr+0xf0/0xf0 [ 3680.122482] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3680.122689] mdc_close+0x53b/0xd70 [mdc] [ 3680.122726] lmv_close+0x18d/0x490 [lmv] [ 3680.122763] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3680.122893] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3680.122979] ll_md_close+0x274/0x800 [lustre] [ 3680.123067] ll_file_release+0x128/0xc50 [lustre] [ 3680.123145] __fput+0x10d/0x440 [ 3680.123153] ____fput+0x12/0x20 [ 3680.123158] task_work_run+0xc6/0x110 [ 3680.123164] exit_to_usermode_loop+0x1dd/0x1f0 [ 3680.123172] do_syscall_64+0x3d6/0x440 [ 3680.123176] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3680.123182] RIP: 0033:0x7fa749552cd8 [ 3680.123188] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3680.123191] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3680.123195] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3680.123197] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3680.123199] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3680.123201] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3680.123202] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3680.935437] ? do_raw_spin_unlock+0x75/0x190 [ 3680.936893] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3680.956269] ? do_wait_intr+0xf0/0xf0 [ 3680.962024] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3680.974571] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3680.981401] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3680.991860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3680.998294] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3681.000900] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3681.006591] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3681.010661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3681.020978] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3681.031799] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3681.037016] lmv_intent_open+0x338/0x1150 [lmv] [ 3681.044752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3681.053749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3681.056462] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3681.060865] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3681.074045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3681.075977] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3681.084978] ? do_raw_spin_unlock+0x75/0x190 [ 3681.094468] ? _raw_spin_unlock+0x12/0x30 [ 3681.103662] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3681.109947] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3681.119512] ll_atomic_open+0x36a/0xd70 [lustre] [ 3681.125769] lookup_open+0x356/0xb40 [ 3681.129878] do_last+0x313/0xfc0 [ 3681.131972] ? nd_jump_root+0xe5/0x160 [ 3681.139603] ? path_init+0x437/0x520 [ 3681.143708] path_openat+0xf7/0x500 [ 3681.144900] ? do_raw_spin_unlock+0x75/0x190 [ 3681.146390] do_filp_open+0x99/0x140 [ 3681.155814] ? getname_flags+0x6e/0x330 [ 3681.159679] ? __check_object_size+0xff/0x256 [ 3681.171353] ? do_raw_spin_unlock+0x75/0x190 [ 3681.175942] ? _raw_spin_unlock+0x12/0x30 [ 3681.185882] do_sys_openat2+0x2b4/0x410 [ 3681.188255] do_sys_open+0x73/0xa0 [ 3681.189317] __x64_sys_openat+0x24/0x30 [ 3681.204554] do_syscall_64+0xc1/0x440 [ 3681.212372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3681.223938] RIP: 0033:0x7fa749552332 [ 3681.228868] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3681.253979] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3681.271697] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3681.274013] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3681.286883] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3681.291813] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3681.305163] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3684.841320] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1767s! [ 3684.847603] Showing busy workqueues and worker pools: [ 3684.851747] workqueue events: flags=0x0 [ 3684.856235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3684.860452] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3684.868171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3684.872435] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3684.877989] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3684.881594] pending: kfree_rcu_monitor [ 3684.883758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3684.888662] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3684.895596] workqueue events_long: flags=0x0 [ 3684.897779] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3684.905286] pending: lu_objects_flush [obdclass] [ 3684.912748] workqueue events_unbound: flags=0x2 [ 3684.918740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3684.929089] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3684.935841] workqueue events_power_efficient: flags=0x80 [ 3684.938948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3684.948586] pending: check_lifetime [ 3684.958928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3684.964848] pending: neigh_periodic_work [ 3684.971359] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3684.975601] pending: neigh_periodic_work [ 3684.979841] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3684.983717] pending: do_cache_clean [ 3684.987031] workqueue mm_percpu_wq: flags=0x8 [ 3684.988643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3684.997141] pending: vmstat_update [ 3685.000415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3685.011264] pending: vmstat_update [ 3685.014428] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3685.021298] pending: vmstat_update [ 3685.024639] workqueue writeback: flags=0x4a [ 3685.031424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3685.035579] pending: wb_workfn, wb_workfn [ 3685.038697] workqueue xprtiod: flags=0xa [ 3685.046288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3685.050487] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3685.059642] workqueue ipv6_addrconf: flags=0x40008 [ 3685.062441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3685.071351] pending: addrconf_verify_work [ 3685.076193] workqueue knbd0-recv: flags=0x1a [ 3685.081293] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3685.086708] in-flight: 651:recv_work [ 3685.091522] workqueue ptlrpc_pinger: flags=0x2 [ 3685.094385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3685.114339] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3685.127681] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1767s workers=4 idle: 37 31846 28897 [ 3685.137520] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3675s workers=2 idle: 653 [ 3692.060567] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3692.064395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3692.109340] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3692.119265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3692.127885] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3692.135365] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3692.143383] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3692.152724] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3692.156229] RAX: 0000000000006c0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3692.159259] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3692.162465] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3692.166165] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3692.171142] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3692.174608] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3692.178716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3692.181132] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3692.197509] Call Trace: [ 3692.199057] [ 3692.201208] ? show_regs.cold.9+0x22/0x2f [ 3692.209741] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3692.213733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3692.215327] ? watchdog+0x40/0x40 [ 3692.217163] ? hrtimer_interrupt+0x128/0x2c0 [ 3692.219118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3692.221710] ? apic_timer_interrupt+0xf/0x20 [ 3692.224860] [ 3692.227396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3692.229591] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3692.232824] do_raw_spin_lock+0x177/0x190 [ 3692.236337] _raw_spin_lock+0x21/0x30 [ 3692.240971] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3692.247063] process_one_work+0x2c8/0x700 [ 3692.249858] worker_thread+0x296/0x6e0 [ 3692.254194] ? rescuer_thread+0x570/0x570 [ 3692.258266] kthread+0x1d1/0x200 [ 3692.263032] ? set_kthread_struct+0x70/0x70 [ 3692.264518] ret_from_fork+0x1f/0x30 [ 3704.058022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3704.061769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3704.076403] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3704.079778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3704.082571] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3704.083946] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3704.089898] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3704.092133] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3704.094284] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3704.097045] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3704.098997] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3704.100972] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3704.102985] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3704.106684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3704.109009] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3704.111342] Call Trace: [ 3704.112024] [ 3704.112516] ? show_regs.cold.9+0x22/0x2f [ 3704.113615] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3704.115431] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3704.118235] ? watchdog+0x40/0x40 [ 3704.119437] ? hrtimer_interrupt+0x128/0x2c0 [ 3704.121487] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3704.122763] ? apic_timer_interrupt+0xf/0x20 [ 3704.123968] [ 3704.124566] ? native_safe_halt+0x2e/0x50 [ 3704.125738] kvm_wait+0xcd/0xe0 [ 3704.126554] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3704.129531] do_raw_spin_lock+0x177/0x190 [ 3704.130868] _raw_spin_lock+0x21/0x30 [ 3704.132219] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3704.134375] obd_mod_free+0x46/0x140 [ptlrpc] [ 3704.136166] mdc_commit_open+0x91/0xb0 [mdc] [ 3704.137475] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3704.139268] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3704.142173] after_reply+0xe2c/0x1780 [ptlrpc] [ 3704.145117] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3704.147485] ? __next_timer_interrupt+0x160/0x160 [ 3704.149721] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3704.152317] ? do_wait_intr+0xf0/0xf0 [ 3704.153502] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3704.155534] kthread+0x1d1/0x200 [ 3704.156591] ? set_kthread_struct+0x70/0x70 [ 3704.159762] ret_from_fork+0x1f/0x30 [ 3708.059265] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3708.064126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3708.087649] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3708.091823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3708.095320] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3708.097079] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3708.104064] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3708.107084] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3708.109922] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3708.112821] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3708.115540] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3708.118095] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3708.121314] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3708.124395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3708.126449] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3708.128873] Call Trace: [ 3708.129765] [ 3708.130859] ? show_regs.cold.9+0x22/0x2f [ 3708.132364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3708.134219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3708.136072] ? watchdog+0x40/0x40 [ 3708.137511] ? hrtimer_interrupt+0x128/0x2c0 [ 3708.139143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3708.141189] ? apic_timer_interrupt+0xf/0x20 [ 3708.142710] [ 3708.143570] ? native_safe_halt+0x2e/0x50 [ 3708.145203] kvm_wait+0xcd/0xe0 [ 3708.146360] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3708.148379] do_raw_spin_lock+0x177/0x190 [ 3708.149781] _raw_spin_lock+0x21/0x30 [ 3708.151407] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3708.153431] ? do_raw_spin_unlock+0x75/0x190 [ 3708.155024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3708.157095] ? do_wait_intr+0xf0/0xf0 [ 3708.158669] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3708.160595] mdc_close+0x53b/0xd70 [mdc] [ 3708.162431] lmv_close+0x18d/0x490 [lmv] [ 3708.163798] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3708.166495] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3708.168727] ll_md_close+0x274/0x800 [lustre] [ 3708.170426] ll_file_release+0x128/0xc50 [lustre] [ 3708.172339] __fput+0x10d/0x440 [ 3708.173601] ____fput+0x12/0x20 [ 3708.174958] task_work_run+0xc6/0x110 [ 3708.176322] exit_to_usermode_loop+0x1dd/0x1f0 [ 3708.177872] do_syscall_64+0x3d6/0x440 [ 3708.179361] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3708.181323] RIP: 0033:0x7fa749552cd8 [ 3708.182662] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3708.189428] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3708.192306] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3708.195109] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3708.197662] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3708.200336] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3708.203014] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3712.060437] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3712.066386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3712.111022] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3712.123214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3712.135625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3712.147186] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3712.164076] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3712.169969] RAX: 0000000000002a0d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3712.176417] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3712.187012] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3712.192578] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3712.203969] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3712.211141] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3712.220675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3712.228489] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3712.235382] Call Trace: [ 3712.236620] [ 3712.239392] ? show_regs.cold.9+0x22/0x2f [ 3712.243768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3712.249526] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3712.255201] ? watchdog+0x40/0x40 [ 3712.261151] ? hrtimer_interrupt+0x128/0x2c0 [ 3712.264617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3712.274967] ? apic_timer_interrupt+0xf/0x20 [ 3712.283369] [ 3712.288959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3712.293670] do_raw_spin_lock+0x177/0x190 [ 3712.313952] _raw_spin_lock+0x21/0x30 [ 3712.318737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3712.336711] ? do_raw_spin_unlock+0x75/0x190 [ 3712.341602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3712.346073] ? do_wait_intr+0xf0/0xf0 [ 3712.347723] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3712.357560] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3712.363851] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3712.367594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.371660] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3712.376400] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3712.379104] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3712.382756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.388970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3712.393959] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3712.400065] lmv_intent_open+0x338/0x1150 [lmv] [ 3712.403405] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.410602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.415670] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3712.417143] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3712.427349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.433370] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3712.440709] ? do_raw_spin_unlock+0x75/0x190 [ 3712.445869] ? _raw_spin_unlock+0x12/0x30 [ 3712.447220] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3712.451392] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3712.458638] ll_atomic_open+0x36a/0xd70 [lustre] [ 3712.471020] lookup_open+0x356/0xb40 [ 3712.474173] do_last+0x313/0xfc0 [ 3712.475313] ? nd_jump_root+0xe5/0x160 [ 3712.480746] ? path_init+0x437/0x520 [ 3712.482606] path_openat+0xf7/0x500 [ 3712.484488] ? do_raw_spin_unlock+0x75/0x190 [ 3712.491008] do_filp_open+0x99/0x140 [ 3712.497181] ? getname_flags+0x6e/0x330 [ 3712.499542] ? __check_object_size+0xff/0x256 [ 3712.504240] ? do_raw_spin_unlock+0x75/0x190 [ 3712.505591] ? _raw_spin_unlock+0x12/0x30 [ 3712.516809] do_sys_openat2+0x2b4/0x410 [ 3712.518109] do_sys_open+0x73/0xa0 [ 3712.523656] __x64_sys_openat+0x24/0x30 [ 3712.524785] do_syscall_64+0xc1/0x440 [ 3712.526052] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3712.531812] RIP: 0033:0x7fa749552332 [ 3712.536324] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3712.557575] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3712.561514] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3712.566579] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3712.572559] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3712.576576] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3712.580165] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3715.553825] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1798s! [ 3715.556422] Showing busy workqueues and worker pools: [ 3715.559019] workqueue events: flags=0x0 [ 3715.560237] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3715.566726] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3715.576302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3715.584766] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3715.591547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.594952] pending: kfree_rcu_monitor [ 3715.598892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3715.605186] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3715.615518] workqueue events_long: flags=0x0 [ 3715.622714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.626583] pending: lu_objects_flush [obdclass] [ 3715.629174] workqueue events_unbound: flags=0x2 [ 3715.633001] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3715.639983] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3715.646717] workqueue events_power_efficient: flags=0x80 [ 3715.649039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.654248] pending: check_lifetime [ 3715.657254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.665717] pending: neigh_periodic_work [ 3715.670439] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.674812] pending: neigh_periodic_work [ 3715.677138] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.682493] pending: do_cache_clean [ 3715.684791] workqueue mm_percpu_wq: flags=0x8 [ 3715.686526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.694695] pending: vmstat_update [ 3715.697701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.699936] pending: vmstat_update [ 3715.701489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3715.706165] pending: vmstat_update [ 3715.711023] workqueue writeback: flags=0x4a [ 3715.713762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3715.719279] pending: wb_workfn, wb_workfn [ 3715.723033] workqueue xprtiod: flags=0xa [ 3715.726368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3715.729765] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3715.736014] workqueue ipv6_addrconf: flags=0x40008 [ 3715.738580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3715.741496] pending: addrconf_verify_work [ 3715.747240] workqueue knbd0-recv: flags=0x1a [ 3715.751009] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3715.755902] in-flight: 651:recv_work [ 3715.758431] workqueue ptlrpc_pinger: flags=0x2 [ 3715.760059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3715.765492] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3715.773113] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1798s workers=4 idle: 37 31846 28897 [ 3715.786124] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3705s workers=2 idle: 653 [ 3720.057057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3720.059337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3720.088125] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3720.094101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3720.102309] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3720.109948] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3720.116719] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3720.144899] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3720.156644] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3720.163902] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3720.170075] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3720.176352] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3720.185768] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3720.190043] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3720.193142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3720.199808] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3720.205775] Call Trace: [ 3720.206361] [ 3720.208009] ? show_regs.cold.9+0x22/0x2f [ 3720.212937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3720.215556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3720.217762] ? watchdog+0x40/0x40 [ 3720.219157] ? hrtimer_interrupt+0x128/0x2c0 [ 3720.226982] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3720.231153] ? apic_timer_interrupt+0xf/0x20 [ 3720.232645] [ 3720.235943] ? native_safe_halt+0x2e/0x50 [ 3720.242371] kvm_wait+0xcd/0xe0 [ 3720.243446] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3720.250156] do_raw_spin_lock+0x177/0x190 [ 3720.251476] _raw_spin_lock+0x21/0x30 [ 3720.254924] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3720.256979] process_one_work+0x2c8/0x700 [ 3720.263841] worker_thread+0x296/0x6e0 [ 3720.265554] ? rescuer_thread+0x570/0x570 [ 3720.268596] kthread+0x1d1/0x200 [ 3720.270077] ? set_kthread_struct+0x70/0x70 [ 3720.273662] ret_from_fork+0x1f/0x30 [ 3732.058485] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3732.064678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.090825] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.094771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.097648] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3732.099758] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3732.107169] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.109591] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3732.114536] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3732.118802] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3732.123790] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3732.135650] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3732.147624] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3732.157172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.164548] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3732.169706] Call Trace: [ 3732.175710] [ 3732.177591] ? show_regs.cold.9+0x22/0x2f [ 3732.180104] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.182215] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.186678] ? watchdog+0x40/0x40 [ 3732.188622] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.190777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.195863] ? apic_timer_interrupt+0xf/0x20 [ 3732.198165] [ 3732.199143] ? native_safe_halt+0x2e/0x50 [ 3732.201016] kvm_wait+0xcd/0xe0 [ 3732.201796] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3732.203533] do_raw_spin_lock+0x177/0x190 [ 3732.205282] _raw_spin_lock+0x21/0x30 [ 3732.206523] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3732.208726] obd_mod_free+0x46/0x140 [ptlrpc] [ 3732.211721] mdc_commit_open+0x91/0xb0 [mdc] [ 3732.213239] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3732.215400] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3732.217681] after_reply+0xe2c/0x1780 [ptlrpc] [ 3732.219761] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3732.221665] ? __next_timer_interrupt+0x160/0x160 [ 3732.223257] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3732.231138] ? do_wait_intr+0xf0/0xf0 [ 3732.236602] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3732.242621] kthread+0x1d1/0x200 [ 3732.246520] ? set_kthread_struct+0x70/0x70 [ 3732.250085] ret_from_fork+0x1f/0x30 [ 3736.060570] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3736.091070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3736.245520] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3736.261896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3736.273204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3736.277349] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3736.321295] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3736.331690] RAX: 0000000000002fbc RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3736.338191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3736.349372] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3736.361188] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3736.372292] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3736.389422] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3736.398252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3736.406135] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3736.418927] Call Trace: [ 3736.419820] [ 3736.420486] ? show_regs.cold.9+0x22/0x2f [ 3736.433045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3736.442434] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3736.450177] ? watchdog+0x40/0x40 [ 3736.456545] ? hrtimer_interrupt+0x128/0x2c0 [ 3736.462520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3736.466949] ? apic_timer_interrupt+0xf/0x20 [ 3736.482389] [ 3736.484047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3736.492136] do_raw_spin_lock+0x177/0x190 [ 3736.493460] _raw_spin_lock+0x21/0x30 [ 3736.494483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3736.501068] ? do_raw_spin_unlock+0x75/0x190 [ 3736.504683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3736.506853] ? do_wait_intr+0xf0/0xf0 [ 3736.510802] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3736.513936] mdc_close+0x53b/0xd70 [mdc] [ 3736.515557] lmv_close+0x18d/0x490 [lmv] [ 3736.520420] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3736.525630] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3736.531760] ll_md_close+0x274/0x800 [lustre] [ 3736.533807] ll_file_release+0x128/0xc50 [lustre] [ 3736.542050] __fput+0x10d/0x440 [ 3736.545607] ____fput+0x12/0x20 [ 3736.547202] task_work_run+0xc6/0x110 [ 3736.551543] exit_to_usermode_loop+0x1dd/0x1f0 [ 3736.559521] do_syscall_64+0x3d6/0x440 [ 3736.562124] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3736.567068] RIP: 0033:0x7fa749552cd8 [ 3736.575127] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3736.603545] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3736.620065] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3736.630773] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3736.639343] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3736.654116] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3736.664248] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3740.059107] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3740.066283] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3740.111959] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3740.121004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3740.130895] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3740.135251] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3740.149151] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3740.156008] RAX: 0000000000001b84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3740.162640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3740.170766] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3740.177320] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3740.184089] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3740.191820] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3740.198331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3740.203983] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3740.213351] Call Trace: [ 3740.216834] [ 3740.217796] ? show_regs.cold.9+0x22/0x2f [ 3740.219422] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3740.221471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3740.224268] ? watchdog+0x40/0x40 [ 3740.225694] ? hrtimer_interrupt+0x128/0x2c0 [ 3740.227835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3740.230586] ? apic_timer_interrupt+0xf/0x20 [ 3740.232657] [ 3740.233692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3740.236278] do_raw_spin_lock+0x177/0x190 [ 3740.238088] _raw_spin_lock+0x21/0x30 [ 3740.239784] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3740.241895] ? do_raw_spin_unlock+0x75/0x190 [ 3740.243994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3740.246401] ? do_wait_intr+0xf0/0xf0 [ 3740.247986] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3740.256765] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3740.262284] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3740.265708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.278383] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3740.286295] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3740.291676] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3740.294856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.306212] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3740.308254] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3740.312492] lmv_intent_open+0x338/0x1150 [lmv] [ 3740.317523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.318996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.321297] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3740.322805] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3740.327565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.330477] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3740.336121] ? do_raw_spin_unlock+0x75/0x190 [ 3740.341003] ? _raw_spin_unlock+0x12/0x30 [ 3740.344986] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3740.350373] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3740.356650] ll_atomic_open+0x36a/0xd70 [lustre] [ 3740.359666] lookup_open+0x356/0xb40 [ 3740.363426] do_last+0x313/0xfc0 [ 3740.365834] ? nd_jump_root+0xe5/0x160 [ 3740.370284] ? path_init+0x437/0x520 [ 3740.373451] path_openat+0xf7/0x500 [ 3740.376083] ? do_raw_spin_unlock+0x75/0x190 [ 3740.379426] do_filp_open+0x99/0x140 [ 3740.383026] ? getname_flags+0x6e/0x330 [ 3740.385828] ? __check_object_size+0xff/0x256 [ 3740.388221] ? do_raw_spin_unlock+0x75/0x190 [ 3740.390101] ? _raw_spin_unlock+0x12/0x30 [ 3740.392021] do_sys_openat2+0x2b4/0x410 [ 3740.393742] do_sys_open+0x73/0xa0 [ 3740.395451] __x64_sys_openat+0x24/0x30 [ 3740.399704] do_syscall_64+0xc1/0x440 [ 3740.404492] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3740.407511] RIP: 0033:0x7fa749552332 [ 3740.409706] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3740.418268] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3740.421831] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3740.424978] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3740.428462] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3740.432079] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3740.435853] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3746.277255] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1829s! [ 3746.286293] Showing busy workqueues and worker pools: [ 3746.290912] workqueue events: flags=0x0 [ 3746.300595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3746.312518] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3746.319734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3746.333130] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3746.344189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.353422] pending: kfree_rcu_monitor [ 3746.357272] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3746.368263] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3746.375775] workqueue events_long: flags=0x0 [ 3746.376944] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.393238] pending: lu_objects_flush [obdclass] [ 3746.405862] workqueue events_unbound: flags=0x2 [ 3746.410574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3746.417887] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3746.424215] workqueue events_power_efficient: flags=0x80 [ 3746.440111] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.452395] pending: check_lifetime [ 3746.456474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.472555] pending: neigh_periodic_work [ 3746.484291] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.490279] pending: neigh_periodic_work [ 3746.491758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.499490] pending: do_cache_clean [ 3746.500569] workqueue mm_percpu_wq: flags=0x8 [ 3746.512329] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.521177] pending: vmstat_update [ 3746.526591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.536874] pending: vmstat_update [ 3746.543540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3746.550678] pending: vmstat_update [ 3746.558823] workqueue writeback: flags=0x4a [ 3746.566190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3746.580014] pending: wb_workfn, wb_workfn [ 3746.591269] workqueue xprtiod: flags=0xa [ 3746.594480] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3746.606362] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3746.617716] workqueue ipv6_addrconf: flags=0x40008 [ 3746.619009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3746.638256] pending: addrconf_verify_work [ 3746.641128] workqueue knbd0-recv: flags=0x1a [ 3746.642796] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3746.645220] in-flight: 651:recv_work [ 3746.646739] workqueue ptlrpc_pinger: flags=0x2 [ 3746.648448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3746.650837] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3746.653408] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1829s workers=4 idle: 37 31846 28897 [ 3746.656467] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3736s workers=2 idle: 653 [ 3748.069922] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3748.075112] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3748.125414] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3748.137545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3748.152753] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3748.155919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3748.157553] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3748.178820] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3748.193849] RAX: 0000000000006e8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3748.200859] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3748.202730] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3748.209346] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3748.217246] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3748.225201] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3748.227175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3748.257809] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3748.269135] Call Trace: [ 3748.273152] [ 3748.274570] ? show_regs.cold.9+0x22/0x2f [ 3748.283386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3748.288136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3748.293127] ? watchdog+0x40/0x40 [ 3748.294984] ? hrtimer_interrupt+0x128/0x2c0 [ 3748.305013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3748.310086] ? apic_timer_interrupt+0xf/0x20 [ 3748.313498] [ 3748.314788] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3748.323546] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3748.324957] do_raw_spin_lock+0x177/0x190 [ 3748.326010] _raw_spin_lock+0x21/0x30 [ 3748.330936] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3748.337507] process_one_work+0x2c8/0x700 [ 3748.341149] worker_thread+0x296/0x6e0 [ 3748.350537] ? rescuer_thread+0x570/0x570 [ 3748.353707] kthread+0x1d1/0x200 [ 3748.355248] ? set_kthread_struct+0x70/0x70 [ 3748.362464] ret_from_fork+0x1f/0x30 [ 3760.058015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3760.061017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.094979] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.105641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.114799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3760.116428] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3760.121666] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3760.123582] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3760.125221] RDX: 00000000000043b0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3760.126772] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3760.128345] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3760.129802] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3760.131532] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3760.133418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.134875] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3760.136647] Call Trace: [ 3760.137381] [ 3760.137951] ? show_regs.cold.9+0x22/0x2f [ 3760.139484] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.141297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.142387] ? watchdog+0x40/0x40 [ 3760.143146] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.144122] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.146179] ? apic_timer_interrupt+0xf/0x20 [ 3760.147920] [ 3760.148696] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3760.150673] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3760.152359] do_raw_spin_lock+0x177/0x190 [ 3760.153232] _raw_spin_lock+0x21/0x30 [ 3760.154476] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3760.156864] obd_mod_free+0x46/0x140 [ptlrpc] [ 3760.162671] mdc_commit_open+0x91/0xb0 [mdc] [ 3760.164712] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3760.167484] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3760.172067] after_reply+0xe2c/0x1780 [ptlrpc] [ 3760.176605] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3760.181183] ? __next_timer_interrupt+0x160/0x160 [ 3760.185331] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3760.191727] ? do_wait_intr+0xf0/0xf0 [ 3760.193959] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3760.196682] kthread+0x1d1/0x200 [ 3760.199110] ? set_kthread_struct+0x70/0x70 [ 3760.202261] ret_from_fork+0x1f/0x30 [ 3764.063814] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3764.071599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3764.123682] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3764.138906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3764.151480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3764.160898] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3764.179354] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3764.183846] RAX: 0000000000003d0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3764.190391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3764.203808] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3764.213081] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3764.230781] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3764.240126] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3764.248565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3764.252802] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3764.263847] Call Trace: [ 3764.270882] [ 3764.275391] ? show_regs.cold.9+0x22/0x2f [ 3764.279270] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3764.284893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3764.295304] ? watchdog+0x40/0x40 [ 3764.297855] ? hrtimer_interrupt+0x128/0x2c0 [ 3764.302099] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3764.305945] ? apic_timer_interrupt+0xf/0x20 [ 3764.311192] [ 3764.312432] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3764.317592] do_raw_spin_lock+0x177/0x190 [ 3764.325080] _raw_spin_lock+0x21/0x30 [ 3764.331245] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3764.337034] ? do_raw_spin_unlock+0x75/0x190 [ 3764.342266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3764.348871] ? do_wait_intr+0xf0/0xf0 [ 3764.355506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3764.360854] mdc_close+0x53b/0xd70 [mdc] [ 3764.366320] lmv_close+0x18d/0x490 [lmv] [ 3764.375896] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3764.383183] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3764.387259] ll_md_close+0x274/0x800 [lustre] [ 3764.392631] ll_file_release+0x128/0xc50 [lustre] [ 3764.398584] __fput+0x10d/0x440 [ 3764.400529] ____fput+0x12/0x20 [ 3764.404551] task_work_run+0xc6/0x110 [ 3764.411633] exit_to_usermode_loop+0x1dd/0x1f0 [ 3764.420472] do_syscall_64+0x3d6/0x440 [ 3764.426764] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3764.430064] RIP: 0033:0x7fa749552cd8 [ 3764.432593] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3764.467337] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3764.481895] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3764.491713] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3764.502213] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3764.510955] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3764.520055] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3772.054388] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3772.061798] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3772.110428] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3772.124714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3772.136738] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3772.140171] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3772.162876] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3772.173468] RAX: 0000000000007313 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3772.182899] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3772.193413] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3772.207946] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3772.226629] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3772.244598] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3772.253829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3772.257234] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3772.260353] Call Trace: [ 3772.261430] [ 3772.262714] ? show_regs.cold.9+0x22/0x2f [ 3772.264317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3772.266275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3772.268144] ? watchdog+0x40/0x40 [ 3772.269813] ? hrtimer_interrupt+0x128/0x2c0 [ 3772.271726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3772.274803] ? apic_timer_interrupt+0xf/0x20 [ 3772.280000] [ 3772.282324] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3772.287366] do_raw_spin_lock+0x177/0x190 [ 3772.291237] _raw_spin_lock+0x21/0x30 [ 3772.295825] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3772.301154] ? do_raw_spin_unlock+0x75/0x190 [ 3772.305660] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3772.311049] ? do_wait_intr+0xf0/0xf0 [ 3772.317268] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3772.323918] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3772.331287] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3772.336842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3772.342833] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3772.347292] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3772.356766] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3772.360652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3772.363575] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3772.367339] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3772.371809] lmv_intent_open+0x338/0x1150 [lmv] [ 3772.375390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3772.379529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3772.387236] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3772.395651] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3772.403804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3772.409847] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3772.414327] ? do_raw_spin_unlock+0x75/0x190 [ 3772.421105] ? _raw_spin_unlock+0x12/0x30 [ 3772.422929] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3772.430640] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3772.440496] ll_atomic_open+0x36a/0xd70 [lustre] [ 3772.442618] lookup_open+0x356/0xb40 [ 3772.446441] do_last+0x313/0xfc0 [ 3772.449594] ? nd_jump_root+0xe5/0x160 [ 3772.453572] ? path_init+0x437/0x520 [ 3772.457474] path_openat+0xf7/0x500 [ 3772.461517] ? do_raw_spin_unlock+0x75/0x190 [ 3772.467653] do_filp_open+0x99/0x140 [ 3772.469680] ? getname_flags+0x6e/0x330 [ 3772.472439] ? __check_object_size+0xff/0x256 [ 3772.477855] ? do_raw_spin_unlock+0x75/0x190 [ 3772.479234] ? _raw_spin_unlock+0x12/0x30 [ 3772.482730] do_sys_openat2+0x2b4/0x410 [ 3772.485656] do_sys_open+0x73/0xa0 [ 3772.487445] __x64_sys_openat+0x24/0x30 [ 3772.489508] do_syscall_64+0xc1/0x440 [ 3772.497339] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3772.504352] RIP: 0033:0x7fa749552332 [ 3772.508597] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3772.534444] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3772.542013] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3772.549957] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3772.554723] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3772.559432] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3772.568893] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3773.732885] rcu: INFO: rcu_sched self-detected stall on CPU [ 3773.741504] rcu: 2-...!: (1007260 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 3773.752904] (t=1860043 jiffies g=258701 q=2114) [ 3773.759750] rcu: rcu_sched kthread starved for 1856838 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3773.774172] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3773.788831] rcu: RCU grace-period kthread stack dump: [ 3773.799481] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3773.817283] Call Trace: [ 3773.818347] __schedule+0x351/0xcb0 [ 3773.819589] schedule+0xc0/0x180 [ 3773.830656] schedule_timeout+0xb4/0x190 [ 3773.834766] ? do_raw_spin_unlock+0x75/0x190 [ 3773.841917] ? __next_timer_interrupt+0x160/0x160 [ 3773.848200] ? prepare_to_swait_event+0xf3/0x140 [ 3773.853413] rcu_gp_kthread+0x8dd/0x15b0 [ 3773.859333] ? force_qs_rnp+0x2c0/0x2c0 [ 3773.867743] kthread+0x1d1/0x200 [ 3773.872957] ? set_kthread_struct+0x70/0x70 [ 3773.882509] ret_from_fork+0x1f/0x30 [ 3773.883867] rcu: Stack dump where RCU GP kthread last ran: [ 3773.896777] NMI backtrace for cpu 2 [ 3773.907378] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3773.922956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3773.934447] Call Trace: [ 3773.935990] [ 3773.937541] ? dump_stack+0xbb/0x10e [ 3773.945753] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3773.948931] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3773.959585] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3773.964679] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3773.972542] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3773.974784] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3773.978454] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 3773.981744] ? update_process_times+0x6d/0xc0 [ 3773.983864] ? tick_sched_handle.isra.18+0x27/0x90 [ 3773.988624] ? tick_sched_timer+0x53/0xb0 [ 3773.993099] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3773.995699] ? tick_sched_handle.isra.18+0x90/0x90 [ 3774.000690] ? hrtimer_interrupt+0x128/0x2c0 [ 3774.004863] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3774.011976] ? apic_timer_interrupt+0xf/0x20 [ 3774.019372] [ 3774.026878] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3774.032740] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3774.044639] ? do_raw_spin_lock+0x177/0x190 [ 3774.051526] ? _raw_spin_lock+0x21/0x30 [ 3774.062621] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3774.070736] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3774.089122] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3774.095281] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3774.103439] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3774.115074] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3774.117127] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3774.128975] ? __next_timer_interrupt+0x160/0x160 [ 3774.130611] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3774.137396] ? do_wait_intr+0xf0/0xf0 [ 3774.142861] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3774.149751] ? kthread+0x1d1/0x200 [ 3774.160736] ? set_kthread_struct+0x70/0x70 [ 3774.167385] ? ret_from_fork+0x1f/0x30 [ 3774.169542] NMI backtrace for cpu 2 [ 3774.175459] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3774.180667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3774.192329] Call Trace: [ 3774.193235] [ 3774.200025] ? dump_stack+0xbb/0x10e [ 3774.201992] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3774.207531] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3774.210682] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3774.224008] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3774.226469] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3774.233962] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3774.238980] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3774.243569] ? update_process_times+0x6d/0xc0 [ 3774.246212] ? tick_sched_handle.isra.18+0x27/0x90 [ 3774.254489] ? tick_sched_timer+0x53/0xb0 [ 3774.264528] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3774.268619] ? tick_sched_handle.isra.18+0x90/0x90 [ 3774.280035] ? hrtimer_interrupt+0x128/0x2c0 [ 3774.285538] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3774.299454] ? apic_timer_interrupt+0xf/0x20 [ 3774.302994] [ 3774.307488] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3774.321414] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3774.324922] ? do_raw_spin_lock+0x177/0x190 [ 3774.330695] ? _raw_spin_lock+0x21/0x30 [ 3774.333218] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3774.340403] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3774.345458] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3774.349021] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3774.359498] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3774.366895] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3774.376699] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3774.380496] ? __next_timer_interrupt+0x160/0x160 [ 3774.383126] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3774.396214] ? do_wait_intr+0xf0/0xf0 [ 3774.404276] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3774.408370] ? kthread+0x1d1/0x200 [ 3774.413649] ? set_kthread_struct+0x70/0x70 [ 3774.422480] ? ret_from_fork+0x1f/0x30 [ 3776.061065] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3776.074328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3776.146200] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3776.165310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3776.178204] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3776.190594] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3776.197576] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3776.236834] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3776.250929] RAX: 0000000000002b10 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3776.259127] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3776.263943] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3776.269953] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3776.275648] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3776.279370] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3776.282839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3776.293693] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3776.297766] Call Trace: [ 3776.299658] [ 3776.301863] ? show_regs.cold.9+0x22/0x2f [ 3776.305867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3776.307574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3776.310502] ? watchdog+0x40/0x40 [ 3776.312291] ? hrtimer_interrupt+0x128/0x2c0 [ 3776.317721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3776.321663] ? apic_timer_interrupt+0xf/0x20 [ 3776.325785] [ 3776.329744] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3776.337441] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3776.342972] do_raw_spin_lock+0x177/0x190 [ 3776.349975] _raw_spin_lock+0x21/0x30 [ 3776.357098] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3776.370959] process_one_work+0x2c8/0x700 [ 3776.375516] worker_thread+0x296/0x6e0 [ 3776.379589] ? rescuer_thread+0x570/0x570 [ 3776.382334] kthread+0x1d1/0x200 [ 3776.385980] ? set_kthread_struct+0x70/0x70 [ 3776.390615] ret_from_fork+0x1f/0x30 [ 3777.002934] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1859s! [ 3777.014744] Showing busy workqueues and worker pools: [ 3777.022640] workqueue events: flags=0x0 [ 3777.034750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3777.042693] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3777.055302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3777.067450] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3777.075564] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.082278] pending: kfree_rcu_monitor [ 3777.094434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3777.107963] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3777.132237] workqueue events_long: flags=0x0 [ 3777.135569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.146484] pending: lu_objects_flush [obdclass] [ 3777.152744] workqueue events_unbound: flags=0x2 [ 3777.160080] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3777.163311] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3777.173331] workqueue events_power_efficient: flags=0x80 [ 3777.192420] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.208641] pending: check_lifetime [ 3777.216468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.227172] pending: neigh_periodic_work [ 3777.228672] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.244289] pending: neigh_periodic_work [ 3777.260358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.272776] pending: do_cache_clean [ 3777.274274] workqueue mm_percpu_wq: flags=0x8 [ 3777.284568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.295257] pending: vmstat_update [ 3777.305652] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.315009] pending: vmstat_update [ 3777.324610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3777.336746] pending: vmstat_update [ 3777.345816] workqueue writeback: flags=0x4a [ 3777.347151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3777.356062] pending: wb_workfn, wb_workfn [ 3777.361935] workqueue xprtiod: flags=0xa [ 3777.368049] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3777.376811] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3777.388250] workqueue ipv6_addrconf: flags=0x40008 [ 3777.402689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3777.414023] pending: addrconf_verify_work [ 3777.415465] workqueue knbd0-recv: flags=0x1a [ 3777.427081] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3777.431870] in-flight: 651:recv_work [ 3777.443290] workqueue ptlrpc_pinger: flags=0x2 [ 3777.452513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3777.454735] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3777.470710] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1860s workers=4 idle: 37 31846 28897 [ 3777.485921] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3767s workers=2 idle: 653 [ 3792.067468] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3792.075838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3792.106672] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3792.116704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3792.125741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3792.130114] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3792.146882] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3792.157668] RAX: 0000000000007b8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3792.162476] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3792.170998] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3792.176748] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3792.183846] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3792.188895] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3792.193642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3792.196793] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3792.200884] Call Trace: [ 3792.202405] [ 3792.203601] ? show_regs.cold.9+0x22/0x2f [ 3792.206402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3792.208973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3792.211515] ? watchdog+0x40/0x40 [ 3792.213374] ? hrtimer_interrupt+0x128/0x2c0 [ 3792.215687] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3792.218425] ? apic_timer_interrupt+0xf/0x20 [ 3792.220824] [ 3792.222018] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3792.225620] do_raw_spin_lock+0x177/0x190 [ 3792.227941] _raw_spin_lock+0x21/0x30 [ 3792.229948] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3792.233027] ? do_raw_spin_unlock+0x75/0x190 [ 3792.235628] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3792.238993] ? do_wait_intr+0xf0/0xf0 [ 3792.241139] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3792.244592] mdc_close+0x53b/0xd70 [mdc] [ 3792.246934] lmv_close+0x18d/0x490 [lmv] [ 3792.249505] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3792.253288] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3792.256880] ll_md_close+0x274/0x800 [lustre] [ 3792.261099] ll_file_release+0x128/0xc50 [lustre] [ 3792.267407] __fput+0x10d/0x440 [ 3792.271454] ____fput+0x12/0x20 [ 3792.274320] task_work_run+0xc6/0x110 [ 3792.278521] exit_to_usermode_loop+0x1dd/0x1f0 [ 3792.281962] do_syscall_64+0x3d6/0x440 [ 3792.285327] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3792.291805] RIP: 0033:0x7fa749552cd8 [ 3792.295526] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3792.308908] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3792.322083] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3792.327955] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3792.335232] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3792.341856] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3792.346977] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3800.060403] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3800.068560] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3800.089273] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3800.095067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3800.098251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3800.101300] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3800.114315] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3800.121947] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3800.126386] RDX: 000000000000232e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3800.132565] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3800.135791] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3800.139083] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3800.144062] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3800.149691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3800.152859] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3800.155737] Call Trace: [ 3800.156601] [ 3800.157515] ? show_regs.cold.9+0x22/0x2f [ 3800.159307] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3800.161093] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3800.163058] ? watchdog+0x40/0x40 [ 3800.164738] ? hrtimer_interrupt+0x128/0x2c0 [ 3800.167153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3800.170619] ? apic_timer_interrupt+0xf/0x20 [ 3800.172763] [ 3800.174197] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3800.176670] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3800.179177] do_raw_spin_lock+0x177/0x190 [ 3800.181051] _raw_spin_lock+0x21/0x30 [ 3800.182733] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3800.185653] obd_mod_free+0x46/0x140 [ptlrpc] [ 3800.189303] mdc_commit_open+0x91/0xb0 [mdc] [ 3800.191800] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3800.195080] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3800.198718] after_reply+0xe2c/0x1780 [ptlrpc] [ 3800.201168] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3800.203985] ? __next_timer_interrupt+0x160/0x160 [ 3800.206300] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3800.208900] ? do_wait_intr+0xf0/0xf0 [ 3800.211044] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3800.215368] kthread+0x1d1/0x200 [ 3800.218289] ? set_kthread_struct+0x70/0x70 [ 3800.220857] ret_from_fork+0x1f/0x30 [ 3804.054240] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3804.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3804.061356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3804.061435] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3804.061443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3804.061446] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3804.061459] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3804.061469] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3804.061475] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3804.061477] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3804.061480] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3804.061482] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3804.061485] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3804.061488] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3804.061491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3804.061494] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3804.062886] Call Trace: [ 3804.062896] [ 3804.062902] ? show_regs.cold.9+0x22/0x2f [ 3804.062914] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3804.062921] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3804.062925] ? watchdog+0x40/0x40 [ 3804.062929] ? hrtimer_interrupt+0x128/0x2c0 [ 3804.062934] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3804.062941] ? apic_timer_interrupt+0xf/0x20 [ 3804.062946] [ 3804.062949] ? native_safe_halt+0x2e/0x50 [ 3804.062953] kvm_wait+0xcd/0xe0 [ 3804.062961] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3804.062971] do_raw_spin_lock+0x177/0x190 [ 3804.062975] _raw_spin_lock+0x21/0x30 [ 3804.062980] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3804.079868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3804.079944] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3804.079952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3804.079959] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3804.080257] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3804.080267] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3804.080271] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3804.080277] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3804.080280] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3804.080282] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3804.080284] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3804.080360] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3804.080364] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3804.080367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3804.080370] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3804.080378] Call Trace: [ 3804.080384] [ 3804.080389] ? show_regs.cold.9+0x22/0x2f [ 3804.080402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3804.080410] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3804.080415] ? watchdog+0x40/0x40 [ 3804.080420] ? hrtimer_interrupt+0x128/0x2c0 [ 3804.080431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3804.080442] ? apic_timer_interrupt+0xf/0x20 [ 3804.080448] [ 3804.080452] ? native_safe_halt+0x2e/0x50 [ 3804.080462] kvm_wait+0xcd/0xe0 [ 3804.080470] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3804.080479] do_raw_spin_lock+0x177/0x190 [ 3804.080483] _raw_spin_lock+0x21/0x30 [ 3804.080488] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3804.080831] process_one_work+0x2c8/0x700 [ 3804.080838] worker_thread+0x296/0x6e0 [ 3804.080842] ? rescuer_thread+0x570/0x570 [ 3804.080846] kthread+0x1d1/0x200 [ 3804.080850] ? set_kthread_struct+0x70/0x70 [ 3804.080854] ret_from_fork+0x1f/0x30 [ 3804.518218] ? do_raw_spin_unlock+0x75/0x190 [ 3804.519843] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3804.524990] ? do_wait_intr+0xf0/0xf0 [ 3804.527440] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3804.541053] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3804.542916] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3804.547824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3804.553657] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3804.558994] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3804.562430] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3804.564333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3804.570199] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3804.575681] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3804.578863] lmv_intent_open+0x338/0x1150 [lmv] [ 3804.580429] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3804.582213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3804.589706] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3804.593918] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3804.596237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3804.607858] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3804.611761] ? do_raw_spin_unlock+0x75/0x190 [ 3804.616451] ? _raw_spin_unlock+0x12/0x30 [ 3804.620936] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3804.626443] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3804.629137] ll_atomic_open+0x36a/0xd70 [lustre] [ 3804.634129] lookup_open+0x356/0xb40 [ 3804.635705] do_last+0x313/0xfc0 [ 3804.638123] ? nd_jump_root+0xe5/0x160 [ 3804.639968] ? path_init+0x437/0x520 [ 3804.644456] path_openat+0xf7/0x500 [ 3804.647216] ? do_raw_spin_unlock+0x75/0x190 [ 3804.655008] do_filp_open+0x99/0x140 [ 3804.656176] ? getname_flags+0x6e/0x330 [ 3804.659204] ? __check_object_size+0xff/0x256 [ 3804.660566] ? do_raw_spin_unlock+0x75/0x190 [ 3804.663200] ? _raw_spin_unlock+0x12/0x30 [ 3804.666848] do_sys_openat2+0x2b4/0x410 [ 3804.671980] do_sys_open+0x73/0xa0 [ 3804.673001] __x64_sys_openat+0x24/0x30 [ 3804.676919] do_syscall_64+0xc1/0x440 [ 3804.678969] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3804.683078] RIP: 0033:0x7fa749552332 [ 3804.685232] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3804.723904] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3804.732435] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3804.751294] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3804.753632] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3804.774425] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3804.787761] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3807.712319] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1890s! [ 3807.742876] Showing busy workqueues and worker pools: [ 3807.763508] workqueue events: flags=0x0 [ 3807.770645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3807.787384] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3807.814937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3807.825802] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3807.850897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3807.864220] pending: kfree_rcu_monitor [ 3807.875901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3807.892336] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3807.904936] workqueue events_long: flags=0x0 [ 3807.911172] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3807.923827] pending: lu_objects_flush [obdclass] [ 3807.935020] workqueue events_unbound: flags=0x2 [ 3807.948227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3807.960633] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3807.982149] workqueue events_power_efficient: flags=0x80 [ 3807.999032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3808.008771] pending: check_lifetime [ 3808.013864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3808.032815] pending: neigh_periodic_work [ 3808.046165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3808.065824] pending: neigh_periodic_work [ 3808.073189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3808.084950] pending: do_cache_clean [ 3808.102806] workqueue mm_percpu_wq: flags=0x8 [ 3808.115296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3808.121758] pending: vmstat_update [ 3808.132375] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3808.150471] pending: vmstat_update [ 3808.158852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3808.167390] pending: vmstat_update [ 3808.180823] workqueue writeback: flags=0x4a [ 3808.191745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3808.208049] pending: wb_workfn, wb_workfn [ 3808.212542] workqueue xprtiod: flags=0xa [ 3808.228069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3808.245976] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3808.267188] workqueue ipv6_addrconf: flags=0x40008 [ 3808.275665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3808.291837] pending: addrconf_verify_work [ 3808.311000] workqueue knbd0-recv: flags=0x1a [ 3808.328271] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3808.347727] in-flight: 651:recv_work [ 3808.349160] workqueue ptlrpc_pinger: flags=0x2 [ 3808.363743] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3808.384392] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3808.389867] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1891s workers=4 idle: 37 31846 28897 [ 3808.409890] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3798s workers=2 idle: 653 [ 3820.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3820.061702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3820.149052] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3820.162366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3820.179273] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3820.188742] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3820.217657] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3820.229836] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3820.237165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3820.250718] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3820.261104] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3820.273480] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3820.288217] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3820.310156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3820.312185] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3820.326294] Call Trace: [ 3820.328523] [ 3820.333478] ? show_regs.cold.9+0x22/0x2f [ 3820.335381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3820.346894] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3820.349119] ? watchdog+0x40/0x40 [ 3820.353140] ? hrtimer_interrupt+0x128/0x2c0 [ 3820.356025] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3820.360518] ? apic_timer_interrupt+0xf/0x20 [ 3820.364406] [ 3820.365322] ? native_safe_halt+0x2e/0x50 [ 3820.367686] kvm_wait+0xcd/0xe0 [ 3820.371646] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3820.384381] do_raw_spin_lock+0x177/0x190 [ 3820.392535] _raw_spin_lock+0x21/0x30 [ 3820.400909] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3820.415382] ? do_raw_spin_unlock+0x75/0x190 [ 3820.427185] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3820.429291] ? do_wait_intr+0xf0/0xf0 [ 3820.445379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3820.450551] mdc_close+0x53b/0xd70 [mdc] [ 3820.457117] lmv_close+0x18d/0x490 [lmv] [ 3820.466534] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3820.469383] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3820.489319] ll_md_close+0x274/0x800 [lustre] [ 3820.499331] ll_file_release+0x128/0xc50 [lustre] [ 3820.501223] __fput+0x10d/0x440 [ 3820.509545] ____fput+0x12/0x20 [ 3820.513452] task_work_run+0xc6/0x110 [ 3820.521014] exit_to_usermode_loop+0x1dd/0x1f0 [ 3820.527862] do_syscall_64+0x3d6/0x440 [ 3820.531807] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3820.536867] RIP: 0033:0x7fa749552cd8 [ 3820.541469] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3820.567873] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3820.577341] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3820.584515] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3820.597335] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3820.608097] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3820.617501] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3828.068521] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3828.076382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3828.149554] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3828.159241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3828.178147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3828.189084] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3828.218619] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3828.224365] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3828.229397] RDX: 0000000000007de6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3828.236274] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3828.250433] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3828.263326] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3828.278297] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3828.289752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3828.292986] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3828.301763] Call Trace: [ 3828.303498] [ 3828.304690] ? show_regs.cold.9+0x22/0x2f [ 3828.309712] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3828.312313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3828.325170] ? watchdog+0x40/0x40 [ 3828.331107] ? hrtimer_interrupt+0x128/0x2c0 [ 3828.341146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3828.349323] ? apic_timer_interrupt+0xf/0x20 [ 3828.352217] [ 3828.352984] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3828.354717] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3828.361035] do_raw_spin_lock+0x177/0x190 [ 3828.363420] _raw_spin_lock+0x21/0x30 [ 3828.367592] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3828.370687] obd_mod_free+0x46/0x140 [ptlrpc] [ 3828.375079] mdc_commit_open+0x91/0xb0 [mdc] [ 3828.377909] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3828.382251] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3828.387721] after_reply+0xe2c/0x1780 [ptlrpc] [ 3828.397956] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3828.399968] ? __next_timer_interrupt+0x160/0x160 [ 3828.411119] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3828.412786] ? do_wait_intr+0xf0/0xf0 [ 3828.420452] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3828.430378] kthread+0x1d1/0x200 [ 3828.435580] ? set_kthread_struct+0x70/0x70 [ 3828.448463] ret_from_fork+0x1f/0x30 [ 3832.057594] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3832.060918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3832.116734] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3832.136290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3832.147938] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3832.156264] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3832.161656] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3832.185279] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3832.193858] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3832.201877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3832.217962] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3832.231509] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3832.233683] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3832.243109] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3832.255537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3832.266042] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3832.281049] Call Trace: [ 3832.283174] [ 3832.286578] ? show_regs.cold.9+0x22/0x2f [ 3832.293683] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3832.303356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3832.304687] ? watchdog+0x40/0x40 [ 3832.305802] ? hrtimer_interrupt+0x128/0x2c0 [ 3832.321910] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3832.323670] ? apic_timer_interrupt+0xf/0x20 [ 3832.329806] [ 3832.332338] ? native_safe_halt+0x2e/0x50 [ 3832.334766] kvm_wait+0xcd/0xe0 [ 3832.337468] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3832.340858] do_raw_spin_lock+0x177/0x190 [ 3832.346633] _raw_spin_lock+0x21/0x30 [ 3832.348307] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3832.351869] process_one_work+0x2c8/0x700 [ 3832.354714] worker_thread+0x296/0x6e0 [ 3832.356885] ? rescuer_thread+0x570/0x570 [ 3832.358921] kthread+0x1d1/0x200 [ 3832.360903] ? set_kthread_struct+0x70/0x70 [ 3832.363287] ret_from_fork+0x1f/0x30 [ 3836.060203] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3836.070820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3836.128465] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3836.147187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3836.158876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3836.167325] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3836.182954] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3836.187678] RAX: 0000000000007e13 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3836.194186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3836.200944] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3836.207978] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3836.212218] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3836.215809] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3836.222444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3836.225074] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3836.228097] Call Trace: [ 3836.229271] [ 3836.230063] ? show_regs.cold.9+0x22/0x2f [ 3836.231726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3836.233423] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3836.235075] ? watchdog+0x40/0x40 [ 3836.236075] ? hrtimer_interrupt+0x128/0x2c0 [ 3836.237677] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3836.239407] ? apic_timer_interrupt+0xf/0x20 [ 3836.241661] [ 3836.242688] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3836.244934] do_raw_spin_lock+0x177/0x190 [ 3836.249536] _raw_spin_lock+0x21/0x30 [ 3836.251174] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3836.253870] ? do_raw_spin_unlock+0x75/0x190 [ 3836.256656] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3836.259380] ? do_wait_intr+0xf0/0xf0 [ 3836.261407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3836.264930] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3836.267674] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3836.273493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3836.275979] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3836.278247] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3836.280290] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3836.282320] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3836.284643] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3836.287447] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3836.289687] lmv_intent_open+0x338/0x1150 [lmv] [ 3836.291343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3836.294180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3836.296820] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3836.298681] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3836.300951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3836.303268] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3836.305649] ? do_raw_spin_unlock+0x75/0x190 [ 3836.307330] ? _raw_spin_unlock+0x12/0x30 [ 3836.308867] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3836.310890] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3836.312939] ll_atomic_open+0x36a/0xd70 [lustre] [ 3836.314873] lookup_open+0x356/0xb40 [ 3836.316106] do_last+0x313/0xfc0 [ 3836.317366] ? nd_jump_root+0xe5/0x160 [ 3836.318944] ? path_init+0x437/0x520 [ 3836.321022] path_openat+0xf7/0x500 [ 3836.322665] ? do_raw_spin_unlock+0x75/0x190 [ 3836.324740] do_filp_open+0x99/0x140 [ 3836.326060] ? getname_flags+0x6e/0x330 [ 3836.327800] ? __check_object_size+0xff/0x256 [ 3836.329595] ? do_raw_spin_unlock+0x75/0x190 [ 3836.331439] ? _raw_spin_unlock+0x12/0x30 [ 3836.333611] do_sys_openat2+0x2b4/0x410 [ 3836.337308] do_sys_open+0x73/0xa0 [ 3836.339497] __x64_sys_openat+0x24/0x30 [ 3836.341159] do_syscall_64+0xc1/0x440 [ 3836.344179] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3836.348854] RIP: 0033:0x7fa749552332 [ 3836.351961] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3836.359438] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3836.362479] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3836.366400] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3836.369508] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3836.372555] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3836.375677] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3838.433147] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1921s! [ 3838.438035] Showing busy workqueues and worker pools: [ 3838.444471] workqueue events: flags=0x0 [ 3838.448301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3838.452730] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3838.459272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3838.462758] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3838.469148] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.473791] pending: kfree_rcu_monitor [ 3838.476143] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3838.478360] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3838.492846] workqueue events_long: flags=0x0 [ 3838.498863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.504552] pending: lu_objects_flush [obdclass] [ 3838.512264] workqueue events_unbound: flags=0x2 [ 3838.516624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3838.521803] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3838.534086] workqueue events_power_efficient: flags=0x80 [ 3838.544180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.548467] pending: check_lifetime [ 3838.551840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.568989] pending: neigh_periodic_work [ 3838.575616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.582991] pending: neigh_periodic_work [ 3838.586971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.596166] pending: do_cache_clean [ 3838.599592] workqueue mm_percpu_wq: flags=0x8 [ 3838.601757] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.604608] pending: vmstat_update [ 3838.606588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.611652] pending: vmstat_update [ 3838.615001] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3838.624299] pending: vmstat_update [ 3838.628112] workqueue writeback: flags=0x4a [ 3838.635860] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3838.648784] pending: wb_workfn, wb_workfn [ 3838.658071] workqueue xprtiod: flags=0xa [ 3838.659556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3838.670493] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3838.682806] workqueue ipv6_addrconf: flags=0x40008 [ 3838.687293] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3838.691277] pending: addrconf_verify_work [ 3838.693337] workqueue knbd0-recv: flags=0x1a [ 3838.695159] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3838.698104] in-flight: 651:recv_work [ 3838.702416] workqueue ptlrpc_pinger: flags=0x2 [ 3838.706337] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3838.712016] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3838.716144] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1921s workers=4 idle: 37 31846 28897 [ 3838.730962] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3828s workers=2 idle: 653 [ 3848.059879] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3848.074424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3848.127856] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3848.136813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3848.142414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3848.147886] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3848.168814] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3848.174181] RAX: 0000000000007a92 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3848.178634] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3848.191119] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3848.202023] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3848.217470] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3848.225240] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3848.233089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3848.237928] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3848.247117] Call Trace: [ 3848.250131] [ 3848.252162] ? show_regs.cold.9+0x22/0x2f [ 3848.268312] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3848.271994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3848.282303] ? watchdog+0x40/0x40 [ 3848.291513] ? hrtimer_interrupt+0x128/0x2c0 [ 3848.297562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3848.302074] ? apic_timer_interrupt+0xf/0x20 [ 3848.306185] [ 3848.307154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3848.313449] do_raw_spin_lock+0x177/0x190 [ 3848.314938] _raw_spin_lock+0x21/0x30 [ 3848.324346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3848.329590] ? do_raw_spin_unlock+0x75/0x190 [ 3848.331170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3848.341761] ? do_wait_intr+0xf0/0xf0 [ 3848.351364] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3848.364225] mdc_close+0x53b/0xd70 [mdc] [ 3848.373175] lmv_close+0x18d/0x490 [lmv] [ 3848.383818] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3848.386886] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3848.395940] ll_md_close+0x274/0x800 [lustre] [ 3848.405448] ll_file_release+0x128/0xc50 [lustre] [ 3848.413197] __fput+0x10d/0x440 [ 3848.414396] ____fput+0x12/0x20 [ 3848.425920] task_work_run+0xc6/0x110 [ 3848.429661] exit_to_usermode_loop+0x1dd/0x1f0 [ 3848.437589] do_syscall_64+0x3d6/0x440 [ 3848.438759] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3848.450022] RIP: 0033:0x7fa749552cd8 [ 3848.453083] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3848.464245] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3848.468153] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3848.476080] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3848.483379] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3848.489942] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3848.495289] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3856.059387] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3856.067651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3856.114240] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3856.126014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3856.136277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3856.142666] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3856.163907] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3856.168803] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3856.173499] RDX: 0000000000002bfd RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3856.176903] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3856.180439] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3856.186019] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3856.195362] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3856.202889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3856.207783] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3856.211024] Call Trace: [ 3856.212112] [ 3856.213890] ? show_regs.cold.9+0x22/0x2f [ 3856.217813] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3856.224274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3856.226529] ? watchdog+0x40/0x40 [ 3856.230027] ? hrtimer_interrupt+0x128/0x2c0 [ 3856.231078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3856.238187] ? apic_timer_interrupt+0xf/0x20 [ 3856.241406] [ 3856.247030] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3856.249294] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3856.251971] do_raw_spin_lock+0x177/0x190 [ 3856.256352] _raw_spin_lock+0x21/0x30 [ 3856.257499] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3856.259114] obd_mod_free+0x46/0x140 [ptlrpc] [ 3856.267845] mdc_commit_open+0x91/0xb0 [mdc] [ 3856.268958] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3856.272377] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3856.276312] after_reply+0xe2c/0x1780 [ptlrpc] [ 3856.278303] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3856.286897] ? __next_timer_interrupt+0x160/0x160 [ 3856.294562] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3856.297930] ? do_wait_intr+0xf0/0xf0 [ 3856.309925] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3856.321799] kthread+0x1d1/0x200 [ 3856.322788] ? set_kthread_struct+0x70/0x70 [ 3856.327831] ret_from_fork+0x1f/0x30 [ 3860.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 3860.063522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3860.095720] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3860.104809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3860.109705] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3860.115351] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3860.122617] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3860.136712] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3860.145437] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3860.152440] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3860.161704] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3860.169956] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3860.178990] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3860.187287] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3860.193976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3860.198269] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3860.207102] Call Trace: [ 3860.208000] [ 3860.211667] ? show_regs.cold.9+0x22/0x2f [ 3860.212993] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3860.216576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3860.221008] ? watchdog+0x40/0x40 [ 3860.223290] ? hrtimer_interrupt+0x128/0x2c0 [ 3860.224939] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3860.227260] ? apic_timer_interrupt+0xf/0x20 [ 3860.230780] [ 3860.231962] ? native_safe_halt+0x2e/0x50 [ 3860.233082] kvm_wait+0xcd/0xe0 [ 3860.234514] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3860.236844] do_raw_spin_lock+0x177/0x190 [ 3860.238552] _raw_spin_lock+0x21/0x30 [ 3860.240837] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3860.245424] process_one_work+0x2c8/0x700 [ 3860.247831] worker_thread+0x296/0x6e0 [ 3860.250513] ? rescuer_thread+0x570/0x570 [ 3860.253525] kthread+0x1d1/0x200 [ 3860.258076] ? set_kthread_struct+0x70/0x70 [ 3860.259937] ret_from_fork+0x1f/0x30 [ 3864.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3864.056867] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3864.075124] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3864.080847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3864.085792] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3864.088665] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3864.095398] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3864.099631] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3864.102897] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3864.109463] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3864.113621] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3864.115692] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3864.117867] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3864.121727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3864.127811] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3864.131691] Call Trace: [ 3864.132594] [ 3864.133150] ? show_regs.cold.9+0x22/0x2f [ 3864.135854] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3864.140165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3864.146768] ? watchdog+0x40/0x40 [ 3864.150745] ? hrtimer_interrupt+0x128/0x2c0 [ 3864.154267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3864.157607] ? apic_timer_interrupt+0xf/0x20 [ 3864.162543] [ 3864.164125] ? native_safe_halt+0x2e/0x50 [ 3864.165966] kvm_wait+0xcd/0xe0 [ 3864.167572] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3864.173190] do_raw_spin_lock+0x177/0x190 [ 3864.175303] _raw_spin_lock+0x21/0x30 [ 3864.177909] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3864.181312] ? do_raw_spin_unlock+0x75/0x190 [ 3864.183409] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3864.187743] ? do_wait_intr+0xf0/0xf0 [ 3864.197820] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3864.200783] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3864.202912] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3864.207283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3864.213145] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3864.218156] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3864.221918] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3864.224248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3864.227548] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3864.231202] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3864.235685] lmv_intent_open+0x338/0x1150 [lmv] [ 3864.238658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3864.242102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3864.250685] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3864.253219] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3864.260555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3864.268175] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3864.273409] ? do_raw_spin_unlock+0x75/0x190 [ 3864.281620] ? _raw_spin_unlock+0x12/0x30 [ 3864.292478] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3864.298210] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3864.307754] ll_atomic_open+0x36a/0xd70 [lustre] [ 3864.317182] lookup_open+0x356/0xb40 [ 3864.322916] do_last+0x313/0xfc0 [ 3864.324808] ? nd_jump_root+0xe5/0x160 [ 3864.330159] ? path_init+0x437/0x520 [ 3864.334925] path_openat+0xf7/0x500 [ 3864.337157] ? do_raw_spin_unlock+0x75/0x190 [ 3864.345144] do_filp_open+0x99/0x140 [ 3864.351923] ? getname_flags+0x6e/0x330 [ 3864.356817] ? __check_object_size+0xff/0x256 [ 3864.359194] ? do_raw_spin_unlock+0x75/0x190 [ 3864.361303] ? _raw_spin_unlock+0x12/0x30 [ 3864.363508] do_sys_openat2+0x2b4/0x410 [ 3864.366435] do_sys_open+0x73/0xa0 [ 3864.368345] __x64_sys_openat+0x24/0x30 [ 3864.370392] do_syscall_64+0xc1/0x440 [ 3864.372566] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3864.377840] RIP: 0033:0x7fa749552332 [ 3864.380180] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3864.391844] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3864.397381] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3864.401847] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3864.411953] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3864.419751] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3864.425933] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3869.152385] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1951s! [ 3869.158441] Showing busy workqueues and worker pools: [ 3869.160490] workqueue events: flags=0x0 [ 3869.162764] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3869.166128] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3869.172022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3869.175083] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3869.178408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.181434] pending: kfree_rcu_monitor [ 3869.183240] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3869.186552] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3869.190861] workqueue events_long: flags=0x0 [ 3869.192550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.196292] pending: lu_objects_flush [obdclass] [ 3869.199123] workqueue events_unbound: flags=0x2 [ 3869.201044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3869.204383] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3869.208915] workqueue events_power_efficient: flags=0x80 [ 3869.213805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.222462] pending: check_lifetime [ 3869.226030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.230583] pending: neigh_periodic_work [ 3869.234063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.236493] pending: neigh_periodic_work [ 3869.237711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.241542] pending: do_cache_clean [ 3869.245504] workqueue mm_percpu_wq: flags=0x8 [ 3869.249100] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.252858] pending: vmstat_update [ 3869.254739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.258031] pending: vmstat_update [ 3869.260411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3869.263507] pending: vmstat_update [ 3869.265448] workqueue writeback: flags=0x4a [ 3869.267538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3869.270343] pending: wb_workfn, wb_workfn [ 3869.272923] workqueue xprtiod: flags=0xa [ 3869.275269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3869.277810] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3869.281830] workqueue ipv6_addrconf: flags=0x40008 [ 3869.287535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3869.290796] pending: addrconf_verify_work [ 3869.293540] workqueue knbd0-recv: flags=0x1a [ 3869.296228] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3869.299843] in-flight: 651:recv_work [ 3869.301874] workqueue ptlrpc_pinger: flags=0x2 [ 3869.305790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3869.308946] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3869.312242] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1952s workers=4 idle: 37 31846 28897 [ 3869.316365] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3859s workers=2 idle: 653 [ 3876.059025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3876.063050] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3876.090987] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3876.096378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3876.100613] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3876.103353] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3876.115505] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3876.121040] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3876.123482] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3876.125920] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3876.128853] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3876.131724] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3876.135264] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3876.137771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3876.140209] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3876.143744] Call Trace: [ 3876.144645] [ 3876.145384] ? show_regs.cold.9+0x22/0x2f [ 3876.146607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3876.148584] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3876.150610] ? watchdog+0x40/0x40 [ 3876.151860] ? hrtimer_interrupt+0x128/0x2c0 [ 3876.153949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3876.156897] ? apic_timer_interrupt+0xf/0x20 [ 3876.158724] [ 3876.162911] ? native_safe_halt+0x2e/0x50 [ 3876.164871] kvm_wait+0xcd/0xe0 [ 3876.166578] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3876.171561] do_raw_spin_lock+0x177/0x190 [ 3876.184222] _raw_spin_lock+0x21/0x30 [ 3876.185377] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3876.186981] ? do_raw_spin_unlock+0x75/0x190 [ 3876.202341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3876.204041] ? do_wait_intr+0xf0/0xf0 [ 3876.214496] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3876.218845] mdc_close+0x53b/0xd70 [mdc] [ 3876.221344] lmv_close+0x18d/0x490 [lmv] [ 3876.223470] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3876.234522] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3876.237878] ll_md_close+0x274/0x800 [lustre] [ 3876.243705] ll_file_release+0x128/0xc50 [lustre] [ 3876.246651] __fput+0x10d/0x440 [ 3876.248749] ____fput+0x12/0x20 [ 3876.250756] task_work_run+0xc6/0x110 [ 3876.254235] exit_to_usermode_loop+0x1dd/0x1f0 [ 3876.258180] do_syscall_64+0x3d6/0x440 [ 3876.266533] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3876.270393] RIP: 0033:0x7fa749552cd8 [ 3876.273735] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3876.287029] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3876.295526] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3876.303831] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3876.308072] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3876.312134] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3876.320553] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3884.058017] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 3884.061028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.102738] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.111109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3884.123140] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3884.126096] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3884.142050] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3884.149976] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3884.156810] RDX: 0000000000005798 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3884.163466] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3884.169577] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3884.174818] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3884.184032] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3884.193834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.198142] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3884.200465] Call Trace: [ 3884.204037] [ 3884.205313] ? show_regs.cold.9+0x22/0x2f [ 3884.207510] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.211522] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.217299] ? watchdog+0x40/0x40 [ 3884.221199] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.225051] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.232103] ? apic_timer_interrupt+0xf/0x20 [ 3884.236498] [ 3884.237969] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3884.239624] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3884.244069] do_raw_spin_lock+0x177/0x190 [ 3884.247207] _raw_spin_lock+0x21/0x30 [ 3884.251190] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3884.256515] obd_mod_free+0x46/0x140 [ptlrpc] [ 3884.262982] mdc_commit_open+0x91/0xb0 [mdc] [ 3884.266342] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3884.272487] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3884.279434] after_reply+0xe2c/0x1780 [ptlrpc] [ 3884.285493] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3884.289282] ? __next_timer_interrupt+0x160/0x160 [ 3884.294153] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3884.298422] ? do_wait_intr+0xf0/0xf0 [ 3884.302377] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3884.308324] kthread+0x1d1/0x200 [ 3884.311719] ? set_kthread_struct+0x70/0x70 [ 3884.312849] ret_from_fork+0x1f/0x30 [ 3888.057016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3888.061114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3888.090328] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3888.095083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3888.100288] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3888.104916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3888.106560] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3888.114168] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3888.117554] RAX: 000000000000227e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3888.121226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3888.124479] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3888.127624] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3888.130906] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3888.134061] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3888.137687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3888.140296] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3888.142985] Call Trace: [ 3888.144476] [ 3888.145402] ? show_regs.cold.9+0x22/0x2f [ 3888.147077] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3888.149074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3888.150765] ? watchdog+0x40/0x40 [ 3888.152187] ? hrtimer_interrupt+0x128/0x2c0 [ 3888.154919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3888.156871] ? apic_timer_interrupt+0xf/0x20 [ 3888.158796] [ 3888.159793] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3888.162323] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3888.164413] do_raw_spin_lock+0x177/0x190 [ 3888.166438] _raw_spin_lock+0x21/0x30 [ 3888.167633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3888.170303] process_one_work+0x2c8/0x700 [ 3888.172642] worker_thread+0x296/0x6e0 [ 3888.174018] ? rescuer_thread+0x570/0x570 [ 3888.175883] kthread+0x1d1/0x200 [ 3888.177060] ? set_kthread_struct+0x70/0x70 [ 3888.178982] ret_from_fork+0x1f/0x30 [ 3896.056660] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3896.061883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3896.109550] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3896.125323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3896.136613] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3896.138830] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3896.156220] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3896.163004] RAX: 0000000000003a49 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3896.167107] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3896.170622] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3896.179222] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3896.182308] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3896.187536] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3896.192012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3896.204662] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3896.209395] Call Trace: [ 3896.211259] [ 3896.212804] ? show_regs.cold.9+0x22/0x2f [ 3896.214794] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3896.221198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3896.224002] ? watchdog+0x40/0x40 [ 3896.225123] ? hrtimer_interrupt+0x128/0x2c0 [ 3896.228241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3896.236788] ? apic_timer_interrupt+0xf/0x20 [ 3896.240610] [ 3896.244681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3896.247920] do_raw_spin_lock+0x177/0x190 [ 3896.249891] _raw_spin_lock+0x21/0x30 [ 3896.252014] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3896.255190] ? do_raw_spin_unlock+0x75/0x190 [ 3896.258066] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3896.260215] ? do_wait_intr+0xf0/0xf0 [ 3896.261587] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3896.267874] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3896.272136] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3896.277887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.281165] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3896.286632] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3896.295229] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3896.300859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.305901] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3896.308993] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3896.311975] lmv_intent_open+0x338/0x1150 [lmv] [ 3896.317562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.326069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.328133] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3896.337818] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3896.342662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.346043] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3896.348765] ? do_raw_spin_unlock+0x75/0x190 [ 3896.351086] ? _raw_spin_unlock+0x12/0x30 [ 3896.356219] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3896.361848] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3896.364736] ll_atomic_open+0x36a/0xd70 [lustre] [ 3896.369892] lookup_open+0x356/0xb40 [ 3896.371502] do_last+0x313/0xfc0 [ 3896.373819] ? nd_jump_root+0xe5/0x160 [ 3896.376417] ? path_init+0x437/0x520 [ 3896.379504] path_openat+0xf7/0x500 [ 3896.386097] ? do_raw_spin_unlock+0x75/0x190 [ 3896.394353] do_filp_open+0x99/0x140 [ 3896.398090] ? getname_flags+0x6e/0x330 [ 3896.399490] ? __check_object_size+0xff/0x256 [ 3896.405125] ? do_raw_spin_unlock+0x75/0x190 [ 3896.406650] ? _raw_spin_unlock+0x12/0x30 [ 3896.410868] do_sys_openat2+0x2b4/0x410 [ 3896.414916] do_sys_open+0x73/0xa0 [ 3896.421812] __x64_sys_openat+0x24/0x30 [ 3896.423257] do_syscall_64+0xc1/0x440 [ 3896.424952] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3896.427252] RIP: 0033:0x7fa749552332 [ 3896.430909] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3896.441846] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3896.454944] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3896.465625] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3896.469834] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3896.474786] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3896.481817] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3899.368814] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1982s! [ 3899.382777] Showing busy workqueues and worker pools: [ 3899.389929] workqueue events: flags=0x0 [ 3899.397947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3899.410742] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3899.435938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3899.447567] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3899.460822] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.472140] pending: kfree_rcu_monitor [ 3899.476841] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3899.500928] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3899.516335] workqueue events_long: flags=0x0 [ 3899.517817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.535605] pending: lu_objects_flush [obdclass] [ 3899.546021] workqueue events_unbound: flags=0x2 [ 3899.552346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3899.563019] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3899.579304] workqueue events_power_efficient: flags=0x80 [ 3899.581239] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.602872] pending: check_lifetime [ 3899.610919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.633795] pending: neigh_periodic_work [ 3899.645554] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.657381] pending: neigh_periodic_work [ 3899.660291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.668019] pending: do_cache_clean [ 3899.669402] workqueue mm_percpu_wq: flags=0x8 [ 3899.673146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.680090] pending: vmstat_update [ 3899.683555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.687938] pending: vmstat_update [ 3899.690626] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3899.698965] pending: vmstat_update [ 3899.701908] workqueue writeback: flags=0x4a [ 3899.708608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3899.714756] pending: wb_workfn, wb_workfn [ 3899.717960] workqueue xprtiod: flags=0xa [ 3899.722957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3899.730948] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3899.738539] workqueue ipv6_addrconf: flags=0x40008 [ 3899.740580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3899.745329] pending: addrconf_verify_work [ 3899.749581] workqueue knbd0-recv: flags=0x1a [ 3899.768933] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3899.790497] in-flight: 651:recv_work [ 3899.797994] workqueue ptlrpc_pinger: flags=0x2 [ 3899.801090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3899.810251] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3899.818412] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1982s workers=4 idle: 37 31846 28897 [ 3899.829915] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3889s workers=2 idle: 653 [ 3904.059122] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 3904.064268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3904.095190] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3904.104261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3904.107958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3904.110566] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3904.124416] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3904.131439] RAX: 0000000000005d41 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3904.137236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3904.143116] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3904.148542] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3904.152155] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3904.155960] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3904.159265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3904.164514] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3904.171485] Call Trace: [ 3904.172366] [ 3904.176160] ? show_regs.cold.9+0x22/0x2f [ 3904.178544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3904.180734] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3904.185320] ? watchdog+0x40/0x40 [ 3904.186819] ? hrtimer_interrupt+0x128/0x2c0 [ 3904.189170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3904.191739] ? apic_timer_interrupt+0xf/0x20 [ 3904.193949] [ 3904.194931] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3904.199661] do_raw_spin_lock+0x177/0x190 [ 3904.201602] _raw_spin_lock+0x21/0x30 [ 3904.205376] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3904.211253] ? do_raw_spin_unlock+0x75/0x190 [ 3904.213159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3904.217629] ? do_wait_intr+0xf0/0xf0 [ 3904.218931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3904.224099] mdc_close+0x53b/0xd70 [mdc] [ 3904.226097] lmv_close+0x18d/0x490 [lmv] [ 3904.227859] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3904.233347] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3904.235535] ll_md_close+0x274/0x800 [lustre] [ 3904.239733] ll_file_release+0x128/0xc50 [lustre] [ 3904.243514] __fput+0x10d/0x440 [ 3904.244690] ____fput+0x12/0x20 [ 3904.246928] task_work_run+0xc6/0x110 [ 3904.248608] exit_to_usermode_loop+0x1dd/0x1f0 [ 3904.251403] do_syscall_64+0x3d6/0x440 [ 3904.253144] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3904.255413] RIP: 0033:0x7fa749552cd8 [ 3904.257245] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3904.267206] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3904.272896] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3904.276987] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3904.282513] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3904.287150] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3904.292189] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3912.058012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3912.061025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.078306] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.082313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.086023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3912.088241] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3912.095477] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.098937] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3912.101941] RDX: 0000000000001993 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3912.105627] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3912.108959] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3912.112362] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3912.115514] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3912.119246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.121810] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3912.124869] Call Trace: [ 3912.126054] [ 3912.126926] ? show_regs.cold.9+0x22/0x2f [ 3912.128602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.130373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.132333] ? watchdog+0x40/0x40 [ 3912.133895] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.135340] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.137121] ? apic_timer_interrupt+0xf/0x20 [ 3912.139024] [ 3912.139962] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3912.142547] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3912.144729] do_raw_spin_lock+0x177/0x190 [ 3912.146506] _raw_spin_lock+0x21/0x30 [ 3912.148330] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3912.151251] obd_mod_free+0x46/0x140 [ptlrpc] [ 3912.153884] mdc_commit_open+0x91/0xb0 [mdc] [ 3912.155624] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3912.158394] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3912.160719] after_reply+0xe2c/0x1780 [ptlrpc] [ 3912.163020] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3912.165728] ? __next_timer_interrupt+0x160/0x160 [ 3912.167906] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3912.170309] ? do_wait_intr+0xf0/0xf0 [ 3912.172051] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3912.174837] kthread+0x1d1/0x200 [ 3912.176244] ? set_kthread_struct+0x70/0x70 [ 3912.178244] ret_from_fork+0x1f/0x30 [ 3916.057501] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3916.074268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3916.119825] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3916.134236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3916.144784] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3916.164279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3916.176088] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3916.213661] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3916.216397] RAX: 0000000000007d67 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3916.231162] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3916.243018] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3916.250245] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3916.254459] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3916.262596] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3916.271798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3916.275560] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3916.278241] Call Trace: [ 3916.280287] [ 3916.281487] ? show_regs.cold.9+0x22/0x2f [ 3916.283990] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3916.287354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3916.292480] ? watchdog+0x40/0x40 [ 3916.300695] ? hrtimer_interrupt+0x128/0x2c0 [ 3916.307442] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3916.318263] ? apic_timer_interrupt+0xf/0x20 [ 3916.326956] [ 3916.329733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3916.339139] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3916.348481] do_raw_spin_lock+0x177/0x190 [ 3916.355296] _raw_spin_lock+0x21/0x30 [ 3916.359007] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3916.367331] process_one_work+0x2c8/0x700 [ 3916.369708] worker_thread+0x296/0x6e0 [ 3916.374844] ? rescuer_thread+0x570/0x570 [ 3916.377841] kthread+0x1d1/0x200 [ 3916.383308] ? set_kthread_struct+0x70/0x70 [ 3916.390195] ret_from_fork+0x1f/0x30 [ 3924.055083] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 3924.072109] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3924.103106] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3924.108497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3924.112669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3924.115709] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3924.125215] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3924.128278] RAX: 000000000000768a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3924.133251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3924.137910] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3924.142198] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3924.147093] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3924.150999] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3924.154919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3924.158186] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3924.161658] Call Trace: [ 3924.162876] [ 3924.164142] ? show_regs.cold.9+0x22/0x2f [ 3924.165801] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3924.168338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3924.170517] ? watchdog+0x40/0x40 [ 3924.172289] ? hrtimer_interrupt+0x128/0x2c0 [ 3924.174203] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3924.176642] ? apic_timer_interrupt+0xf/0x20 [ 3924.178651] [ 3924.180112] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3924.182612] do_raw_spin_lock+0x177/0x190 [ 3924.184804] _raw_spin_lock+0x21/0x30 [ 3924.186547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3924.189233] ? do_raw_spin_unlock+0x75/0x190 [ 3924.191382] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3924.193661] ? do_wait_intr+0xf0/0xf0 [ 3924.195452] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3924.198244] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3924.201168] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3924.204318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.206995] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3924.209386] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3924.211911] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3924.213702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.216384] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3924.219645] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3924.222539] lmv_intent_open+0x338/0x1150 [lmv] [ 3924.224779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.227599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.231142] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3924.233617] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3924.236559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.239667] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3924.242091] ? do_raw_spin_unlock+0x75/0x190 [ 3924.244152] ? _raw_spin_unlock+0x12/0x30 [ 3924.246648] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3924.249240] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3924.251861] ll_atomic_open+0x36a/0xd70 [lustre] [ 3924.254299] lookup_open+0x356/0xb40 [ 3924.255841] do_last+0x313/0xfc0 [ 3924.257392] ? nd_jump_root+0xe5/0x160 [ 3924.259516] ? path_init+0x437/0x520 [ 3924.261658] path_openat+0xf7/0x500 [ 3924.263236] ? do_raw_spin_unlock+0x75/0x190 [ 3924.265232] do_filp_open+0x99/0x140 [ 3924.266889] ? getname_flags+0x6e/0x330 [ 3924.268922] ? __check_object_size+0xff/0x256 [ 3924.270671] ? do_raw_spin_unlock+0x75/0x190 [ 3924.273109] ? _raw_spin_unlock+0x12/0x30 [ 3924.275066] do_sys_openat2+0x2b4/0x410 [ 3924.276923] do_sys_open+0x73/0xa0 [ 3924.278818] __x64_sys_openat+0x24/0x30 [ 3924.281307] do_syscall_64+0xc1/0x440 [ 3924.282838] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3924.285515] RIP: 0033:0x7fa749552332 [ 3924.287303] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3924.295755] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3924.298843] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3924.302584] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3924.305768] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3924.308785] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3924.311793] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3930.086162] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2012s! [ 3930.096052] Showing busy workqueues and worker pools: [ 3930.105112] workqueue events: flags=0x0 [ 3930.107351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3930.115366] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3930.125581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3930.134412] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3930.141070] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.148502] pending: kfree_rcu_monitor [ 3930.155082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3930.168780] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3930.175277] workqueue events_long: flags=0x0 [ 3930.182266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.189493] pending: lu_objects_flush [obdclass] [ 3930.195565] workqueue events_unbound: flags=0x2 [ 3930.206698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3930.222348] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3930.231261] workqueue events_power_efficient: flags=0x80 [ 3930.235189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.244453] pending: check_lifetime [ 3930.248662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.255936] pending: neigh_periodic_work [ 3930.260230] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.266334] pending: neigh_periodic_work [ 3930.271693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.283159] pending: do_cache_clean [ 3930.284608] workqueue mm_percpu_wq: flags=0x8 [ 3930.288449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.303254] pending: vmstat_update [ 3930.309913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.321921] pending: vmstat_update [ 3930.324319] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3930.332582] pending: vmstat_update [ 3930.334031] workqueue writeback: flags=0x4a [ 3930.336450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3930.347544] pending: wb_workfn, wb_workfn [ 3930.351126] workqueue xprtiod: flags=0xa [ 3930.358435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3930.360595] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3930.371758] workqueue ipv6_addrconf: flags=0x40008 [ 3930.375492] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3930.383268] pending: addrconf_verify_work [ 3930.388296] workqueue knbd0-recv: flags=0x1a [ 3930.391299] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3930.400762] in-flight: 651:recv_work [ 3930.403394] workqueue ptlrpc_pinger: flags=0x2 [ 3930.407546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3930.411847] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3930.413996] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2013s workers=4 idle: 37 31846 28897 [ 3930.427837] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3920s workers=2 idle: 653 [ 3932.061978] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3932.065149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3932.098958] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3932.109914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3932.118967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3932.121200] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3932.138540] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3932.143632] RAX: 0000000000006d10 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3932.152243] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3932.156255] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3932.166321] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3932.170751] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3932.177140] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3932.181010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3932.184582] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3932.192107] Call Trace: [ 3932.193958] [ 3932.196296] ? show_regs.cold.9+0x22/0x2f [ 3932.200112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3932.206061] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3932.207719] ? watchdog+0x40/0x40 [ 3932.217870] ? hrtimer_interrupt+0x128/0x2c0 [ 3932.219882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3932.222230] ? apic_timer_interrupt+0xf/0x20 [ 3932.224050] [ 3932.225260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3932.230349] do_raw_spin_lock+0x177/0x190 [ 3932.236379] _raw_spin_lock+0x21/0x30 [ 3932.239241] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3932.242413] ? do_raw_spin_unlock+0x75/0x190 [ 3932.248464] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3932.256427] ? do_wait_intr+0xf0/0xf0 [ 3932.257745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3932.263434] mdc_close+0x53b/0xd70 [mdc] [ 3932.266783] lmv_close+0x18d/0x490 [lmv] [ 3932.277691] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3932.281670] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3932.283906] ll_md_close+0x274/0x800 [lustre] [ 3932.286615] ll_file_release+0x128/0xc50 [lustre] [ 3932.289456] __fput+0x10d/0x440 [ 3932.291130] ____fput+0x12/0x20 [ 3932.294974] task_work_run+0xc6/0x110 [ 3932.297049] exit_to_usermode_loop+0x1dd/0x1f0 [ 3932.304235] do_syscall_64+0x3d6/0x440 [ 3932.305773] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3932.307615] RIP: 0033:0x7fa749552cd8 [ 3932.309744] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3932.336916] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3932.346611] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3932.357471] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3932.371005] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3932.378304] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3932.389703] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3940.058224] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3940.062350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3940.088482] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3940.097365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3940.102285] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3940.104846] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3940.117477] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3940.121527] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3940.125440] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3940.129827] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3940.133903] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3940.138276] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3940.145275] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3940.155452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3940.158722] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3940.162482] Call Trace: [ 3940.163765] [ 3940.164919] ? show_regs.cold.9+0x22/0x2f [ 3940.167357] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3940.170815] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3940.173368] ? watchdog+0x40/0x40 [ 3940.175937] ? hrtimer_interrupt+0x128/0x2c0 [ 3940.180539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3940.183468] ? apic_timer_interrupt+0xf/0x20 [ 3940.186109] [ 3940.187331] ? native_safe_halt+0x2e/0x50 [ 3940.190285] kvm_wait+0xcd/0xe0 [ 3940.192709] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3940.195345] do_raw_spin_lock+0x177/0x190 [ 3940.197354] _raw_spin_lock+0x21/0x30 [ 3940.201364] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3940.204490] obd_mod_free+0x46/0x140 [ptlrpc] [ 3940.207287] mdc_commit_open+0x91/0xb0 [mdc] [ 3940.209636] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3940.212513] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3940.215551] after_reply+0xe2c/0x1780 [ptlrpc] [ 3940.218506] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3940.221777] ? __next_timer_interrupt+0x160/0x160 [ 3940.224378] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3940.226820] ? do_wait_intr+0xf0/0xf0 [ 3940.228705] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3940.231458] kthread+0x1d1/0x200 [ 3940.233307] ? set_kthread_struct+0x70/0x70 [ 3940.235186] ret_from_fork+0x1f/0x30 [ 3944.062005] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3944.072814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3944.137007] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3944.158196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3944.169838] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3944.179327] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3944.181691] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3944.189774] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3944.193646] RAX: 0000000000002c91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3944.197141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3944.205179] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3944.213602] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3944.219210] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3944.232813] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3944.245851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3944.254166] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3944.262322] Call Trace: [ 3944.263249] [ 3944.268091] ? show_regs.cold.9+0x22/0x2f [ 3944.269614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3944.277349] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3944.281072] ? watchdog+0x40/0x40 [ 3944.285443] ? hrtimer_interrupt+0x128/0x2c0 [ 3944.291360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3944.298866] ? apic_timer_interrupt+0xf/0x20 [ 3944.302550] [ 3944.305059] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3944.310169] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3944.317201] do_raw_spin_lock+0x177/0x190 [ 3944.323707] _raw_spin_lock+0x21/0x30 [ 3944.326530] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3944.335038] process_one_work+0x2c8/0x700 [ 3944.340524] worker_thread+0x296/0x6e0 [ 3944.346230] ? rescuer_thread+0x570/0x570 [ 3944.352436] kthread+0x1d1/0x200 [ 3944.357886] ? set_kthread_struct+0x70/0x70 [ 3944.365292] ret_from_fork+0x1f/0x30 [ 3953.735425] rcu: INFO: rcu_sched self-detected stall on CPU [ 3953.742120] rcu: 2-...!: (1105621 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 3953.759203] (t=2040046 jiffies g=258701 q=2166) [ 3953.768061] rcu: rcu_sched kthread starved for 2036841 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 3953.784912] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3953.799877] rcu: RCU grace-period kthread stack dump: [ 3953.811460] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3953.828331] Call Trace: [ 3953.829197] __schedule+0x351/0xcb0 [ 3953.830473] schedule+0xc0/0x180 [ 3953.843238] schedule_timeout+0xb4/0x190 [ 3953.855174] ? do_raw_spin_unlock+0x75/0x190 [ 3953.862935] ? __next_timer_interrupt+0x160/0x160 [ 3953.874036] ? prepare_to_swait_event+0xf3/0x140 [ 3953.879153] rcu_gp_kthread+0x8dd/0x15b0 [ 3953.882508] ? force_qs_rnp+0x2c0/0x2c0 [ 3953.884834] kthread+0x1d1/0x200 [ 3953.889525] ? set_kthread_struct+0x70/0x70 [ 3953.894315] ret_from_fork+0x1f/0x30 [ 3953.899572] rcu: Stack dump where RCU GP kthread last ran: [ 3953.904251] Sending NMI from CPU 2 to CPUs 0: [ 3953.909512] NMI backtrace for cpu 0 [ 3953.909519] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3953.909522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3953.909524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3953.909525] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3953.909528] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [ 3953.909530] RAX: 0000000000004299 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3953.909532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3953.909533] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3953.909535] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3953.909536] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3953.909538] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3953.909539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3953.909540] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3953.909542] Call Trace: [ 3953.909543] [ 3953.909543] ? show_regs.cold.9+0x22/0x2f [ 3953.909544] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3953.909545] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3953.909546] ? nmi_handle+0xbd/0x210 [ 3953.909547] ? default_do_nmi+0x4b/0x1b0 [ 3953.909548] ? do_nmi+0x280/0x380 [ 3953.909549] ? end_repeat_nmi+0x16/0x1f [ 3953.909550] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3953.909551] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3953.909552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3953.909553] [ 3953.909554] do_raw_spin_lock+0x177/0x190 [ 3953.909555] _raw_spin_lock+0x21/0x30 [ 3953.909556] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3953.909557] ? do_raw_spin_unlock+0x75/0x190 [ 3953.909559] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3953.909560] ? do_wait_intr+0xf0/0xf0 [ 3953.909561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3953.909562] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3953.909563] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3953.909564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3953.909565] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3953.909566] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3953.909567] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3953.909568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3953.909570] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3953.909571] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3953.909625] lmv_intent_open+0x338/0x1150 [lmv] [ 3953.909627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3953.909629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3953.909630] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3953.909631] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3953.909632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3953.909633] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3953.909634] ? do_raw_spin_unlock+0x75/0x190 [ 3953.909635] ? _raw_spin_unlock+0x12/0x30 [ 3953.909637] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3953.909638] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3953.909639] ll_atomic_open+0x36a/0xd70 [lustre] [ 3953.909640] lookup_open+0x356/0xb40 [ 3953.909641] do_last+0x313/0xfc0 [ 3953.909642] ? nd_jump_root+0xe5/0x160 [ 3953.909643] ? path_init+0x437/0x520 [ 3953.909644] path_openat+0xf7/0x500 [ 3953.909645] ? do_raw_spin_unlock+0x75/0x190 [ 3953.909646] do_filp_open+0x99/0x140 [ 3953.909648] ? getname_flags+0x6e/0x330 [ 3953.909649] ? __check_object_size+0xff/0x256 [ 3953.909650] ? do_raw_spin_unlock+0x75/0x190 [ 3953.909651] ? _raw_spin_unlock+0x12/0x30 [ 3953.909652] do_sys_openat2+0x2b4/0x410 [ 3953.909653] do_sys_open+0x73/0xa0 [ 3953.909654] __x64_sys_openat+0x24/0x30 [ 3953.909655] do_syscall_64+0xc1/0x440 [ 3953.909656] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3953.909657] RIP: 0033:0x7fa749552332 [ 3953.909658] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3953.909660] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3953.909663] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3953.909664] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3953.909665] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3953.909667] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3953.909668] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3953.912989] NMI backtrace for cpu 2 [ 3954.300481] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3954.324087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3954.335139] Call Trace: [ 3954.335919] [ 3954.336566] ? dump_stack+0xbb/0x10e [ 3954.343520] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3954.345784] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3954.350708] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3954.362427] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3954.365194] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3954.372297] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3954.375875] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3954.379513] ? update_process_times+0x6d/0xc0 [ 3954.382040] ? tick_sched_handle.isra.18+0x27/0x90 [ 3954.385751] ? tick_sched_timer+0x53/0xb0 [ 3954.392541] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3954.395511] ? tick_sched_handle.isra.18+0x90/0x90 [ 3954.397674] ? hrtimer_interrupt+0x128/0x2c0 [ 3954.399545] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3954.400846] ? apic_timer_interrupt+0xf/0x20 [ 3954.403505] [ 3954.404398] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3954.408800] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3954.412208] ? do_raw_spin_lock+0x177/0x190 [ 3954.414209] ? _raw_spin_lock+0x21/0x30 [ 3954.415641] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3954.420152] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3954.422055] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3954.424675] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3954.428294] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3954.433335] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3954.438467] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3954.441282] ? __next_timer_interrupt+0x160/0x160 [ 3954.443551] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3954.449021] ? do_wait_intr+0xf0/0xf0 [ 3954.451512] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3954.454608] ? kthread+0x1d1/0x200 [ 3954.456128] ? set_kthread_struct+0x70/0x70 [ 3954.458216] ? ret_from_fork+0x1f/0x30 [ 3960.059021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3960.062223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3960.092829] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3960.100896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3960.105364] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3960.107895] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3960.121425] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3960.125686] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3960.131097] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3960.134677] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3960.138534] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3960.142662] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3960.146835] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3960.151700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3960.157329] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3960.161224] Call Trace: [ 3960.163089] [ 3960.164190] ? show_regs.cold.9+0x22/0x2f [ 3960.168068] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3960.171060] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3960.173659] ? watchdog+0x40/0x40 [ 3960.177004] ? hrtimer_interrupt+0x128/0x2c0 [ 3960.182949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3960.189480] ? apic_timer_interrupt+0xf/0x20 [ 3960.191053] [ 3960.191859] ? native_safe_halt+0x2e/0x50 [ 3960.206130] kvm_wait+0xcd/0xe0 [ 3960.207015] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3960.210970] do_raw_spin_lock+0x177/0x190 [ 3960.213552] _raw_spin_lock+0x21/0x30 [ 3960.215291] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3960.218717] ? do_raw_spin_unlock+0x75/0x190 [ 3960.220713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3960.230906] ? do_wait_intr+0xf0/0xf0 [ 3960.236071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3960.242656] mdc_close+0x53b/0xd70 [mdc] [ 3960.248737] lmv_close+0x18d/0x490 [lmv] [ 3960.256331] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3960.266038] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3960.272633] ll_md_close+0x274/0x800 [lustre] [ 3960.278137] ll_file_release+0x128/0xc50 [lustre] [ 3960.285146] __fput+0x10d/0x440 [ 3960.287112] ____fput+0x12/0x20 [ 3960.289291] task_work_run+0xc6/0x110 [ 3960.293279] exit_to_usermode_loop+0x1dd/0x1f0 [ 3960.296268] do_syscall_64+0x3d6/0x440 [ 3960.302088] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3960.306758] RIP: 0033:0x7fa749552cd8 [ 3960.312869] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3960.330511] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3960.339183] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3960.349694] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3960.356526] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3960.362089] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3960.365989] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3960.801034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2043s! [ 3960.805367] Showing busy workqueues and worker pools: [ 3960.815424] workqueue events: flags=0x0 [ 3960.820235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3960.827795] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3960.844588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3960.863958] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3960.888724] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3960.903490] pending: kfree_rcu_monitor [ 3960.906655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3960.915273] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3960.931899] workqueue events_long: flags=0x0 [ 3960.943065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3960.956592] pending: lu_objects_flush [obdclass] [ 3960.971545] workqueue events_unbound: flags=0x2 [ 3960.982369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3961.000645] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3961.013320] workqueue events_power_efficient: flags=0x80 [ 3961.035245] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3961.050773] pending: check_lifetime [ 3961.057848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3961.072861] pending: neigh_periodic_work [ 3961.074470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3961.089496] pending: neigh_periodic_work [ 3961.097960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3961.105912] pending: do_cache_clean [ 3961.115908] workqueue mm_percpu_wq: flags=0x8 [ 3961.126990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3961.139673] pending: vmstat_update [ 3961.144434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3961.156547] pending: vmstat_update [ 3961.162268] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3961.178630] pending: vmstat_update [ 3961.186903] workqueue writeback: flags=0x4a [ 3961.197806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3961.206845] pending: wb_workfn, wb_workfn [ 3961.218135] workqueue xprtiod: flags=0xa [ 3961.223836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3961.235265] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3961.246808] workqueue ipv6_addrconf: flags=0x40008 [ 3961.257293] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3961.267253] pending: addrconf_verify_work [ 3961.276164] workqueue knbd0-recv: flags=0x1a [ 3961.279563] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3961.283191] in-flight: 651:recv_work [ 3961.286244] workqueue ptlrpc_pinger: flags=0x2 [ 3961.293991] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3961.309271] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3961.323893] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2044s workers=4 idle: 37 31846 28897 [ 3961.336916] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3951s workers=2 idle: 653 [ 3972.058515] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 3972.066473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3972.118838] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3972.134642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3972.138670] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3972.154645] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3972.162846] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3972.185852] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3972.191288] RAX: 0000000000005f07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3972.198626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 3972.204561] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3972.209487] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 3972.219878] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 3972.226608] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 3972.231710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3972.235160] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 3972.237840] Call Trace: [ 3972.238831] [ 3972.239741] ? show_regs.cold.9+0x22/0x2f [ 3972.241475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3972.243151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3972.244846] ? watchdog+0x40/0x40 [ 3972.246223] ? hrtimer_interrupt+0x128/0x2c0 [ 3972.247816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3972.250337] ? apic_timer_interrupt+0xf/0x20 [ 3972.252089] [ 3972.252913] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3972.255307] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3972.257481] do_raw_spin_lock+0x177/0x190 [ 3972.259078] _raw_spin_lock+0x21/0x30 [ 3972.260683] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3972.263017] process_one_work+0x2c8/0x700 [ 3972.264718] worker_thread+0x296/0x6e0 [ 3972.266719] ? rescuer_thread+0x570/0x570 [ 3972.268279] kthread+0x1d1/0x200 [ 3972.269577] ? set_kthread_struct+0x70/0x70 [ 3972.271317] ret_from_fork+0x1f/0x30 [ 3980.058086] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 3980.062207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3980.081073] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3980.086470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3980.093923] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3980.101711] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3980.121364] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3980.123721] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 3980.125732] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 3980.127786] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 3980.130103] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 3980.132727] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3980.135336] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 3980.138711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3980.141077] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 3980.144283] Call Trace: [ 3980.145297] [ 3980.146068] ? show_regs.cold.9+0x22/0x2f [ 3980.147252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3980.151485] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3980.155108] ? watchdog+0x40/0x40 [ 3980.156274] ? hrtimer_interrupt+0x128/0x2c0 [ 3980.162780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3980.166196] ? apic_timer_interrupt+0xf/0x20 [ 3980.169527] [ 3980.170311] ? native_safe_halt+0x2e/0x50 [ 3980.171803] kvm_wait+0xcd/0xe0 [ 3980.173768] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3980.176671] do_raw_spin_lock+0x177/0x190 [ 3980.178455] _raw_spin_lock+0x21/0x30 [ 3980.180776] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3980.183154] obd_mod_free+0x46/0x140 [ptlrpc] [ 3980.185095] mdc_commit_open+0x91/0xb0 [mdc] [ 3980.187935] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3980.190505] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3980.194155] after_reply+0xe2c/0x1780 [ptlrpc] [ 3980.196869] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3980.200372] ? __next_timer_interrupt+0x160/0x160 [ 3980.204707] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3980.207233] ? do_wait_intr+0xf0/0xf0 [ 3980.209003] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3980.213129] kthread+0x1d1/0x200 [ 3980.215729] ? set_kthread_struct+0x70/0x70 [ 3980.221582] ret_from_fork+0x1f/0x30 [ 3988.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 3988.056124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3988.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 3988.086004] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3988.086022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3988.086026] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3988.086040] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3988.086044] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3988.086049] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3988.086051] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 3988.086053] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 3988.086055] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 3988.086057] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 3988.086060] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 3988.086062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3988.086064] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 3988.086073] Call Trace: [ 3988.086078] [ 3988.086083] ? show_regs.cold.9+0x22/0x2f [ 3988.086091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3988.086097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3988.086112] ? watchdog+0x40/0x40 [ 3988.086116] ? hrtimer_interrupt+0x128/0x2c0 [ 3988.086120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3988.086128] ? apic_timer_interrupt+0xf/0x20 [ 3988.086133] [ 3988.086136] ? native_safe_halt+0x2e/0x50 [ 3988.086140] kvm_wait+0xcd/0xe0 [ 3988.086147] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3988.086158] do_raw_spin_lock+0x177/0x190 [ 3988.086163] _raw_spin_lock+0x21/0x30 [ 3988.086167] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3988.086496] ? do_raw_spin_unlock+0x75/0x190 [ 3988.086503] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3988.086706] ? do_wait_intr+0xf0/0xf0 [ 3988.086715] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3988.086988] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3988.087273] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3988.095036] Modules linked in: [ 3988.099991] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3988.109049] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3988.109199] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3988.109206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3988.109209] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3988.109223] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3988.109229] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3988.109234] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 3988.109236] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 3988.109238] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3988.109240] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 3988.109242] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 3988.109245] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 3988.109248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3988.109250] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 3988.109259] Call Trace: [ 3988.109265] [ 3988.109271] ? show_regs.cold.9+0x22/0x2f [ 3988.114706] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3988.114760] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3988.114791] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3988.130740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3988.133715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3988.133852] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3988.140077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3988.140096] ? watchdog+0x40/0x40 [ 3988.140113] ? hrtimer_interrupt+0x128/0x2c0 [ 3988.140117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3988.140125] ? apic_timer_interrupt+0xf/0x20 [ 3988.140129] [ 3988.140133] ? native_safe_halt+0x2e/0x50 [ 3988.140137] kvm_wait+0xcd/0xe0 [ 3988.140146] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3988.140154] do_raw_spin_lock+0x177/0x190 [ 3988.140158] _raw_spin_lock+0x21/0x30 [ 3988.140162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3988.141762] ? do_raw_spin_unlock+0x75/0x190 [ 3988.141770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3988.145565] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3988.146039] lmv_intent_open+0x338/0x1150 [lmv] [ 3988.146077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3988.146215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3988.146298] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3988.148982] ? do_wait_intr+0xf0/0xf0 [ 3988.149002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3988.150114] mdc_close+0x53b/0xd70 [mdc] [ 3988.150154] lmv_close+0x18d/0x490 [lmv] [ 3988.150184] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3988.150303] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3988.154592] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3988.154718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3988.154799] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3988.158306] ll_md_close+0x274/0x800 [lustre] [ 3988.158797] ll_file_release+0x128/0xc50 [lustre] [ 3988.158996] __fput+0x10d/0x440 [ 3988.159006] ____fput+0x12/0x20 [ 3988.159011] task_work_run+0xc6/0x110 [ 3988.159019] exit_to_usermode_loop+0x1dd/0x1f0 [ 3988.159026] do_syscall_64+0x3d6/0x440 [ 3988.159031] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3988.159040] RIP: 0033:0x7fa749552cd8 [ 3988.159047] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3988.159053] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3988.159058] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 3988.159060] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 3988.159062] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 3988.159064] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 3988.159066] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3988.609103] ? do_raw_spin_unlock+0x75/0x190 [ 3988.611013] ? _raw_spin_unlock+0x12/0x30 [ 3988.612245] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3988.615178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3988.617960] ll_atomic_open+0x36a/0xd70 [lustre] [ 3988.619933] lookup_open+0x356/0xb40 [ 3988.624207] do_last+0x313/0xfc0 [ 3988.627248] ? nd_jump_root+0xe5/0x160 [ 3988.629578] ? path_init+0x437/0x520 [ 3988.632109] path_openat+0xf7/0x500 [ 3988.637043] ? do_raw_spin_unlock+0x75/0x190 [ 3988.639225] do_filp_open+0x99/0x140 [ 3988.641911] ? getname_flags+0x6e/0x330 [ 3988.643586] ? __check_object_size+0xff/0x256 [ 3988.646786] ? do_raw_spin_unlock+0x75/0x190 [ 3988.652665] ? _raw_spin_unlock+0x12/0x30 [ 3988.654260] do_sys_openat2+0x2b4/0x410 [ 3988.656192] do_sys_open+0x73/0xa0 [ 3988.657300] __x64_sys_openat+0x24/0x30 [ 3988.658779] do_syscall_64+0xc1/0x440 [ 3988.660295] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3988.663157] RIP: 0033:0x7fa749552332 [ 3988.664649] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3988.687906] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3988.698350] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 3988.710208] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 3988.723174] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 3988.731425] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 3988.747927] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 3991.527706] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2074s! [ 3991.537653] Showing busy workqueues and worker pools: [ 3991.545266] workqueue events: flags=0x0 [ 3991.553478] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3991.563504] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 3991.573032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3991.576557] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3991.587963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.593771] pending: kfree_rcu_monitor [ 3991.598989] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3991.607098] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 3991.626846] workqueue events_long: flags=0x0 [ 3991.629200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.632998] pending: lu_objects_flush [obdclass] [ 3991.643293] workqueue events_unbound: flags=0x2 [ 3991.649422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 3991.655059] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3991.663779] workqueue events_power_efficient: flags=0x80 [ 3991.667033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.670174] pending: check_lifetime [ 3991.672089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.675843] pending: neigh_periodic_work [ 3991.677874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.680856] pending: neigh_periodic_work [ 3991.682732] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.685330] pending: do_cache_clean [ 3991.686738] workqueue mm_percpu_wq: flags=0x8 [ 3991.688377] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.690968] pending: vmstat_update [ 3991.692681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.695250] pending: vmstat_update [ 3991.696561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3991.699068] pending: vmstat_update [ 3991.700499] workqueue writeback: flags=0x4a [ 3991.702126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3991.704496] pending: wb_workfn, wb_workfn [ 3991.706115] workqueue xprtiod: flags=0xa [ 3991.707738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3991.710026] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3991.713109] workqueue ipv6_addrconf: flags=0x40008 [ 3991.714946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3991.717988] pending: addrconf_verify_work [ 3991.719580] workqueue knbd0-recv: flags=0x1a [ 3991.720977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3991.723206] in-flight: 651:recv_work [ 3991.724678] workqueue ptlrpc_pinger: flags=0x2 [ 3991.726419] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3991.728668] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3991.731089] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2074s workers=4 idle: 37 31846 28897 [ 3991.733675] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3981s workers=2 idle: 653 [ 4000.063255] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4000.067902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4000.106783] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4000.119027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4000.127397] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4000.135600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4000.140237] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4000.153487] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4000.157202] RAX: 0000000000007c09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4000.161803] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4000.175800] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4000.178947] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4000.182975] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4000.190830] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4000.199962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4000.201789] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4000.206835] Call Trace: [ 4000.209517] [ 4000.210443] ? show_regs.cold.9+0x22/0x2f [ 4000.211630] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4000.213498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4000.215249] ? watchdog+0x40/0x40 [ 4000.216946] ? hrtimer_interrupt+0x128/0x2c0 [ 4000.219665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4000.222245] ? apic_timer_interrupt+0xf/0x20 [ 4000.224124] [ 4000.225058] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4000.229324] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4000.231861] do_raw_spin_lock+0x177/0x190 [ 4000.240980] _raw_spin_lock+0x21/0x30 [ 4000.243413] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4000.253292] process_one_work+0x2c8/0x700 [ 4000.257281] worker_thread+0x296/0x6e0 [ 4000.259954] ? rescuer_thread+0x570/0x570 [ 4000.262173] kthread+0x1d1/0x200 [ 4000.263560] ? set_kthread_struct+0x70/0x70 [ 4000.267520] ret_from_fork+0x1f/0x30 [ 4008.058064] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4008.060351] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4008.085906] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4008.090381] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4008.092300] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4008.093829] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4008.098540] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4008.100203] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4008.102112] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4008.106550] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4008.110020] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4008.112571] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4008.116055] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4008.119783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4008.123818] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4008.127437] Call Trace: [ 4008.128630] [ 4008.132931] ? show_regs.cold.9+0x22/0x2f [ 4008.135480] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4008.139727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4008.143699] ? watchdog+0x40/0x40 [ 4008.145337] ? hrtimer_interrupt+0x128/0x2c0 [ 4008.147686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4008.149591] ? apic_timer_interrupt+0xf/0x20 [ 4008.151701] [ 4008.152871] ? native_safe_halt+0x2e/0x50 [ 4008.158171] kvm_wait+0xcd/0xe0 [ 4008.160325] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4008.163617] do_raw_spin_lock+0x177/0x190 [ 4008.165813] _raw_spin_lock+0x21/0x30 [ 4008.167144] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4008.172919] obd_mod_free+0x46/0x140 [ptlrpc] [ 4008.175860] mdc_commit_open+0x91/0xb0 [mdc] [ 4008.178884] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4008.183356] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4008.188175] after_reply+0xe2c/0x1780 [ptlrpc] [ 4008.192428] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4008.194513] ? __next_timer_interrupt+0x160/0x160 [ 4008.199358] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4008.201865] ? do_wait_intr+0xf0/0xf0 [ 4008.203828] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4008.207659] kthread+0x1d1/0x200 [ 4008.209711] ? set_kthread_struct+0x70/0x70 [ 4008.213437] ret_from_fork+0x1f/0x30 [ 4016.054368] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4016.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4016.061904] Modules linked in: [ 4016.067012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4016.067077] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4016.067083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4016.067085] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4016.067096] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4016.067100] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4016.067104] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4016.067106] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4016.067108] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4016.067109] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4016.067111] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4016.067113] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4016.067116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4016.067117] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4016.067124] Call Trace: [ 4016.067130] [ 4016.067136] ? show_regs.cold.9+0x22/0x2f [ 4016.067146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4016.067152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4016.067154] ? watchdog+0x40/0x40 [ 4016.067157] ? hrtimer_interrupt+0x128/0x2c0 [ 4016.067160] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4016.067165] ? apic_timer_interrupt+0xf/0x20 [ 4016.067168] [ 4016.067171] ? native_safe_halt+0x2e/0x50 [ 4016.067173] kvm_wait+0xcd/0xe0 [ 4016.067180] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4016.067186] do_raw_spin_lock+0x177/0x190 [ 4016.067189] _raw_spin_lock+0x21/0x30 [ 4016.067192] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4016.067485] ? do_raw_spin_unlock+0x75/0x190 [ 4016.067489] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4016.067654] ? do_wait_intr+0xf0/0xf0 [ 4016.067662] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4016.068305] mdc_close+0x53b/0xd70 [mdc] [ 4016.068407] lmv_close+0x18d/0x490 [lmv] [ 4016.068535] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4016.068737] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4016.068968] ll_md_close+0x274/0x800 [lustre] [ 4016.069031] ll_file_release+0x128/0xc50 [lustre] [ 4016.069084] __fput+0x10d/0x440 [ 4016.069092] ____fput+0x12/0x20 [ 4016.069095] task_work_run+0xc6/0x110 [ 4016.069101] exit_to_usermode_loop+0x1dd/0x1f0 [ 4016.069107] do_syscall_64+0x3d6/0x440 [ 4016.069110] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4016.069115] RIP: 0033:0x7fa749552cd8 [ 4016.069121] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4016.069124] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4016.069127] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4016.069128] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4016.069130] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4016.069131] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4016.069133] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4016.264411] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4016.292174] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4016.299052] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4016.305366] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4016.306474] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4016.319128] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4016.325044] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4016.329101] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4016.331952] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4016.337825] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4016.341540] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4016.346952] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4016.350219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4016.352187] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4016.356110] Call Trace: [ 4016.357220] [ 4016.358514] ? show_regs.cold.9+0x22/0x2f [ 4016.361153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4016.364017] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4016.368186] ? watchdog+0x40/0x40 [ 4016.370024] ? hrtimer_interrupt+0x128/0x2c0 [ 4016.373853] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4016.377400] ? apic_timer_interrupt+0xf/0x20 [ 4016.380057] [ 4016.382669] ? native_safe_halt+0x2e/0x50 [ 4016.384867] kvm_wait+0xcd/0xe0 [ 4016.387942] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4016.391496] do_raw_spin_lock+0x177/0x190 [ 4016.393861] _raw_spin_lock+0x21/0x30 [ 4016.396101] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4016.400869] ? do_raw_spin_unlock+0x75/0x190 [ 4016.403952] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4016.405405] ? do_wait_intr+0xf0/0xf0 [ 4016.407056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4016.409963] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4016.413713] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4016.420594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4016.427406] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4016.434257] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4016.437720] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4016.443154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4016.448149] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4016.454193] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4016.457596] lmv_intent_open+0x338/0x1150 [lmv] [ 4016.460102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4016.465145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4016.473076] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4016.475574] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4016.477727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4016.482121] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4016.488330] ? do_raw_spin_unlock+0x75/0x190 [ 4016.493100] ? _raw_spin_unlock+0x12/0x30 [ 4016.495583] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4016.498759] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4016.501528] ll_atomic_open+0x36a/0xd70 [lustre] [ 4016.507089] lookup_open+0x356/0xb40 [ 4016.508762] do_last+0x313/0xfc0 [ 4016.512733] ? nd_jump_root+0xe5/0x160 [ 4016.516172] ? path_init+0x437/0x520 [ 4016.519103] path_openat+0xf7/0x500 [ 4016.521188] ? do_raw_spin_unlock+0x75/0x190 [ 4016.523219] do_filp_open+0x99/0x140 [ 4016.530341] ? getname_flags+0x6e/0x330 [ 4016.533086] ? __check_object_size+0xff/0x256 [ 4016.535961] ? do_raw_spin_unlock+0x75/0x190 [ 4016.542059] ? _raw_spin_unlock+0x12/0x30 [ 4016.545747] do_sys_openat2+0x2b4/0x410 [ 4016.551978] do_sys_open+0x73/0xa0 [ 4016.554552] __x64_sys_openat+0x24/0x30 [ 4016.558276] do_syscall_64+0xc1/0x440 [ 4016.560375] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4016.565156] RIP: 0033:0x7fa749552332 [ 4016.566859] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4016.584878] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4016.587488] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4016.592972] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4016.598524] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4016.601877] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4016.605292] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4022.247081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2105s! [ 4022.252640] Showing busy workqueues and worker pools: [ 4022.255579] workqueue events: flags=0x0 [ 4022.257099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4022.259619] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4022.267132] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4022.270474] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4022.276295] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.280198] pending: kfree_rcu_monitor [ 4022.282706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4022.286807] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4022.293875] workqueue events_long: flags=0x0 [ 4022.297362] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.302888] pending: lu_objects_flush [obdclass] [ 4022.304704] workqueue events_unbound: flags=0x2 [ 4022.309112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4022.311927] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4022.317598] workqueue events_power_efficient: flags=0x80 [ 4022.320591] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.331187] pending: check_lifetime [ 4022.335341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.338373] pending: neigh_periodic_work [ 4022.345762] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.351229] pending: neigh_periodic_work [ 4022.354757] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.360402] pending: do_cache_clean [ 4022.362521] workqueue mm_percpu_wq: flags=0x8 [ 4022.364535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.367820] pending: vmstat_update [ 4022.369732] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.373187] pending: vmstat_update [ 4022.375151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4022.378420] pending: vmstat_update [ 4022.380343] workqueue writeback: flags=0x4a [ 4022.382435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4022.385390] pending: wb_workfn, wb_workfn [ 4022.387696] workqueue xprtiod: flags=0xa [ 4022.390305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4022.393521] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4022.397806] workqueue ipv6_addrconf: flags=0x40008 [ 4022.400388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4022.403797] pending: addrconf_verify_work [ 4022.406164] workqueue knbd0-recv: flags=0x1a [ 4022.409185] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4022.413303] in-flight: 651:recv_work [ 4022.415707] workqueue ptlrpc_pinger: flags=0x2 [ 4022.418399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4022.424312] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4022.430593] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2105s workers=4 idle: 37 31846 28897 [ 4022.435251] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4012s workers=2 idle: 653 [ 4028.057069] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4028.060333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4028.085464] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4028.090081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4028.093856] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4028.096755] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4028.098820] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4028.108475] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4028.112885] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4028.117694] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4028.127355] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4028.135020] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4028.137520] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4028.141573] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4028.145865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4028.148568] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4028.151647] Call Trace: [ 4028.153225] [ 4028.154091] ? show_regs.cold.9+0x22/0x2f [ 4028.156446] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4028.158452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4028.160732] ? watchdog+0x40/0x40 [ 4028.162575] ? hrtimer_interrupt+0x128/0x2c0 [ 4028.166559] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4028.169385] ? apic_timer_interrupt+0xf/0x20 [ 4028.174530] [ 4028.176257] ? native_safe_halt+0x2e/0x50 [ 4028.180077] kvm_wait+0xcd/0xe0 [ 4028.183713] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4028.189561] do_raw_spin_lock+0x177/0x190 [ 4028.193020] _raw_spin_lock+0x21/0x30 [ 4028.195014] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4028.199625] process_one_work+0x2c8/0x700 [ 4028.201464] worker_thread+0x296/0x6e0 [ 4028.206801] ? rescuer_thread+0x570/0x570 [ 4028.208375] kthread+0x1d1/0x200 [ 4028.210552] ? set_kthread_struct+0x70/0x70 [ 4028.212202] ret_from_fork+0x1f/0x30 [ 4036.058023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4036.061928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.090189] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.098139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4036.115320] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4036.122992] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4036.144846] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4036.156706] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4036.166302] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4036.177929] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4036.183111] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4036.193622] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4036.203416] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4036.217319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.218727] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4036.229460] Call Trace: [ 4036.230957] [ 4036.231839] ? show_regs.cold.9+0x22/0x2f [ 4036.235230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.239275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.242791] ? watchdog+0x40/0x40 [ 4036.245313] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.251278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.262248] ? apic_timer_interrupt+0xf/0x20 [ 4036.265331] [ 4036.268472] ? native_safe_halt+0x2e/0x50 [ 4036.276441] kvm_wait+0xcd/0xe0 [ 4036.280177] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4036.284637] do_raw_spin_lock+0x177/0x190 [ 4036.288274] _raw_spin_lock+0x21/0x30 [ 4036.290672] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4036.293662] obd_mod_free+0x46/0x140 [ptlrpc] [ 4036.299036] mdc_commit_open+0x91/0xb0 [mdc] [ 4036.305780] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4036.310370] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4036.315390] after_reply+0xe2c/0x1780 [ptlrpc] [ 4036.319240] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4036.323421] ? __next_timer_interrupt+0x160/0x160 [ 4036.326490] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4036.330483] ? do_wait_intr+0xf0/0xf0 [ 4036.333680] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4036.336612] kthread+0x1d1/0x200 [ 4036.338547] ? set_kthread_struct+0x70/0x70 [ 4036.345524] ret_from_fork+0x1f/0x30 [ 4044.059015] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4044.062212] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4044.099817] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4044.114520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4044.123323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4044.127949] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4044.140446] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4044.148054] RAX: 00000000000030da RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4044.152037] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4044.155646] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4044.159628] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4044.164276] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4044.174453] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4044.187579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4044.195797] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4044.200340] Call Trace: [ 4044.201644] [ 4044.202692] ? show_regs.cold.9+0x22/0x2f [ 4044.205806] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4044.209032] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4044.211313] ? watchdog+0x40/0x40 [ 4044.213139] ? hrtimer_interrupt+0x128/0x2c0 [ 4044.215241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4044.217975] ? apic_timer_interrupt+0xf/0x20 [ 4044.219949] [ 4044.220946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4044.223462] do_raw_spin_lock+0x177/0x190 [ 4044.225430] _raw_spin_lock+0x21/0x30 [ 4044.227182] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4044.229947] ? do_raw_spin_unlock+0x75/0x190 [ 4044.231627] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4044.234040] ? do_wait_intr+0xf0/0xf0 [ 4044.235348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4044.238244] mdc_close+0x53b/0xd70 [mdc] [ 4044.240442] lmv_close+0x18d/0x490 [lmv] [ 4044.242583] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4044.246113] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4044.249406] ll_md_close+0x274/0x800 [lustre] [ 4044.252086] ll_file_release+0x128/0xc50 [lustre] [ 4044.253628] __fput+0x10d/0x440 [ 4044.254563] ____fput+0x12/0x20 [ 4044.256064] task_work_run+0xc6/0x110 [ 4044.257336] exit_to_usermode_loop+0x1dd/0x1f0 [ 4044.258768] do_syscall_64+0x3d6/0x440 [ 4044.260559] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4044.264284] RIP: 0033:0x7fa749552cd8 [ 4044.266483] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4044.278155] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4044.283792] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4044.287645] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4044.293225] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4044.296539] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4044.300641] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4048.054076] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4048.059968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.117722] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.131217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.139858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4048.143156] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4048.172896] RSP: 0018:ffffb893c1493500 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [ 4048.178799] RAX: 0000000000001a00 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4048.186559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4048.196202] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4048.199598] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4048.202950] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4048.207612] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4048.212172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.220719] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4048.227804] Call Trace: [ 4048.233396] [ 4048.234564] ? show_regs.cold.9+0x22/0x2f [ 4048.237147] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.242041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.247280] ? watchdog+0x40/0x40 [ 4048.248881] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.254846] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.260978] ? apic_timer_interrupt+0xf/0x20 [ 4048.268759] [ 4048.270675] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4048.278597] do_raw_spin_lock+0x177/0x190 [ 4048.285092] _raw_spin_lock+0x21/0x30 [ 4048.289526] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4048.293924] ? do_raw_spin_unlock+0x75/0x190 [ 4048.298554] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4048.305481] ? do_wait_intr+0xf0/0xf0 [ 4048.307863] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4048.315197] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4048.319951] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4048.322948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.326169] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4048.329984] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4048.341705] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4048.347963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.351691] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4048.358613] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4048.361602] lmv_intent_open+0x338/0x1150 [lmv] [ 4048.366357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.373362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.377782] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4048.380828] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4048.384180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.392676] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4048.397521] ? do_raw_spin_unlock+0x75/0x190 [ 4048.398737] ? _raw_spin_unlock+0x12/0x30 [ 4048.406937] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4048.410995] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4048.414476] ll_atomic_open+0x36a/0xd70 [lustre] [ 4048.417101] lookup_open+0x356/0xb40 [ 4048.421724] do_last+0x313/0xfc0 [ 4048.424976] ? nd_jump_root+0xe5/0x160 [ 4048.427533] ? path_init+0x437/0x520 [ 4048.429913] path_openat+0xf7/0x500 [ 4048.433093] ? do_raw_spin_unlock+0x75/0x190 [ 4048.436122] do_filp_open+0x99/0x140 [ 4048.437977] ? getname_flags+0x6e/0x330 [ 4048.440173] ? __check_object_size+0xff/0x256 [ 4048.443015] ? do_raw_spin_unlock+0x75/0x190 [ 4048.444714] ? _raw_spin_unlock+0x12/0x30 [ 4048.447733] do_sys_openat2+0x2b4/0x410 [ 4048.451701] do_sys_open+0x73/0xa0 [ 4048.456282] __x64_sys_openat+0x24/0x30 [ 4048.457588] do_syscall_64+0xc1/0x440 [ 4048.471269] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4048.473043] RIP: 0033:0x7fa749552332 [ 4048.481460] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4048.501083] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4048.514507] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4048.516997] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4048.536778] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4048.543655] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4048.549598] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4052.449052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2135s! [ 4052.454340] Showing busy workqueues and worker pools: [ 4052.457416] workqueue events: flags=0x0 [ 4052.459899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4052.463789] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4052.472235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4052.475825] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4052.481903] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.486098] pending: kfree_rcu_monitor [ 4052.488422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4052.491867] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4052.497366] workqueue events_long: flags=0x0 [ 4052.500337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.503454] pending: lu_objects_flush [obdclass] [ 4052.506711] workqueue events_unbound: flags=0x2 [ 4052.509100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4052.512665] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4052.518099] workqueue events_power_efficient: flags=0x80 [ 4052.520899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.524131] pending: check_lifetime [ 4052.526857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.531173] pending: neigh_periodic_work [ 4052.533762] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.537379] pending: neigh_periodic_work [ 4052.540426] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.545768] pending: do_cache_clean [ 4052.548483] workqueue mm_percpu_wq: flags=0x8 [ 4052.551759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.555340] pending: vmstat_update [ 4052.556560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.559144] pending: vmstat_update [ 4052.560668] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4052.566302] pending: vmstat_update [ 4052.569472] workqueue writeback: flags=0x4a [ 4052.571560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4052.574263] pending: wb_workfn, wb_workfn [ 4052.576520] workqueue xprtiod: flags=0xa [ 4052.578891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4052.581878] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4052.585301] workqueue ipv6_addrconf: flags=0x40008 [ 4052.587283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4052.589982] pending: addrconf_verify_work [ 4052.591968] workqueue knbd0-recv: flags=0x1a [ 4052.593833] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4052.597128] in-flight: 651:recv_work [ 4052.598850] workqueue ptlrpc_pinger: flags=0x2 [ 4052.600695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4052.604345] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4052.607308] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2135s workers=4 idle: 37 31846 28897 [ 4052.610854] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4042s workers=2 idle: 653 [ 4056.058135] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4056.065789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4056.116598] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4056.125288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4056.138234] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4056.147010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4056.151542] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4056.172212] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4056.183502] RAX: 0000000000006084 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4056.190691] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4056.200309] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4056.212142] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4056.223009] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4056.233107] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4056.242516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4056.247751] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4056.254476] Call Trace: [ 4056.256481] [ 4056.258281] ? show_regs.cold.9+0x22/0x2f [ 4056.260476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4056.263418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4056.267292] ? watchdog+0x40/0x40 [ 4056.271299] ? hrtimer_interrupt+0x128/0x2c0 [ 4056.274979] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4056.277975] ? apic_timer_interrupt+0xf/0x20 [ 4056.281169] [ 4056.283055] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4056.286259] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4056.290131] do_raw_spin_lock+0x177/0x190 [ 4056.292464] _raw_spin_lock+0x21/0x30 [ 4056.294706] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4056.298063] process_one_work+0x2c8/0x700 [ 4056.300112] worker_thread+0x296/0x6e0 [ 4056.302398] ? rescuer_thread+0x570/0x570 [ 4056.305700] kthread+0x1d1/0x200 [ 4056.308054] ? set_kthread_struct+0x70/0x70 [ 4056.310455] ret_from_fork+0x1f/0x30 [ 4064.058118] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4064.061911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.085157] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.090715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4064.094927] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4064.097677] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4064.107250] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4064.111734] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4064.115592] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4064.119194] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4064.122722] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4064.126735] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4064.131134] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4064.135044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.137943] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4064.140481] Call Trace: [ 4064.141761] [ 4064.143494] ? show_regs.cold.9+0x22/0x2f [ 4064.145578] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.148466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.151087] ? watchdog+0x40/0x40 [ 4064.152901] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.155480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.158107] ? apic_timer_interrupt+0xf/0x20 [ 4064.160694] [ 4064.161719] ? native_safe_halt+0x2e/0x50 [ 4064.163683] kvm_wait+0xcd/0xe0 [ 4064.165414] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4064.167894] do_raw_spin_lock+0x177/0x190 [ 4064.169835] _raw_spin_lock+0x21/0x30 [ 4064.171635] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4064.174276] obd_mod_free+0x46/0x140 [ptlrpc] [ 4064.176857] mdc_commit_open+0x91/0xb0 [mdc] [ 4064.178639] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4064.181793] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4064.185147] after_reply+0xe2c/0x1780 [ptlrpc] [ 4064.187839] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4064.190986] ? __next_timer_interrupt+0x160/0x160 [ 4064.193600] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4064.196071] ? do_wait_intr+0xf0/0xf0 [ 4064.198423] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4064.201364] kthread+0x1d1/0x200 [ 4064.203121] ? set_kthread_struct+0x70/0x70 [ 4064.205120] ret_from_fork+0x1f/0x30 [ 4072.060477] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4072.063413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4072.111932] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4072.120039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4072.129042] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4072.135272] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4072.152132] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4072.157681] RAX: 0000000000001c8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4072.163885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4072.169160] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4072.186452] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4072.200157] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4072.207460] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4072.212086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4072.215914] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4072.219870] Call Trace: [ 4072.221714] [ 4072.222766] ? show_regs.cold.9+0x22/0x2f [ 4072.225186] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4072.228414] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4072.231179] ? watchdog+0x40/0x40 [ 4072.233004] ? hrtimer_interrupt+0x128/0x2c0 [ 4072.235617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4072.238449] ? apic_timer_interrupt+0xf/0x20 [ 4072.242444] [ 4072.248326] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4072.254233] do_raw_spin_lock+0x177/0x190 [ 4072.262285] _raw_spin_lock+0x21/0x30 [ 4072.268234] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4072.276870] ? do_raw_spin_unlock+0x75/0x190 [ 4072.284323] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4072.293865] ? do_wait_intr+0xf0/0xf0 [ 4072.296749] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4072.306181] mdc_close+0x53b/0xd70 [mdc] [ 4072.314235] lmv_close+0x18d/0x490 [lmv] [ 4072.316200] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4072.317638] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4072.326212] ll_md_close+0x274/0x800 [lustre] [ 4072.328956] ll_file_release+0x128/0xc50 [lustre] [ 4072.335064] __fput+0x10d/0x440 [ 4072.336523] ____fput+0x12/0x20 [ 4072.341195] task_work_run+0xc6/0x110 [ 4072.342365] exit_to_usermode_loop+0x1dd/0x1f0 [ 4072.348028] do_syscall_64+0x3d6/0x440 [ 4072.352986] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4072.359471] RIP: 0033:0x7fa749552cd8 [ 4072.361206] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4072.389125] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4072.397554] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4072.407222] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4072.414867] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4072.417036] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4072.429747] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4080.062249] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4080.077874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4080.123486] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4080.139534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4080.143720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4080.145770] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4080.154281] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4080.158556] RAX: 0000000000001b84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4080.166036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4080.170142] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4080.173750] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4080.177278] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4080.183554] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4080.187476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4080.189999] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4080.193752] Call Trace: [ 4080.195054] [ 4080.199191] ? show_regs.cold.9+0x22/0x2f [ 4080.201459] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4080.203109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4080.206016] ? watchdog+0x40/0x40 [ 4080.207462] ? hrtimer_interrupt+0x128/0x2c0 [ 4080.209640] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4080.211900] ? apic_timer_interrupt+0xf/0x20 [ 4080.213838] [ 4080.214701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4080.217913] do_raw_spin_lock+0x177/0x190 [ 4080.220478] _raw_spin_lock+0x21/0x30 [ 4080.222083] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4080.226629] ? do_raw_spin_unlock+0x75/0x190 [ 4080.228624] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4080.231636] ? do_wait_intr+0xf0/0xf0 [ 4080.233576] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4080.236271] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4080.242122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4080.245080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4080.247694] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4080.253213] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4080.256628] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4080.258798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4080.261823] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4080.265168] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4080.268254] lmv_intent_open+0x338/0x1150 [lmv] [ 4080.270651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4080.273100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4080.277055] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4080.279029] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4080.281107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4080.283694] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4080.285639] ? do_raw_spin_unlock+0x75/0x190 [ 4080.293219] ? _raw_spin_unlock+0x12/0x30 [ 4080.297093] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4080.300640] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4080.303539] ll_atomic_open+0x36a/0xd70 [lustre] [ 4080.306068] lookup_open+0x356/0xb40 [ 4080.307773] do_last+0x313/0xfc0 [ 4080.310011] ? nd_jump_root+0xe5/0x160 [ 4080.312053] ? path_init+0x437/0x520 [ 4080.314366] path_openat+0xf7/0x500 [ 4080.315991] ? do_raw_spin_unlock+0x75/0x190 [ 4080.318778] do_filp_open+0x99/0x140 [ 4080.320436] ? getname_flags+0x6e/0x330 [ 4080.322268] ? __check_object_size+0xff/0x256 [ 4080.325954] ? do_raw_spin_unlock+0x75/0x190 [ 4080.328562] ? _raw_spin_unlock+0x12/0x30 [ 4080.330238] do_sys_openat2+0x2b4/0x410 [ 4080.331991] do_sys_open+0x73/0xa0 [ 4080.335960] __x64_sys_openat+0x24/0x30 [ 4080.338030] do_syscall_64+0xc1/0x440 [ 4080.340103] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4080.343850] RIP: 0033:0x7fa749552332 [ 4080.346247] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4080.363803] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4080.367607] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4080.370453] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4080.375685] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4080.382527] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4080.389036] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4082.660918] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2165s! [ 4082.679870] Showing busy workqueues and worker pools: [ 4082.683897] workqueue events: flags=0x0 [ 4082.686771] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4082.694020] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4082.703225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4082.712003] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4082.719967] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.728980] pending: kfree_rcu_monitor [ 4082.732847] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4082.737461] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4082.745878] workqueue events_long: flags=0x0 [ 4082.750910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.755227] pending: lu_objects_flush [obdclass] [ 4082.759651] workqueue events_unbound: flags=0x2 [ 4082.763288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4082.768458] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4082.786491] workqueue events_power_efficient: flags=0x80 [ 4082.791895] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.796897] pending: check_lifetime [ 4082.807250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.819871] pending: neigh_periodic_work [ 4082.826254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.832809] pending: neigh_periodic_work [ 4082.835467] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.844508] pending: do_cache_clean [ 4082.846334] workqueue mm_percpu_wq: flags=0x8 [ 4082.852952] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.872290] pending: vmstat_update [ 4082.874924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.880702] pending: vmstat_update [ 4082.882759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4082.889034] pending: vmstat_update [ 4082.890388] workqueue writeback: flags=0x4a [ 4082.913559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4082.920669] pending: wb_workfn, wb_workfn [ 4082.935102] workqueue xprtiod: flags=0xa [ 4082.939695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4082.941747] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4082.949313] workqueue ipv6_addrconf: flags=0x40008 [ 4082.952869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4082.964092] pending: addrconf_verify_work [ 4082.969667] workqueue knbd0-recv: flags=0x1a [ 4082.974928] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4082.979540] in-flight: 651:recv_work [ 4082.981932] workqueue ptlrpc_pinger: flags=0x2 [ 4082.985278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4082.995594] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4083.008502] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2165s workers=4 idle: 37 31846 28897 [ 4083.017632] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4072s workers=2 idle: 653 [ 4084.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4084.059668] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4084.082963] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4084.087802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4084.091754] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4084.095195] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4084.097405] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4084.105601] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4084.109527] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4084.112408] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4084.115649] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4084.118707] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4084.121720] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4084.124324] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4084.127666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4084.130353] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4084.133669] Call Trace: [ 4084.135184] [ 4084.136041] ? show_regs.cold.9+0x22/0x2f [ 4084.137612] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4084.139094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4084.141011] ? watchdog+0x40/0x40 [ 4084.142304] ? hrtimer_interrupt+0x128/0x2c0 [ 4084.143763] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4084.145663] ? apic_timer_interrupt+0xf/0x20 [ 4084.147261] [ 4084.147917] ? native_safe_halt+0x2e/0x50 [ 4084.149596] kvm_wait+0xcd/0xe0 [ 4084.150891] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4084.153052] do_raw_spin_lock+0x177/0x190 [ 4084.154844] _raw_spin_lock+0x21/0x30 [ 4084.156255] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4084.158499] process_one_work+0x2c8/0x700 [ 4084.160081] worker_thread+0x296/0x6e0 [ 4084.161752] ? rescuer_thread+0x570/0x570 [ 4084.163324] kthread+0x1d1/0x200 [ 4084.164903] ? set_kthread_struct+0x70/0x70 [ 4084.166490] ret_from_fork+0x1f/0x30 [ 4092.059325] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4092.067760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4092.115415] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4092.127476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4092.136859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4092.154208] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4092.169879] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4092.175387] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4092.182787] RDX: 0000000000007b7a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4092.190404] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4092.196309] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4092.201792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4092.206389] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4092.213770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4092.217238] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4092.227086] Call Trace: [ 4092.229678] [ 4092.230809] ? show_regs.cold.9+0x22/0x2f [ 4092.234416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4092.237395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4092.239971] ? watchdog+0x40/0x40 [ 4092.242118] ? hrtimer_interrupt+0x128/0x2c0 [ 4092.244212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4092.247246] ? apic_timer_interrupt+0xf/0x20 [ 4092.249758] [ 4092.251123] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4092.254357] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4092.258055] do_raw_spin_lock+0x177/0x190 [ 4092.262913] _raw_spin_lock+0x21/0x30 [ 4092.265890] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4092.270941] obd_mod_free+0x46/0x140 [ptlrpc] [ 4092.274171] mdc_commit_open+0x91/0xb0 [mdc] [ 4092.276588] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4092.288106] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4092.294581] after_reply+0xe2c/0x1780 [ptlrpc] [ 4092.298109] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4092.302027] ? __next_timer_interrupt+0x160/0x160 [ 4092.304776] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4092.309792] ? do_wait_intr+0xf0/0xf0 [ 4092.314641] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4092.320329] kthread+0x1d1/0x200 [ 4092.322817] ? set_kthread_struct+0x70/0x70 [ 4092.325840] ret_from_fork+0x1f/0x30 [ 4100.068271] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4100.073294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4100.156966] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4100.175267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4100.182555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4100.193817] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4100.221039] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4100.228794] RAX: 00000000000078bb RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4100.235439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4100.248836] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4100.265646] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4100.280874] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4100.289773] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4100.302330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4100.306762] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4100.314066] Call Trace: [ 4100.316768] [ 4100.318383] ? show_regs.cold.9+0x22/0x2f [ 4100.319901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4100.323078] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4100.325675] ? watchdog+0x40/0x40 [ 4100.327757] ? hrtimer_interrupt+0x128/0x2c0 [ 4100.331826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4100.335210] ? apic_timer_interrupt+0xf/0x20 [ 4100.341055] [ 4100.341779] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4100.348782] do_raw_spin_lock+0x177/0x190 [ 4100.351014] _raw_spin_lock+0x21/0x30 [ 4100.354278] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4100.358155] ? do_raw_spin_unlock+0x75/0x190 [ 4100.360671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4100.363914] ? do_wait_intr+0xf0/0xf0 [ 4100.366923] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4100.372265] mdc_close+0x53b/0xd70 [mdc] [ 4100.374785] lmv_close+0x18d/0x490 [lmv] [ 4100.381565] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4100.391764] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4100.395549] ll_md_close+0x274/0x800 [lustre] [ 4100.403340] ll_file_release+0x128/0xc50 [lustre] [ 4100.412650] __fput+0x10d/0x440 [ 4100.421010] ____fput+0x12/0x20 [ 4100.426985] task_work_run+0xc6/0x110 [ 4100.431465] exit_to_usermode_loop+0x1dd/0x1f0 [ 4100.433024] do_syscall_64+0x3d6/0x440 [ 4100.446256] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4100.454024] RIP: 0033:0x7fa749552cd8 [ 4100.455946] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4100.479061] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4100.488751] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4100.499718] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4100.510158] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4100.520028] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4100.532790] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4108.065116] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4108.082214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4108.142359] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4108.161473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4108.176289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4108.186095] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4108.213762] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4108.226723] RAX: 000000000000080d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4108.235882] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4108.246639] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4108.263157] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4108.275373] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4108.283580] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4108.294893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4108.306685] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4108.317744] Call Trace: [ 4108.318971] [ 4108.321849] ? show_regs.cold.9+0x22/0x2f [ 4108.329847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4108.332491] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4108.342482] ? watchdog+0x40/0x40 [ 4108.345164] ? hrtimer_interrupt+0x128/0x2c0 [ 4108.349322] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4108.354434] ? apic_timer_interrupt+0xf/0x20 [ 4108.363803] [ 4108.366057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4108.368901] do_raw_spin_lock+0x177/0x190 [ 4108.375332] _raw_spin_lock+0x21/0x30 [ 4108.384096] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4108.401205] ? do_raw_spin_unlock+0x75/0x190 [ 4108.411381] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4108.422212] ? do_wait_intr+0xf0/0xf0 [ 4108.425204] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4108.433844] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4108.451841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4108.461534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4108.472158] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4108.477686] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4108.483691] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4108.491191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4108.495211] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4108.506183] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4108.510220] lmv_intent_open+0x338/0x1150 [lmv] [ 4108.512903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4108.516085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4108.519340] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4108.521805] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4108.523960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4108.526540] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4108.529186] ? do_raw_spin_unlock+0x75/0x190 [ 4108.531103] ? _raw_spin_unlock+0x12/0x30 [ 4108.533237] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4108.536698] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4108.540106] ll_atomic_open+0x36a/0xd70 [lustre] [ 4108.543618] lookup_open+0x356/0xb40 [ 4108.545493] do_last+0x313/0xfc0 [ 4108.547314] ? nd_jump_root+0xe5/0x160 [ 4108.549528] ? path_init+0x437/0x520 [ 4108.551525] path_openat+0xf7/0x500 [ 4108.554385] ? do_raw_spin_unlock+0x75/0x190 [ 4108.557174] do_filp_open+0x99/0x140 [ 4108.560629] ? getname_flags+0x6e/0x330 [ 4108.562536] ? __check_object_size+0xff/0x256 [ 4108.565110] ? do_raw_spin_unlock+0x75/0x190 [ 4108.568023] ? _raw_spin_unlock+0x12/0x30 [ 4108.570586] do_sys_openat2+0x2b4/0x410 [ 4108.572849] do_sys_open+0x73/0xa0 [ 4108.574694] __x64_sys_openat+0x24/0x30 [ 4108.580123] do_syscall_64+0xc1/0x440 [ 4108.582061] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4108.585011] RIP: 0033:0x7fa749552332 [ 4108.586155] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4108.603372] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4108.611397] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4108.618049] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4108.624893] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4108.633750] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4108.640201] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4112.057014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4112.061496] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4112.092528] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4112.103015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4112.123436] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4112.136629] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4112.148040] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4112.168214] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4112.173078] RAX: 0000000000001ba7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4112.187610] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4112.197707] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4112.204796] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4112.211879] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4112.219274] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4112.226871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4112.232052] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4112.238994] Call Trace: [ 4112.240352] [ 4112.241901] ? show_regs.cold.9+0x22/0x2f [ 4112.245912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4112.250990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4112.252746] ? watchdog+0x40/0x40 [ 4112.255270] ? hrtimer_interrupt+0x128/0x2c0 [ 4112.257080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4112.263389] ? apic_timer_interrupt+0xf/0x20 [ 4112.268468] [ 4112.271857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4112.277340] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4112.279494] do_raw_spin_lock+0x177/0x190 [ 4112.282080] _raw_spin_lock+0x21/0x30 [ 4112.285572] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4112.290374] process_one_work+0x2c8/0x700 [ 4112.293357] worker_thread+0x296/0x6e0 [ 4112.296717] ? rescuer_thread+0x570/0x570 [ 4112.300438] kthread+0x1d1/0x200 [ 4112.302076] ? set_kthread_struct+0x70/0x70 [ 4112.304362] ret_from_fork+0x1f/0x30 [ 4113.381247] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2196s! [ 4113.388953] Showing busy workqueues and worker pools: [ 4113.390785] workqueue events: flags=0x0 [ 4113.394732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4113.399344] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4113.405012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4113.408828] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4113.416831] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.424798] pending: kfree_rcu_monitor [ 4113.427825] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4113.434474] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4113.453532] workqueue events_long: flags=0x0 [ 4113.459188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.465946] pending: lu_objects_flush [obdclass] [ 4113.467835] workqueue events_unbound: flags=0x2 [ 4113.479391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4113.486769] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4113.491184] workqueue events_power_efficient: flags=0x80 [ 4113.493211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.496920] pending: check_lifetime [ 4113.498802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.501311] pending: neigh_periodic_work [ 4113.503169] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.511258] pending: neigh_periodic_work [ 4113.513472] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.530227] pending: do_cache_clean [ 4113.534510] workqueue mm_percpu_wq: flags=0x8 [ 4113.539883] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.542519] pending: vmstat_update [ 4113.545424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.550022] pending: vmstat_update [ 4113.552036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4113.556379] pending: vmstat_update [ 4113.558332] workqueue writeback: flags=0x4a [ 4113.561028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4113.564307] pending: wb_workfn, wb_workfn [ 4113.566402] workqueue xprtiod: flags=0xa [ 4113.570842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4113.576263] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4113.583014] workqueue ipv6_addrconf: flags=0x40008 [ 4113.585446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4113.590616] pending: addrconf_verify_work [ 4113.596180] workqueue knbd0-recv: flags=0x1a [ 4113.601481] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4113.603770] in-flight: 651:recv_work [ 4113.613094] workqueue ptlrpc_pinger: flags=0x2 [ 4113.618363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4113.622293] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4113.625816] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2196s workers=4 idle: 37 31846 28897 [ 4113.632014] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4103s workers=2 idle: 653 [ 4120.058950] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4120.062057] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4120.099468] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4120.105716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4120.110367] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4120.112388] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4120.124433] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4120.135375] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4120.138910] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4120.145433] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4120.151705] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4120.155408] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4120.159343] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4120.162974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4120.166618] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4120.170358] Call Trace: [ 4120.171568] [ 4120.172928] ? show_regs.cold.9+0x22/0x2f [ 4120.174699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4120.178251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4120.179691] ? watchdog+0x40/0x40 [ 4120.181690] ? hrtimer_interrupt+0x128/0x2c0 [ 4120.183527] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4120.185930] ? apic_timer_interrupt+0xf/0x20 [ 4120.187725] [ 4120.189193] ? native_safe_halt+0x2e/0x50 [ 4120.191351] kvm_wait+0xcd/0xe0 [ 4120.193139] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4120.195605] do_raw_spin_lock+0x177/0x190 [ 4120.197818] _raw_spin_lock+0x21/0x30 [ 4120.199357] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4120.202267] obd_mod_free+0x46/0x140 [ptlrpc] [ 4120.205436] mdc_commit_open+0x91/0xb0 [mdc] [ 4120.207670] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4120.210707] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4120.215429] after_reply+0xe2c/0x1780 [ptlrpc] [ 4120.218121] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4120.220256] ? __next_timer_interrupt+0x160/0x160 [ 4120.223346] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4120.225930] ? do_wait_intr+0xf0/0xf0 [ 4120.228280] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4120.231705] kthread+0x1d1/0x200 [ 4120.233361] ? set_kthread_struct+0x70/0x70 [ 4120.235665] ret_from_fork+0x1f/0x30 [ 4128.059763] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4128.063702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4128.085468] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4128.092049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4128.096257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4128.099202] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4128.108028] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4128.110565] RAX: 0000000000001c3f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4128.114292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4128.118292] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4128.121609] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4128.125054] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4128.127288] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4128.131406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4128.134061] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4128.138328] Call Trace: [ 4128.140166] [ 4128.142490] ? show_regs.cold.9+0x22/0x2f [ 4128.144462] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4128.146558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4128.148463] ? watchdog+0x40/0x40 [ 4128.149989] ? hrtimer_interrupt+0x128/0x2c0 [ 4128.151445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4128.153551] ? apic_timer_interrupt+0xf/0x20 [ 4128.155461] [ 4128.156691] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4128.159024] do_raw_spin_lock+0x177/0x190 [ 4128.161119] _raw_spin_lock+0x21/0x30 [ 4128.163004] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4128.166396] ? do_raw_spin_unlock+0x75/0x190 [ 4128.168773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4128.172201] ? do_wait_intr+0xf0/0xf0 [ 4128.174467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4128.177478] mdc_close+0x53b/0xd70 [mdc] [ 4128.179783] lmv_close+0x18d/0x490 [lmv] [ 4128.182224] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4128.184944] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4128.187433] ll_md_close+0x274/0x800 [lustre] [ 4128.189756] ll_file_release+0x128/0xc50 [lustre] [ 4128.192132] __fput+0x10d/0x440 [ 4128.193910] ____fput+0x12/0x20 [ 4128.195888] task_work_run+0xc6/0x110 [ 4128.197795] exit_to_usermode_loop+0x1dd/0x1f0 [ 4128.200180] do_syscall_64+0x3d6/0x440 [ 4128.202014] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4128.204428] RIP: 0033:0x7fa749552cd8 [ 4128.207204] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4128.219722] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4128.225148] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4128.227566] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4128.231291] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4128.236411] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4128.241680] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4133.738012] rcu: INFO: rcu_sched self-detected stall on CPU [ 4133.740975] rcu: 2-...!: (1206599 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 4133.751483] (t=2220049 jiffies g=258701 q=2246) [ 4133.756040] rcu: rcu_sched kthread starved for 2216844 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4133.765057] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4133.773350] rcu: RCU grace-period kthread stack dump: [ 4133.777632] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4133.789207] Call Trace: [ 4133.791145] __schedule+0x351/0xcb0 [ 4133.793034] schedule+0xc0/0x180 [ 4133.795069] schedule_timeout+0xb4/0x190 [ 4133.798276] ? do_raw_spin_unlock+0x75/0x190 [ 4133.805937] ? __next_timer_interrupt+0x160/0x160 [ 4133.816813] ? prepare_to_swait_event+0xf3/0x140 [ 4133.818430] rcu_gp_kthread+0x8dd/0x15b0 [ 4133.826888] ? force_qs_rnp+0x2c0/0x2c0 [ 4133.830401] kthread+0x1d1/0x200 [ 4133.836654] ? set_kthread_struct+0x70/0x70 [ 4133.841179] ret_from_fork+0x1f/0x30 [ 4133.842493] rcu: Stack dump where RCU GP kthread last ran: [ 4133.850250] Sending NMI from CPU 2 to CPUs 0: [ 4133.861612] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 6.633 msecs [ 4133.864407] NMI backtrace for cpu 0 [ 4133.864416] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4133.864419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4133.864420] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 4133.864422] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [ 4133.864424] RSP: 0018:ffffb893c1493500 EFLAGS: 00000282 [ 4133.864426] RAX: 00000000000019fb RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4133.864427] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4133.864429] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4133.864430] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4133.864432] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4133.864433] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4133.864434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4133.864435] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4133.864436] Call Trace: [ 4133.864437] [ 4133.864438] ? show_regs.cold.9+0x22/0x2f [ 4133.864439] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4133.864440] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4133.864441] ? nmi_handle+0xbd/0x210 [ 4133.864442] ? default_do_nmi+0x4b/0x1b0 [ 4133.864443] ? do_nmi+0x280/0x380 [ 4133.864444] ? end_repeat_nmi+0x16/0x1f [ 4133.864445] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 4133.864446] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 4133.864447] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 4133.864448] [ 4133.864449] do_raw_spin_lock+0x177/0x190 [ 4133.864450] _raw_spin_lock+0x21/0x30 [ 4133.864452] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4133.864453] ? do_raw_spin_unlock+0x75/0x190 [ 4133.864454] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4133.864455] ? do_wait_intr+0xf0/0xf0 [ 4133.864457] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4133.864458] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4133.864459] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4133.864461] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4133.864462] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4133.864463] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4133.864464] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4133.864465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4133.864466] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4133.864468] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4133.864469] lmv_intent_open+0x338/0x1150 [lmv] [ 4133.864470] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4133.864471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4133.864472] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4133.864473] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4133.864474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4133.864475] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4133.864477] ? do_raw_spin_unlock+0x75/0x190 [ 4133.864477] ? _raw_spin_unlock+0x12/0x30 [ 4133.864478] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4133.864480] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4133.864481] ll_atomic_open+0x36a/0xd70 [lustre] [ 4133.864482] lookup_open+0x356/0xb40 [ 4133.864483] do_last+0x313/0xfc0 [ 4133.864484] ? nd_jump_root+0xe5/0x160 [ 4133.864485] ? path_init+0x437/0x520 [ 4133.864486] path_openat+0xf7/0x500 [ 4133.864487] ? do_raw_spin_unlock+0x75/0x190 [ 4133.864488] do_filp_open+0x99/0x140 [ 4133.864489] ? getname_flags+0x6e/0x330 [ 4133.864490] ? __check_object_size+0xff/0x256 [ 4133.864491] ? do_raw_spin_unlock+0x75/0x190 [ 4133.864492] ? _raw_spin_unlock+0x12/0x30 [ 4133.864493] do_sys_openat2+0x2b4/0x410 [ 4133.864494] do_sys_open+0x73/0xa0 [ 4133.864495] __x64_sys_openat+0x24/0x30 [ 4133.864497] do_syscall_64+0xc1/0x440 [ 4133.864498] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4133.864499] RIP: 0033:0x7fa749552332 [ 4133.864500] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4133.864503] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4133.864506] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4133.864507] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4133.864508] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4133.864510] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4133.864511] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4133.864517] NMI backtrace for cpu 2 [ 4134.287159] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4134.292163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4134.296028] Call Trace: [ 4134.296871] [ 4134.297786] ? dump_stack+0xbb/0x10e [ 4134.299555] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4134.301250] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4134.314741] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4134.317954] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4134.323775] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4134.330577] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4134.336383] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4134.338738] ? update_process_times+0x6d/0xc0 [ 4134.350144] ? tick_sched_handle.isra.18+0x27/0x90 [ 4134.359491] ? tick_sched_timer+0x53/0xb0 [ 4134.362737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4134.375930] ? tick_sched_handle.isra.18+0x90/0x90 [ 4134.387359] ? hrtimer_interrupt+0x128/0x2c0 [ 4134.392804] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4134.402234] ? apic_timer_interrupt+0xf/0x20 [ 4134.407743] [ 4134.409114] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4134.416733] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4134.425219] ? do_raw_spin_lock+0x177/0x190 [ 4134.430561] ? _raw_spin_lock+0x21/0x30 [ 4134.435277] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4134.439746] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4134.447792] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4134.454558] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4134.463327] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4134.469916] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4134.473517] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4134.480761] ? __next_timer_interrupt+0x160/0x160 [ 4134.484368] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4134.489772] ? do_wait_intr+0xf0/0xf0 [ 4134.496491] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4134.504951] ? kthread+0x1d1/0x200 [ 4134.511420] ? set_kthread_struct+0x70/0x70 [ 4134.513163] ? ret_from_fork+0x1f/0x30 [ 4140.058515] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4140.069772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4140.102062] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4140.107854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4140.118931] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4140.122331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4140.125725] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4140.143828] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4140.148495] RAX: 0000000000006537 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4140.152660] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4140.155768] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4140.159882] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4140.164031] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4140.170350] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4140.176783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4140.178856] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4140.181969] Call Trace: [ 4140.182754] [ 4140.183601] ? show_regs.cold.9+0x22/0x2f [ 4140.186325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4140.190512] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4140.193426] ? watchdog+0x40/0x40 [ 4140.195845] ? hrtimer_interrupt+0x128/0x2c0 [ 4140.197894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4140.200339] ? apic_timer_interrupt+0xf/0x20 [ 4140.203873] [ 4140.204766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4140.206967] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4140.209714] do_raw_spin_lock+0x177/0x190 [ 4140.212954] _raw_spin_lock+0x21/0x30 [ 4140.214576] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4140.219519] process_one_work+0x2c8/0x700 [ 4140.221318] worker_thread+0x296/0x6e0 [ 4140.222874] ? rescuer_thread+0x570/0x570 [ 4140.227328] kthread+0x1d1/0x200 [ 4140.229641] ? set_kthread_struct+0x70/0x70 [ 4140.233424] ret_from_fork+0x1f/0x30 [ 4144.101081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2226s! [ 4144.104148] Showing busy workqueues and worker pools: [ 4144.110790] workqueue events: flags=0x0 [ 4144.116965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4144.122968] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4144.136754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4144.145387] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4144.157800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.165748] pending: kfree_rcu_monitor [ 4144.176852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4144.187630] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4144.198818] workqueue events_long: flags=0x0 [ 4144.207920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.216976] pending: lu_objects_flush [obdclass] [ 4144.233825] workqueue events_unbound: flags=0x2 [ 4144.242444] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4144.255623] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4144.286132] workqueue events_power_efficient: flags=0x80 [ 4144.287969] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.311538] pending: check_lifetime [ 4144.319034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.329262] pending: neigh_periodic_work [ 4144.330511] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.342576] pending: neigh_periodic_work [ 4144.351626] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.353974] pending: do_cache_clean [ 4144.361873] workqueue mm_percpu_wq: flags=0x8 [ 4144.363601] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.372741] pending: vmstat_update [ 4144.379946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.384862] pending: vmstat_update [ 4144.386902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4144.390196] pending: vmstat_update [ 4144.396198] workqueue writeback: flags=0x4a [ 4144.404480] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4144.412462] pending: wb_workfn, wb_workfn [ 4144.416188] workqueue xprtiod: flags=0xa [ 4144.429990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4144.446912] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4144.460187] workqueue ipv6_addrconf: flags=0x40008 [ 4144.462812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4144.469992] pending: addrconf_verify_work [ 4144.477809] workqueue knbd0-recv: flags=0x1a [ 4144.484049] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4144.490842] in-flight: 651:recv_work [ 4144.496445] workqueue ptlrpc_pinger: flags=0x2 [ 4144.499142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4144.505033] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4144.509720] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2227s workers=4 idle: 37 31846 28897 [ 4144.533917] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4134s workers=2 idle: 653 [ 4156.059115] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4156.062378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4156.085275] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4156.089041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4156.093197] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4156.094979] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4156.102838] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4156.105661] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4156.108670] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4156.113942] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4156.116577] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4156.118715] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4156.124374] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4156.129504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4156.133632] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4156.137552] Call Trace: [ 4156.138519] [ 4156.140286] ? show_regs.cold.9+0x22/0x2f [ 4156.141972] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4156.144521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4156.150699] ? watchdog+0x40/0x40 [ 4156.155233] ? hrtimer_interrupt+0x128/0x2c0 [ 4156.160959] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4156.168282] ? apic_timer_interrupt+0xf/0x20 [ 4156.176292] [ 4156.177661] ? native_safe_halt+0x2e/0x50 [ 4156.183103] kvm_wait+0xcd/0xe0 [ 4156.187628] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4156.195704] do_raw_spin_lock+0x177/0x190 [ 4156.205648] _raw_spin_lock+0x21/0x30 [ 4156.208254] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4156.214404] ? do_raw_spin_unlock+0x75/0x190 [ 4156.219683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4156.224578] ? do_wait_intr+0xf0/0xf0 [ 4156.228059] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4156.230516] mdc_close+0x53b/0xd70 [mdc] [ 4156.232242] lmv_close+0x18d/0x490 [lmv] [ 4156.237480] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4156.242636] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4156.247172] ll_md_close+0x274/0x800 [lustre] [ 4156.254602] ll_file_release+0x128/0xc50 [lustre] [ 4156.256523] __fput+0x10d/0x440 [ 4156.260090] ____fput+0x12/0x20 [ 4156.265631] task_work_run+0xc6/0x110 [ 4156.268555] exit_to_usermode_loop+0x1dd/0x1f0 [ 4156.271216] do_syscall_64+0x3d6/0x440 [ 4156.272482] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4156.274541] RIP: 0033:0x7fa749552cd8 [ 4156.275688] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4156.289010] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4156.291989] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4156.295047] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4156.298439] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4156.305382] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4156.308694] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4160.058045] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4160.067177] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.092691] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.097494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.107669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4160.116328] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4160.132678] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4160.136246] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4160.143179] RDX: 0000000000005580 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4160.152667] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4160.159714] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4160.169083] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4160.173254] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4160.179667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.181685] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4160.190626] Call Trace: [ 4160.191921] [ 4160.199077] ? show_regs.cold.9+0x22/0x2f [ 4160.203145] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.210184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.217955] ? watchdog+0x40/0x40 [ 4160.222622] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.225012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.227595] ? apic_timer_interrupt+0xf/0x20 [ 4160.229799] [ 4160.231002] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4160.233912] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4160.237912] do_raw_spin_lock+0x177/0x190 [ 4160.240949] _raw_spin_lock+0x21/0x30 [ 4160.243142] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4160.246964] obd_mod_free+0x46/0x140 [ptlrpc] [ 4160.249742] mdc_commit_open+0x91/0xb0 [mdc] [ 4160.252944] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4160.257187] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4160.260780] after_reply+0xe2c/0x1780 [ptlrpc] [ 4160.269944] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4160.275525] ? __next_timer_interrupt+0x160/0x160 [ 4160.278978] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4160.284932] ? do_wait_intr+0xf0/0xf0 [ 4160.288014] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4160.299474] kthread+0x1d1/0x200 [ 4160.302890] ? set_kthread_struct+0x70/0x70 [ 4160.307936] ret_from_fork+0x1f/0x30 [ 4168.057014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4168.063326] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4168.104139] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4168.109618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4168.117552] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4168.121607] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4168.129842] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4168.140047] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4168.143481] RAX: 00000000000055f6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4168.148030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4168.158737] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4168.163902] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4168.169045] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4168.180603] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4168.184615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4168.187443] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4168.191620] Call Trace: [ 4168.193297] [ 4168.194514] ? show_regs.cold.9+0x22/0x2f [ 4168.197490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4168.201487] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4168.203423] ? watchdog+0x40/0x40 [ 4168.207638] ? hrtimer_interrupt+0x128/0x2c0 [ 4168.213502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4168.216183] ? apic_timer_interrupt+0xf/0x20 [ 4168.219224] [ 4168.220462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4168.223600] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4168.227419] do_raw_spin_lock+0x177/0x190 [ 4168.230004] _raw_spin_lock+0x21/0x30 [ 4168.234451] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4168.236698] process_one_work+0x2c8/0x700 [ 4168.238293] worker_thread+0x296/0x6e0 [ 4168.239432] ? rescuer_thread+0x570/0x570 [ 4168.240709] kthread+0x1d1/0x200 [ 4168.242103] ? set_kthread_struct+0x70/0x70 [ 4168.243626] ret_from_fork+0x1f/0x30 [ 4172.058207] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4172.064568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4172.110628] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4172.127421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4172.140183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4172.148964] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4172.166704] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4172.171439] RAX: 0000000000004d8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4172.175477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4172.179496] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4172.183843] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4172.192474] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4172.201738] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4172.204632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4172.216848] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4172.226215] Call Trace: [ 4172.228288] [ 4172.230283] ? show_regs.cold.9+0x22/0x2f [ 4172.234151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4172.239151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4172.243387] ? watchdog+0x40/0x40 [ 4172.246378] ? hrtimer_interrupt+0x128/0x2c0 [ 4172.254189] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4172.260216] ? apic_timer_interrupt+0xf/0x20 [ 4172.271783] [ 4172.272715] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4172.278698] do_raw_spin_lock+0x177/0x190 [ 4172.282471] _raw_spin_lock+0x21/0x30 [ 4172.287150] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4172.289305] ? do_raw_spin_unlock+0x75/0x190 [ 4172.298396] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4172.305446] ? do_wait_intr+0xf0/0xf0 [ 4172.306821] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4172.316315] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4172.324823] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4172.334863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4172.341254] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4172.350866] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4172.358070] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4172.363362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4172.370224] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4172.380986] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4172.389926] lmv_intent_open+0x338/0x1150 [lmv] [ 4172.393943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4172.401389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4172.411836] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4172.416897] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4172.425331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4172.437103] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4172.438829] ? do_raw_spin_unlock+0x75/0x190 [ 4172.444898] ? _raw_spin_unlock+0x12/0x30 [ 4172.448066] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4172.454157] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4172.460675] ll_atomic_open+0x36a/0xd70 [lustre] [ 4172.464369] lookup_open+0x356/0xb40 [ 4172.470383] do_last+0x313/0xfc0 [ 4172.476110] ? nd_jump_root+0xe5/0x160 [ 4172.483927] ? path_init+0x437/0x520 [ 4172.488130] path_openat+0xf7/0x500 [ 4172.493832] ? do_raw_spin_unlock+0x75/0x190 [ 4172.499709] do_filp_open+0x99/0x140 [ 4172.504714] ? getname_flags+0x6e/0x330 [ 4172.513150] ? __check_object_size+0xff/0x256 [ 4172.520901] ? do_raw_spin_unlock+0x75/0x190 [ 4172.534043] ? _raw_spin_unlock+0x12/0x30 [ 4172.537197] do_sys_openat2+0x2b4/0x410 [ 4172.543432] do_sys_open+0x73/0xa0 [ 4172.547820] __x64_sys_openat+0x24/0x30 [ 4172.549291] do_syscall_64+0xc1/0x440 [ 4172.550608] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4172.564351] RIP: 0033:0x7fa749552332 [ 4172.569864] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4172.594888] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4172.607397] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4172.615087] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4172.625699] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4172.636283] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4172.645049] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4174.823506] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2257s! [ 4174.842993] Showing busy workqueues and worker pools: [ 4174.849797] workqueue events: flags=0x0 [ 4174.851313] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4174.860983] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4174.875809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4174.888339] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4174.913397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4174.918623] pending: kfree_rcu_monitor [ 4174.925502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4174.934102] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4174.944320] workqueue events_long: flags=0x0 [ 4174.958233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4174.960444] pending: lu_objects_flush [obdclass] [ 4174.968556] workqueue events_unbound: flags=0x2 [ 4174.977280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4174.984100] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4174.999334] workqueue events_power_efficient: flags=0x80 [ 4175.005887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4175.022326] pending: check_lifetime [ 4175.032176] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4175.048124] pending: neigh_periodic_work [ 4175.051332] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4175.067234] pending: neigh_periodic_work [ 4175.068758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4175.083377] pending: do_cache_clean [ 4175.085863] workqueue mm_percpu_wq: flags=0x8 [ 4175.090307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4175.106526] pending: vmstat_update [ 4175.114513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4175.127093] pending: vmstat_update [ 4175.131423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4175.142068] pending: vmstat_update [ 4175.149562] workqueue writeback: flags=0x4a [ 4175.157334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4175.164538] pending: wb_workfn, wb_workfn [ 4175.170486] workqueue xprtiod: flags=0xa [ 4175.171531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4175.173223] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4175.189487] workqueue ipv6_addrconf: flags=0x40008 [ 4175.197144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4175.209539] pending: addrconf_verify_work [ 4175.212144] workqueue knbd0-recv: flags=0x1a [ 4175.223055] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4175.231493] in-flight: 651:recv_work [ 4175.235981] workqueue ptlrpc_pinger: flags=0x2 [ 4175.244263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4175.254501] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4175.264395] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2258s workers=4 idle: 37 31846 28897 [ 4175.278134] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4165s workers=2 idle: 653 [ 4184.061767] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4184.071598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4184.113820] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4184.128582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4184.138831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4184.147726] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4184.162722] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4184.170418] RAX: 0000000000006689 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4184.177891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4184.185450] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4184.191924] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4184.199871] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4184.202536] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4184.214501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4184.222545] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4184.230032] Call Trace: [ 4184.232164] [ 4184.234822] ? show_regs.cold.9+0x22/0x2f [ 4184.238512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4184.243248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4184.248296] ? watchdog+0x40/0x40 [ 4184.252088] ? hrtimer_interrupt+0x128/0x2c0 [ 4184.256600] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4184.263259] ? apic_timer_interrupt+0xf/0x20 [ 4184.267633] [ 4184.271504] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4184.276572] do_raw_spin_lock+0x177/0x190 [ 4184.280361] _raw_spin_lock+0x21/0x30 [ 4184.284083] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4184.293660] ? do_raw_spin_unlock+0x75/0x190 [ 4184.296091] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4184.302172] ? do_wait_intr+0xf0/0xf0 [ 4184.305636] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4184.310563] mdc_close+0x53b/0xd70 [mdc] [ 4184.317643] lmv_close+0x18d/0x490 [lmv] [ 4184.319765] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4184.326215] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4184.329220] ll_md_close+0x274/0x800 [lustre] [ 4184.335291] ll_file_release+0x128/0xc50 [lustre] [ 4184.341158] __fput+0x10d/0x440 [ 4184.342712] ____fput+0x12/0x20 [ 4184.344168] task_work_run+0xc6/0x110 [ 4184.346709] exit_to_usermode_loop+0x1dd/0x1f0 [ 4184.352097] do_syscall_64+0x3d6/0x440 [ 4184.355165] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4184.361344] RIP: 0033:0x7fa749552cd8 [ 4184.362866] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4184.398710] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4184.410600] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4184.419581] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4184.424731] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4184.427575] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4184.431491] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4188.058037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4188.062190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.085884] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.091185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.096672] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4188.099252] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4188.109105] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4188.113528] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4188.118772] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4188.124530] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4188.128907] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4188.132616] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4188.137130] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4188.142257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.145184] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4188.149902] Call Trace: [ 4188.151514] [ 4188.152617] ? show_regs.cold.9+0x22/0x2f [ 4188.154975] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.156721] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.163954] ? watchdog+0x40/0x40 [ 4188.169641] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.174257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.178310] ? apic_timer_interrupt+0xf/0x20 [ 4188.187730] [ 4188.194002] ? native_safe_halt+0x2e/0x50 [ 4188.199222] kvm_wait+0xcd/0xe0 [ 4188.205641] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4188.215956] do_raw_spin_lock+0x177/0x190 [ 4188.221516] _raw_spin_lock+0x21/0x30 [ 4188.226221] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4188.236960] obd_mod_free+0x46/0x140 [ptlrpc] [ 4188.250560] mdc_commit_open+0x91/0xb0 [mdc] [ 4188.262784] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4188.282224] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4188.297230] after_reply+0xe2c/0x1780 [ptlrpc] [ 4188.311958] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4188.319842] ? __next_timer_interrupt+0x160/0x160 [ 4188.324261] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4188.335605] ? do_wait_intr+0xf0/0xf0 [ 4188.346866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4188.357989] kthread+0x1d1/0x200 [ 4188.359155] ? set_kthread_struct+0x70/0x70 [ 4188.370535] ret_from_fork+0x1f/0x30 [ 4196.057371] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4196.064171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4196.085116] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4196.090866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4196.095922] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4196.098891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4196.101185] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4196.109926] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4196.113394] RAX: 0000000000007f33 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4196.116598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4196.119168] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4196.121976] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4196.125008] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4196.128473] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4196.132024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4196.134946] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4196.137995] Call Trace: [ 4196.139323] [ 4196.140395] ? show_regs.cold.9+0x22/0x2f [ 4196.142243] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4196.144012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4196.146338] ? watchdog+0x40/0x40 [ 4196.147571] ? hrtimer_interrupt+0x128/0x2c0 [ 4196.149848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4196.151686] ? apic_timer_interrupt+0xf/0x20 [ 4196.153046] [ 4196.153570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4196.159586] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4196.161925] do_raw_spin_lock+0x177/0x190 [ 4196.163511] _raw_spin_lock+0x21/0x30 [ 4196.164962] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4196.167128] process_one_work+0x2c8/0x700 [ 4196.169254] worker_thread+0x296/0x6e0 [ 4196.170540] ? rescuer_thread+0x570/0x570 [ 4196.172474] kthread+0x1d1/0x200 [ 4196.173715] ? set_kthread_struct+0x70/0x70 [ 4196.175516] ret_from_fork+0x1f/0x30 [ 4200.055430] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4200.061257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4200.109741] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4200.121537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4200.130228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4200.135855] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4200.154338] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4200.162722] RAX: 0000000000004609 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4200.170794] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4200.177883] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4200.186740] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4200.193758] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4200.204921] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4200.213841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4200.221164] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4200.229382] Call Trace: [ 4200.232465] [ 4200.234945] ? show_regs.cold.9+0x22/0x2f [ 4200.240783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4200.249422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4200.253651] ? watchdog+0x40/0x40 [ 4200.257665] ? hrtimer_interrupt+0x128/0x2c0 [ 4200.262895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4200.268762] ? apic_timer_interrupt+0xf/0x20 [ 4200.274568] [ 4200.276735] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4200.283239] do_raw_spin_lock+0x177/0x190 [ 4200.288839] _raw_spin_lock+0x21/0x30 [ 4200.293283] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4200.300287] ? do_raw_spin_unlock+0x75/0x190 [ 4200.307222] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4200.310243] ? do_wait_intr+0xf0/0xf0 [ 4200.312589] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4200.316593] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4200.321838] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4200.334750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4200.342270] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4200.360078] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4200.361777] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4200.375604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4200.379558] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4200.384173] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4200.387649] lmv_intent_open+0x338/0x1150 [lmv] [ 4200.390009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4200.394044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4200.398308] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4200.400137] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4200.403586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4200.407189] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4200.410478] ? do_raw_spin_unlock+0x75/0x190 [ 4200.414364] ? _raw_spin_unlock+0x12/0x30 [ 4200.418732] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4200.422479] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4200.425270] ll_atomic_open+0x36a/0xd70 [lustre] [ 4200.432673] lookup_open+0x356/0xb40 [ 4200.434637] do_last+0x313/0xfc0 [ 4200.437440] ? nd_jump_root+0xe5/0x160 [ 4200.438884] ? path_init+0x437/0x520 [ 4200.441273] path_openat+0xf7/0x500 [ 4200.443028] ? do_raw_spin_unlock+0x75/0x190 [ 4200.449332] do_filp_open+0x99/0x140 [ 4200.451750] ? getname_flags+0x6e/0x330 [ 4200.453911] ? __check_object_size+0xff/0x256 [ 4200.456547] ? do_raw_spin_unlock+0x75/0x190 [ 4200.459053] ? _raw_spin_unlock+0x12/0x30 [ 4200.461376] do_sys_openat2+0x2b4/0x410 [ 4200.463461] do_sys_open+0x73/0xa0 [ 4200.465835] __x64_sys_openat+0x24/0x30 [ 4200.467661] do_syscall_64+0xc1/0x440 [ 4200.470408] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4200.472558] RIP: 0033:0x7fa749552332 [ 4200.476285] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4200.487860] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4200.491363] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4200.494705] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4200.498954] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4200.502709] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4200.506295] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4205.538039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2288s! [ 4205.544893] Showing busy workqueues and worker pools: [ 4205.547503] workqueue events: flags=0x0 [ 4205.550532] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4205.555289] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4205.566859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4205.572913] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4205.576759] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.579709] pending: kfree_rcu_monitor [ 4205.581804] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4205.584510] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4205.590853] workqueue events_long: flags=0x0 [ 4205.592010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.593805] pending: lu_objects_flush [obdclass] [ 4205.596550] workqueue events_unbound: flags=0x2 [ 4205.598621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4205.601789] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4205.608425] workqueue events_power_efficient: flags=0x80 [ 4205.612503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.642305] pending: check_lifetime [ 4205.645104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.648386] pending: neigh_periodic_work [ 4205.650608] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.653475] pending: neigh_periodic_work [ 4205.655476] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.663097] pending: do_cache_clean [ 4205.664757] workqueue mm_percpu_wq: flags=0x8 [ 4205.666434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.669447] pending: vmstat_update [ 4205.670933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.673423] pending: vmstat_update [ 4205.674843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4205.676970] pending: vmstat_update [ 4205.678201] workqueue writeback: flags=0x4a [ 4205.679735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4205.682318] pending: wb_workfn, wb_workfn [ 4205.683955] workqueue xprtiod: flags=0xa [ 4205.685324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4205.688347] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4205.691720] workqueue ipv6_addrconf: flags=0x40008 [ 4205.693461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4205.695844] pending: addrconf_verify_work [ 4205.697547] workqueue knbd0-recv: flags=0x1a [ 4205.699039] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4205.701597] in-flight: 651:recv_work [ 4205.703052] workqueue ptlrpc_pinger: flags=0x2 [ 4205.704757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4205.707018] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4205.709175] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2288s workers=4 idle: 37 31846 28897 [ 4205.718721] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4195s workers=2 idle: 653 [ 4212.065646] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4212.071438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4212.114811] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4212.126063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4212.130665] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4212.134006] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4212.146967] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4212.151237] RAX: 0000000000002981 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4212.157103] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4212.162238] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4212.167067] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4212.170904] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4212.175122] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4212.179115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4212.182727] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4212.185750] Call Trace: [ 4212.186764] [ 4212.188076] ? show_regs.cold.9+0x22/0x2f [ 4212.190038] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4212.192159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4212.194527] ? watchdog+0x40/0x40 [ 4212.196447] ? hrtimer_interrupt+0x128/0x2c0 [ 4212.198836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4212.201519] ? apic_timer_interrupt+0xf/0x20 [ 4212.203878] [ 4212.205862] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4212.208895] do_raw_spin_lock+0x177/0x190 [ 4212.211484] _raw_spin_lock+0x21/0x30 [ 4212.213498] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4212.217704] ? do_raw_spin_unlock+0x75/0x190 [ 4212.219476] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4212.222138] ? do_wait_intr+0xf0/0xf0 [ 4212.223834] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4212.227433] mdc_close+0x53b/0xd70 [mdc] [ 4212.229505] lmv_close+0x18d/0x490 [lmv] [ 4212.231637] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4212.235644] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4212.240099] ll_md_close+0x274/0x800 [lustre] [ 4212.242414] ll_file_release+0x128/0xc50 [lustre] [ 4212.244208] __fput+0x10d/0x440 [ 4212.246043] ____fput+0x12/0x20 [ 4212.248137] task_work_run+0xc6/0x110 [ 4212.250388] exit_to_usermode_loop+0x1dd/0x1f0 [ 4212.253057] do_syscall_64+0x3d6/0x440 [ 4212.255753] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4212.258452] RIP: 0033:0x7fa749552cd8 [ 4212.260670] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4212.270620] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4212.275059] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4212.279812] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4212.285839] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4212.291965] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4212.295734] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4216.061023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4216.071452] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.114376] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.124972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.133229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4216.136915] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4216.156799] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4216.158454] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4216.170473] RDX: 0000000000003cab RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4216.177930] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4216.185520] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4216.193572] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4216.199411] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4216.207147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.213018] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4216.219508] Call Trace: [ 4216.222481] [ 4216.225169] ? show_regs.cold.9+0x22/0x2f [ 4216.228948] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.231456] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.234748] ? watchdog+0x40/0x40 [ 4216.237467] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.242371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.246528] ? apic_timer_interrupt+0xf/0x20 [ 4216.250362] [ 4216.255229] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4216.260044] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4216.265789] do_raw_spin_lock+0x177/0x190 [ 4216.271172] _raw_spin_lock+0x21/0x30 [ 4216.274269] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4216.277472] obd_mod_free+0x46/0x140 [ptlrpc] [ 4216.282849] mdc_commit_open+0x91/0xb0 [mdc] [ 4216.286807] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4216.297036] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4216.303278] after_reply+0xe2c/0x1780 [ptlrpc] [ 4216.308416] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4216.313922] ? __next_timer_interrupt+0x160/0x160 [ 4216.319038] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4216.325285] ? do_wait_intr+0xf0/0xf0 [ 4216.326467] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4216.338896] kthread+0x1d1/0x200 [ 4216.342053] ? set_kthread_struct+0x70/0x70 [ 4216.344748] ret_from_fork+0x1f/0x30 [ 4224.058042] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4224.072484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4224.184740] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4224.199247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4224.208563] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4224.215318] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4224.219641] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4224.234476] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4224.238562] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4224.244548] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4224.249445] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4224.256211] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4224.261355] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4224.270586] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4224.278323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4224.281382] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4224.285400] Call Trace: [ 4224.287019] [ 4224.287614] ? show_regs.cold.9+0x22/0x2f [ 4224.288895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4224.298912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4224.307796] ? watchdog+0x40/0x40 [ 4224.314742] ? hrtimer_interrupt+0x128/0x2c0 [ 4224.324282] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4224.328904] ? apic_timer_interrupt+0xf/0x20 [ 4224.333205] [ 4224.337563] ? native_safe_halt+0x2e/0x50 [ 4224.340632] kvm_wait+0xcd/0xe0 [ 4224.341778] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4224.351064] do_raw_spin_lock+0x177/0x190 [ 4224.354807] _raw_spin_lock+0x21/0x30 [ 4224.356740] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4224.368577] process_one_work+0x2c8/0x700 [ 4224.372464] worker_thread+0x296/0x6e0 [ 4224.378144] ? rescuer_thread+0x570/0x570 [ 4224.381202] kthread+0x1d1/0x200 [ 4224.387474] ? set_kthread_struct+0x70/0x70 [ 4224.394149] ret_from_fork+0x1f/0x30 [ 4232.056677] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4232.060097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4232.131758] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4232.142009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4232.156332] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4232.164651] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4232.187667] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4232.200078] RAX: 0000000000002614 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4232.204271] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4232.224111] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4232.233185] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4232.244913] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4232.248461] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4232.256464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4232.265228] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4232.271675] Call Trace: [ 4232.273928] [ 4232.275193] ? show_regs.cold.9+0x22/0x2f [ 4232.276999] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4232.283897] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4232.291609] ? watchdog+0x40/0x40 [ 4232.294653] ? hrtimer_interrupt+0x128/0x2c0 [ 4232.303261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4232.306063] ? apic_timer_interrupt+0xf/0x20 [ 4232.311945] [ 4232.313380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4232.316700] do_raw_spin_lock+0x177/0x190 [ 4232.322624] _raw_spin_lock+0x21/0x30 [ 4232.324322] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4232.331796] ? do_raw_spin_unlock+0x75/0x190 [ 4232.339029] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4232.342297] ? do_wait_intr+0xf0/0xf0 [ 4232.345955] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4232.352372] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4232.356939] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4232.360841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4232.365066] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4232.368512] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4232.370896] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4232.372993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4232.375790] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4232.378383] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4232.380840] lmv_intent_open+0x338/0x1150 [lmv] [ 4232.383040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4232.385099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4232.387414] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4232.390044] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4232.392030] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4232.395237] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4232.397084] ? do_raw_spin_unlock+0x75/0x190 [ 4232.398736] ? _raw_spin_unlock+0x12/0x30 [ 4232.400264] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4232.402393] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4232.404440] ll_atomic_open+0x36a/0xd70 [lustre] [ 4232.408283] lookup_open+0x356/0xb40 [ 4232.415878] do_last+0x313/0xfc0 [ 4232.417866] ? nd_jump_root+0xe5/0x160 [ 4232.420814] ? path_init+0x437/0x520 [ 4232.424312] path_openat+0xf7/0x500 [ 4232.426261] ? do_raw_spin_unlock+0x75/0x190 [ 4232.430621] do_filp_open+0x99/0x140 [ 4232.432770] ? getname_flags+0x6e/0x330 [ 4232.435976] ? __check_object_size+0xff/0x256 [ 4232.439099] ? do_raw_spin_unlock+0x75/0x190 [ 4232.442501] ? _raw_spin_unlock+0x12/0x30 [ 4232.447921] do_sys_openat2+0x2b4/0x410 [ 4232.455966] do_sys_open+0x73/0xa0 [ 4232.460800] __x64_sys_openat+0x24/0x30 [ 4232.465535] do_syscall_64+0xc1/0x440 [ 4232.469440] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4232.476791] RIP: 0033:0x7fa749552332 [ 4232.482050] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4232.499641] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4232.503774] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4232.507736] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4232.511506] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4232.515553] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4232.520190] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4235.745042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2318s! [ 4235.747208] Showing busy workqueues and worker pools: [ 4235.749584] workqueue events: flags=0x0 [ 4235.762307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4235.773734] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4235.790332] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4235.795210] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4235.805268] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.813231] pending: kfree_rcu_monitor [ 4235.816202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4235.820129] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4235.831628] workqueue events_long: flags=0x0 [ 4235.833474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.839691] pending: lu_objects_flush [obdclass] [ 4235.843069] workqueue events_unbound: flags=0x2 [ 4235.846016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4235.850556] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4235.857229] workqueue events_power_efficient: flags=0x80 [ 4235.860312] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.870215] pending: check_lifetime [ 4235.873963] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.877837] pending: neigh_periodic_work [ 4235.880735] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.885455] pending: neigh_periodic_work [ 4235.887265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.893413] pending: do_cache_clean [ 4235.896344] workqueue mm_percpu_wq: flags=0x8 [ 4235.904220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.911319] pending: vmstat_update [ 4235.912449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.915413] pending: vmstat_update [ 4235.923241] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4235.927390] pending: vmstat_update [ 4235.928735] workqueue writeback: flags=0x4a [ 4235.930468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4235.933285] pending: wb_workfn, wb_workfn [ 4235.934561] workqueue xprtiod: flags=0xa [ 4235.936285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4235.938842] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4235.947112] workqueue ipv6_addrconf: flags=0x40008 [ 4235.950434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4235.956779] pending: addrconf_verify_work [ 4235.961091] workqueue knbd0-recv: flags=0x1a [ 4235.963649] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4235.970066] in-flight: 651:recv_work [ 4235.971301] workqueue ptlrpc_pinger: flags=0x2 [ 4235.975188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4235.981445] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4235.988701] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2318s workers=4 idle: 37 31846 28897 [ 4235.994853] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4225s workers=2 idle: 653 [ 4240.059031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4240.067217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4240.104211] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4240.117738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4240.126163] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4240.131886] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4240.151975] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4240.160324] RAX: 0000000000006147 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4240.163725] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4240.166498] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4240.169956] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4240.172864] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4240.177070] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4240.182269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4240.187306] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4240.191010] Call Trace: [ 4240.192150] [ 4240.194277] ? show_regs.cold.9+0x22/0x2f [ 4240.199962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4240.203167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4240.205280] ? watchdog+0x40/0x40 [ 4240.210228] ? hrtimer_interrupt+0x128/0x2c0 [ 4240.211994] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4240.213315] ? apic_timer_interrupt+0xf/0x20 [ 4240.214894] [ 4240.215673] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4240.217237] do_raw_spin_lock+0x177/0x190 [ 4240.218928] _raw_spin_lock+0x21/0x30 [ 4240.223000] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4240.226522] ? do_raw_spin_unlock+0x75/0x190 [ 4240.228237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4240.230547] ? do_wait_intr+0xf0/0xf0 [ 4240.231968] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4240.234313] mdc_close+0x53b/0xd70 [mdc] [ 4240.236094] lmv_close+0x18d/0x490 [lmv] [ 4240.237960] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4240.240562] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4240.242634] ll_md_close+0x274/0x800 [lustre] [ 4240.246739] ll_file_release+0x128/0xc50 [lustre] [ 4240.252201] __fput+0x10d/0x440 [ 4240.253979] ____fput+0x12/0x20 [ 4240.255246] task_work_run+0xc6/0x110 [ 4240.257828] exit_to_usermode_loop+0x1dd/0x1f0 [ 4240.259570] do_syscall_64+0x3d6/0x440 [ 4240.261215] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4240.265096] RIP: 0033:0x7fa749552cd8 [ 4240.266804] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4240.277239] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4240.281098] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4240.284520] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4240.287726] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4240.291611] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4240.298868] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4244.058088] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4244.061501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.098425] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.103795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4244.107438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4244.110287] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4244.128287] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4244.134963] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4244.141358] RDX: 0000000000007acc RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4244.149160] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4244.151681] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4244.154244] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4244.156578] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4244.166263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.171851] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4244.178659] Call Trace: [ 4244.183081] [ 4244.186004] ? show_regs.cold.9+0x22/0x2f [ 4244.187916] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4244.191730] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4244.196711] ? watchdog+0x40/0x40 [ 4244.199759] ? hrtimer_interrupt+0x128/0x2c0 [ 4244.203716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4244.209156] ? apic_timer_interrupt+0xf/0x20 [ 4244.212718] [ 4244.216054] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4244.219967] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4244.222778] do_raw_spin_lock+0x177/0x190 [ 4244.224248] _raw_spin_lock+0x21/0x30 [ 4244.225862] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4244.227739] obd_mod_free+0x46/0x140 [ptlrpc] [ 4244.230052] mdc_commit_open+0x91/0xb0 [mdc] [ 4244.231980] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4244.234781] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4244.237457] after_reply+0xe2c/0x1780 [ptlrpc] [ 4244.243853] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4244.246703] ? __next_timer_interrupt+0x160/0x160 [ 4244.249041] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4244.252037] ? do_wait_intr+0xf0/0xf0 [ 4244.254904] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4244.258315] kthread+0x1d1/0x200 [ 4244.259871] ? set_kthread_struct+0x70/0x70 [ 4244.263720] ret_from_fork+0x1f/0x30 [ 4252.057110] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4252.067236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4252.094106] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4252.098879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4252.103165] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4252.106735] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4252.113368] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4252.126436] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4252.131958] RAX: 0000000000007821 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4252.137913] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4252.144384] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4252.152599] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4252.159569] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4252.167592] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4252.180135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4252.183959] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4252.191348] Call Trace: [ 4252.192825] [ 4252.197064] ? show_regs.cold.9+0x22/0x2f [ 4252.198871] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4252.204416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4252.206656] ? watchdog+0x40/0x40 [ 4252.208029] ? hrtimer_interrupt+0x128/0x2c0 [ 4252.209879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4252.211814] ? apic_timer_interrupt+0xf/0x20 [ 4252.213749] [ 4252.214665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4252.216948] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4252.219388] do_raw_spin_lock+0x177/0x190 [ 4252.222409] _raw_spin_lock+0x21/0x30 [ 4252.230605] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4252.235452] process_one_work+0x2c8/0x700 [ 4252.238978] worker_thread+0x296/0x6e0 [ 4252.243370] ? rescuer_thread+0x570/0x570 [ 4252.245851] kthread+0x1d1/0x200 [ 4252.249282] ? set_kthread_struct+0x70/0x70 [ 4252.252001] ret_from_fork+0x1f/0x30 [ 4260.058241] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4260.065631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4260.097085] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4260.102453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4260.106863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4260.110124] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4260.120193] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4260.124446] RAX: 0000000000005a8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4260.129999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4260.134416] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4260.137943] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4260.141245] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4260.145666] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4260.149907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4260.153964] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4260.157443] Call Trace: [ 4260.158517] [ 4260.159611] ? show_regs.cold.9+0x22/0x2f [ 4260.161030] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4260.163322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4260.165280] ? watchdog+0x40/0x40 [ 4260.166835] ? hrtimer_interrupt+0x128/0x2c0 [ 4260.171431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4260.173943] ? apic_timer_interrupt+0xf/0x20 [ 4260.175836] [ 4260.176623] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4260.178847] do_raw_spin_lock+0x177/0x190 [ 4260.180707] _raw_spin_lock+0x21/0x30 [ 4260.182278] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4260.184675] ? do_raw_spin_unlock+0x75/0x190 [ 4260.186726] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4260.189086] ? do_wait_intr+0xf0/0xf0 [ 4260.190494] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4260.192980] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4260.195105] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4260.198231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4260.200431] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4260.202612] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4260.204671] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4260.206520] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4260.208995] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4260.212165] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4260.216199] lmv_intent_open+0x338/0x1150 [lmv] [ 4260.219474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4260.222179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4260.225552] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4260.227533] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4260.228914] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4260.231413] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4260.233403] ? do_raw_spin_unlock+0x75/0x190 [ 4260.235192] ? _raw_spin_unlock+0x12/0x30 [ 4260.236967] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4260.239217] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4260.241509] ll_atomic_open+0x36a/0xd70 [lustre] [ 4260.243556] lookup_open+0x356/0xb40 [ 4260.244976] do_last+0x313/0xfc0 [ 4260.246311] ? nd_jump_root+0xe5/0x160 [ 4260.247699] ? path_init+0x437/0x520 [ 4260.249267] path_openat+0xf7/0x500 [ 4260.250828] ? do_raw_spin_unlock+0x75/0x190 [ 4260.252535] do_filp_open+0x99/0x140 [ 4260.254198] ? getname_flags+0x6e/0x330 [ 4260.255808] ? __check_object_size+0xff/0x256 [ 4260.257587] ? do_raw_spin_unlock+0x75/0x190 [ 4260.259559] ? _raw_spin_unlock+0x12/0x30 [ 4260.260884] do_sys_openat2+0x2b4/0x410 [ 4260.262619] do_sys_open+0x73/0xa0 [ 4260.264170] __x64_sys_openat+0x24/0x30 [ 4260.265822] do_syscall_64+0xc1/0x440 [ 4260.267289] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4260.269154] RIP: 0033:0x7fa749552332 [ 4260.270556] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4260.277511] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4260.280314] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4260.282909] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4260.285715] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4260.287868] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4260.290384] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4266.466498] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2349s! [ 4266.473862] Showing busy workqueues and worker pools: [ 4266.477190] workqueue events: flags=0x0 [ 4266.482733] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4266.493153] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4266.511563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4266.514561] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4266.519020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.523295] pending: kfree_rcu_monitor [ 4266.524465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4266.529756] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4266.541838] workqueue events_long: flags=0x0 [ 4266.543200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.547776] pending: lu_objects_flush [obdclass] [ 4266.549521] workqueue events_unbound: flags=0x2 [ 4266.551502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4266.555749] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4266.561606] workqueue events_power_efficient: flags=0x80 [ 4266.563893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.575917] pending: check_lifetime [ 4266.577765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.585744] pending: neigh_periodic_work [ 4266.587517] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.590599] pending: neigh_periodic_work [ 4266.598713] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.604203] pending: do_cache_clean [ 4266.605682] workqueue mm_percpu_wq: flags=0x8 [ 4266.609562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.612753] pending: vmstat_update [ 4266.616680] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.618526] pending: vmstat_update [ 4266.620276] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4266.622826] pending: vmstat_update [ 4266.626845] workqueue writeback: flags=0x4a [ 4266.629400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4266.632729] pending: wb_workfn, wb_workfn [ 4266.633881] workqueue xprtiod: flags=0xa [ 4266.635637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4266.639179] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4266.642454] workqueue ipv6_addrconf: flags=0x40008 [ 4266.643923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4266.649249] pending: addrconf_verify_work [ 4266.651168] workqueue knbd0-recv: flags=0x1a [ 4266.652655] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4266.656908] in-flight: 651:recv_work [ 4266.659927] workqueue ptlrpc_pinger: flags=0x2 [ 4266.662635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4266.664140] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4266.669589] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2349s workers=4 idle: 37 31846 28897 [ 4266.672368] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4256s workers=2 idle: 653 [ 4268.064040] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4268.073162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4268.130514] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4268.147805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4268.158430] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4268.160756] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4268.174433] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4268.177561] RAX: 000000000000092f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4268.180280] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4268.183089] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4268.185783] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4268.197385] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4268.202835] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4268.211868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4268.216129] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4268.225623] Call Trace: [ 4268.230586] [ 4268.231323] ? show_regs.cold.9+0x22/0x2f [ 4268.233397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4268.235224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4268.237585] ? watchdog+0x40/0x40 [ 4268.238669] ? hrtimer_interrupt+0x128/0x2c0 [ 4268.240173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4268.242363] ? apic_timer_interrupt+0xf/0x20 [ 4268.243879] [ 4268.248460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4268.250396] do_raw_spin_lock+0x177/0x190 [ 4268.252676] _raw_spin_lock+0x21/0x30 [ 4268.253976] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4268.256374] ? do_raw_spin_unlock+0x75/0x190 [ 4268.259743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4268.262320] ? do_wait_intr+0xf0/0xf0 [ 4268.268626] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4268.270680] mdc_close+0x53b/0xd70 [mdc] [ 4268.274074] lmv_close+0x18d/0x490 [lmv] [ 4268.276250] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4268.279603] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4268.281737] ll_md_close+0x274/0x800 [lustre] [ 4268.284038] ll_file_release+0x128/0xc50 [lustre] [ 4268.286213] __fput+0x10d/0x440 [ 4268.287492] ____fput+0x12/0x20 [ 4268.288821] task_work_run+0xc6/0x110 [ 4268.290481] exit_to_usermode_loop+0x1dd/0x1f0 [ 4268.292429] do_syscall_64+0x3d6/0x440 [ 4268.294044] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4268.296447] RIP: 0033:0x7fa749552cd8 [ 4268.300184] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4268.314859] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4268.321439] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4268.326600] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4268.330841] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4268.334130] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4268.338472] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4272.059305] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4272.069598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4272.093197] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4272.097186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4272.100242] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4272.102504] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4272.114948] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4272.121322] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4272.127369] RDX: 0000000000005d27 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4272.134021] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4272.138739] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4272.146148] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4272.150492] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4272.159010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4272.162321] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4272.167747] Call Trace: [ 4272.169072] [ 4272.169832] ? show_regs.cold.9+0x22/0x2f [ 4272.171703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4272.173931] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4272.175761] ? watchdog+0x40/0x40 [ 4272.177208] ? hrtimer_interrupt+0x128/0x2c0 [ 4272.178870] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4272.181036] ? apic_timer_interrupt+0xf/0x20 [ 4272.183031] [ 4272.183801] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4272.186239] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4272.188451] do_raw_spin_lock+0x177/0x190 [ 4272.190189] _raw_spin_lock+0x21/0x30 [ 4272.191673] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4272.194312] obd_mod_free+0x46/0x140 [ptlrpc] [ 4272.197488] mdc_commit_open+0x91/0xb0 [mdc] [ 4272.199583] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4272.202272] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4272.206510] after_reply+0xe2c/0x1780 [ptlrpc] [ 4272.208795] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4272.211578] ? __next_timer_interrupt+0x160/0x160 [ 4272.214479] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4272.220648] ? do_wait_intr+0xf0/0xf0 [ 4272.225621] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4272.231632] kthread+0x1d1/0x200 [ 4272.232815] ? set_kthread_struct+0x70/0x70 [ 4272.239342] ret_from_fork+0x1f/0x30 [ 4280.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4280.060962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4280.109249] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4280.118305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4280.124052] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4280.127941] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4280.131825] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4280.140864] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4280.144775] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4280.148974] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4280.153031] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4280.158812] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4280.163814] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4280.167331] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4280.172568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4280.175541] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4280.179664] Call Trace: [ 4280.181090] [ 4280.182709] ? show_regs.cold.9+0x22/0x2f [ 4280.184485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4280.188675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4280.193031] ? watchdog+0x40/0x40 [ 4280.195846] ? hrtimer_interrupt+0x128/0x2c0 [ 4280.198457] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4280.201158] ? apic_timer_interrupt+0xf/0x20 [ 4280.203336] [ 4280.204614] ? native_safe_halt+0x2e/0x50 [ 4280.208609] kvm_wait+0xcd/0xe0 [ 4280.210434] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4280.213802] do_raw_spin_lock+0x177/0x190 [ 4280.219864] _raw_spin_lock+0x21/0x30 [ 4280.223544] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4280.233743] process_one_work+0x2c8/0x700 [ 4280.246632] worker_thread+0x296/0x6e0 [ 4280.248066] ? rescuer_thread+0x570/0x570 [ 4280.257675] kthread+0x1d1/0x200 [ 4280.259343] ? set_kthread_struct+0x70/0x70 [ 4280.267101] ret_from_fork+0x1f/0x30 [ 4292.055867] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4292.065691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4292.112616] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4292.117736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4292.121676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4292.124159] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4292.132252] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4292.135064] RAX: 0000000000007c7f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4292.137702] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4292.140309] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4292.142916] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4292.153411] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4292.166138] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4292.175440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4292.182768] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4292.188445] Call Trace: [ 4292.190791] [ 4292.192955] ? show_regs.cold.9+0x22/0x2f [ 4292.197242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4292.201090] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4292.203111] ? watchdog+0x40/0x40 [ 4292.204326] ? hrtimer_interrupt+0x128/0x2c0 [ 4292.212765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4292.220095] ? apic_timer_interrupt+0xf/0x20 [ 4292.226405] [ 4292.227456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4292.232091] do_raw_spin_lock+0x177/0x190 [ 4292.237389] _raw_spin_lock+0x21/0x30 [ 4292.241449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4292.250349] ? do_raw_spin_unlock+0x75/0x190 [ 4292.252487] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4292.259104] ? do_wait_intr+0xf0/0xf0 [ 4292.266149] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4292.268864] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4292.272152] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4292.277061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4292.280861] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4292.284863] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4292.294464] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4292.299796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4292.302724] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4292.310125] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4292.318007] lmv_intent_open+0x338/0x1150 [lmv] [ 4292.321138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4292.330715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4292.337428] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4292.341747] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4292.347374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4292.351306] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4292.355302] ? do_raw_spin_unlock+0x75/0x190 [ 4292.358440] ? _raw_spin_unlock+0x12/0x30 [ 4292.360838] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4292.364462] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4292.369358] ll_atomic_open+0x36a/0xd70 [lustre] [ 4292.376836] lookup_open+0x356/0xb40 [ 4292.379363] do_last+0x313/0xfc0 [ 4292.380620] ? nd_jump_root+0xe5/0x160 [ 4292.382044] ? path_init+0x437/0x520 [ 4292.389483] path_openat+0xf7/0x500 [ 4292.392919] ? do_raw_spin_unlock+0x75/0x190 [ 4292.395908] do_filp_open+0x99/0x140 [ 4292.398716] ? getname_flags+0x6e/0x330 [ 4292.401348] ? __check_object_size+0xff/0x256 [ 4292.404167] ? do_raw_spin_unlock+0x75/0x190 [ 4292.411448] ? _raw_spin_unlock+0x12/0x30 [ 4292.418548] do_sys_openat2+0x2b4/0x410 [ 4292.421797] do_sys_open+0x73/0xa0 [ 4292.429434] __x64_sys_openat+0x24/0x30 [ 4292.434203] do_syscall_64+0xc1/0x440 [ 4292.438737] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4292.444723] RIP: 0033:0x7fa749552332 [ 4292.446556] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4292.461709] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4292.469262] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4292.473427] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4292.477711] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4292.482207] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4292.493986] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4296.059425] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4296.071042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4296.101006] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4296.111857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4296.118164] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4296.120749] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4296.129976] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4296.141064] RAX: 0000000000005e26 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4296.147281] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4296.149055] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4296.151139] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4296.153802] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4296.156789] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4296.160346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4296.162537] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4296.166020] Call Trace: [ 4296.167237] [ 4296.168214] ? show_regs.cold.9+0x22/0x2f [ 4296.170256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4296.172203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4296.174211] ? watchdog+0x40/0x40 [ 4296.175987] ? hrtimer_interrupt+0x128/0x2c0 [ 4296.178033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4296.180222] ? apic_timer_interrupt+0xf/0x20 [ 4296.182576] [ 4296.183500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4296.185816] do_raw_spin_lock+0x177/0x190 [ 4296.187304] _raw_spin_lock+0x21/0x30 [ 4296.189433] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4296.192265] ? do_raw_spin_unlock+0x75/0x190 [ 4296.195017] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4296.199017] ? do_wait_intr+0xf0/0xf0 [ 4296.200920] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4296.204741] mdc_close+0x53b/0xd70 [mdc] [ 4296.206534] lmv_close+0x18d/0x490 [lmv] [ 4296.208041] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4296.210388] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4296.212570] ll_md_close+0x274/0x800 [lustre] [ 4296.215326] ll_file_release+0x128/0xc50 [lustre] [ 4296.217361] __fput+0x10d/0x440 [ 4296.218415] ____fput+0x12/0x20 [ 4296.219735] task_work_run+0xc6/0x110 [ 4296.220633] exit_to_usermode_loop+0x1dd/0x1f0 [ 4296.222498] do_syscall_64+0x3d6/0x440 [ 4296.223928] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4296.225549] RIP: 0033:0x7fa749552cd8 [ 4296.226919] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4296.232330] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4296.234512] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4296.236328] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4296.238273] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4296.240212] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4296.242077] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4297.191546] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2380s! [ 4297.200060] Showing busy workqueues and worker pools: [ 4297.203215] workqueue events: flags=0x0 [ 4297.204961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4297.213273] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4297.226672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4297.237845] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4297.248800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.259046] pending: kfree_rcu_monitor [ 4297.273516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4297.276404] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4297.297290] workqueue events_long: flags=0x0 [ 4297.306235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.311625] pending: lu_objects_flush [obdclass] [ 4297.316329] workqueue events_unbound: flags=0x2 [ 4297.319517] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4297.328855] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4297.339558] workqueue events_power_efficient: flags=0x80 [ 4297.348378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.350554] pending: check_lifetime [ 4297.354961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.360937] pending: neigh_periodic_work [ 4297.367778] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.381569] pending: neigh_periodic_work [ 4297.383547] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.391045] pending: do_cache_clean [ 4297.393617] workqueue mm_percpu_wq: flags=0x8 [ 4297.395990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.399821] pending: vmstat_update [ 4297.401850] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.410961] pending: vmstat_update [ 4297.412442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4297.416183] pending: vmstat_update [ 4297.418496] workqueue writeback: flags=0x4a [ 4297.421285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4297.426339] pending: wb_workfn, wb_workfn [ 4297.428441] workqueue xprtiod: flags=0xa [ 4297.429500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4297.431497] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4297.436849] workqueue ipv6_addrconf: flags=0x40008 [ 4297.438919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4297.442841] pending: addrconf_verify_work [ 4297.444494] workqueue knbd0-recv: flags=0x1a [ 4297.452285] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4297.456192] in-flight: 651:recv_work [ 4297.457938] workqueue ptlrpc_pinger: flags=0x2 [ 4297.465131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4297.468734] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4297.474774] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2380s workers=4 idle: 37 31846 28897 [ 4297.480037] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4287s workers=2 idle: 653 [ 4300.058014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4300.062510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4300.089003] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4300.096032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4300.100231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4300.103892] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4300.113944] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4300.117328] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4300.122996] RDX: 000000000000141d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4300.126027] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4300.129090] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4300.132107] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4300.135491] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4300.138912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4300.141618] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4300.144937] Call Trace: [ 4300.145926] [ 4300.146756] ? show_regs.cold.9+0x22/0x2f [ 4300.148456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4300.150494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4300.152426] ? watchdog+0x40/0x40 [ 4300.153766] ? hrtimer_interrupt+0x128/0x2c0 [ 4300.155720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4300.157766] ? apic_timer_interrupt+0xf/0x20 [ 4300.159843] [ 4300.160866] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4300.163222] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4300.165517] do_raw_spin_lock+0x177/0x190 [ 4300.167066] _raw_spin_lock+0x21/0x30 [ 4300.168954] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4300.171943] obd_mod_free+0x46/0x140 [ptlrpc] [ 4300.174212] mdc_commit_open+0x91/0xb0 [mdc] [ 4300.176224] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4300.178920] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4300.181489] after_reply+0xe2c/0x1780 [ptlrpc] [ 4300.183829] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4300.186683] ? __next_timer_interrupt+0x160/0x160 [ 4300.189947] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4300.192344] ? do_wait_intr+0xf0/0xf0 [ 4300.194147] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4300.196940] kthread+0x1d1/0x200 [ 4300.198307] ? set_kthread_struct+0x70/0x70 [ 4300.200045] ret_from_fork+0x1f/0x30 [ 4308.057020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4308.063919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4308.121720] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4308.143309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4308.155880] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4308.158214] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4308.164887] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4308.196198] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4308.214514] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4308.226588] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4308.238867] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4308.247714] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4308.268258] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4308.270642] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4308.288864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4308.292188] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4308.305045] Call Trace: [ 4308.305966] [ 4308.309638] ? show_regs.cold.9+0x22/0x2f [ 4308.316803] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4308.324027] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4308.325571] ? watchdog+0x40/0x40 [ 4308.331625] ? hrtimer_interrupt+0x128/0x2c0 [ 4308.333140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4308.347222] ? apic_timer_interrupt+0xf/0x20 [ 4308.348837] [ 4308.354163] ? native_safe_halt+0x2e/0x50 [ 4308.355640] kvm_wait+0xcd/0xe0 [ 4308.356817] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4308.370940] do_raw_spin_lock+0x177/0x190 [ 4308.376985] _raw_spin_lock+0x21/0x30 [ 4308.380211] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4308.382358] process_one_work+0x2c8/0x700 [ 4308.401722] worker_thread+0x296/0x6e0 [ 4308.409008] ? rescuer_thread+0x570/0x570 [ 4308.415435] kthread+0x1d1/0x200 [ 4308.418463] ? set_kthread_struct+0x70/0x70 [ 4308.428051] ret_from_fork+0x1f/0x30 [ 4313.741083] rcu: INFO: rcu_sched self-detected stall on CPU [ 4313.744008] rcu: 2-...!: (1309023 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 4313.748847] (t=2400052 jiffies g=258701 q=2290) [ 4313.750555] rcu: rcu_sched kthread starved for 2396847 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4313.755676] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4313.759896] rcu: RCU grace-period kthread stack dump: [ 4313.762549] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4313.767347] Call Trace: [ 4313.768634] __schedule+0x351/0xcb0 [ 4313.770575] schedule+0xc0/0x180 [ 4313.772432] schedule_timeout+0xb4/0x190 [ 4313.774445] ? do_raw_spin_unlock+0x75/0x190 [ 4313.776683] ? __next_timer_interrupt+0x160/0x160 [ 4313.778742] ? prepare_to_swait_event+0xf3/0x140 [ 4313.781058] rcu_gp_kthread+0x8dd/0x15b0 [ 4313.782971] ? force_qs_rnp+0x2c0/0x2c0 [ 4313.784915] kthread+0x1d1/0x200 [ 4313.786654] ? set_kthread_struct+0x70/0x70 [ 4313.788593] ret_from_fork+0x1f/0x30 [ 4313.789943] rcu: Stack dump where RCU GP kthread last ran: [ 4313.792524] Sending NMI from CPU 2 to CPUs 3: [ 4313.794684] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 4313.798988] NMI backtrace for cpu 2 [ 4313.801600] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4313.807789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4313.812427] Call Trace: [ 4313.813248] [ 4313.813828] ? dump_stack+0xbb/0x10e [ 4313.815773] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4313.818783] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4313.820818] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4313.823604] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4313.826520] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4313.828968] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4313.831275] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4313.835034] ? update_process_times+0x6d/0xc0 [ 4313.837847] ? tick_sched_handle.isra.18+0x27/0x90 [ 4313.840678] ? tick_sched_timer+0x53/0xb0 [ 4313.842654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4313.844937] ? tick_sched_handle.isra.18+0x90/0x90 [ 4313.847668] ? hrtimer_interrupt+0x128/0x2c0 [ 4313.850807] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4313.853797] ? apic_timer_interrupt+0xf/0x20 [ 4313.856430] [ 4313.857358] ? native_safe_halt+0x2e/0x50 [ 4313.859494] ? kvm_wait+0xcd/0xe0 [ 4313.860824] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4313.863044] ? do_raw_spin_lock+0x177/0x190 [ 4313.864758] ? _raw_spin_lock+0x21/0x30 [ 4313.866673] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4313.868963] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4313.871392] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4313.873520] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4313.875885] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4313.878750] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4313.881286] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4313.884332] ? __next_timer_interrupt+0x160/0x160 [ 4313.886434] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4313.888548] ? do_wait_intr+0xf0/0xf0 [ 4313.890436] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4313.892988] ? kthread+0x1d1/0x200 [ 4313.894760] ? set_kthread_struct+0x70/0x70 [ 4313.896185] ? ret_from_fork+0x1f/0x30 [ 4324.055680] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4324.067439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4324.105505] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4324.114645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4324.121343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4324.122821] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4324.139688] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4324.149195] RAX: 0000000000006412 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4324.159094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4324.170160] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4324.175105] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4324.186754] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4324.198525] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4324.218295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4324.226031] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4324.231535] Call Trace: [ 4324.241128] [ 4324.241743] ? show_regs.cold.9+0x22/0x2f [ 4324.249584] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4324.255216] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4324.260266] ? watchdog+0x40/0x40 [ 4324.263441] ? hrtimer_interrupt+0x128/0x2c0 [ 4324.271771] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4324.277793] ? apic_timer_interrupt+0xf/0x20 [ 4324.285193] [ 4324.289673] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4324.296746] do_raw_spin_lock+0x177/0x190 [ 4324.302117] _raw_spin_lock+0x21/0x30 [ 4324.308549] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4324.315599] ? do_raw_spin_unlock+0x75/0x190 [ 4324.321122] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4324.334485] ? do_wait_intr+0xf0/0xf0 [ 4324.335864] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4324.345874] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4324.353620] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4324.364140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4324.376041] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4324.379666] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4324.384537] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4324.392748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4324.399875] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4324.403161] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4324.415481] lmv_intent_open+0x338/0x1150 [lmv] [ 4324.418715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4324.424286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4324.427628] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4324.430289] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4324.433324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4324.437080] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4324.440818] ? do_raw_spin_unlock+0x75/0x190 [ 4324.443143] ? _raw_spin_unlock+0x12/0x30 [ 4324.445787] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4324.449171] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4324.452658] ll_atomic_open+0x36a/0xd70 [lustre] [ 4324.457681] lookup_open+0x356/0xb40 [ 4324.461724] do_last+0x313/0xfc0 [ 4324.464691] ? nd_jump_root+0xe5/0x160 [ 4324.467032] ? path_init+0x437/0x520 [ 4324.468543] path_openat+0xf7/0x500 [ 4324.469937] ? do_raw_spin_unlock+0x75/0x190 [ 4324.476719] do_filp_open+0x99/0x140 [ 4324.481660] ? getname_flags+0x6e/0x330 [ 4324.485531] ? __check_object_size+0xff/0x256 [ 4324.489272] ? do_raw_spin_unlock+0x75/0x190 [ 4324.492600] ? _raw_spin_unlock+0x12/0x30 [ 4324.495429] do_sys_openat2+0x2b4/0x410 [ 4324.497555] do_sys_open+0x73/0xa0 [ 4324.500182] __x64_sys_openat+0x24/0x30 [ 4324.503233] do_syscall_64+0xc1/0x440 [ 4324.506865] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4324.511573] RIP: 0033:0x7fa749552332 [ 4324.513652] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4324.528515] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4324.532933] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4324.541663] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4324.546349] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4324.553481] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4324.559204] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4327.906483] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2410s! [ 4327.911603] Showing busy workqueues and worker pools: [ 4327.924139] workqueue events: flags=0x0 [ 4327.926097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4327.934360] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4327.950039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4327.956494] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4327.963948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4327.968672] pending: kfree_rcu_monitor [ 4327.973314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4327.979251] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4327.986515] workqueue events_long: flags=0x0 [ 4327.987930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4327.997782] pending: lu_objects_flush [obdclass] [ 4328.002438] workqueue events_unbound: flags=0x2 [ 4328.006069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4328.010026] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4328.014630] workqueue events_power_efficient: flags=0x80 [ 4328.017570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4328.023073] pending: check_lifetime [ 4328.025925] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4328.032300] pending: neigh_periodic_work [ 4328.035253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4328.037352] pending: neigh_periodic_work [ 4328.039952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4328.043129] pending: do_cache_clean [ 4328.044835] workqueue mm_percpu_wq: flags=0x8 [ 4328.047659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4328.053838] pending: vmstat_update [ 4328.055719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4328.059935] pending: vmstat_update [ 4328.061677] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4328.065107] pending: vmstat_update [ 4328.066825] workqueue writeback: flags=0x4a [ 4328.068825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4328.072331] pending: wb_workfn, wb_workfn [ 4328.073845] workqueue xprtiod: flags=0xa [ 4328.075679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4328.078588] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4328.082136] workqueue ipv6_addrconf: flags=0x40008 [ 4328.084315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4328.094714] pending: addrconf_verify_work [ 4328.098672] workqueue knbd0-recv: flags=0x1a [ 4328.099939] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4328.105432] in-flight: 651:recv_work [ 4328.107973] workqueue ptlrpc_pinger: flags=0x2 [ 4328.110342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4328.113703] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4328.115418] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2410s workers=4 idle: 37 31846 28897 [ 4328.118633] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4318s workers=2 idle: 653 [ 4336.057035] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4336.066330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4336.094225] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4336.105859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4336.110963] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4336.112751] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4336.120064] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4336.133286] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4336.139885] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4336.145748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4336.150285] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4336.153833] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4336.157369] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4336.160639] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4336.168170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4336.176690] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4336.182486] Call Trace: [ 4336.185300] [ 4336.188026] ? show_regs.cold.9+0x22/0x2f [ 4336.191247] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4336.196226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4336.199182] ? watchdog+0x40/0x40 [ 4336.204345] ? hrtimer_interrupt+0x128/0x2c0 [ 4336.209551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4336.211803] ? apic_timer_interrupt+0xf/0x20 [ 4336.216363] [ 4336.217136] ? native_safe_halt+0x2e/0x50 [ 4336.219164] kvm_wait+0xcd/0xe0 [ 4336.220402] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4336.223023] do_raw_spin_lock+0x177/0x190 [ 4336.224991] _raw_spin_lock+0x21/0x30 [ 4336.227035] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4336.229637] process_one_work+0x2c8/0x700 [ 4336.231881] worker_thread+0x296/0x6e0 [ 4336.233205] ? rescuer_thread+0x570/0x570 [ 4336.235047] kthread+0x1d1/0x200 [ 4336.236093] ? set_kthread_struct+0x70/0x70 [ 4336.237143] ret_from_fork+0x1f/0x30 [ 4340.058006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4340.059014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4340.071347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 4340.076679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.076782] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.076793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.076796] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.076818] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4340.076823] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.076829] RAX: 0000000000001ed9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4340.076832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4340.076834] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4340.076836] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4340.076848] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4340.076852] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4340.076855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.076857] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4340.076865] Call Trace: [ 4340.076873] [ 4340.076881] ? show_regs.cold.9+0x22/0x2f [ 4340.076892] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.118973] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.119029] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.149607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.149630] ? watchdog+0x40/0x40 [ 4340.149637] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.149641] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.149651] ? apic_timer_interrupt+0xf/0x20 [ 4340.149656] [ 4340.149661] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.149672] do_raw_spin_lock+0x177/0x190 [ 4340.149680] _raw_spin_lock+0x21/0x30 [ 4340.149686] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4340.150399] ? do_raw_spin_unlock+0x75/0x190 [ 4340.150407] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4340.150780] ? do_wait_intr+0xf0/0xf0 [ 4340.150790] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4340.151224] mdc_close+0x53b/0xd70 [mdc] [ 4340.151414] lmv_close+0x18d/0x490 [lmv] [ 4340.151459] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4340.151726] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4340.151818] ll_md_close+0x274/0x800 [lustre] [ 4340.154114] ll_file_release+0x128/0xc50 [lustre] [ 4340.154466] __fput+0x10d/0x440 [ 4340.154477] ____fput+0x12/0x20 [ 4340.154482] task_work_run+0xc6/0x110 [ 4340.154490] exit_to_usermode_loop+0x1dd/0x1f0 [ 4340.154497] do_syscall_64+0x3d6/0x440 [ 4340.154502] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4340.154513] RIP: 0033:0x7fa749552cd8 [ 4340.154523] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4340.154528] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4340.154535] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4340.154538] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4340.154540] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4340.154542] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4340.154544] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4340.567117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.574577] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4340.584227] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4340.600450] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.610500] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4340.620127] RDX: 00000000000007f4 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4340.625703] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4340.630446] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4340.636184] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4340.640095] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4340.654566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.663026] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4340.675206] Call Trace: [ 4340.679094] [ 4340.681505] ? show_regs.cold.9+0x22/0x2f [ 4340.684463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.688157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.689693] ? watchdog+0x40/0x40 [ 4340.698543] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.700006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.703123] ? apic_timer_interrupt+0xf/0x20 [ 4340.707162] [ 4340.708404] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4340.711847] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4340.718824] do_raw_spin_lock+0x177/0x190 [ 4340.720885] _raw_spin_lock+0x21/0x30 [ 4340.724445] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4340.734311] obd_mod_free+0x46/0x140 [ptlrpc] [ 4340.741801] mdc_commit_open+0x91/0xb0 [mdc] [ 4340.743308] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4340.749189] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4340.752560] after_reply+0xe2c/0x1780 [ptlrpc] [ 4340.757826] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4340.766810] ? __next_timer_interrupt+0x160/0x160 [ 4340.775182] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4340.781736] ? do_wait_intr+0xf0/0xf0 [ 4340.783433] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4340.792549] kthread+0x1d1/0x200 [ 4340.795570] ? set_kthread_struct+0x70/0x70 [ 4340.797818] ret_from_fork+0x1f/0x30 [ 4356.057600] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4356.063394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4356.092745] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4356.104233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4356.108186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4356.111491] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4356.129832] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4356.136437] RAX: 0000000000006485 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4356.141794] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4356.144816] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4356.148399] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4356.157188] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4356.161600] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4356.167069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4356.176341] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4356.183388] Call Trace: [ 4356.184467] [ 4356.185585] ? show_regs.cold.9+0x22/0x2f [ 4356.189223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4356.192680] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4356.195462] ? watchdog+0x40/0x40 [ 4356.198469] ? hrtimer_interrupt+0x128/0x2c0 [ 4356.200519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4356.203649] ? apic_timer_interrupt+0xf/0x20 [ 4356.206614] [ 4356.207960] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4356.210628] do_raw_spin_lock+0x177/0x190 [ 4356.212983] _raw_spin_lock+0x21/0x30 [ 4356.215495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4356.219098] ? do_raw_spin_unlock+0x75/0x190 [ 4356.221616] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4356.225905] ? do_wait_intr+0xf0/0xf0 [ 4356.229051] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4356.232352] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4356.237097] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4356.242150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.247636] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4356.251112] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4356.254066] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4356.256538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.259490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4356.263016] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4356.265483] lmv_intent_open+0x338/0x1150 [lmv] [ 4356.267993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.271095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.274160] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4356.276482] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4356.279569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.282294] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4356.285542] ? do_raw_spin_unlock+0x75/0x190 [ 4356.288514] ? _raw_spin_unlock+0x12/0x30 [ 4356.290294] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4356.292865] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4356.295864] ll_atomic_open+0x36a/0xd70 [lustre] [ 4356.298577] lookup_open+0x356/0xb40 [ 4356.300315] do_last+0x313/0xfc0 [ 4356.301577] ? nd_jump_root+0xe5/0x160 [ 4356.303512] ? path_init+0x437/0x520 [ 4356.305508] path_openat+0xf7/0x500 [ 4356.307114] ? do_raw_spin_unlock+0x75/0x190 [ 4356.309110] do_filp_open+0x99/0x140 [ 4356.311280] ? getname_flags+0x6e/0x330 [ 4356.313310] ? __check_object_size+0xff/0x256 [ 4356.315554] ? do_raw_spin_unlock+0x75/0x190 [ 4356.319471] ? _raw_spin_unlock+0x12/0x30 [ 4356.321567] do_sys_openat2+0x2b4/0x410 [ 4356.323665] do_sys_open+0x73/0xa0 [ 4356.325432] __x64_sys_openat+0x24/0x30 [ 4356.327268] do_syscall_64+0xc1/0x440 [ 4356.329273] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4356.332170] RIP: 0033:0x7fa749552332 [ 4356.334269] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4356.344288] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4356.348290] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4356.352102] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4356.356082] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4356.360443] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4356.364472] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4358.629824] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2441s! [ 4358.636511] Showing busy workqueues and worker pools: [ 4358.638943] workqueue events: flags=0x0 [ 4358.641038] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4358.648040] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4358.660787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4358.667903] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4358.674569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.678996] pending: kfree_rcu_monitor [ 4358.682108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4358.684246] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4358.688663] workqueue events_long: flags=0x0 [ 4358.692468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.696165] pending: lu_objects_flush [obdclass] [ 4358.697693] workqueue events_unbound: flags=0x2 [ 4358.700101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4358.704165] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4358.713157] workqueue events_power_efficient: flags=0x80 [ 4358.717038] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.721814] pending: check_lifetime [ 4358.725705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.730912] pending: neigh_periodic_work [ 4358.735158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.739130] pending: neigh_periodic_work [ 4358.741503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.754236] pending: do_cache_clean [ 4358.760043] workqueue mm_percpu_wq: flags=0x8 [ 4358.761537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.765519] pending: vmstat_update [ 4358.768861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.772838] pending: vmstat_update [ 4358.777868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4358.782731] pending: vmstat_update [ 4358.784077] workqueue writeback: flags=0x4a [ 4358.788538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4358.791806] pending: wb_workfn, wb_workfn [ 4358.795882] workqueue xprtiod: flags=0xa [ 4358.799832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4358.807011] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4358.812924] workqueue ipv6_addrconf: flags=0x40008 [ 4358.816604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4358.822791] pending: addrconf_verify_work [ 4358.826214] workqueue knbd0-recv: flags=0x1a [ 4358.830238] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4358.835520] in-flight: 651:recv_work [ 4358.840171] workqueue ptlrpc_pinger: flags=0x2 [ 4358.843687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4358.847785] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4358.853721] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2441s workers=4 idle: 37 31846 28897 [ 4358.856777] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4348s workers=2 idle: 653 [ 4364.057037] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4364.060380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4364.104741] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4364.110308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4364.115170] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4364.118410] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4364.122624] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4364.144467] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4364.149141] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4364.152072] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4364.154308] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4364.158298] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4364.165641] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4364.168574] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4364.178252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4364.180803] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4364.190199] Call Trace: [ 4364.198480] [ 4364.198948] ? show_regs.cold.9+0x22/0x2f [ 4364.200968] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4364.202664] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4364.204589] ? watchdog+0x40/0x40 [ 4364.205699] ? hrtimer_interrupt+0x128/0x2c0 [ 4364.207508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4364.209359] ? apic_timer_interrupt+0xf/0x20 [ 4364.211149] [ 4364.212210] ? native_safe_halt+0x2e/0x50 [ 4364.213684] kvm_wait+0xcd/0xe0 [ 4364.215397] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4364.217631] do_raw_spin_lock+0x177/0x190 [ 4364.220026] _raw_spin_lock+0x21/0x30 [ 4364.221718] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4364.224481] process_one_work+0x2c8/0x700 [ 4364.226428] worker_thread+0x296/0x6e0 [ 4364.227833] ? rescuer_thread+0x570/0x570 [ 4364.229679] kthread+0x1d1/0x200 [ 4364.230952] ? set_kthread_struct+0x70/0x70 [ 4364.232800] ret_from_fork+0x1f/0x30 [ 4368.061692] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4368.067259] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4368.070002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.070080] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.070089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.070091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.070105] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4368.070109] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4368.070113] RAX: 0000000000000588 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4368.070115] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4368.070117] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4368.070119] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4368.070121] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4368.070124] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4368.070127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.070130] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4368.070138] Call Trace: [ 4368.070144] [ 4368.070150] ? show_regs.cold.9+0x22/0x2f [ 4368.070159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.070165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.070169] ? watchdog+0x40/0x40 [ 4368.070173] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.070177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.070186] ? apic_timer_interrupt+0xf/0x20 [ 4368.070190] [ 4368.070192] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.070198] do_raw_spin_lock+0x177/0x190 [ 4368.070203] _raw_spin_lock+0x21/0x30 [ 4368.070207] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4368.074075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.074146] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.074153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.074155] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4368.074170] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4368.074175] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4368.074180] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4368.074182] RDX: 0000000000006073 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4368.074184] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4368.074186] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4368.074188] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4368.074191] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4368.074193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.074195] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4368.074204] Call Trace: [ 4368.074209] [ 4368.074216] ? show_regs.cold.9+0x22/0x2f [ 4368.074225] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.074232] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.075126] ? watchdog+0x40/0x40 [ 4368.075132] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.075135] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.075143] ? apic_timer_interrupt+0xf/0x20 [ 4368.075147] [ 4368.075151] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4368.075156] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4368.075218] do_raw_spin_lock+0x177/0x190 [ 4368.075224] _raw_spin_lock+0x21/0x30 [ 4368.075232] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4368.076919] obd_mod_free+0x46/0x140 [ptlrpc] [ 4368.077148] mdc_commit_open+0x91/0xb0 [mdc] [ 4368.079752] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4368.080020] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4368.082300] after_reply+0xe2c/0x1780 [ptlrpc] [ 4368.082751] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4368.082942] ? __next_timer_interrupt+0x160/0x160 [ 4368.082953] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4368.083153] ? do_wait_intr+0xf0/0xf0 [ 4368.083338] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4368.126633] ? do_raw_spin_unlock+0x75/0x190 [ 4368.126656] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4368.126948] ? do_wait_intr+0xf0/0xf0 [ 4368.126956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4368.127152] mdc_close+0x53b/0xd70 [mdc] [ 4368.127195] lmv_close+0x18d/0x490 [lmv] [ 4368.127226] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4368.127344] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4368.127421] ll_md_close+0x274/0x800 [lustre] [ 4368.131638] ll_file_release+0x128/0xc50 [lustre] [ 4368.131721] __fput+0x10d/0x440 [ 4368.131733] ____fput+0x12/0x20 [ 4368.131737] task_work_run+0xc6/0x110 [ 4368.131746] exit_to_usermode_loop+0x1dd/0x1f0 [ 4368.131752] do_syscall_64+0x3d6/0x440 [ 4368.131757] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4368.131767] RIP: 0033:0x7fa749552cd8 [ 4368.131775] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4368.131780] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4368.131785] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4368.131787] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4368.131789] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4368.131791] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4368.131793] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4368.605078] kthread+0x1d1/0x200 [ 4368.611061] ? set_kthread_struct+0x70/0x70 [ 4368.613883] ret_from_fork+0x1f/0x30 [ 4384.054077] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4384.059212] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4384.115338] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4384.135154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4384.140424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4384.160136] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4384.177219] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4384.197316] RAX: 0000000000003d83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4384.208039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4384.218593] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4384.223812] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4384.237665] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4384.247729] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4384.252559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4384.254551] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4384.259079] Call Trace: [ 4384.260279] [ 4384.262274] ? show_regs.cold.9+0x22/0x2f [ 4384.265840] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4384.270316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4384.276056] ? watchdog+0x40/0x40 [ 4384.277844] ? hrtimer_interrupt+0x128/0x2c0 [ 4384.279286] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4384.282691] ? apic_timer_interrupt+0xf/0x20 [ 4384.285138] [ 4384.286274] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4384.292042] do_raw_spin_lock+0x177/0x190 [ 4384.295833] _raw_spin_lock+0x21/0x30 [ 4384.299301] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4384.302855] ? do_raw_spin_unlock+0x75/0x190 [ 4384.305125] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4384.312358] ? do_wait_intr+0xf0/0xf0 [ 4384.316295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4384.322488] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4384.330059] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4384.347894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.352454] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4384.361870] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4384.369677] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4384.373275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.376826] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4384.382206] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4384.387313] lmv_intent_open+0x338/0x1150 [lmv] [ 4384.390073] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.395160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.401708] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4384.403460] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4384.407778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.415247] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4384.419764] ? do_raw_spin_unlock+0x75/0x190 [ 4384.422004] ? _raw_spin_unlock+0x12/0x30 [ 4384.424177] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4384.427209] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4384.430092] ll_atomic_open+0x36a/0xd70 [lustre] [ 4384.437220] lookup_open+0x356/0xb40 [ 4384.441109] do_last+0x313/0xfc0 [ 4384.450587] ? nd_jump_root+0xe5/0x160 [ 4384.458793] ? path_init+0x437/0x520 [ 4384.463849] path_openat+0xf7/0x500 [ 4384.470410] ? do_raw_spin_unlock+0x75/0x190 [ 4384.473253] do_filp_open+0x99/0x140 [ 4384.478033] ? getname_flags+0x6e/0x330 [ 4384.481746] ? __check_object_size+0xff/0x256 [ 4384.483731] ? do_raw_spin_unlock+0x75/0x190 [ 4384.486766] ? _raw_spin_unlock+0x12/0x30 [ 4384.489744] do_sys_openat2+0x2b4/0x410 [ 4384.492775] do_sys_open+0x73/0xa0 [ 4384.494335] __x64_sys_openat+0x24/0x30 [ 4384.496849] do_syscall_64+0xc1/0x440 [ 4384.499057] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4384.502076] RIP: 0033:0x7fa749552332 [ 4384.503818] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4384.511521] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4384.515386] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4384.518507] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4384.520977] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4384.523801] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4384.526452] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4389.344052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2472s! [ 4389.350807] Showing busy workqueues and worker pools: [ 4389.358017] workqueue events: flags=0x0 [ 4389.363287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4389.370652] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4389.418920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4389.421444] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4389.425202] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.428144] pending: kfree_rcu_monitor [ 4389.429691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4389.432333] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4389.435885] workqueue events_long: flags=0x0 [ 4389.437637] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.440399] pending: lu_objects_flush [obdclass] [ 4389.442097] workqueue events_unbound: flags=0x2 [ 4389.443295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4389.445463] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4389.449146] workqueue events_power_efficient: flags=0x80 [ 4389.452436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.463575] pending: check_lifetime [ 4389.464605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.471995] pending: neigh_periodic_work [ 4389.477211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.479558] pending: neigh_periodic_work [ 4389.484650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.487755] pending: do_cache_clean [ 4389.490192] workqueue mm_percpu_wq: flags=0x8 [ 4389.495436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.499417] pending: vmstat_update [ 4389.500872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.504032] pending: vmstat_update [ 4389.506371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4389.510655] pending: vmstat_update [ 4389.515682] workqueue writeback: flags=0x4a [ 4389.518773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4389.530530] pending: wb_workfn, wb_workfn [ 4389.532174] workqueue xprtiod: flags=0xa [ 4389.536097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4389.540375] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4389.547609] workqueue ipv6_addrconf: flags=0x40008 [ 4389.552924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4389.560083] pending: addrconf_verify_work [ 4389.565222] workqueue knbd0-recv: flags=0x1a [ 4389.567982] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4389.574733] in-flight: 651:recv_work [ 4389.577713] workqueue ptlrpc_pinger: flags=0x2 [ 4389.582340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4389.585595] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4389.590751] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2472s workers=4 idle: 37 31846 28897 [ 4389.597985] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4379s workers=2 idle: 653 [ 4392.059704] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4392.069590] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4392.130739] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4392.137530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4392.151254] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4392.158098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4392.162346] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4392.180786] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4392.187509] RAX: 0000000000007a09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4392.192729] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4392.195796] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4392.199279] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4392.202543] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4392.207680] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4392.224063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4392.229921] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4392.232594] Call Trace: [ 4392.238608] [ 4392.239307] ? show_regs.cold.9+0x22/0x2f [ 4392.240554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4392.249822] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4392.259159] ? watchdog+0x40/0x40 [ 4392.260344] ? hrtimer_interrupt+0x128/0x2c0 [ 4392.261912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4392.272792] ? apic_timer_interrupt+0xf/0x20 [ 4392.276615] [ 4392.277391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4392.280320] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4392.286651] do_raw_spin_lock+0x177/0x190 [ 4392.290648] _raw_spin_lock+0x21/0x30 [ 4392.292329] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4392.295289] process_one_work+0x2c8/0x700 [ 4392.296783] worker_thread+0x296/0x6e0 [ 4392.300718] ? rescuer_thread+0x570/0x570 [ 4392.303774] kthread+0x1d1/0x200 [ 4392.305678] ? set_kthread_struct+0x70/0x70 [ 4392.312544] ret_from_fork+0x1f/0x30 [ 4396.060264] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4396.061020] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4396.065655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4396.065750] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4396.065757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4396.065760] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4396.065773] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4396.065778] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4396.065783] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4396.065785] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4396.065788] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4396.065790] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4396.065792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4396.065796] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4396.065799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4396.065801] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4396.065810] Call Trace: [ 4396.065817] [ 4396.065823] ? show_regs.cold.9+0x22/0x2f [ 4396.065834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4396.065841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4396.065845] ? watchdog+0x40/0x40 [ 4396.065849] ? hrtimer_interrupt+0x128/0x2c0 [ 4396.065860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4396.065867] ? apic_timer_interrupt+0xf/0x20 [ 4396.065872] [ 4396.065875] ? native_safe_halt+0x2e/0x50 [ 4396.065879] kvm_wait+0xcd/0xe0 [ 4396.065886] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4396.065893] do_raw_spin_lock+0x177/0x190 [ 4396.065898] _raw_spin_lock+0x21/0x30 [ 4396.065909] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4396.066226] obd_mod_free+0x46/0x140 [ptlrpc] [ 4396.066516] mdc_commit_open+0x91/0xb0 [mdc] [ 4396.066555] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4396.067380] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4396.067635] after_reply+0xe2c/0x1780 [ptlrpc] [ 4396.067849] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4396.073088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4396.073170] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4396.073178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4396.073181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.073198] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4396.073203] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4396.073208] RAX: 00000000000071ee RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4396.073210] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4396.073212] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4396.073214] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4396.073217] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4396.073220] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4396.073223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4396.073226] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4396.073234] Call Trace: [ 4396.073239] [ 4396.073245] ? show_regs.cold.9+0x22/0x2f [ 4396.073254] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4396.073261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4396.073265] ? watchdog+0x40/0x40 [ 4396.073269] ? hrtimer_interrupt+0x128/0x2c0 [ 4396.073273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4396.073282] ? apic_timer_interrupt+0xf/0x20 [ 4396.073287] [ 4396.073290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.073297] do_raw_spin_lock+0x177/0x190 [ 4396.073302] _raw_spin_lock+0x21/0x30 [ 4396.073307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4396.073770] ? do_raw_spin_unlock+0x75/0x190 [ 4396.073777] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4396.073988] ? do_wait_intr+0xf0/0xf0 [ 4396.073998] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4396.074204] mdc_close+0x53b/0xd70 [mdc] [ 4396.074254] lmv_close+0x18d/0x490 [lmv] [ 4396.074289] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4396.074418] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4396.074636] ll_md_close+0x274/0x800 [lustre] [ 4396.128634] ? __next_timer_interrupt+0x160/0x160 [ 4396.128656] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4396.129107] ? do_wait_intr+0xf0/0xf0 [ 4396.129117] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4396.135368] ll_file_release+0x128/0xc50 [lustre] [ 4396.147441] kthread+0x1d1/0x200 [ 4396.147474] ? set_kthread_struct+0x70/0x70 [ 4396.147487] ret_from_fork+0x1f/0x30 [ 4396.150298] __fput+0x10d/0x440 [ 4396.523725] ____fput+0x12/0x20 [ 4396.525354] task_work_run+0xc6/0x110 [ 4396.527293] exit_to_usermode_loop+0x1dd/0x1f0 [ 4396.531849] do_syscall_64+0x3d6/0x440 [ 4396.535570] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4396.540940] RIP: 0033:0x7fa749552cd8 [ 4396.544464] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4396.560697] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4396.565926] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4396.573630] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4396.580279] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4396.586650] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4396.591594] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4416.058209] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4416.073684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4416.118360] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4416.136453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4416.179241] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4416.191133] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4416.224688] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4416.238504] RAX: 0000000000006611 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4416.260972] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4416.271377] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4416.284485] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4416.301233] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4416.312010] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4416.331513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4416.346168] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4416.367760] Call Trace: [ 4416.369540] [ 4416.374693] ? show_regs.cold.9+0x22/0x2f [ 4416.389721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4416.403500] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4416.408630] ? watchdog+0x40/0x40 [ 4416.430269] ? hrtimer_interrupt+0x128/0x2c0 [ 4416.446148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4416.463139] ? apic_timer_interrupt+0xf/0x20 [ 4416.483666] [ 4416.498502] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4416.516251] do_raw_spin_lock+0x177/0x190 [ 4416.527410] _raw_spin_lock+0x21/0x30 [ 4416.535518] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4416.545814] ? do_raw_spin_unlock+0x75/0x190 [ 4416.558038] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4416.566450] ? do_wait_intr+0xf0/0xf0 [ 4416.574463] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4416.579836] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4416.591281] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4416.602750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4416.612724] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4416.620914] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4416.633291] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4416.637997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4416.645266] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4416.660836] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4416.668967] lmv_intent_open+0x338/0x1150 [lmv] [ 4416.674286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4416.683118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4416.690686] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4416.694641] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4416.697724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4416.701890] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4416.705689] ? do_raw_spin_unlock+0x75/0x190 [ 4416.709432] ? _raw_spin_unlock+0x12/0x30 [ 4416.712766] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4416.715845] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4416.721688] ll_atomic_open+0x36a/0xd70 [lustre] [ 4416.726104] lookup_open+0x356/0xb40 [ 4416.732260] do_last+0x313/0xfc0 [ 4416.740300] ? nd_jump_root+0xe5/0x160 [ 4416.747729] ? path_init+0x437/0x520 [ 4416.759718] path_openat+0xf7/0x500 [ 4416.767392] ? do_raw_spin_unlock+0x75/0x190 [ 4416.788014] do_filp_open+0x99/0x140 [ 4416.798237] ? getname_flags+0x6e/0x330 [ 4416.804652] ? __check_object_size+0xff/0x256 [ 4416.819103] ? do_raw_spin_unlock+0x75/0x190 [ 4416.828264] ? _raw_spin_unlock+0x12/0x30 [ 4416.832296] do_sys_openat2+0x2b4/0x410 [ 4416.839219] do_sys_open+0x73/0xa0 [ 4416.845977] __x64_sys_openat+0x24/0x30 [ 4416.855229] do_syscall_64+0xc1/0x440 [ 4416.860566] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4416.863570] RIP: 0033:0x7fa749552332 [ 4416.872091] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4416.900426] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4416.903073] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4416.909724] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4416.918057] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4416.921689] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4416.933740] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4420.057043] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4420.060053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4420.069222] BUG: workqueue lockup - pool [ 4420.082714] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4420.084888] cpus=0-3 flags=0x4 nice=0 stuck for 2502s! [ 4420.084907] Showing busy workqueues and worker pools: [ 4420.084910] workqueue events: flags=0x0 [ 4420.084914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4420.084923] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4420.085370] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4420.085380] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4420.085907] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.085917] pending: kfree_rcu_monitor [ 4420.085926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4420.085934] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4420.086368] workqueue events_long: flags=0x0 [ 4420.086373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.086379] pending: lu_objects_flush [obdclass] [ 4420.093000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4420.093014] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4420.093320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4420.093332] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4420.093336] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4420.093341] RAX: 00000000000039f1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4420.093343] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4420.093345] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4420.093347] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4420.093350] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4420.093353] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4420.093356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4420.093358] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4420.093367] Call Trace: [ 4420.093373] [ 4420.093378] ? show_regs.cold.9+0x22/0x2f [ 4420.093388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4420.093395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4420.093399] ? watchdog+0x40/0x40 [ 4420.093404] ? hrtimer_interrupt+0x128/0x2c0 [ 4420.093408] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4420.093416] ? apic_timer_interrupt+0xf/0x20 [ 4420.093421] [ 4420.093424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4420.093429] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4420.093435] do_raw_spin_lock+0x177/0x190 [ 4420.093440] _raw_spin_lock+0x21/0x30 [ 4420.093445] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4420.093662] process_one_work+0x2c8/0x700 [ 4420.093669] worker_thread+0x296/0x6e0 [ 4420.093674] ? rescuer_thread+0x570/0x570 [ 4420.093678] kthread+0x1d1/0x200 [ 4420.093812] ? set_kthread_struct+0x70/0x70 [ 4420.093816] ret_from_fork+0x1f/0x30 [ 4420.293212] [ 4420.295170] workqueue events_unbound: flags=0x2 [ 4420.298616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4420.301956] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4420.309584] workqueue events_power_efficient: flags=0x80 [ 4420.315369] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.322615] pending: check_lifetime [ 4420.324165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.332023] pending: neigh_periodic_work [ 4420.334217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.339877] pending: neigh_periodic_work [ 4420.344992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.348851] pending: do_cache_clean [ 4420.350512] workqueue mm_percpu_wq: flags=0x8 [ 4420.355523] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.360006] pending: vmstat_update [ 4420.362224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.368407] pending: vmstat_update [ 4420.370026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4420.376695] pending: vmstat_update [ 4420.381068] workqueue writeback: flags=0x4a [ 4420.383575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4420.389546] pending: wb_workfn, wb_workfn [ 4420.391157] workqueue xprtiod: flags=0xa [ 4420.393934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4420.401752] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4420.408478] workqueue ipv6_addrconf: flags=0x40008 [ 4420.411498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4420.420338] pending: addrconf_verify_work [ 4420.424080] workqueue knbd0-recv: flags=0x1a [ 4420.429330] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4420.438620] in-flight: 651:recv_work [ 4420.439929] workqueue ptlrpc_pinger: flags=0x2 [ 4420.445711] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4420.447916] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4420.457579] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2503s workers=4 idle: 37 31846 28897 [ 4420.466895] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4410s workers=2 idle: 653 [ 4424.058717] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4424.064580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.072439] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4424.132043] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.132058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4424.132060] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4424.132075] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4424.132079] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4424.132084] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4424.132086] RDX: 0000000000006631 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4424.132088] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4424.132090] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4424.132092] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4424.132095] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4424.132097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.132099] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4424.132108] Call Trace: [ 4424.132114] [ 4424.132120] ? show_regs.cold.9+0x22/0x2f [ 4424.132130] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.132135] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.132139] ? watchdog+0x40/0x40 [ 4424.132143] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.132148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.132155] ? apic_timer_interrupt+0xf/0x20 [ 4424.132160] [ 4424.132163] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4424.132168] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4424.132172] do_raw_spin_lock+0x177/0x190 [ 4424.132176] _raw_spin_lock+0x21/0x30 [ 4424.136725] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4424.137084] obd_mod_free+0x46/0x140 [ptlrpc] [ 4424.139298] mdc_commit_open+0x91/0xb0 [mdc] [ 4424.139338] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4424.139725] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4424.139913] after_reply+0xe2c/0x1780 [ptlrpc] [ 4424.144877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.144952] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.144960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4424.144963] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.144977] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4424.144981] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4424.144986] RAX: 0000000000004b0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4424.144988] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4424.144990] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4424.144992] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4424.144995] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4424.144998] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4424.145001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.145003] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4424.145012] Call Trace: [ 4424.145018] [ 4424.145025] ? show_regs.cold.9+0x22/0x2f [ 4424.153706] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.153713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.153718] ? watchdog+0x40/0x40 [ 4424.153722] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.153725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.153734] ? apic_timer_interrupt+0xf/0x20 [ 4424.153739] [ 4424.153743] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.153753] do_raw_spin_lock+0x177/0x190 [ 4424.153759] _raw_spin_lock+0x21/0x30 [ 4424.153764] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4424.154032] ? do_raw_spin_unlock+0x75/0x190 [ 4424.154037] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4424.154326] ? do_wait_intr+0xf0/0xf0 [ 4424.154334] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4424.154570] mdc_close+0x53b/0xd70 [mdc] [ 4424.154609] lmv_close+0x18d/0x490 [lmv] [ 4424.154655] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4424.154970] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4424.157298] ll_md_close+0x274/0x800 [lustre] [ 4424.157404] ll_file_release+0x128/0xc50 [lustre] [ 4424.157494] __fput+0x10d/0x440 [ 4424.157503] ____fput+0x12/0x20 [ 4424.157508] task_work_run+0xc6/0x110 [ 4424.157517] exit_to_usermode_loop+0x1dd/0x1f0 [ 4424.157524] do_syscall_64+0x3d6/0x440 [ 4424.157528] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4424.157537] RIP: 0033:0x7fa749552cd8 [ 4424.157545] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4424.157549] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4424.157555] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4424.157557] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4424.157558] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4424.157561] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4424.157563] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4424.909967] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4424.920962] ? __next_timer_interrupt+0x160/0x160 [ 4424.938999] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4424.944074] ? do_wait_intr+0xf0/0xf0 [ 4424.945337] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4424.955896] kthread+0x1d1/0x200 [ 4424.958433] ? set_kthread_struct+0x70/0x70 [ 4424.969596] ret_from_fork+0x1f/0x30 [ 4448.055855] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4448.061544] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4448.076294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 4448.079730] Modules linked in: lustre(O) osp(O) ofd(O) [ 4448.093105] osc(O) lmv(O) ec(O) fid(O) fld(O) [ 4448.096359] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4448.096446] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4448.096453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4448.096460] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4448.097063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4448.097078] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4448.097082] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4448.097087] RAX: 0000000000002498 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4448.097089] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4448.097091] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4448.097094] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4448.097096] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4448.097099] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4448.097101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4448.097103] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4448.097112] Call Trace: [ 4448.097118] [ 4448.097124] ? show_regs.cold.9+0x22/0x2f [ 4448.099125] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4448.099154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4448.099160] ? watchdog+0x40/0x40 [ 4448.099165] ? hrtimer_interrupt+0x128/0x2c0 [ 4448.099172] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4448.099184] ? apic_timer_interrupt+0xf/0x20 [ 4448.099189] [ 4448.099193] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4448.099199] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4448.099210] do_raw_spin_lock+0x177/0x190 [ 4448.099217] _raw_spin_lock+0x21/0x30 [ 4448.099222] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4448.103202] process_one_work+0x2c8/0x700 [ 4448.105255] ptlrpc_gss(O) [ 4448.120923] worker_thread+0x296/0x6e0 [ 4448.126946] ptlrpc(O) [ 4448.130859] ? rescuer_thread+0x570/0x570 [ 4448.132425] obdclass(O) [ 4448.135076] kthread+0x1d1/0x200 [ 4448.142829] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4448.142884] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4448.142892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4448.142896] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4448.142911] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4448.142916] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4448.142920] RAX: 0000000000007b18 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4448.142923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4448.142925] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4448.142928] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4448.142930] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4448.142934] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4448.142937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4448.142940] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4448.142949] Call Trace: [ 4448.142955] [ 4448.142961] ? show_regs.cold.9+0x22/0x2f [ 4448.142973] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4448.142982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4448.142987] ? watchdog+0x40/0x40 [ 4448.142992] ? hrtimer_interrupt+0x128/0x2c0 [ 4448.142996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4448.143005] ? apic_timer_interrupt+0xf/0x20 [ 4448.143010] [ 4448.143013] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4448.143022] do_raw_spin_lock+0x177/0x190 [ 4448.145286] ? set_kthread_struct+0x70/0x70 [ 4448.148738] _raw_spin_lock+0x21/0x30 [ 4448.151583] ret_from_fork+0x1f/0x30 [ 4448.154152] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4448.316435] ? do_raw_spin_unlock+0x75/0x190 [ 4448.319506] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4448.321902] ? do_wait_intr+0xf0/0xf0 [ 4448.323702] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4448.330812] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4448.343559] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4448.345040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4448.346218] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4448.347472] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4448.348423] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4448.349312] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4448.350422] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4448.351811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4448.352884] lmv_intent_open+0x338/0x1150 [lmv] [ 4448.353812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4448.363391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4448.367184] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4448.370689] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4448.374469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4448.377723] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4448.379540] ? do_raw_spin_unlock+0x75/0x190 [ 4448.381126] ? _raw_spin_unlock+0x12/0x30 [ 4448.382650] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4448.386763] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4448.388021] ll_atomic_open+0x36a/0xd70 [lustre] [ 4448.393271] lookup_open+0x356/0xb40 [ 4448.394363] do_last+0x313/0xfc0 [ 4448.395723] ? nd_jump_root+0xe5/0x160 [ 4448.397217] ? path_init+0x437/0x520 [ 4448.399471] path_openat+0xf7/0x500 [ 4448.403009] ? do_raw_spin_unlock+0x75/0x190 [ 4448.404478] do_filp_open+0x99/0x140 [ 4448.406343] ? getname_flags+0x6e/0x330 [ 4448.408834] ? __check_object_size+0xff/0x256 [ 4448.411603] ? do_raw_spin_unlock+0x75/0x190 [ 4448.413253] ? _raw_spin_unlock+0x12/0x30 [ 4448.414427] do_sys_openat2+0x2b4/0x410 [ 4448.415612] do_sys_open+0x73/0xa0 [ 4448.416935] __x64_sys_openat+0x24/0x30 [ 4448.418348] do_syscall_64+0xc1/0x440 [ 4448.419616] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4448.421587] RIP: 0033:0x7fa749552332 [ 4448.423323] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4448.441548] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4448.445890] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4448.448234] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4448.451302] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4448.454326] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4448.456654] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4450.789279] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2533s! [ 4450.799408] Showing busy workqueues and worker pools: [ 4450.807201] workqueue events: flags=0x0 [ 4450.810261] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4450.815488] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4450.824520] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4450.828048] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4450.833705] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.837629] pending: kfree_rcu_monitor [ 4450.839647] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4450.843057] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4450.851287] workqueue events_long: flags=0x0 [ 4450.854118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.862408] pending: lu_objects_flush [obdclass] [ 4450.866144] workqueue events_unbound: flags=0x2 [ 4450.875074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4450.881614] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4450.890071] workqueue events_power_efficient: flags=0x80 [ 4450.893045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.896326] pending: check_lifetime [ 4450.897510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.905067] pending: neigh_periodic_work [ 4450.911493] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.926054] pending: neigh_periodic_work [ 4450.930669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.940586] pending: do_cache_clean [ 4450.943679] workqueue mm_percpu_wq: flags=0x8 [ 4450.950340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.955567] pending: vmstat_update [ 4450.959474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.970539] pending: vmstat_update [ 4450.981655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4450.991697] pending: vmstat_update [ 4450.993272] workqueue writeback: flags=0x4a [ 4450.995691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4451.004614] pending: wb_workfn, wb_workfn [ 4451.008979] workqueue xprtiod: flags=0xa [ 4451.012606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4451.016276] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4451.021341] workqueue ipv6_addrconf: flags=0x40008 [ 4451.023596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4451.026394] pending: addrconf_verify_work [ 4451.030807] workqueue knbd0-recv: flags=0x1a [ 4451.032311] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4451.041061] in-flight: 651:recv_work [ 4451.043897] workqueue ptlrpc_pinger: flags=0x2 [ 4451.047308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4451.053340] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4451.065754] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2533s workers=4 idle: 37 31846 28897 [ 4451.079817] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4440s workers=2 idle: 653 [ 4452.059249] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4452.061575] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4452.062800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.062872] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.062880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4452.062882] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.062897] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4452.062901] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4452.062906] RAX: 00000000000050ca RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4452.062908] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4452.062910] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4452.062913] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4452.062915] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4452.062919] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4452.062922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.062924] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4452.062933] Call Trace: [ 4452.062939] [ 4452.062945] ? show_regs.cold.9+0x22/0x2f [ 4452.062954] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.062960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.062964] ? watchdog+0x40/0x40 [ 4452.062968] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.062972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.062993] ? apic_timer_interrupt+0xf/0x20 [ 4452.062997] [ 4452.062999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.063006] do_raw_spin_lock+0x177/0x190 [ 4452.063011] _raw_spin_lock+0x21/0x30 [ 4452.063015] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4452.063319] ? do_raw_spin_unlock+0x75/0x190 [ 4452.063325] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4452.063521] ? do_wait_intr+0xf0/0xf0 [ 4452.063529] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4452.063721] mdc_close+0x53b/0xd70 [mdc] [ 4452.063760] lmv_close+0x18d/0x490 [lmv] [ 4452.063792] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4452.063910] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4452.064638] ll_md_close+0x274/0x800 [lustre] [ 4452.065045] ll_file_release+0x128/0xc50 [lustre] [ 4452.065125] __fput+0x10d/0x440 [ 4452.065134] ____fput+0x12/0x20 [ 4452.065138] task_work_run+0xc6/0x110 [ 4452.065145] exit_to_usermode_loop+0x1dd/0x1f0 [ 4452.065152] do_syscall_64+0x3d6/0x440 [ 4452.065157] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4452.065163] RIP: 0033:0x7fa749552cd8 [ 4452.065171] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4452.065175] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4452.065180] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4452.065181] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4452.065183] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4452.065186] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4452.065188] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4452.355502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.391876] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.397723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4452.405994] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4452.411347] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4452.426204] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4452.432520] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4452.439437] RDX: 0000000000006757 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4452.449589] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4452.455874] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4452.465114] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4452.472117] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4452.476509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.480778] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4452.483662] Call Trace: [ 4452.485316] [ 4452.486149] ? show_regs.cold.9+0x22/0x2f [ 4452.487401] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.489969] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.493892] ? watchdog+0x40/0x40 [ 4452.497554] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.499788] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.501456] ? apic_timer_interrupt+0xf/0x20 [ 4452.503982] [ 4452.504700] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4452.506553] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4452.511028] do_raw_spin_lock+0x177/0x190 [ 4452.513817] _raw_spin_lock+0x21/0x30 [ 4452.515842] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4452.517953] obd_mod_free+0x46/0x140 [ptlrpc] [ 4452.520382] mdc_commit_open+0x91/0xb0 [mdc] [ 4452.524887] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4452.528997] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4452.533823] after_reply+0xe2c/0x1780 [ptlrpc] [ 4452.536778] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4452.538525] ? __next_timer_interrupt+0x160/0x160 [ 4452.542309] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4452.545592] ? do_wait_intr+0xf0/0xf0 [ 4452.552433] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4452.558014] kthread+0x1d1/0x200 [ 4452.560791] ? set_kthread_struct+0x70/0x70 [ 4452.563047] ret_from_fork+0x1f/0x30 [ 4476.054208] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4476.057606] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4476.061324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4476.061403] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4476.061411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4476.061414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4476.061431] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4476.061435] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4476.061441] RAX: 000000000000363d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4476.061443] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4476.061445] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4476.061447] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4476.061450] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4476.061454] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4476.061457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4476.061460] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4476.061470] Call Trace: [ 4476.061479] [ 4476.061487] ? show_regs.cold.9+0x22/0x2f [ 4476.061500] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4476.061507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4476.061511] ? watchdog+0x40/0x40 [ 4476.061516] ? hrtimer_interrupt+0x128/0x2c0 [ 4476.061521] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4476.061531] ? apic_timer_interrupt+0xf/0x20 [ 4476.061536] [ 4476.061539] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4476.061546] do_raw_spin_lock+0x177/0x190 [ 4476.061552] _raw_spin_lock+0x21/0x30 [ 4476.061557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4476.062903] ? do_raw_spin_unlock+0x75/0x190 [ 4476.062914] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4476.063148] ? do_wait_intr+0xf0/0xf0 [ 4476.063159] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4476.063863] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4476.064343] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4476.070251] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 4476.087776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4476.098684] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4476.098760] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4476.098767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4476.098774] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4476.106787] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4476.110228] [ 4476.110238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4476.110258] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4476.110262] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4476.110266] RAX: 0000000000007afa RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4476.110268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4476.110270] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4476.110321] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4476.110323] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4476.110326] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4476.110328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4476.110330] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4476.110338] Call Trace: [ 4476.110344] [ 4476.110351] ? show_regs.cold.9+0x22/0x2f [ 4476.110361] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4476.110369] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4476.110374] ? watchdog+0x40/0x40 [ 4476.110377] ? hrtimer_interrupt+0x128/0x2c0 [ 4476.110380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4476.110453] ? apic_timer_interrupt+0xf/0x20 [ 4476.117253] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4476.117486] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4476.117512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4476.117660] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4476.118047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4476.118076] lmv_intent_open+0x338/0x1150 [lmv] [ 4476.118106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4476.118190] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4476.118331] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4476.118350] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4476.118464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4476.118599] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4476.118683] ? do_raw_spin_unlock+0x75/0x190 [ 4476.118691] ? _raw_spin_unlock+0x12/0x30 [ 4476.118697] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4476.118724] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4476.118746] ll_atomic_open+0x36a/0xd70 [lustre] [ 4476.118828] lookup_open+0x356/0xb40 [ 4476.118837] do_last+0x313/0xfc0 [ 4476.118842] ? nd_jump_root+0xe5/0x160 [ 4476.118846] ? path_init+0x437/0x520 [ 4476.118850] path_openat+0xf7/0x500 [ 4476.118855] ? do_raw_spin_unlock+0x75/0x190 [ 4476.118859] do_filp_open+0x99/0x140 [ 4476.118865] ? getname_flags+0x6e/0x330 [ 4476.118869] ? __check_object_size+0xff/0x256 [ 4476.118875] ? do_raw_spin_unlock+0x75/0x190 [ 4476.118879] ? _raw_spin_unlock+0x12/0x30 [ 4476.118883] do_sys_openat2+0x2b4/0x410 [ 4476.118887] do_sys_open+0x73/0xa0 [ 4476.118891] __x64_sys_openat+0x24/0x30 [ 4476.118894] do_syscall_64+0xc1/0x440 [ 4476.118900] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4476.118906] RIP: 0033:0x7fa749552332 [ 4476.118913] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4476.118917] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4476.118922] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4476.118924] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4476.118926] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4476.118928] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4476.118931] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4476.320001] [ 4476.320885] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4476.323256] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4476.325490] do_raw_spin_lock+0x177/0x190 [ 4476.326674] _raw_spin_lock+0x21/0x30 [ 4476.327789] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4476.329563] process_one_work+0x2c8/0x700 [ 4476.331009] worker_thread+0x296/0x6e0 [ 4476.332159] ? rescuer_thread+0x570/0x570 [ 4476.333782] kthread+0x1d1/0x200 [ 4476.334860] ? set_kthread_struct+0x70/0x70 [ 4476.336225] ret_from_fork+0x1f/0x30 [ 4480.058020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4480.059282] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4480.061640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.061913] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.061922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4480.061925] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4480.061944] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4480.061948] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4480.061953] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4480.061955] RDX: 0000000000004ec7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4480.061957] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4480.061960] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4480.061963] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4480.061967] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4480.061970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.061973] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4480.061984] Call Trace: [ 4480.061990] [ 4480.061996] ? show_regs.cold.9+0x22/0x2f [ 4480.062008] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.062015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.062020] ? watchdog+0x40/0x40 [ 4480.062024] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.062029] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.062041] ? apic_timer_interrupt+0xf/0x20 [ 4480.062046] [ 4480.062050] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4480.062054] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4480.062058] do_raw_spin_lock+0x177/0x190 [ 4480.062063] _raw_spin_lock+0x21/0x30 [ 4480.062070] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4480.062416] obd_mod_free+0x46/0x140 [ptlrpc] [ 4480.062632] mdc_commit_open+0x91/0xb0 [mdc] [ 4480.062757] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4480.062949] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4480.063298] after_reply+0xe2c/0x1780 [ptlrpc] [ 4480.063517] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4480.063913] ? __next_timer_interrupt+0x160/0x160 [ 4480.063985] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4480.068475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 4480.087710] ? do_wait_intr+0xf0/0xf0 [ 4480.087736] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4480.088079] kthread+0x1d1/0x200 [ 4480.088087] ? set_kthread_struct+0x70/0x70 [ 4480.088091] ret_from_fork+0x1f/0x30 [ 4480.259374] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.274780] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.281467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4480.291009] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4480.294013] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4480.302338] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4480.308038] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4480.317813] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4480.323102] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4480.330617] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4480.333149] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4480.340612] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4480.345835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.347254] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4480.353846] Call Trace: [ 4480.354667] [ 4480.356111] ? show_regs.cold.9+0x22/0x2f [ 4480.359427] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.365314] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.370459] ? watchdog+0x40/0x40 [ 4480.372015] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.374257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.377990] ? apic_timer_interrupt+0xf/0x20 [ 4480.385092] [ 4480.385922] ? native_safe_halt+0x2e/0x50 [ 4480.387729] kvm_wait+0xcd/0xe0 [ 4480.389482] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4480.394161] do_raw_spin_lock+0x177/0x190 [ 4480.397716] _raw_spin_lock+0x21/0x30 [ 4480.398992] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4480.401808] ? do_raw_spin_unlock+0x75/0x190 [ 4480.406369] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4480.408090] ? do_wait_intr+0xf0/0xf0 [ 4480.410611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4480.414257] mdc_close+0x53b/0xd70 [mdc] [ 4480.419302] lmv_close+0x18d/0x490 [lmv] [ 4480.420390] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4480.424471] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4480.426697] ll_md_close+0x274/0x800 [lustre] [ 4480.428608] ll_file_release+0x128/0xc50 [lustre] [ 4480.433441] __fput+0x10d/0x440 [ 4480.435448] ____fput+0x12/0x20 [ 4480.436473] task_work_run+0xc6/0x110 [ 4480.438519] exit_to_usermode_loop+0x1dd/0x1f0 [ 4480.443391] do_syscall_64+0x3d6/0x440 [ 4480.444939] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4480.449189] RIP: 0033:0x7fa749552cd8 [ 4480.450499] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4480.470164] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4480.476604] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4480.487279] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4480.489200] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4480.492472] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4480.497456] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4481.509262] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2564s! [ 4481.512477] Showing busy workqueues and worker pools: [ 4481.513955] workqueue events: flags=0x0 [ 4481.515380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4481.518410] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4481.525285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4481.528612] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4481.533716] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.536789] pending: kfree_rcu_monitor [ 4481.538637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4481.542174] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4481.546707] workqueue events_long: flags=0x0 [ 4481.548838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.552039] pending: lu_objects_flush [obdclass] [ 4481.554584] workqueue events_unbound: flags=0x2 [ 4481.556914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4481.560310] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4481.564056] workqueue events_power_efficient: flags=0x80 [ 4481.566551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.569566] pending: check_lifetime [ 4481.571140] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.574437] pending: neigh_periodic_work [ 4481.576710] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.579595] pending: neigh_periodic_work [ 4481.581347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.584307] pending: do_cache_clean [ 4481.586357] workqueue mm_percpu_wq: flags=0x8 [ 4481.588086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.591026] pending: vmstat_update [ 4481.592869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.595890] pending: vmstat_update [ 4481.597410] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4481.600239] pending: vmstat_update [ 4481.602318] workqueue writeback: flags=0x4a [ 4481.603914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4481.605871] pending: wb_workfn, wb_workfn [ 4481.607578] workqueue xprtiod: flags=0xa [ 4481.608837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4481.611376] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4481.614681] workqueue ipv6_addrconf: flags=0x40008 [ 4481.616635] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4481.620032] pending: addrconf_verify_work [ 4481.621928] workqueue knbd0-recv: flags=0x1a [ 4481.623704] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4481.626387] in-flight: 651:recv_work [ 4481.627608] workqueue ptlrpc_pinger: flags=0x2 [ 4481.629486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4481.631638] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4481.634600] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2564s workers=4 idle: 37 31846 28897 [ 4481.637980] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4471s workers=2 idle: 653 [ 4493.747439] rcu: INFO: rcu_sched self-detected stall on CPU [ 4493.759933] rcu: 2-...!: (1406770 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 4493.778606] (t=2580058 jiffies g=258701 q=2365) [ 4493.783584] rcu: rcu_sched kthread starved for 2576853 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4493.799927] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4493.821216] rcu: RCU grace-period kthread stack dump: [ 4493.836242] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4493.865804] Call Trace: [ 4493.870112] __schedule+0x351/0xcb0 [ 4493.880342] schedule+0xc0/0x180 [ 4493.883945] schedule_timeout+0xb4/0x190 [ 4493.886161] ? do_raw_spin_unlock+0x75/0x190 [ 4493.890100] ? __next_timer_interrupt+0x160/0x160 [ 4493.896113] ? prepare_to_swait_event+0xf3/0x140 [ 4493.901169] rcu_gp_kthread+0x8dd/0x15b0 [ 4493.909423] ? force_qs_rnp+0x2c0/0x2c0 [ 4493.912456] kthread+0x1d1/0x200 [ 4493.919784] ? set_kthread_struct+0x70/0x70 [ 4493.927495] ret_from_fork+0x1f/0x30 [ 4493.933955] rcu: Stack dump where RCU GP kthread last ran: [ 4493.952964] Sending NMI from CPU 2 to CPUs 3: [ 4493.957241] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 4493.958391] NMI backtrace for cpu 2 [ 4493.968573] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4493.987046] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4494.006479] Call Trace: [ 4494.008923] [ 4494.011945] ? dump_stack+0xbb/0x10e [ 4494.014889] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4494.016670] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4494.021596] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4494.036497] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4494.041713] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4494.043618] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4494.057488] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4494.061166] ? update_process_times+0x6d/0xc0 [ 4494.073375] ? tick_sched_handle.isra.18+0x27/0x90 [ 4494.079030] ? tick_sched_timer+0x53/0xb0 [ 4494.090749] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4494.092655] ? tick_sched_handle.isra.18+0x90/0x90 [ 4494.101979] ? hrtimer_interrupt+0x128/0x2c0 [ 4494.103113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4494.112551] ? apic_timer_interrupt+0xf/0x20 [ 4494.115861] [ 4494.118397] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4494.122606] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4494.131199] ? do_raw_spin_lock+0x177/0x190 [ 4494.140189] ? _raw_spin_lock+0x21/0x30 [ 4494.146653] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4494.157290] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4494.165191] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4494.168546] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4494.171776] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4494.183873] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4494.187024] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4494.193565] ? __next_timer_interrupt+0x160/0x160 [ 4494.199483] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4494.203845] ? do_wait_intr+0xf0/0xf0 [ 4494.209314] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4494.221131] ? kthread+0x1d1/0x200 [ 4494.227556] ? set_kthread_struct+0x70/0x70 [ 4494.230131] ? ret_from_fork+0x1f/0x30 [ 4504.059534] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4504.069909] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4504.116754] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4504.130710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4504.139209] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4504.146625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4504.154052] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4504.171605] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4504.180605] RAX: 0000000000006809 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4504.188577] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4504.193779] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4504.197359] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4504.201700] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4504.205359] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4504.209665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4504.212694] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4504.216641] Call Trace: [ 4504.218130] [ 4504.219099] ? show_regs.cold.9+0x22/0x2f [ 4504.221000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4504.223290] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4504.225621] ? watchdog+0x40/0x40 [ 4504.227237] ? hrtimer_interrupt+0x128/0x2c0 [ 4504.229501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4504.231504] ? apic_timer_interrupt+0xf/0x20 [ 4504.233693] [ 4504.234690] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4504.237360] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4504.240164] do_raw_spin_lock+0x177/0x190 [ 4504.242021] _raw_spin_lock+0x21/0x30 [ 4504.244040] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4504.247513] process_one_work+0x2c8/0x700 [ 4504.249654] worker_thread+0x296/0x6e0 [ 4504.251665] ? rescuer_thread+0x570/0x570 [ 4504.253936] kthread+0x1d1/0x200 [ 4504.255424] ? set_kthread_struct+0x70/0x70 [ 4504.257626] ret_from_fork+0x1f/0x30 [ 4508.054934] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4508.069145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4508.118545] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4508.124366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4508.127827] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4508.130192] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4508.145809] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4508.148839] RAX: 000000000000675b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4508.151575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4508.154331] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4508.157283] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4508.160378] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4508.163198] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4508.166118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4508.167508] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4508.169954] Call Trace: [ 4508.173242] [ 4508.174008] ? show_regs.cold.9+0x22/0x2f [ 4508.176456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4508.178566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4508.180636] ? watchdog+0x40/0x40 [ 4508.182236] ? hrtimer_interrupt+0x128/0x2c0 [ 4508.185331] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4508.187526] ? apic_timer_interrupt+0xf/0x20 [ 4508.189140] [ 4508.192207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4508.193887] do_raw_spin_lock+0x177/0x190 [ 4508.196219] _raw_spin_lock+0x21/0x30 [ 4508.197471] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4508.199941] ? do_raw_spin_unlock+0x75/0x190 [ 4508.201276] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4508.203047] ? do_wait_intr+0xf0/0xf0 [ 4508.204179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4508.205909] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4508.207461] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4508.212920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4508.215299] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4508.217411] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4508.222219] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4508.224210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4508.226467] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4508.228641] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4508.230415] lmv_intent_open+0x338/0x1150 [lmv] [ 4508.232163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4508.233968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4508.239178] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4508.240547] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4508.241629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4508.244677] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4508.247322] ? do_raw_spin_unlock+0x75/0x190 [ 4508.248897] ? _raw_spin_unlock+0x12/0x30 [ 4508.250043] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4508.252676] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4508.255144] ll_atomic_open+0x36a/0xd70 [lustre] [ 4508.256682] lookup_open+0x356/0xb40 [ 4508.257929] do_last+0x313/0xfc0 [ 4508.260285] ? nd_jump_root+0xe5/0x160 [ 4508.265885] ? path_init+0x437/0x520 [ 4508.267158] path_openat+0xf7/0x500 [ 4508.268013] ? do_raw_spin_unlock+0x75/0x190 [ 4508.268986] do_filp_open+0x99/0x140 [ 4508.281012] ? getname_flags+0x6e/0x330 [ 4508.284297] ? __check_object_size+0xff/0x256 [ 4508.286317] ? do_raw_spin_unlock+0x75/0x190 [ 4508.289392] ? _raw_spin_unlock+0x12/0x30 [ 4508.294088] do_sys_openat2+0x2b4/0x410 [ 4508.295490] do_sys_open+0x73/0xa0 [ 4508.296638] __x64_sys_openat+0x24/0x30 [ 4508.301468] do_syscall_64+0xc1/0x440 [ 4508.303844] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4508.306281] RIP: 0033:0x7fa749552332 [ 4508.310310] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4508.331584] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4508.339359] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4508.341951] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4508.356661] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4508.372949] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4508.385253] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4511.718105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2594s! [ 4511.726921] Showing busy workqueues and worker pools: [ 4511.736012] workqueue events: flags=0x0 [ 4511.738187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4511.752189] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4511.767476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4511.773319] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4511.784912] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.788202] pending: kfree_rcu_monitor [ 4511.790983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4511.794539] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4511.808631] workqueue events_long: flags=0x0 [ 4511.814204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.826687] pending: lu_objects_flush [obdclass] [ 4511.834219] workqueue events_unbound: flags=0x2 [ 4511.836566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4511.844300] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4511.853476] workqueue events_power_efficient: flags=0x80 [ 4511.856663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.860158] pending: check_lifetime [ 4511.868243] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.876868] pending: neigh_periodic_work [ 4511.881721] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.887896] pending: neigh_periodic_work [ 4511.892170] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.896766] pending: do_cache_clean [ 4511.899292] workqueue mm_percpu_wq: flags=0x8 [ 4511.905016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.907703] pending: vmstat_update [ 4511.910275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.923137] pending: vmstat_update [ 4511.927402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4511.935145] pending: vmstat_update [ 4511.940345] workqueue writeback: flags=0x4a [ 4511.942344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4511.947206] pending: wb_workfn, wb_workfn [ 4511.950841] workqueue xprtiod: flags=0xa [ 4511.952225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4511.957035] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4511.968531] workqueue ipv6_addrconf: flags=0x40008 [ 4511.971797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4511.977151] pending: addrconf_verify_work [ 4511.978510] workqueue knbd0-recv: flags=0x1a [ 4511.980373] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4511.983530] in-flight: 651:recv_work [ 4511.985230] workqueue ptlrpc_pinger: flags=0x2 [ 4511.991018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4511.993399] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4511.997292] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2594s workers=4 idle: 37 31846 28897 [ 4512.008077] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4501s workers=2 idle: 653 [ 4520.058507] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4520.059087] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4520.064873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4520.064956] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4520.064965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4520.064969] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4520.064984] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4520.064989] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4520.064993] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4520.064996] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4520.064998] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4520.065001] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4520.065003] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4520.065006] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4520.065009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4520.065011] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4520.065021] Call Trace: [ 4520.065028] [ 4520.065035] ? show_regs.cold.9+0x22/0x2f [ 4520.065049] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4520.065056] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4520.065060] ? watchdog+0x40/0x40 [ 4520.065064] ? hrtimer_interrupt+0x128/0x2c0 [ 4520.065068] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4520.065075] ? apic_timer_interrupt+0xf/0x20 [ 4520.065079] [ 4520.065082] ? native_safe_halt+0x2e/0x50 [ 4520.065086] kvm_wait+0xcd/0xe0 [ 4520.065093] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4520.065101] do_raw_spin_lock+0x177/0x190 [ 4520.065105] _raw_spin_lock+0x21/0x30 [ 4520.065111] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4520.074019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4520.074098] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4520.074106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4520.074109] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4520.074134] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4520.074139] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4520.074144] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4520.074147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4520.074149] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4520.074151] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4520.074154] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4520.074157] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4520.074160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4520.074162] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4520.074171] Call Trace: [ 4520.074345] [ 4520.074355] ? show_regs.cold.9+0x22/0x2f [ 4520.074375] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4520.074383] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4520.074387] ? watchdog+0x40/0x40 [ 4520.074391] ? hrtimer_interrupt+0x128/0x2c0 [ 4520.074395] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4520.074402] ? apic_timer_interrupt+0xf/0x20 [ 4520.074408] [ 4520.074410] ? native_safe_halt+0x2e/0x50 [ 4520.074414] kvm_wait+0xcd/0xe0 [ 4520.074423] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4520.074432] do_raw_spin_lock+0x177/0x190 [ 4520.074436] _raw_spin_lock+0x21/0x30 [ 4520.074441] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4520.111324] obd_mod_free+0x46/0x140 [ptlrpc] [ 4520.119937] ? do_raw_spin_unlock+0x75/0x190 [ 4520.119960] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4520.120387] ? do_wait_intr+0xf0/0xf0 [ 4520.120396] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4520.120696] mdc_close+0x53b/0xd70 [mdc] [ 4520.120739] lmv_close+0x18d/0x490 [lmv] [ 4520.120766] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4520.120904] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4520.120985] ll_md_close+0x274/0x800 [lustre] [ 4520.121069] ll_file_release+0x128/0xc50 [lustre] [ 4520.121434] __fput+0x10d/0x440 [ 4520.121444] ____fput+0x12/0x20 [ 4520.121447] task_work_run+0xc6/0x110 [ 4520.121453] exit_to_usermode_loop+0x1dd/0x1f0 [ 4520.121460] do_syscall_64+0x3d6/0x440 [ 4520.121464] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4520.121472] RIP: 0033:0x7fa749552cd8 [ 4520.121480] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4520.121484] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4520.121488] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4520.121490] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4520.121492] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4520.121494] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4520.121496] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4520.630729] mdc_commit_open+0x91/0xb0 [mdc] [ 4520.632273] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4520.640505] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4520.647238] after_reply+0xe2c/0x1780 [ptlrpc] [ 4520.650448] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4520.653221] ? __next_timer_interrupt+0x160/0x160 [ 4520.662380] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4520.669668] ? do_wait_intr+0xf0/0xf0 [ 4520.671416] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4520.678173] kthread+0x1d1/0x200 [ 4520.679255] ? set_kthread_struct+0x70/0x70 [ 4520.683753] ret_from_fork+0x1f/0x30 [ 4532.057084] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4532.060749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4532.085562] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4532.091530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4532.096136] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4532.098281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4532.100098] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4532.110093] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4532.115968] RAX: 0000000000003d4e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4532.122378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4532.130224] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4532.133526] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4532.142232] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4532.145319] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4532.158084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4532.166186] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4532.176146] Call Trace: [ 4532.178662] [ 4532.182607] ? show_regs.cold.9+0x22/0x2f [ 4532.184825] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4532.188204] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4532.192069] ? watchdog+0x40/0x40 [ 4532.194157] ? hrtimer_interrupt+0x128/0x2c0 [ 4532.197840] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4532.200697] ? apic_timer_interrupt+0xf/0x20 [ 4532.209537] [ 4532.214176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4532.221377] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4532.223958] do_raw_spin_lock+0x177/0x190 [ 4532.226869] _raw_spin_lock+0x21/0x30 [ 4532.229615] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4532.234457] process_one_work+0x2c8/0x700 [ 4532.239200] worker_thread+0x296/0x6e0 [ 4532.241478] ? rescuer_thread+0x570/0x570 [ 4532.243822] kthread+0x1d1/0x200 [ 4532.245049] ? set_kthread_struct+0x70/0x70 [ 4532.246555] ret_from_fork+0x1f/0x30 [ 4536.054037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4536.058036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4536.089449] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4536.096190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4536.106563] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4536.109122] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4536.124727] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4536.128712] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4536.132172] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4536.136696] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4536.141347] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4536.146154] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4536.149778] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4536.154302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4536.156972] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4536.161596] Call Trace: [ 4536.163483] [ 4536.164690] ? show_regs.cold.9+0x22/0x2f [ 4536.167289] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4536.170077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4536.172426] ? watchdog+0x40/0x40 [ 4536.173969] ? hrtimer_interrupt+0x128/0x2c0 [ 4536.175597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4536.179919] ? apic_timer_interrupt+0xf/0x20 [ 4536.182755] [ 4536.183788] ? native_safe_halt+0x2e/0x50 [ 4536.185641] kvm_wait+0xcd/0xe0 [ 4536.188876] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4536.192215] do_raw_spin_lock+0x177/0x190 [ 4536.194476] _raw_spin_lock+0x21/0x30 [ 4536.197491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4536.200254] ? do_raw_spin_unlock+0x75/0x190 [ 4536.202799] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4536.206420] ? do_wait_intr+0xf0/0xf0 [ 4536.208106] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4536.212338] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4536.217391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4536.222084] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4536.224576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4536.228330] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4536.231088] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4536.233627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4536.239129] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4536.241884] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4536.244257] lmv_intent_open+0x338/0x1150 [lmv] [ 4536.247711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4536.251793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4536.254473] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4536.258879] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4536.263384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4536.266617] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4536.269094] ? do_raw_spin_unlock+0x75/0x190 [ 4536.271162] ? _raw_spin_unlock+0x12/0x30 [ 4536.273725] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4536.276866] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4536.280441] ll_atomic_open+0x36a/0xd70 [lustre] [ 4536.282873] lookup_open+0x356/0xb40 [ 4536.285198] do_last+0x313/0xfc0 [ 4536.287099] ? nd_jump_root+0xe5/0x160 [ 4536.288987] ? path_init+0x437/0x520 [ 4536.291929] path_openat+0xf7/0x500 [ 4536.293718] ? do_raw_spin_unlock+0x75/0x190 [ 4536.298616] do_filp_open+0x99/0x140 [ 4536.299981] ? getname_flags+0x6e/0x330 [ 4536.301815] ? __check_object_size+0xff/0x256 [ 4536.303769] ? do_raw_spin_unlock+0x75/0x190 [ 4536.305822] ? _raw_spin_unlock+0x12/0x30 [ 4536.307363] do_sys_openat2+0x2b4/0x410 [ 4536.309165] do_sys_open+0x73/0xa0 [ 4536.311122] __x64_sys_openat+0x24/0x30 [ 4536.313173] do_syscall_64+0xc1/0x440 [ 4536.314835] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4536.319950] RIP: 0033:0x7fa749552332 [ 4536.321972] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4536.333224] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4536.339273] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4536.343973] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4536.350320] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4536.353981] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4536.357189] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4542.432277] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2625s! [ 4542.457508] Showing busy workqueues and worker pools: [ 4542.461156] workqueue events: flags=0x0 [ 4542.471020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4542.480815] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4542.497486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4542.512030] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4542.524855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.532958] pending: kfree_rcu_monitor [ 4542.534935] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4542.542615] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4542.553217] workqueue events_long: flags=0x0 [ 4542.563051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.575946] pending: lu_objects_flush [obdclass] [ 4542.580946] workqueue events_unbound: flags=0x2 [ 4542.591313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4542.595823] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4542.609511] workqueue events_power_efficient: flags=0x80 [ 4542.613279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.617753] pending: check_lifetime [ 4542.621780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.627240] pending: neigh_periodic_work [ 4542.631691] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.635017] pending: neigh_periodic_work [ 4542.637435] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.643837] pending: do_cache_clean [ 4542.652613] workqueue mm_percpu_wq: flags=0x8 [ 4542.659029] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.668806] pending: vmstat_update [ 4542.673620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.675286] pending: vmstat_update [ 4542.676191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4542.692625] pending: vmstat_update [ 4542.695425] workqueue writeback: flags=0x4a [ 4542.696617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4542.703983] pending: wb_workfn, wb_workfn [ 4542.706914] workqueue xprtiod: flags=0xa [ 4542.712505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4542.727237] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4542.733362] workqueue ipv6_addrconf: flags=0x40008 [ 4542.740138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4542.750502] pending: addrconf_verify_work [ 4542.755579] workqueue knbd0-recv: flags=0x1a [ 4542.757420] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4542.766373] in-flight: 651:recv_work [ 4542.772541] workqueue ptlrpc_pinger: flags=0x2 [ 4542.779086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4542.786462] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4542.790291] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2625s workers=4 idle: 37 31846 28897 [ 4542.796429] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4532s workers=2 idle: 653 [ 4548.060271] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4548.061337] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4548.067036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4548.067116] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4548.067124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4548.067128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4548.067142] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4548.067146] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4548.067150] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4548.067152] RDX: 0000000000005bcd RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4548.067154] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4548.067156] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4548.067159] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4548.067162] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4548.067165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4548.067167] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4548.067177] Call Trace: [ 4548.067182] [ 4548.067187] ? show_regs.cold.9+0x22/0x2f [ 4548.067197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4548.067203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4548.067206] ? watchdog+0x40/0x40 [ 4548.067210] ? hrtimer_interrupt+0x128/0x2c0 [ 4548.067214] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4548.067223] ? apic_timer_interrupt+0xf/0x20 [ 4548.067228] [ 4548.067231] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4548.067237] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4548.067253] do_raw_spin_lock+0x177/0x190 [ 4548.067259] _raw_spin_lock+0x21/0x30 [ 4548.067266] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4548.067595] obd_mod_free+0x46/0x140 [ptlrpc] [ 4548.068139] mdc_commit_open+0x91/0xb0 [mdc] [ 4548.068180] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4548.068400] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4548.068892] after_reply+0xe2c/0x1780 [ptlrpc] [ 4548.069150] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4548.069454] ? __next_timer_interrupt+0x160/0x160 [ 4548.069466] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4548.076085] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4548.076176] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4548.076184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4548.076187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4548.076205] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4548.076210] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4548.076218] RAX: 0000000000007b8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4548.076220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4548.076223] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4548.076225] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4548.076228] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4548.076232] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4548.076235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4548.076237] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4548.076246] Call Trace: [ 4548.076252] [ 4548.076259] ? show_regs.cold.9+0x22/0x2f [ 4548.076270] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4548.076277] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4548.076281] ? watchdog+0x40/0x40 [ 4548.076286] ? hrtimer_interrupt+0x128/0x2c0 [ 4548.076289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4548.076298] ? apic_timer_interrupt+0xf/0x20 [ 4548.109565] ? do_wait_intr+0xf0/0xf0 [ 4548.109596] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4548.110336] kthread+0x1d1/0x200 [ 4548.110343] ? set_kthread_struct+0x70/0x70 [ 4548.110346] ret_from_fork+0x1f/0x30 [ 4548.375412] [ 4548.377249] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4548.379826] do_raw_spin_lock+0x177/0x190 [ 4548.382184] _raw_spin_lock+0x21/0x30 [ 4548.384062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4548.386832] ? do_raw_spin_unlock+0x75/0x190 [ 4548.389291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4548.391850] ? do_wait_intr+0xf0/0xf0 [ 4548.393828] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4548.396772] mdc_close+0x53b/0xd70 [mdc] [ 4548.398571] lmv_close+0x18d/0x490 [lmv] [ 4548.400547] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4548.403552] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4548.406567] ll_md_close+0x274/0x800 [lustre] [ 4548.409106] ll_file_release+0x128/0xc50 [lustre] [ 4548.412558] __fput+0x10d/0x440 [ 4548.414103] ____fput+0x12/0x20 [ 4548.415403] task_work_run+0xc6/0x110 [ 4548.417590] exit_to_usermode_loop+0x1dd/0x1f0 [ 4548.419710] do_syscall_64+0x3d6/0x440 [ 4548.422447] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4548.425693] RIP: 0033:0x7fa749552cd8 [ 4548.428428] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4548.438051] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4548.441658] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4548.445546] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4548.452301] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4548.455383] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4548.458916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4560.057410] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4560.061849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4560.087819] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4560.094424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4560.102033] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4560.105350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4560.107555] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4560.119062] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4560.122535] RAX: 0000000000005106 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4560.126457] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4560.131257] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4560.133996] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4560.136495] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4560.139386] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4560.142763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4560.145456] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4560.148582] Call Trace: [ 4560.149470] [ 4560.150117] ? show_regs.cold.9+0x22/0x2f [ 4560.151421] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4560.152874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4560.155482] ? watchdog+0x40/0x40 [ 4560.156611] ? hrtimer_interrupt+0x128/0x2c0 [ 4560.158157] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4560.159797] ? apic_timer_interrupt+0xf/0x20 [ 4560.161393] [ 4560.162014] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4560.164237] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4560.166273] do_raw_spin_lock+0x177/0x190 [ 4560.167626] _raw_spin_lock+0x21/0x30 [ 4560.169145] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4560.172721] process_one_work+0x2c8/0x700 [ 4560.174929] worker_thread+0x296/0x6e0 [ 4560.178518] ? rescuer_thread+0x570/0x570 [ 4560.181733] kthread+0x1d1/0x200 [ 4560.183443] ? set_kthread_struct+0x70/0x70 [ 4560.186278] ret_from_fork+0x1f/0x30 [ 4568.058728] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4568.066805] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4568.132228] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4568.152262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4568.163433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4568.178468] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4568.206205] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4568.213318] RAX: 0000000000000c64 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4568.226136] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4568.237559] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4568.250184] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4568.261580] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4568.274368] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4568.284135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4568.288825] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4568.298189] Call Trace: [ 4568.298995] [ 4568.299728] ? show_regs.cold.9+0x22/0x2f [ 4568.310596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4568.317343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4568.328674] ? watchdog+0x40/0x40 [ 4568.334844] ? hrtimer_interrupt+0x128/0x2c0 [ 4568.339412] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4568.348828] ? apic_timer_interrupt+0xf/0x20 [ 4568.350350] [ 4568.350875] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4568.369557] do_raw_spin_lock+0x177/0x190 [ 4568.371424] _raw_spin_lock+0x21/0x30 [ 4568.380012] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4568.396597] ? do_raw_spin_unlock+0x75/0x190 [ 4568.402523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4568.414800] ? do_wait_intr+0xf0/0xf0 [ 4568.418610] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4568.427553] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4568.440271] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4568.458175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4568.469409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4568.471365] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4568.490438] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4568.499087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4568.501183] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4568.520356] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4568.538343] lmv_intent_open+0x338/0x1150 [lmv] [ 4568.539969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4568.557218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4568.564549] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4568.570601] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4568.576493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4568.591667] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4568.593434] ? do_raw_spin_unlock+0x75/0x190 [ 4568.610623] ? _raw_spin_unlock+0x12/0x30 [ 4568.615270] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4568.624458] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4568.638312] ll_atomic_open+0x36a/0xd70 [lustre] [ 4568.640114] lookup_open+0x356/0xb40 [ 4568.659294] do_last+0x313/0xfc0 [ 4568.660528] ? nd_jump_root+0xe5/0x160 [ 4568.669652] ? path_init+0x437/0x520 [ 4568.674463] path_openat+0xf7/0x500 [ 4568.675670] ? do_raw_spin_unlock+0x75/0x190 [ 4568.685725] do_filp_open+0x99/0x140 [ 4568.690411] ? getname_flags+0x6e/0x330 [ 4568.698527] ? __check_object_size+0xff/0x256 [ 4568.708211] ? do_raw_spin_unlock+0x75/0x190 [ 4568.709772] ? _raw_spin_unlock+0x12/0x30 [ 4568.719214] do_sys_openat2+0x2b4/0x410 [ 4568.720546] do_sys_open+0x73/0xa0 [ 4568.733457] __x64_sys_openat+0x24/0x30 [ 4568.740365] do_syscall_64+0xc1/0x440 [ 4568.749208] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4568.754749] RIP: 0033:0x7fa749552332 [ 4568.765544] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4568.804973] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4568.807677] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4568.828050] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4568.835758] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4568.837871] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4568.840016] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4573.154815] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2656s! [ 4573.163360] Showing busy workqueues and worker pools: [ 4573.168493] workqueue events: flags=0x0 [ 4573.172803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4573.185163] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4573.196819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4573.205273] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4573.213975] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.222723] pending: kfree_rcu_monitor [ 4573.231258] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4573.241880] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4573.256776] workqueue events_long: flags=0x0 [ 4573.278348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.288890] pending: lu_objects_flush [obdclass] [ 4573.299238] workqueue events_unbound: flags=0x2 [ 4573.308737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4573.322938] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4573.348022] workqueue events_power_efficient: flags=0x80 [ 4573.355681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.369600] pending: check_lifetime [ 4573.377248] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.389254] pending: neigh_periodic_work [ 4573.390815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.407900] pending: neigh_periodic_work [ 4573.412396] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.420538] pending: do_cache_clean [ 4573.426423] workqueue mm_percpu_wq: flags=0x8 [ 4573.430600] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.440104] pending: vmstat_update [ 4573.446769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.459491] pending: vmstat_update [ 4573.463420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4573.481530] pending: vmstat_update [ 4573.488913] workqueue writeback: flags=0x4a [ 4573.492769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4573.500800] pending: wb_workfn, wb_workfn [ 4573.515446] workqueue xprtiod: flags=0xa [ 4573.520429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4573.535903] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4573.548446] workqueue ipv6_addrconf: flags=0x40008 [ 4573.557560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4573.566107] pending: addrconf_verify_work [ 4573.577473] workqueue knbd0-recv: flags=0x1a [ 4573.586428] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4573.604971] in-flight: 651:recv_work [ 4573.612277] workqueue ptlrpc_pinger: flags=0x2 [ 4573.621216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4573.636802] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4573.644490] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2656s workers=4 idle: 37 31846 28897 [ 4573.663094] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4563s workers=2 idle: 653 [ 4576.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4576.060866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4576.062122] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4576.079638] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4576.079653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4576.079657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4576.079672] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4576.079677] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4576.079683] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4576.079686] RDX: 0000000000000ab8 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4576.079688] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4576.079691] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4576.079694] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4576.079697] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4576.079700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4576.079703] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4576.079712] Call Trace: [ 4576.079719] [ 4576.079726] ? show_regs.cold.9+0x22/0x2f [ 4576.079736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4576.079743] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4576.079748] ? watchdog+0x40/0x40 [ 4576.079753] ? hrtimer_interrupt+0x128/0x2c0 [ 4576.079757] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4576.079769] ? apic_timer_interrupt+0xf/0x20 [ 4576.079774] [ 4576.079777] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4576.079782] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4576.079787] do_raw_spin_lock+0x177/0x190 [ 4576.079792] _raw_spin_lock+0x21/0x30 [ 4576.079801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4576.083055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4576.083136] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4576.083143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4576.083147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4576.083160] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4576.083165] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4576.083170] RAX: 0000000000005f5d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4576.083172] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4576.083175] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4576.083177] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4576.083179] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4576.083183] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4576.083186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4576.083188] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4576.083199] Call Trace: [ 4576.083206] [ 4576.083212] ? show_regs.cold.9+0x22/0x2f [ 4576.083225] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4576.083234] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4576.083239] ? watchdog+0x40/0x40 [ 4576.083243] ? hrtimer_interrupt+0x128/0x2c0 [ 4576.083247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4576.083256] ? apic_timer_interrupt+0xf/0x20 [ 4576.083261] [ 4576.083264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4576.083270] do_raw_spin_lock+0x177/0x190 [ 4576.083274] _raw_spin_lock+0x21/0x30 [ 4576.083278] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4576.083541] ? do_raw_spin_unlock+0x75/0x190 [ 4576.083547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4576.083737] ? do_wait_intr+0xf0/0xf0 [ 4576.083745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4576.084202] mdc_close+0x53b/0xd70 [mdc] [ 4576.084247] lmv_close+0x18d/0x490 [lmv] [ 4576.084282] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4576.084414] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4576.084497] ll_md_close+0x274/0x800 [lustre] [ 4576.084595] ll_file_release+0x128/0xc50 [lustre] [ 4576.084674] __fput+0x10d/0x440 [ 4576.084685] ____fput+0x12/0x20 [ 4576.084689] task_work_run+0xc6/0x110 [ 4576.084695] exit_to_usermode_loop+0x1dd/0x1f0 [ 4576.084702] do_syscall_64+0x3d6/0x440 [ 4576.084706] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4576.084713] RIP: 0033:0x7fa749552cd8 [ 4576.084720] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4576.084724] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4576.084730] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4576.084732] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4576.084734] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4576.084736] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4576.084738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4576.319322] obd_mod_free+0x46/0x140 [ptlrpc] [ 4576.321311] mdc_commit_open+0x91/0xb0 [mdc] [ 4576.323098] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4576.325408] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4576.327514] after_reply+0xe2c/0x1780 [ptlrpc] [ 4576.329959] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4576.332485] ? __next_timer_interrupt+0x160/0x160 [ 4576.334060] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4576.335839] ? do_wait_intr+0xf0/0xf0 [ 4576.336795] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4576.339199] kthread+0x1d1/0x200 [ 4576.340780] ? set_kthread_struct+0x70/0x70 [ 4576.342288] ret_from_fork+0x1f/0x30 [ 4588.060582] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4588.065187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4588.087693] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4588.093125] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4588.097130] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4588.100222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.102866] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4588.111064] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4588.114309] RAX: 0000000000001d84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4588.117300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4588.120449] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4588.123444] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4588.126635] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4588.129695] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4588.133269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4588.136204] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4588.138917] Call Trace: [ 4588.140001] [ 4588.140720] ? show_regs.cold.9+0x22/0x2f [ 4588.142517] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4588.144919] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4588.146964] ? watchdog+0x40/0x40 [ 4588.148095] ? hrtimer_interrupt+0x128/0x2c0 [ 4588.149760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4588.151652] ? apic_timer_interrupt+0xf/0x20 [ 4588.153438] [ 4588.154297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.156366] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4588.158545] do_raw_spin_lock+0x177/0x190 [ 4588.160202] _raw_spin_lock+0x21/0x30 [ 4588.161832] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4588.166092] process_one_work+0x2c8/0x700 [ 4588.168265] worker_thread+0x296/0x6e0 [ 4588.169764] ? rescuer_thread+0x570/0x570 [ 4588.171053] kthread+0x1d1/0x200 [ 4588.172197] ? set_kthread_struct+0x70/0x70 [ 4588.174063] ret_from_fork+0x1f/0x30 [ 4600.055293] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4600.059277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4600.095626] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4600.105469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4600.114229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4600.122148] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4600.134463] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4600.139267] RAX: 0000000000007f79 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4600.144876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4600.149310] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4600.152848] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4600.159139] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4600.163362] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4600.173499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4600.182355] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4600.194989] Call Trace: [ 4600.195902] [ 4600.196632] ? show_regs.cold.9+0x22/0x2f [ 4600.201908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4600.208605] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4600.210381] ? watchdog+0x40/0x40 [ 4600.214000] ? hrtimer_interrupt+0x128/0x2c0 [ 4600.217093] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4600.230001] ? apic_timer_interrupt+0xf/0x20 [ 4600.234370] [ 4600.235311] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4600.240285] do_raw_spin_lock+0x177/0x190 [ 4600.243357] _raw_spin_lock+0x21/0x30 [ 4600.245404] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4600.248982] ? do_raw_spin_unlock+0x75/0x190 [ 4600.253636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4600.258817] ? do_wait_intr+0xf0/0xf0 [ 4600.263858] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4600.268893] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4600.271574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4600.279425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4600.283904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4600.293091] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4600.295587] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4600.298340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4600.303075] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4600.308834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4600.316420] lmv_intent_open+0x338/0x1150 [lmv] [ 4600.318074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4600.326230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4600.332487] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4600.336393] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4600.340628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4600.345208] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4600.348395] ? do_raw_spin_unlock+0x75/0x190 [ 4600.351434] ? _raw_spin_unlock+0x12/0x30 [ 4600.352969] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4600.355452] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4600.364397] ll_atomic_open+0x36a/0xd70 [lustre] [ 4600.366513] lookup_open+0x356/0xb40 [ 4600.370058] do_last+0x313/0xfc0 [ 4600.371579] ? nd_jump_root+0xe5/0x160 [ 4600.373498] ? path_init+0x437/0x520 [ 4600.375227] path_openat+0xf7/0x500 [ 4600.377303] ? do_raw_spin_unlock+0x75/0x190 [ 4600.379447] do_filp_open+0x99/0x140 [ 4600.383627] ? getname_flags+0x6e/0x330 [ 4600.388039] ? __check_object_size+0xff/0x256 [ 4600.391494] ? do_raw_spin_unlock+0x75/0x190 [ 4600.393676] ? _raw_spin_unlock+0x12/0x30 [ 4600.395740] do_sys_openat2+0x2b4/0x410 [ 4600.404184] do_sys_open+0x73/0xa0 [ 4600.406395] __x64_sys_openat+0x24/0x30 [ 4600.407532] do_syscall_64+0xc1/0x440 [ 4600.408652] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4600.411981] RIP: 0033:0x7fa749552332 [ 4600.413496] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4600.433330] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4600.440173] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4600.445005] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4600.449041] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4600.453369] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4600.457282] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4603.873050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2686s! [ 4603.876012] Showing busy workqueues and worker pools: [ 4603.882383] workqueue events: flags=0x0 [ 4603.884130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4603.887480] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4603.900749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4603.906651] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4603.913540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4603.919783] pending: kfree_rcu_monitor [ 4603.926254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4603.930210] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4603.933827] workqueue events_long: flags=0x0 [ 4603.942055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4603.949338] pending: lu_objects_flush [obdclass] [ 4603.957862] workqueue events_unbound: flags=0x2 [ 4603.959466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4603.964635] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4603.970573] workqueue events_power_efficient: flags=0x80 [ 4603.973641] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4603.977700] pending: check_lifetime [ 4603.979468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4603.988155] pending: neigh_periodic_work [ 4603.992871] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4603.997566] pending: neigh_periodic_work [ 4604.001171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4604.005783] pending: do_cache_clean [ 4604.011203] workqueue mm_percpu_wq: flags=0x8 [ 4604.019898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4604.026753] pending: vmstat_update [ 4604.028037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4604.031960] pending: vmstat_update [ 4604.036741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4604.047826] pending: vmstat_update [ 4604.054220] workqueue writeback: flags=0x4a [ 4604.057502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4604.058229] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4604.059031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4604.059042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4604.059127] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4604.059132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4604.059136] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4604.059152] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4604.059157] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4604.059161] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4604.059163] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4604.059166] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4604.059168] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4604.059175] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4604.059179] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4604.059182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4604.059184] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4604.059192] Call Trace: [ 4604.059199] [ 4604.059207] ? show_regs.cold.9+0x22/0x2f [ 4604.059219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4604.059226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4604.059232] ? watchdog+0x40/0x40 [ 4604.059245] ? hrtimer_interrupt+0x128/0x2c0 [ 4604.059249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4604.059260] ? apic_timer_interrupt+0xf/0x20 [ 4604.059265] [ 4604.059267] ? native_safe_halt+0x2e/0x50 [ 4604.059273] kvm_wait+0xcd/0xe0 [ 4604.059283] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4604.059294] do_raw_spin_lock+0x177/0x190 [ 4604.059299] _raw_spin_lock+0x21/0x30 [ 4604.059303] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4604.059925] ? do_raw_spin_unlock+0x75/0x190 [ 4604.059932] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4604.060263] ? do_wait_intr+0xf0/0xf0 [ 4604.060273] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4604.060566] mdc_close+0x53b/0xd70 [mdc] [ 4604.060605] lmv_close+0x18d/0x490 [lmv] [ 4604.060638] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4604.060970] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4604.061058] ll_md_close+0x274/0x800 [lustre] [ 4604.061298] ll_file_release+0x128/0xc50 [lustre] [ 4604.061401] __fput+0x10d/0x440 [ 4604.061571] ____fput+0x12/0x20 [ 4604.061576] task_work_run+0xc6/0x110 [ 4604.061587] exit_to_usermode_loop+0x1dd/0x1f0 [ 4604.061597] do_syscall_64+0x3d6/0x440 [ 4604.061603] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4604.061613] RIP: 0033:0x7fa749552cd8 [ 4604.061623] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4604.061627] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4604.061633] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4604.061636] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4604.061640] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4604.061642] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4604.061730] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4604.067164] pending: [ 4604.074064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4604.074164] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4604.074172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4604.074175] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4604.074195] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4604.074199] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4604.074203] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4604.074206] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4604.074208] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4604.074214] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4604.074217] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4604.074220] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4604.074223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4604.074225] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4604.074233] Call Trace: [ 4604.074245] [ 4604.074253] ? show_regs.cold.9+0x22/0x2f [ 4604.074265] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4604.074275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4604.074279] ? watchdog+0x40/0x40 [ 4604.074284] ? hrtimer_interrupt+0x128/0x2c0 [ 4604.080280] wb_workfn [ 4604.099208] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4604.105760] , wb_workfn [ 4604.105815] workqueue xprtiod: flags=0xa [ 4604.105819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4604.105831] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4604.105852] workqueue ipv6_addrconf: flags=0x40008 [ 4604.105855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4604.105864] pending: addrconf_verify_work [ 4604.105877] workqueue knbd0-recv: flags=0x1a [ 4604.105880] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4604.105887] in-flight: 651:recv_work [ 4604.105898] workqueue ptlrpc_pinger: flags=0x2 [ 4604.105900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4604.105906] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4604.106220] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2686s workers=4 idle: 37 31846 28897 [ 4604.106234] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4593s workers=2 idle: 653 [ 4604.407023] ? apic_timer_interrupt+0xf/0x20 [ 4604.408650] [ 4604.409364] ? native_safe_halt+0x2e/0x50 [ 4604.410591] kvm_wait+0xcd/0xe0 [ 4604.411580] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4604.413472] do_raw_spin_lock+0x177/0x190 [ 4604.415020] _raw_spin_lock+0x21/0x30 [ 4604.416377] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4604.418378] obd_mod_free+0x46/0x140 [ptlrpc] [ 4604.420381] mdc_commit_open+0x91/0xb0 [mdc] [ 4604.421976] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4604.424366] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4604.430871] after_reply+0xe2c/0x1780 [ptlrpc] [ 4604.433687] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4604.437152] ? __next_timer_interrupt+0x160/0x160 [ 4604.438818] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4604.440894] ? do_wait_intr+0xf0/0xf0 [ 4604.442736] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4604.446438] kthread+0x1d1/0x200 [ 4604.449281] ? set_kthread_struct+0x70/0x70 [ 4604.451785] ret_from_fork+0x1f/0x30 [ 4616.057530] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4616.061763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.130779] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.143610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4616.155155] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4616.162255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.171082] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4616.201930] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4616.208679] RAX: 0000000000006a16 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4616.227651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4616.234328] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4616.249913] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4616.261142] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4616.263668] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4616.268876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.272479] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4616.277278] Call Trace: [ 4616.278836] [ 4616.280100] ? show_regs.cold.9+0x22/0x2f [ 4616.283654] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.285794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.288093] ? watchdog+0x40/0x40 [ 4616.289801] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.292040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.295753] ? apic_timer_interrupt+0xf/0x20 [ 4616.299676] [ 4616.301053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.309119] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4616.312162] do_raw_spin_lock+0x177/0x190 [ 4616.317559] _raw_spin_lock+0x21/0x30 [ 4616.319674] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4616.324138] process_one_work+0x2c8/0x700 [ 4616.326337] worker_thread+0x296/0x6e0 [ 4616.331127] ? rescuer_thread+0x570/0x570 [ 4616.336313] kthread+0x1d1/0x200 [ 4616.338774] ? set_kthread_struct+0x70/0x70 [ 4616.340674] ret_from_fork+0x1f/0x30 [ 4632.054159] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4632.060580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4632.070665] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4632.070964] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4632.070970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4632.071042] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4632.071048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4632.071051] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4632.071066] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4632.071070] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4632.071075] RAX: 0000000000007d07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4632.071077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4632.071080] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4632.071081] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4632.071084] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4632.071087] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4632.071090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4632.071093] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4632.071101] Call Trace: [ 4632.071107] [ 4632.071113] ? show_regs.cold.9+0x22/0x2f [ 4632.071123] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4632.071131] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4632.071136] ? watchdog+0x40/0x40 [ 4632.071141] ? hrtimer_interrupt+0x128/0x2c0 [ 4632.071146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4632.071157] ? apic_timer_interrupt+0xf/0x20 [ 4632.071164] [ 4632.071167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4632.071174] do_raw_spin_lock+0x177/0x190 [ 4632.071181] _raw_spin_lock+0x21/0x30 [ 4632.071187] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4632.071461] ? do_raw_spin_unlock+0x75/0x190 [ 4632.071466] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4632.072907] ? do_wait_intr+0xf0/0xf0 [ 4632.072918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4632.073613] mdc_close+0x53b/0xd70 [mdc] [ 4632.073658] lmv_close+0x18d/0x490 [lmv] [ 4632.073690] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4632.073820] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4632.073912] ll_md_close+0x274/0x800 [lustre] [ 4632.074006] ll_file_release+0x128/0xc50 [lustre] [ 4632.074089] __fput+0x10d/0x440 [ 4632.074101] ____fput+0x12/0x20 [ 4632.074105] task_work_run+0xc6/0x110 [ 4632.074113] exit_to_usermode_loop+0x1dd/0x1f0 [ 4632.074120] do_syscall_64+0x3d6/0x440 [ 4632.074125] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4632.074134] RIP: 0033:0x7fa749552cd8 [ 4632.074179] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4632.074184] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4632.074189] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4632.074192] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4632.074194] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4632.074196] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4632.074198] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4632.096801] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4632.096815] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4632.096818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4632.096841] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4632.096845] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4632.096851] RAX: 0000000000004596 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4632.096854] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4632.096856] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4632.096858] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4632.096864] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4632.096868] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4632.096870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4632.096873] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4632.096882] Call Trace: [ 4632.096888] [ 4632.096894] ? show_regs.cold.9+0x22/0x2f [ 4632.096904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4632.096911] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4632.096925] ? watchdog+0x40/0x40 [ 4632.096930] ? hrtimer_interrupt+0x128/0x2c0 [ 4632.096934] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4632.096944] ? apic_timer_interrupt+0xf/0x20 [ 4632.096949] [ 4632.096951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4632.096961] do_raw_spin_lock+0x177/0x190 [ 4632.096966] _raw_spin_lock+0x21/0x30 [ 4632.096972] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4632.097285] ? do_raw_spin_unlock+0x75/0x190 [ 4632.097291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4632.097993] ? do_wait_intr+0xf0/0xf0 [ 4632.098003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4632.098418] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4632.098780] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4632.099078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.099414] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4632.099463] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4632.099493] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4632.099520] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.105706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [ 4632.110579] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4632.110960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4632.111005] lmv_intent_open+0x338/0x1150 [lmv] [ 4632.111264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.111378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.111484] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4632.111508] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4632.111593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.176395] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4632.176447] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4632.176456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4632.176459] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4632.176474] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4632.176479] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4632.176484] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4632.176486] RDX: 0000000000007c1c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4632.176488] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4632.176490] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4632.176493] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4632.176497] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4632.176500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4632.176503] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4632.176511] Call Trace: [ 4632.176516] [ 4632.176523] ? show_regs.cold.9+0x22/0x2f [ 4632.184734] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4632.185017] ? do_raw_spin_unlock+0x75/0x190 [ 4632.185027] ? _raw_spin_unlock+0x12/0x30 [ 4632.185035] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4632.185228] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4632.185259] ll_atomic_open+0x36a/0xd70 [lustre] [ 4632.185571] lookup_open+0x356/0xb40 [ 4632.185579] do_last+0x313/0xfc0 [ 4632.185584] ? nd_jump_root+0xe5/0x160 [ 4632.185589] ? path_init+0x437/0x520 [ 4632.185594] path_openat+0xf7/0x500 [ 4632.185599] ? do_raw_spin_unlock+0x75/0x190 [ 4632.185706] do_filp_open+0x99/0x140 [ 4632.185714] ? getname_flags+0x6e/0x330 [ 4632.185719] ? __check_object_size+0xff/0x256 [ 4632.185725] ? do_raw_spin_unlock+0x75/0x190 [ 4632.185729] ? _raw_spin_unlock+0x12/0x30 [ 4632.185733] do_sys_openat2+0x2b4/0x410 [ 4632.185737] do_sys_open+0x73/0xa0 [ 4632.185741] __x64_sys_openat+0x24/0x30 [ 4632.185745] do_syscall_64+0xc1/0x440 [ 4632.185752] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4632.185759] RIP: 0033:0x7fa749552332 [ 4632.185818] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4632.185823] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4632.185840] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4632.185842] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4632.185844] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4632.185846] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4632.185848] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4633.191778] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4633.194746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4633.197255] ? watchdog+0x40/0x40 [ 4633.198789] ? hrtimer_interrupt+0x128/0x2c0 [ 4633.201306] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4633.203780] ? apic_timer_interrupt+0xf/0x20 [ 4633.206800] [ 4633.212726] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4633.219220] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4633.222582] do_raw_spin_lock+0x177/0x190 [ 4633.229502] _raw_spin_lock+0x21/0x30 [ 4633.237751] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4633.243240] obd_mod_free+0x46/0x140 [ptlrpc] [ 4633.250837] mdc_commit_open+0x91/0xb0 [mdc] [ 4633.259960] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4633.266665] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4633.268762] after_reply+0xe2c/0x1780 [ptlrpc] [ 4633.277792] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4633.286702] ? __next_timer_interrupt+0x160/0x160 [ 4633.290020] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4633.293780] ? do_wait_intr+0xf0/0xf0 [ 4633.296274] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4633.307401] kthread+0x1d1/0x200 [ 4633.309102] ? set_kthread_struct+0x70/0x70 [ 4633.311945] ret_from_fork+0x1f/0x30 [ 4634.081110] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2716s! [ 4634.084982] Showing busy workqueues and worker pools: [ 4634.087434] workqueue events: flags=0x0 [ 4634.089581] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4634.093359] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4634.100110] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4634.102747] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4634.106552] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.109135] pending: kfree_rcu_monitor [ 4634.110679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4634.113149] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4634.116638] workqueue events_long: flags=0x0 [ 4634.118163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.120645] pending: lu_objects_flush [obdclass] [ 4634.122631] workqueue events_unbound: flags=0x2 [ 4634.124841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4634.127755] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4634.132016] workqueue events_power_efficient: flags=0x80 [ 4634.134340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.136968] pending: check_lifetime [ 4634.138459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.141369] pending: neigh_periodic_work [ 4634.142944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.145673] pending: neigh_periodic_work [ 4634.147192] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.149752] pending: do_cache_clean [ 4634.151189] workqueue mm_percpu_wq: flags=0x8 [ 4634.152831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.155257] pending: vmstat_update [ 4634.156765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.159115] pending: vmstat_update [ 4634.160612] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4634.163076] pending: vmstat_update [ 4634.164478] workqueue writeback: flags=0x4a [ 4634.166109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4634.168994] pending: wb_workfn, wb_workfn [ 4634.170587] workqueue xprtiod: flags=0xa [ 4634.172432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4634.175457] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4634.178934] workqueue ipv6_addrconf: flags=0x40008 [ 4634.180770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4634.183754] pending: addrconf_verify_work [ 4634.186127] workqueue knbd0-recv: flags=0x1a [ 4634.188263] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4634.190939] in-flight: 651:recv_work [ 4634.192448] workqueue ptlrpc_pinger: flags=0x2 [ 4634.194293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4634.197383] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4634.200262] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2717s workers=4 idle: 37 31846 28897 [ 4634.204109] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4624s workers=2 idle: 653 [ 4644.057303] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4644.060642] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4644.094516] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4644.103491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4644.109017] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4644.115255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4644.118922] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4644.134447] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4644.138523] RAX: 0000000000002312 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4644.142058] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4644.145643] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4644.151380] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4644.155469] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4644.158891] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4644.162226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4644.164524] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4644.167830] Call Trace: [ 4644.168937] [ 4644.169705] ? show_regs.cold.9+0x22/0x2f [ 4644.172856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4644.178573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4644.183710] ? watchdog+0x40/0x40 [ 4644.189828] ? hrtimer_interrupt+0x128/0x2c0 [ 4644.191773] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4644.200603] ? apic_timer_interrupt+0xf/0x20 [ 4644.207952] [ 4644.208738] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4644.215310] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4644.217705] do_raw_spin_lock+0x177/0x190 [ 4644.223491] _raw_spin_lock+0x21/0x30 [ 4644.224809] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4644.232880] process_one_work+0x2c8/0x700 [ 4644.237423] worker_thread+0x296/0x6e0 [ 4644.238868] ? rescuer_thread+0x570/0x570 [ 4644.243947] kthread+0x1d1/0x200 [ 4644.247932] ? set_kthread_struct+0x70/0x70 [ 4644.252506] ret_from_fork+0x1f/0x30 [ 4660.055101] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4660.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4660.058630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 4660.060881] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4660.060891] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4660.060982] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4660.060988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4660.060991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4660.061007] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4660.061012] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4660.061017] RAX: 0000000000003a91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4660.061019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4660.061021] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4660.061024] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4660.061026] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4660.061030] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4660.061033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4660.061036] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4660.061045] Call Trace: [ 4660.061060] [ 4660.061067] ? show_regs.cold.9+0x22/0x2f [ 4660.061076] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4660.061084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4660.061089] ? watchdog+0x40/0x40 [ 4660.061093] ? hrtimer_interrupt+0x128/0x2c0 [ 4660.061098] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4660.061105] ? apic_timer_interrupt+0xf/0x20 [ 4660.061109] [ 4660.061112] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4660.061119] do_raw_spin_lock+0x177/0x190 [ 4660.061124] _raw_spin_lock+0x21/0x30 [ 4660.061128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4660.061490] ? do_raw_spin_unlock+0x75/0x190 [ 4660.061496] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4660.061865] ? do_wait_intr+0xf0/0xf0 [ 4660.061873] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4660.062121] mdc_close+0x53b/0xd70 [mdc] [ 4660.062177] lmv_close+0x18d/0x490 [lmv] [ 4660.062210] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4660.062357] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4660.062451] ll_md_close+0x274/0x800 [lustre] [ 4660.062547] ll_file_release+0x128/0xc50 [lustre] [ 4660.064199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4660.064271] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4660.064278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4660.064281] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4660.064293] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4660.064297] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4660.064301] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4660.064304] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4660.064306] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4660.064308] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4660.064311] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4660.064313] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4660.064317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4660.064319] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4660.064328] Call Trace: [ 4660.064332] [ 4660.064336] ? show_regs.cold.9+0x22/0x2f [ 4660.064725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4660.064732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4660.064736] ? watchdog+0x40/0x40 [ 4660.064739] ? hrtimer_interrupt+0x128/0x2c0 [ 4660.064743] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4660.064750] ? apic_timer_interrupt+0xf/0x20 [ 4660.064754] [ 4660.064757] ? native_safe_halt+0x2e/0x50 [ 4660.064762] kvm_wait+0xcd/0xe0 [ 4660.064769] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4660.064776] do_raw_spin_lock+0x177/0x190 [ 4660.064781] _raw_spin_lock+0x21/0x30 [ 4660.064786] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4660.065051] obd_mod_free+0x46/0x140 [ptlrpc] [ 4660.065320] mdc_commit_open+0x91/0xb0 [mdc] [ 4660.065400] __fput+0x10d/0x440 [ 4660.065412] ____fput+0x12/0x20 [ 4660.065416] task_work_run+0xc6/0x110 [ 4660.065424] exit_to_usermode_loop+0x1dd/0x1f0 [ 4660.065430] do_syscall_64+0x3d6/0x440 [ 4660.065435] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4660.065441] RIP: 0033:0x7fa749552cd8 [ 4660.065447] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4660.065451] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4660.065456] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4660.065459] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4660.065461] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4660.065464] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4660.065369] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4660.065468] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4660.075132] ec(O) [ 4660.078266] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4660.078600] after_reply+0xe2c/0x1780 [ptlrpc] [ 4660.078790] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4660.078982] ? __next_timer_interrupt+0x160/0x160 [ 4660.078994] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4660.079197] ? do_wait_intr+0xf0/0xf0 [ 4660.079206] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4660.079411] kthread+0x1d1/0x200 [ 4660.079417] ? set_kthread_struct+0x70/0x70 [ 4660.079421] ret_from_fork+0x1f/0x30 [ 4660.899802] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4660.932004] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4660.939848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4660.948425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4660.957306] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4660.999936] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4661.010292] RAX: 0000000000007e85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4661.015375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4661.031853] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4661.036876] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4661.050766] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4661.055542] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4661.065855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4661.070789] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4661.075162] Call Trace: [ 4661.079712] [ 4661.081433] ? show_regs.cold.9+0x22/0x2f [ 4661.087788] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4661.090901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4661.092537] ? watchdog+0x40/0x40 [ 4661.098526] ? hrtimer_interrupt+0x128/0x2c0 [ 4661.100052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4661.104750] ? apic_timer_interrupt+0xf/0x20 [ 4661.108624] [ 4661.109621] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4661.113879] do_raw_spin_lock+0x177/0x190 [ 4661.118740] _raw_spin_lock+0x21/0x30 [ 4661.123426] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4661.130423] ? do_raw_spin_unlock+0x75/0x190 [ 4661.142079] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4661.144126] ? do_wait_intr+0xf0/0xf0 [ 4661.153666] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4661.160484] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4661.165268] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4661.170443] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4661.173961] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4661.182957] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4661.186489] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4661.191056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4661.192809] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4661.205021] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4661.210855] lmv_intent_open+0x338/0x1150 [lmv] [ 4661.215920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4661.221932] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4661.225468] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4661.229884] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4661.234544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4661.240880] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4661.248589] ? do_raw_spin_unlock+0x75/0x190 [ 4661.251474] ? _raw_spin_unlock+0x12/0x30 [ 4661.262462] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4661.272082] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4661.278715] ll_atomic_open+0x36a/0xd70 [lustre] [ 4661.285019] lookup_open+0x356/0xb40 [ 4661.288793] do_last+0x313/0xfc0 [ 4661.291130] ? nd_jump_root+0xe5/0x160 [ 4661.297240] ? path_init+0x437/0x520 [ 4661.301183] path_openat+0xf7/0x500 [ 4661.302549] ? do_raw_spin_unlock+0x75/0x190 [ 4661.311624] do_filp_open+0x99/0x140 [ 4661.312867] ? getname_flags+0x6e/0x330 [ 4661.314165] ? __check_object_size+0xff/0x256 [ 4661.323620] ? do_raw_spin_unlock+0x75/0x190 [ 4661.330540] ? _raw_spin_unlock+0x12/0x30 [ 4661.334110] do_sys_openat2+0x2b4/0x410 [ 4661.343785] do_sys_open+0x73/0xa0 [ 4661.345011] __x64_sys_openat+0x24/0x30 [ 4661.346289] do_syscall_64+0xc1/0x440 [ 4661.357947] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4661.367866] RIP: 0033:0x7fa749552332 [ 4661.371503] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4661.410738] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4661.429468] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4661.436461] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4661.443103] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4661.452940] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4661.457381] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4664.288047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2747s! [ 4664.291357] Showing busy workqueues and worker pools: [ 4664.293363] workqueue events: flags=0x0 [ 4664.298358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4664.300413] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4664.310178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4664.315539] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4664.323083] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.329194] pending: kfree_rcu_monitor [ 4664.331058] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4664.336670] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4664.343352] workqueue events_long: flags=0x0 [ 4664.345749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.352108] pending: lu_objects_flush [obdclass] [ 4664.354941] workqueue events_unbound: flags=0x2 [ 4664.360855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4664.367120] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4664.382176] workqueue events_power_efficient: flags=0x80 [ 4664.387241] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.392971] pending: check_lifetime [ 4664.399138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.406761] pending: neigh_periodic_work [ 4664.409657] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.415918] pending: neigh_periodic_work [ 4664.420610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.428364] pending: do_cache_clean [ 4664.431168] workqueue mm_percpu_wq: flags=0x8 [ 4664.436093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.442271] pending: vmstat_update [ 4664.447403] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.461851] pending: vmstat_update [ 4664.464053] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4664.469762] pending: vmstat_update [ 4664.471240] workqueue writeback: flags=0x4a [ 4664.477091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4664.482863] pending: wb_workfn, wb_workfn [ 4664.486069] workqueue xprtiod: flags=0xa [ 4664.494221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4664.503295] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4664.515957] workqueue ipv6_addrconf: flags=0x40008 [ 4664.522694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4664.534147] pending: addrconf_verify_work [ 4664.540120] workqueue knbd0-recv: flags=0x1a [ 4664.551155] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4664.556827] in-flight: 651:recv_work [ 4664.559975] workqueue ptlrpc_pinger: flags=0x2 [ 4664.565293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4664.575600] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4664.586299] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2747s workers=4 idle: 37 31846 28897 [ 4664.591666] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4654s workers=2 idle: 653 [ 4672.057461] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4672.063340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4672.126822] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4672.145914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4672.162739] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4672.176796] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4672.180871] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4672.205322] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4672.213082] RAX: 00000000000007d6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4672.223294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4672.237770] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4672.251444] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4672.258993] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4672.268630] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4672.274356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4672.282197] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4672.285836] Call Trace: [ 4672.287187] [ 4672.287933] ? show_regs.cold.9+0x22/0x2f [ 4672.289443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4672.294713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4672.304645] ? watchdog+0x40/0x40 [ 4672.305864] ? hrtimer_interrupt+0x128/0x2c0 [ 4672.307325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4672.318883] ? apic_timer_interrupt+0xf/0x20 [ 4672.322539] [ 4672.326660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4672.329510] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4672.332480] do_raw_spin_lock+0x177/0x190 [ 4672.344552] _raw_spin_lock+0x21/0x30 [ 4672.347041] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4672.353916] process_one_work+0x2c8/0x700 [ 4672.358405] worker_thread+0x296/0x6e0 [ 4672.360224] ? rescuer_thread+0x570/0x570 [ 4672.365676] kthread+0x1d1/0x200 [ 4672.366777] ? set_kthread_struct+0x70/0x70 [ 4672.371524] ret_from_fork+0x1f/0x30 [ 4673.750983] rcu: INFO: rcu_sched self-detected stall on CPU [ 4673.760331] rcu: 2-...!: (1507787 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 4673.774546] (t=2760061 jiffies g=258701 q=2373) [ 4673.776333] rcu: rcu_sched kthread starved for 2756856 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4673.787296] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4673.802713] rcu: RCU grace-period kthread stack dump: [ 4673.812356] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4673.823322] Call Trace: [ 4673.831683] __schedule+0x351/0xcb0 [ 4673.840185] schedule+0xc0/0x180 [ 4673.849516] schedule_timeout+0xb4/0x190 [ 4673.855548] ? do_raw_spin_unlock+0x75/0x190 [ 4673.862028] ? __next_timer_interrupt+0x160/0x160 [ 4673.869158] ? prepare_to_swait_event+0xf3/0x140 [ 4673.875516] rcu_gp_kthread+0x8dd/0x15b0 [ 4673.884250] ? force_qs_rnp+0x2c0/0x2c0 [ 4673.885710] kthread+0x1d1/0x200 [ 4673.886864] ? set_kthread_struct+0x70/0x70 [ 4673.900091] ret_from_fork+0x1f/0x30 [ 4673.910262] rcu: Stack dump where RCU GP kthread last ran: [ 4673.919271] Sending NMI from CPU 2 to CPUs 3: [ 4673.932857] NMI backtrace for cpu 3 [ 4673.932866] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4673.932868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4673.932870] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4673.932872] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4673.932874] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 [ 4673.932877] RAX: 0000000000000b8c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4673.932878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4673.932879] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4673.932881] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4673.932883] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4673.932884] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4673.932886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4673.932887] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4673.932889] Call Trace: [ 4673.932890] [ 4673.932890] ? show_regs.cold.9+0x22/0x2f [ 4673.932892] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4673.932893] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4673.932894] ? nmi_handle+0xbd/0x210 [ 4673.932895] ? default_do_nmi+0x4b/0x1b0 [ 4673.932896] ? do_nmi+0x280/0x380 [ 4673.932897] ? end_repeat_nmi+0x16/0x1f [ 4673.932899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4673.932900] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4673.932901] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4673.932903] [ 4673.932903] do_raw_spin_lock+0x177/0x190 [ 4673.932904] _raw_spin_lock+0x21/0x30 [ 4673.932905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4673.932908] ? do_raw_spin_unlock+0x75/0x190 [ 4673.932909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4673.932910] ? do_wait_intr+0xf0/0xf0 [ 4673.932911] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4673.932913] mdc_close+0x53b/0xd70 [mdc] [ 4673.932914] lmv_close+0x18d/0x490 [lmv] [ 4673.932915] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4673.932917] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4673.932918] ll_md_close+0x274/0x800 [lustre] [ 4673.932918] ll_file_release+0x128/0xc50 [lustre] [ 4673.932919] __fput+0x10d/0x440 [ 4673.932920] ____fput+0x12/0x20 [ 4673.932921] task_work_run+0xc6/0x110 [ 4673.932922] exit_to_usermode_loop+0x1dd/0x1f0 [ 4673.932923] do_syscall_64+0x3d6/0x440 [ 4673.932924] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4673.932926] RIP: 0033:0x7fa749552cd8 [ 4673.932927] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4673.932929] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4673.932932] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4673.932934] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4673.932935] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4673.932936] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4673.932938] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4673.940831] NMI backtrace for cpu 2 [ 4674.417911] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4674.424961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4674.431230] Call Trace: [ 4674.433032] [ 4674.433910] ? dump_stack+0xbb/0x10e [ 4674.439633] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4674.444103] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4674.459330] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4674.468535] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4674.483420] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4674.494108] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4674.503782] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4674.515318] ? update_process_times+0x6d/0xc0 [ 4674.523103] ? tick_sched_handle.isra.18+0x27/0x90 [ 4674.530496] ? tick_sched_timer+0x53/0xb0 [ 4674.532925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4674.547665] ? tick_sched_handle.isra.18+0x90/0x90 [ 4674.557269] ? hrtimer_interrupt+0x128/0x2c0 [ 4674.565274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4674.573998] ? apic_timer_interrupt+0xf/0x20 [ 4674.582803] [ 4674.584011] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4674.601024] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4674.607876] ? do_raw_spin_lock+0x177/0x190 [ 4674.616261] ? _raw_spin_lock+0x21/0x30 [ 4674.625232] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4674.627844] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4674.640033] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4674.647082] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4674.651254] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4674.658136] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4674.667637] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4674.674174] ? __next_timer_interrupt+0x160/0x160 [ 4674.691920] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4674.693798] ? do_wait_intr+0xf0/0xf0 [ 4674.704123] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4674.720189] ? kthread+0x1d1/0x200 [ 4674.730700] ? set_kthread_struct+0x70/0x70 [ 4674.742147] ? ret_from_fork+0x1f/0x30 [ 4692.057626] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4692.061354] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4692.083614] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4692.087621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4692.091644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4692.094123] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4692.111152] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4692.115010] RAX: 00000000000052ff RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4692.117773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4692.120831] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4692.123583] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4692.129085] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4692.132171] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4692.135398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4692.137428] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4692.140727] Call Trace: [ 4692.141830] [ 4692.142787] ? show_regs.cold.9+0x22/0x2f [ 4692.145346] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4692.147272] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4692.150658] ? watchdog+0x40/0x40 [ 4692.153136] ? hrtimer_interrupt+0x128/0x2c0 [ 4692.154972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4692.161095] ? apic_timer_interrupt+0xf/0x20 [ 4692.167203] [ 4692.167973] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4692.171648] do_raw_spin_lock+0x177/0x190 [ 4692.173394] _raw_spin_lock+0x21/0x30 [ 4692.176201] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4692.181281] ? do_raw_spin_unlock+0x75/0x190 [ 4692.182932] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4692.186292] ? do_wait_intr+0xf0/0xf0 [ 4692.188564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4692.191437] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4692.194146] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4692.200118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.203384] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4692.208014] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4692.212054] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4692.215170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.218002] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4692.222680] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4692.227328] lmv_intent_open+0x338/0x1150 [lmv] [ 4692.232922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.238009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.240416] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4692.241901] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4692.245726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.249926] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4692.254644] ? do_raw_spin_unlock+0x75/0x190 [ 4692.257016] ? _raw_spin_unlock+0x12/0x30 [ 4692.260784] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4692.263846] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4692.266209] ll_atomic_open+0x36a/0xd70 [lustre] [ 4692.268388] lookup_open+0x356/0xb40 [ 4692.269885] do_last+0x313/0xfc0 [ 4692.271357] ? nd_jump_root+0xe5/0x160 [ 4692.272680] ? path_init+0x437/0x520 [ 4692.274100] path_openat+0xf7/0x500 [ 4692.275840] ? do_raw_spin_unlock+0x75/0x190 [ 4692.278439] do_filp_open+0x99/0x140 [ 4692.279977] ? getname_flags+0x6e/0x330 [ 4692.282649] ? __check_object_size+0xff/0x256 [ 4692.284295] ? do_raw_spin_unlock+0x75/0x190 [ 4692.286638] ? _raw_spin_unlock+0x12/0x30 [ 4692.288123] do_sys_openat2+0x2b4/0x410 [ 4692.290592] do_sys_open+0x73/0xa0 [ 4692.291451] __x64_sys_openat+0x24/0x30 [ 4692.292390] do_syscall_64+0xc1/0x440 [ 4692.296399] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4692.297647] RIP: 0033:0x7fa749552332 [ 4692.298539] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4692.308504] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4692.313259] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4692.320375] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4692.331685] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4692.335651] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4692.338972] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4695.008283] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2777s! [ 4695.013068] Showing busy workqueues and worker pools: [ 4695.017768] workqueue events: flags=0x0 [ 4695.021544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4695.025108] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4695.029532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4695.032476] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4695.036681] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.039850] pending: kfree_rcu_monitor [ 4695.041865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4695.044992] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4695.049681] workqueue events_long: flags=0x0 [ 4695.051837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.054954] pending: lu_objects_flush [obdclass] [ 4695.057635] workqueue events_unbound: flags=0x2 [ 4695.060105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4695.062991] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4695.067789] workqueue events_power_efficient: flags=0x80 [ 4695.070365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.073610] pending: check_lifetime [ 4695.075436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.080649] pending: neigh_periodic_work [ 4695.082977] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.085926] pending: neigh_periodic_work [ 4695.088423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.092498] pending: do_cache_clean [ 4695.094759] workqueue mm_percpu_wq: flags=0x8 [ 4695.097611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.101544] pending: vmstat_update [ 4695.103749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.107827] pending: vmstat_update [ 4695.110497] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4695.114106] pending: vmstat_update [ 4695.121634] workqueue writeback: flags=0x4a [ 4695.126620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4695.133807] pending: wb_workfn, wb_workfn [ 4695.137414] workqueue xprtiod: flags=0xa [ 4695.140605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4695.144677] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4695.148855] workqueue ipv6_addrconf: flags=0x40008 [ 4695.150127] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4695.156452] pending: addrconf_verify_work [ 4695.158988] workqueue knbd0-recv: flags=0x1a [ 4695.161802] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4695.164867] in-flight: 651:recv_work [ 4695.167256] workqueue ptlrpc_pinger: flags=0x2 [ 4695.170670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4695.173672] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4695.176231] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2778s workers=4 idle: 37 31846 28897 [ 4695.179767] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4685s workers=2 idle: 653 [ 4700.057011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4700.059066] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4700.064131] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4700.064149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4700.064987] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4700.065705] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4700.065712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4700.066212] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4700.066292] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4700.066297] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4700.066301] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4700.066303] RDX: 00000000000048d3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4700.066305] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4700.066308] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4700.066310] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4700.066313] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4700.066316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4700.066319] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4700.066327] Call Trace: [ 4700.066333] [ 4700.066339] ? show_regs.cold.9+0x22/0x2f [ 4700.066350] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4700.066358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4700.066363] ? watchdog+0x40/0x40 [ 4700.066367] ? hrtimer_interrupt+0x128/0x2c0 [ 4700.066371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4700.066380] ? apic_timer_interrupt+0xf/0x20 [ 4700.066385] [ 4700.066390] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4700.066395] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4700.066399] do_raw_spin_lock+0x177/0x190 [ 4700.066404] _raw_spin_lock+0x21/0x30 [ 4700.066411] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4700.066847] obd_mod_free+0x46/0x140 [ptlrpc] [ 4700.067796] mdc_commit_open+0x91/0xb0 [mdc] [ 4700.067836] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4700.068928] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4700.070064] after_reply+0xe2c/0x1780 [ptlrpc] [ 4700.070380] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4700.070600] ? __next_timer_interrupt+0x160/0x160 [ 4700.070612] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4700.070852] ? do_wait_intr+0xf0/0xf0 [ 4700.070862] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4700.071076] kthread+0x1d1/0x200 [ 4700.071082] ? set_kthread_struct+0x70/0x70 [ 4700.071086] ret_from_fork+0x1f/0x30 [ 4700.083117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4700.083201] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4700.083209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4700.083212] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4700.083225] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4700.083229] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4700.083233] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4700.083235] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4700.083237] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4700.083239] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4700.083241] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4700.083245] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4700.083248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4700.083250] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4700.083258] Call Trace: [ 4700.083264] [ 4700.083269] ? show_regs.cold.9+0x22/0x2f [ 4700.083279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4700.083287] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4700.083291] ? watchdog+0x40/0x40 [ 4700.083295] ? hrtimer_interrupt+0x128/0x2c0 [ 4700.083299] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4700.083307] ? apic_timer_interrupt+0xf/0x20 [ 4700.083311] [ 4700.083314] ? native_safe_halt+0x2e/0x50 [ 4700.083317] kvm_wait+0xcd/0xe0 [ 4700.083325] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4700.083334] do_raw_spin_lock+0x177/0x190 [ 4700.083339] _raw_spin_lock+0x21/0x30 [ 4700.083344] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4700.083706] ? do_raw_spin_unlock+0x75/0x190 [ 4700.083712] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4700.102416] [ 4700.102426] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4700.102435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4700.102447] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4700.136443] ? do_wait_intr+0xf0/0xf0 [ 4700.136476] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4700.137007] mdc_close+0x53b/0xd70 [mdc] [ 4700.137050] lmv_close+0x18d/0x490 [lmv] [ 4700.137084] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4700.137223] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4700.137302] ll_md_close+0x274/0x800 [lustre] [ 4700.137385] ll_file_release+0x128/0xc50 [lustre] [ 4700.137905] __fput+0x10d/0x440 [ 4700.137915] ____fput+0x12/0x20 [ 4700.137920] task_work_run+0xc6/0x110 [ 4700.137928] exit_to_usermode_loop+0x1dd/0x1f0 [ 4700.137938] do_syscall_64+0x3d6/0x440 [ 4700.137943] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4700.137954] RIP: 0033:0x7fa749552cd8 [ 4700.137962] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4700.137966] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4700.137973] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4700.137975] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4700.137978] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4700.137981] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4700.137983] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4700.738579] [ 4700.739123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4700.741039] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4700.746996] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4700.749647] RAX: 00000000000044c7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4700.751835] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4700.754143] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4700.756744] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4700.759420] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4700.761930] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4700.764526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4700.766560] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4700.768810] Call Trace: [ 4700.769647] [ 4700.770321] ? show_regs.cold.9+0x22/0x2f [ 4700.771653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4700.773048] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4700.774784] ? watchdog+0x40/0x40 [ 4700.776013] ? hrtimer_interrupt+0x128/0x2c0 [ 4700.777725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4700.779541] ? apic_timer_interrupt+0xf/0x20 [ 4700.780973] [ 4700.781764] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4700.783864] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4700.786996] do_raw_spin_lock+0x177/0x190 [ 4700.788445] _raw_spin_lock+0x21/0x30 [ 4700.790065] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4700.792014] process_one_work+0x2c8/0x700 [ 4700.793506] worker_thread+0x296/0x6e0 [ 4700.794528] ? rescuer_thread+0x570/0x570 [ 4700.796089] kthread+0x1d1/0x200 [ 4700.797402] ? set_kthread_struct+0x70/0x70 [ 4700.800083] ret_from_fork+0x1f/0x30 [ 4720.057423] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4720.067681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4720.117860] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4720.128278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4720.135739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4720.142902] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4720.155546] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4720.158676] RAX: 0000000000002215 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4720.161531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4720.164479] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4720.167345] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4720.170223] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4720.173114] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4720.176253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4720.180559] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4720.185588] Call Trace: [ 4720.187645] [ 4720.188530] ? show_regs.cold.9+0x22/0x2f [ 4720.189818] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4720.191956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4720.193735] ? watchdog+0x40/0x40 [ 4720.195114] ? hrtimer_interrupt+0x128/0x2c0 [ 4720.196977] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4720.199654] ? apic_timer_interrupt+0xf/0x20 [ 4720.202850] [ 4720.203858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4720.207759] do_raw_spin_lock+0x177/0x190 [ 4720.209848] _raw_spin_lock+0x21/0x30 [ 4720.211641] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4720.216538] ? do_raw_spin_unlock+0x75/0x190 [ 4720.220141] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4720.222861] ? do_wait_intr+0xf0/0xf0 [ 4720.224179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4720.227482] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4720.232138] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4720.234875] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.236979] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4720.239939] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4720.242257] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4720.243801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.246182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4720.248366] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4720.250264] lmv_intent_open+0x338/0x1150 [lmv] [ 4720.258887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.261777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.263765] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4720.265598] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4720.267346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.269239] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4720.270866] ? do_raw_spin_unlock+0x75/0x190 [ 4720.272310] ? _raw_spin_unlock+0x12/0x30 [ 4720.273687] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4720.275554] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4720.279246] ll_atomic_open+0x36a/0xd70 [lustre] [ 4720.281433] lookup_open+0x356/0xb40 [ 4720.284722] do_last+0x313/0xfc0 [ 4720.286230] ? nd_jump_root+0xe5/0x160 [ 4720.287698] ? path_init+0x437/0x520 [ 4720.289652] path_openat+0xf7/0x500 [ 4720.290981] ? do_raw_spin_unlock+0x75/0x190 [ 4720.295564] do_filp_open+0x99/0x140 [ 4720.296823] ? getname_flags+0x6e/0x330 [ 4720.299243] ? __check_object_size+0xff/0x256 [ 4720.300754] ? do_raw_spin_unlock+0x75/0x190 [ 4720.308474] ? _raw_spin_unlock+0x12/0x30 [ 4720.309773] do_sys_openat2+0x2b4/0x410 [ 4720.314960] do_sys_open+0x73/0xa0 [ 4720.317151] __x64_sys_openat+0x24/0x30 [ 4720.319691] do_syscall_64+0xc1/0x440 [ 4720.320977] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4720.323904] RIP: 0033:0x7fa749552332 [ 4720.326733] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4720.337516] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4720.341090] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4720.344410] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4720.346801] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4720.353341] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4720.361797] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4725.220747] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2808s! [ 4725.231123] Showing busy workqueues and worker pools: [ 4725.235656] workqueue events: flags=0x0 [ 4725.239559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4725.247613] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4725.260154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4725.267221] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4725.276963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.283846] pending: kfree_rcu_monitor [ 4725.287151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4725.294283] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4725.301269] workqueue events_long: flags=0x0 [ 4725.303715] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.306365] pending: lu_objects_flush [obdclass] [ 4725.310814] workqueue events_unbound: flags=0x2 [ 4725.314596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4725.322497] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4725.331328] workqueue events_power_efficient: flags=0x80 [ 4725.335321] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.339679] pending: check_lifetime [ 4725.342581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.347422] pending: neigh_periodic_work [ 4725.348621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.359534] pending: neigh_periodic_work [ 4725.366213] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.371621] pending: do_cache_clean [ 4725.375904] workqueue mm_percpu_wq: flags=0x8 [ 4725.381693] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.391284] pending: vmstat_update [ 4725.395840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.404025] pending: vmstat_update [ 4725.409261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4725.411697] pending: vmstat_update [ 4725.417521] workqueue writeback: flags=0x4a [ 4725.420736] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4725.427868] pending: wb_workfn, wb_workfn [ 4725.433229] workqueue xprtiod: flags=0xa [ 4725.436856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4725.443892] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4725.454229] workqueue ipv6_addrconf: flags=0x40008 [ 4725.459423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4725.461044] pending: addrconf_verify_work [ 4725.469896] workqueue knbd0-recv: flags=0x1a [ 4725.478656] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4725.488098] in-flight: 651:recv_work [ 4725.493273] workqueue ptlrpc_pinger: flags=0x2 [ 4725.496653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4725.500084] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4725.513329] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2808s workers=4 idle: 37 31846 28897 [ 4725.527763] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4715s workers=2 idle: 653 [ 4728.057348] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4728.058136] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4728.063789] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4728.063804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4728.063892] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4728.063897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4728.063901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.063921] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4728.063926] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4728.063930] RAX: 0000000000007a90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4728.063936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4728.063939] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4728.063941] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4728.063943] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4728.063946] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4728.063949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4728.063951] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4728.063960] Call Trace: [ 4728.063965] [ 4728.063970] ? show_regs.cold.9+0x22/0x2f [ 4728.063985] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4728.063991] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4728.063996] ? watchdog+0x40/0x40 [ 4728.064000] ? hrtimer_interrupt+0x128/0x2c0 [ 4728.064006] Modules linked in: lustre(O) [ 4728.064006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4728.064014] osp(O) ofd(O) [ 4728.064020] ? apic_timer_interrupt+0xf/0x20 [ 4728.064030] [ 4728.064034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.064040] do_raw_spin_lock+0x177/0x190 [ 4728.064045] _raw_spin_lock+0x21/0x30 [ 4728.064050] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4728.067106] ? do_raw_spin_unlock+0x75/0x190 [ 4728.067117] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4728.067338] ? do_wait_intr+0xf0/0xf0 [ 4728.067352] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4728.068203] mdc_close+0x53b/0xd70 [mdc] [ 4728.068503] lmv_close+0x18d/0x490 [lmv] [ 4728.068547] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4728.068692] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4728.070975] ll_md_close+0x274/0x800 [lustre] [ 4728.071096] ll_file_release+0x128/0xc50 [lustre] [ 4728.071194] __fput+0x10d/0x440 [ 4728.071203] ____fput+0x12/0x20 [ 4728.071207] task_work_run+0xc6/0x110 [ 4728.071215] exit_to_usermode_loop+0x1dd/0x1f0 [ 4728.071233] do_syscall_64+0x3d6/0x440 [ 4728.071238] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4728.071247] RIP: 0033:0x7fa749552cd8 [ 4728.071255] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4728.071259] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4728.071266] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4728.071268] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4728.071270] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4728.071272] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4728.071274] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4728.072483] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4728.073266] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4728.073275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4728.073278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4728.073293] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4728.073297] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4728.073302] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4728.073305] RDX: 0000000000006c2a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4728.073307] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4728.073309] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4728.073311] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4728.073315] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4728.073319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4728.073321] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4728.073330] Call Trace: [ 4728.073337] [ 4728.073343] ? show_regs.cold.9+0x22/0x2f [ 4728.073359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4728.073365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4728.073370] ? watchdog+0x40/0x40 [ 4728.073376] ? hrtimer_interrupt+0x128/0x2c0 [ 4728.073380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4728.073389] ? apic_timer_interrupt+0xf/0x20 [ 4728.073393] [ 4728.073396] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4728.073401] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4728.073405] do_raw_spin_lock+0x177/0x190 [ 4728.073411] _raw_spin_lock+0x21/0x30 [ 4728.073418] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4728.074746] obd_mod_free+0x46/0x140 [ptlrpc] [ 4728.075688] mdc_commit_open+0x91/0xb0 [mdc] [ 4728.075734] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4728.076095] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4728.076301] after_reply+0xe2c/0x1780 [ptlrpc] [ 4728.076493] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4728.086394] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4728.086470] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4728.086477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4728.086484] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4728.123805] ? __next_timer_interrupt+0x160/0x160 [ 4728.123833] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4728.131636] [ 4728.134033] ? do_wait_intr+0xf0/0xf0 [ 4728.147035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.147075] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4728.147080] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4728.147092] RAX: 0000000000007884 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4728.147095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4728.147097] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4728.147099] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4728.147102] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4728.147105] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4728.147108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4728.147110] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4728.147122] Call Trace: [ 4728.147129] [ 4728.147135] ? show_regs.cold.9+0x22/0x2f [ 4728.147146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4728.147164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4728.147168] ? watchdog+0x40/0x40 [ 4728.147173] ? hrtimer_interrupt+0x128/0x2c0 [ 4728.147178] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4728.147192] ? apic_timer_interrupt+0xf/0x20 [ 4728.147197] [ 4728.147200] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.147205] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4728.156357] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4728.156781] kthread+0x1d1/0x200 [ 4728.156789] ? set_kthread_struct+0x70/0x70 [ 4728.156794] ret_from_fork+0x1f/0x30 [ 4728.835757] do_raw_spin_lock+0x177/0x190 [ 4728.841170] _raw_spin_lock+0x21/0x30 [ 4728.851817] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4728.857000] process_one_work+0x2c8/0x700 [ 4728.861647] worker_thread+0x296/0x6e0 [ 4728.865806] ? rescuer_thread+0x570/0x570 [ 4728.869995] kthread+0x1d1/0x200 [ 4728.873107] ? set_kthread_struct+0x70/0x70 [ 4728.879623] ret_from_fork+0x1f/0x30 [ 4752.055140] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4752.065284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4752.133149] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4752.147840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4752.155286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4752.159308] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4752.179694] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4752.187943] RAX: 000000000000788e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4752.199300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4752.202844] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4752.217237] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4752.228937] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4752.239407] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4752.255077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4752.264941] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4752.271637] Call Trace: [ 4752.274269] [ 4752.281293] ? show_regs.cold.9+0x22/0x2f [ 4752.282852] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4752.284571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4752.299913] ? watchdog+0x40/0x40 [ 4752.302104] ? hrtimer_interrupt+0x128/0x2c0 [ 4752.304118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4752.307043] ? apic_timer_interrupt+0xf/0x20 [ 4752.316950] [ 4752.318994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4752.323508] do_raw_spin_lock+0x177/0x190 [ 4752.324559] _raw_spin_lock+0x21/0x30 [ 4752.330841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4752.338113] ? do_raw_spin_unlock+0x75/0x190 [ 4752.341127] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4752.344325] ? do_wait_intr+0xf0/0xf0 [ 4752.346429] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4752.349137] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4752.353704] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4752.356937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4752.359244] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4752.363354] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4752.368011] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4752.373630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4752.378793] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4752.386908] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4752.391276] lmv_intent_open+0x338/0x1150 [lmv] [ 4752.394314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4752.398174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4752.401497] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4752.406270] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4752.409862] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4752.413067] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4752.419502] ? do_raw_spin_unlock+0x75/0x190 [ 4752.425626] ? _raw_spin_unlock+0x12/0x30 [ 4752.428838] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4752.432287] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4752.435650] ll_atomic_open+0x36a/0xd70 [lustre] [ 4752.437042] lookup_open+0x356/0xb40 [ 4752.438664] do_last+0x313/0xfc0 [ 4752.440892] ? nd_jump_root+0xe5/0x160 [ 4752.445188] ? path_init+0x437/0x520 [ 4752.450292] path_openat+0xf7/0x500 [ 4752.452184] ? do_raw_spin_unlock+0x75/0x190 [ 4752.454783] do_filp_open+0x99/0x140 [ 4752.456983] ? getname_flags+0x6e/0x330 [ 4752.464140] ? __check_object_size+0xff/0x256 [ 4752.465760] ? do_raw_spin_unlock+0x75/0x190 [ 4752.468050] ? _raw_spin_unlock+0x12/0x30 [ 4752.472919] do_sys_openat2+0x2b4/0x410 [ 4752.477311] do_sys_open+0x73/0xa0 [ 4752.479795] __x64_sys_openat+0x24/0x30 [ 4752.485221] do_syscall_64+0xc1/0x440 [ 4752.490173] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4752.496033] RIP: 0033:0x7fa749552332 [ 4752.503836] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4752.535017] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4752.539323] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4752.546797] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4752.555789] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4752.564715] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4752.567339] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4755.937038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2838s! [ 4755.942290] Showing busy workqueues and worker pools: [ 4755.945126] workqueue events: flags=0x0 [ 4755.956890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4755.958619] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4755.970304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4755.977550] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4755.985701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4755.990452] pending: kfree_rcu_monitor [ 4755.993956] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4755.996264] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4756.007643] workqueue events_long: flags=0x0 [ 4756.012041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4756.019461] pending: lu_objects_flush [obdclass] [ 4756.027935] workqueue events_unbound: flags=0x2 [ 4756.029454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4756.033274] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4756.038252] workqueue events_power_efficient: flags=0x80 [ 4756.041166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4756.048586] pending: check_lifetime [ 4756.049593] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4756.057088] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4756.059114] pending: neigh_periodic_work [ 4756.059185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4756.059196] pending: neigh_periodic_work [ 4756.059202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4756.059210] pending: do_cache_clean [ 4756.059224] workqueue mm_percpu_wq: flags=0x8 [ 4756.059226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4756.059233] pending: vmstat_update [ 4756.059240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4756.059247] pending: vmstat_update [ 4756.059251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4756.059257] pending: vmstat_update [ 4756.059264] workqueue writeback: flags=0x4a [ 4756.059267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4756.059275] pending: wb_workfn, wb_workfn [ 4756.059292] workqueue xprtiod: flags=0xa [ 4756.059295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4756.059300] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4756.059319] workqueue ipv6_addrconf: flags=0x40008 [ 4756.059322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4756.059328] pending: addrconf_verify_work [ 4756.059340] workqueue knbd0-recv: flags=0x1a [ 4756.059343] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4756.059350] in-flight: 651:recv_work [ 4756.059360] workqueue ptlrpc_pinger: flags=0x2 [ 4756.059362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4756.059365] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4756.059670] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4756.059677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4756.059748] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4756.059754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4756.059757] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4756.059771] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4756.059774] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4756.059778] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4756.059781] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4756.059784] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4756.059786] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4756.059788] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4756.059791] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4756.059794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4756.059796] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4756.059805] Call Trace: [ 4756.059813] [ 4756.059819] ? show_regs.cold.9+0x22/0x2f [ 4756.059829] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4756.059836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4756.059841] ? watchdog+0x40/0x40 [ 4756.059845] ? hrtimer_interrupt+0x128/0x2c0 [ 4756.059849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4756.059857] ? apic_timer_interrupt+0xf/0x20 [ 4756.059862] [ 4756.059864] ? native_safe_halt+0x2e/0x50 [ 4756.059868] kvm_wait+0xcd/0xe0 [ 4756.059877] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4756.059885] do_raw_spin_lock+0x177/0x190 [ 4756.059890] _raw_spin_lock+0x21/0x30 [ 4756.059896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4756.060153] ? do_raw_spin_unlock+0x75/0x190 [ 4756.060217] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4756.060224] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [ 4756.060159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4756.060261] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4756.060281] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4756.060285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4756.060287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4756.060294] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4756.060299] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4756.060303] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4756.060305] RDX: 0000000000007f84 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4756.060307] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4756.060309] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4756.060311] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4756.060313] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4756.060316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4756.060318] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4756.060325] Call Trace: [ 4756.060329] [ 4756.060331] ? show_regs.cold.9+0x22/0x2f [ 4756.060339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4756.060343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4756.060347] ? watchdog+0x40/0x40 [ 4756.060351] ? hrtimer_interrupt+0x128/0x2c0 [ 4756.060356] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4756.060366] ? apic_timer_interrupt+0xf/0x20 [ 4756.060365] ? do_wait_intr+0xf0/0xf0 [ 4756.060375] [ 4756.060382] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4756.060388] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4756.060392] do_raw_spin_lock+0x177/0x190 [ 4756.060396] _raw_spin_lock+0x21/0x30 [ 4756.060401] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4756.060380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4756.061125] mdc_close+0x53b/0xd70 [mdc] [ 4756.061167] lmv_close+0x18d/0x490 [lmv] [ 4756.061201] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4756.061316] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4756.062140] [ 4756.062189] pool 8: [ 4756.061794] ll_md_close+0x274/0x800 [lustre] [ 4756.062446] ll_file_release+0x128/0xc50 [lustre] [ 4756.062500] __fput+0x10d/0x440 [ 4756.062507] ____fput+0x12/0x20 [ 4756.062510] task_work_run+0xc6/0x110 [ 4756.062516] exit_to_usermode_loop+0x1dd/0x1f0 [ 4756.062522] do_syscall_64+0x3d6/0x440 [ 4756.062525] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4756.062532] RIP: 0033:0x7fa749552cd8 [ 4756.062538] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4756.062541] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4756.062545] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4756.062547] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4756.062549] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4756.062552] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4756.062555] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4756.061792] obd_mod_free+0x46/0x140 [ptlrpc] [ 4756.064859] mdc_commit_open+0x91/0xb0 [mdc] [ 4756.064903] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4756.070419] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4756.071588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4756.071676] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4756.071683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4756.071692] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4756.070692] after_reply+0xe2c/0x1780 [ptlrpc] [ 4756.072123] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4756.072969] [ 4756.072973] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4756.073032] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4756.073038] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4756.073034] ? __next_timer_interrupt+0x160/0x160 [ 4756.073043] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4756.073046] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4756.073049] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4756.073051] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4756.073053] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4756.073056] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4756.073059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4756.073061] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4756.073047] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4756.073195] Call Trace: [ 4756.073202] [ 4756.073205] ? show_regs.cold.9+0x22/0x2f [ 4756.073216] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4756.073236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4756.073240] ? watchdog+0x40/0x40 [ 4756.073244] ? hrtimer_interrupt+0x128/0x2c0 [ 4756.073249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4756.073257] ? apic_timer_interrupt+0xf/0x20 [ 4756.073261] [ 4756.073264] ? native_safe_halt+0x2e/0x50 [ 4756.073267] kvm_wait+0xcd/0xe0 [ 4756.073275] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4756.073283] do_raw_spin_lock+0x177/0x190 [ 4756.073287] _raw_spin_lock+0x21/0x30 [ 4756.073291] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4756.073580] process_one_work+0x2c8/0x700 [ 4756.073587] worker_thread+0x296/0x6e0 [ 4756.073592] ? rescuer_thread+0x570/0x570 [ 4756.073596] kthread+0x1d1/0x200 [ 4756.073601] ? set_kthread_struct+0x70/0x70 [ 4756.073605] ret_from_fork+0x1f/0x30 [ 4756.073839] ? do_wait_intr+0xf0/0xf0 [ 4756.073847] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4756.074068] kthread+0x1d1/0x200 [ 4756.074072] ? set_kthread_struct+0x70/0x70 [ 4756.074076] ret_from_fork+0x1f/0x30 [ 4757.024037] cpus=0-3 flags=0x4 nice=0 hung=2839s workers=4 idle: 37 31846 28897 [ 4757.029315] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4746s workers=2 idle: 653 [ 4784.054318] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4784.058006] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4784.058021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4784.058034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4784.058105] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4784.058111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4784.058114] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4784.058128] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4784.058132] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4784.058137] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4784.058139] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4784.058141] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4784.058143] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4784.058145] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4784.058148] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4784.058151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4784.058153] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4784.058162] Call Trace: [ 4784.058180] [ 4784.058187] ? show_regs.cold.9+0x22/0x2f [ 4784.058197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4784.058203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4784.058208] ? watchdog+0x40/0x40 [ 4784.058212] ? hrtimer_interrupt+0x128/0x2c0 [ 4784.058216] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4784.058223] ? apic_timer_interrupt+0xf/0x20 [ 4784.058228] [ 4784.058231] ? native_safe_halt+0x2e/0x50 [ 4784.058235] kvm_wait+0xcd/0xe0 [ 4784.058244] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4784.058252] do_raw_spin_lock+0x177/0x190 [ 4784.058256] _raw_spin_lock+0x21/0x30 [ 4784.058261] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4784.058984] obd_mod_free+0x46/0x140 [ptlrpc] [ 4784.059204] mdc_commit_open+0x91/0xb0 [mdc] [ 4784.059399] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4784.059406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4784.059461] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4784.059465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4784.059467] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.059476] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4784.059480] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4784.059484] RAX: 00000000000005cb RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4784.059486] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4784.059488] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4784.059244] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4784.059815] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4784.059490] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4784.060064] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4784.060067] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4784.060069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4784.060072] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4784.060080] Call Trace: [ 4784.060085] [ 4784.060087] ? show_regs.cold.9+0x22/0x2f [ 4784.060097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4784.060102] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4784.060107] ? watchdog+0x40/0x40 [ 4784.060111] ? hrtimer_interrupt+0x128/0x2c0 [ 4784.060114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4784.060121] ? apic_timer_interrupt+0xf/0x20 [ 4784.060125] [ 4784.060128] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.060134] do_raw_spin_lock+0x177/0x190 [ 4784.060138] _raw_spin_lock+0x21/0x30 [ 4784.060142] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4784.060713] ? do_raw_spin_unlock+0x75/0x190 [ 4784.060820] Modules linked in: [ 4784.060720] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4784.060823] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4784.060887] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4784.060893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4784.060896] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.060906] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4784.060910] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4784.060915] RAX: 0000000000003293 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4784.060917] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4784.060920] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4784.060922] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4784.060924] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4784.060927] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4784.060929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4784.060931] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4784.060939] Call Trace: [ 4784.060943] [ 4784.060946] ? show_regs.cold.9+0x22/0x2f [ 4784.060955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4784.060962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4784.060966] ? watchdog+0x40/0x40 [ 4784.060971] ? hrtimer_interrupt+0x128/0x2c0 [ 4784.060975] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4784.060981] ? apic_timer_interrupt+0xf/0x20 [ 4784.060986] [ 4784.060989] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.060996] do_raw_spin_lock+0x177/0x190 [ 4784.061000] _raw_spin_lock+0x21/0x30 [ 4784.060471] after_reply+0xe2c/0x1780 [ptlrpc] [ 4784.061388] ? do_wait_intr+0xf0/0xf0 [ 4784.061395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4784.061829] mdc_close+0x53b/0xd70 [mdc] [ 4784.061867] lmv_close+0x18d/0x490 [lmv] [ 4784.062008] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4784.062236] ? __next_timer_interrupt+0x160/0x160 [ 4784.061896] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4784.062245] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4784.064252] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4784.064413] ll_md_close+0x274/0x800 [lustre] [ 4784.061005] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4784.066100] ? do_raw_spin_unlock+0x75/0x190 [ 4784.066111] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4784.068070] ? do_wait_intr+0xf0/0xf0 [ 4784.068082] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4784.068311] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4784.068539] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4784.068765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.068875] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4784.068913] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4784.068942] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4784.068971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.069159] ll_file_release+0x128/0xc50 [lustre] [ 4784.069265] __fput+0x10d/0x440 [ 4784.069274] ____fput+0x12/0x20 [ 4784.069279] task_work_run+0xc6/0x110 [ 4784.069287] exit_to_usermode_loop+0x1dd/0x1f0 [ 4784.069294] do_syscall_64+0x3d6/0x440 [ 4784.069299] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4784.069381] RIP: 0033:0x7fa749552cd8 [ 4784.069388] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4784.069392] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4784.069398] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4784.069400] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4784.069402] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4784.069404] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4784.069406] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4784.070614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 [ 4784.069058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4784.071390] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4784.071426] lmv_intent_open+0x338/0x1150 [lmv] [ 4784.071458] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.071570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.071653] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4784.071675] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4784.071758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.071840] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4784.071925] ? do_raw_spin_unlock+0x75/0x190 [ 4784.071933] ? _raw_spin_unlock+0x12/0x30 [ 4784.071940] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4784.070663] pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror [ 4784.071966] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4784.071974] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4784.071983] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4784.071988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4784.071986] ll_atomic_open+0x36a/0xd70 [lustre] [ 4784.071994] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4784.072362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.072370] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4784.072374] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4784.072378] RAX: 0000000000001696 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4784.072381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4784.072383] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4784.072385] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4784.072387] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4784.072390] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4784.072393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4784.072395] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4784.072402] Call Trace: [ 4784.072407] [ 4784.072410] ? show_regs.cold.9+0x22/0x2f [ 4784.072420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4784.072426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4784.072430] ? watchdog+0x40/0x40 [ 4784.072434] ? hrtimer_interrupt+0x128/0x2c0 [ 4784.072438] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4784.072444] ? apic_timer_interrupt+0xf/0x20 [ 4784.072449] [ 4784.072451] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.072455] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4784.072460] do_raw_spin_lock+0x177/0x190 [ 4784.072464] _raw_spin_lock+0x21/0x30 [ 4784.072469] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4784.075342] process_one_work+0x2c8/0x700 [ 4784.075353] worker_thread+0x296/0x6e0 [ 4784.075358] ? rescuer_thread+0x570/0x570 [ 4784.075362] kthread+0x1d1/0x200 [ 4784.075366] ? set_kthread_struct+0x70/0x70 [ 4784.075370] ret_from_fork+0x1f/0x30 [ 4784.079192] ? do_wait_intr+0xf0/0xf0 [ 4784.079213] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4784.123862] lookup_open+0x356/0xb40 [ 4784.135795] kthread+0x1d1/0x200 [ 4784.135814] ? set_kthread_struct+0x70/0x70 [ 4784.135819] ret_from_fork+0x1f/0x30 [ 4785.242766] do_last+0x313/0xfc0 [ 4785.247323] ? nd_jump_root+0xe5/0x160 [ 4785.252603] ? path_init+0x437/0x520 [ 4785.257336] path_openat+0xf7/0x500 [ 4785.264218] ? do_raw_spin_unlock+0x75/0x190 [ 4785.274314] do_filp_open+0x99/0x140 [ 4785.277501] ? getname_flags+0x6e/0x330 [ 4785.281842] ? __check_object_size+0xff/0x256 [ 4785.286230] ? do_raw_spin_unlock+0x75/0x190 [ 4785.289779] ? _raw_spin_unlock+0x12/0x30 [ 4785.293187] do_sys_openat2+0x2b4/0x410 [ 4785.295807] do_sys_open+0x73/0xa0 [ 4785.298502] __x64_sys_openat+0x24/0x30 [ 4785.300386] do_syscall_64+0xc1/0x440 [ 4785.302409] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4785.305242] RIP: 0033:0x7fa749552332 [ 4785.307700] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4785.326223] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4785.339055] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4785.347706] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4785.352392] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4785.359981] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4785.371542] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4787.169081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2870s! [ 4787.188640] Showing busy workqueues and worker pools: [ 4787.190491] workqueue events: flags=0x0 [ 4787.195664] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4787.209277] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4787.225870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4787.239680] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4787.253934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.265164] pending: kfree_rcu_monitor [ 4787.272242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4787.279582] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4787.286140] workqueue events_long: flags=0x0 [ 4787.289251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.298949] pending: lu_objects_flush [obdclass] [ 4787.307144] workqueue events_unbound: flags=0x2 [ 4787.310576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4787.314545] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4787.324093] workqueue events_power_efficient: flags=0x80 [ 4787.337902] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.357427] pending: check_lifetime [ 4787.365256] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.378498] pending: neigh_periodic_work [ 4787.382167] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.387198] pending: neigh_periodic_work [ 4787.390623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.397037] pending: do_cache_clean [ 4787.400032] workqueue mm_percpu_wq: flags=0x8 [ 4787.402561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.406964] pending: vmstat_update [ 4787.409626] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.419797] pending: vmstat_update [ 4787.425777] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4787.441387] pending: vmstat_update [ 4787.448101] workqueue writeback: flags=0x4a [ 4787.455178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4787.468486] pending: wb_workfn, wb_workfn [ 4787.474929] workqueue xprtiod: flags=0xa [ 4787.480159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4787.486893] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4787.494962] workqueue ipv6_addrconf: flags=0x40008 [ 4787.505079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4787.517606] pending: addrconf_verify_work [ 4787.528361] workqueue knbd0-recv: flags=0x1a [ 4787.529887] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4787.538125] in-flight: 651:recv_work [ 4787.544486] workqueue ptlrpc_pinger: flags=0x2 [ 4787.551191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4787.560224] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4787.566746] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2870s workers=4 idle: 37 31846 28897 [ 4787.581980] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4777s workers=2 idle: 653 [ 4812.054129] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4812.058668] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4812.058863] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4812.058870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4812.059109] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4812.059115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4812.059118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4812.059134] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4812.059231] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4812.059244] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4812.059309] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4812.059314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4812.059316] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4812.059330] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4812.059334] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4812.059339] RAX: 0000000000007e0e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4812.059342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4812.059344] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4812.059347] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4812.059350] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4812.059354] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4812.059357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4812.059360] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4812.059370] Call Trace: [ 4812.059382] [ 4812.059383] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 [ 4812.059395] ? show_regs.cold.9+0x22/0x2f [ 4812.059391] ORIG_RAX: ffffffffffffff13 [ 4812.059412] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4812.059419] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4812.059428] RDX: 0000000000007edc RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4812.059427] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4812.059434] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4812.059445] ? watchdog+0x40/0x40 [ 4812.059446] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4812.059458] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4812.059462] ? hrtimer_interrupt+0x128/0x2c0 [ 4812.059465] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4812.059476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4812.059480] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4812.059474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4812.059488] Call Trace: [ 4812.059490] ? apic_timer_interrupt+0xf/0x20 [ 4812.059498] [ 4812.059500] [ 4812.059511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4812.059510] ? show_regs.cold.9+0x22/0x2f [ 4812.059524] do_raw_spin_lock+0x177/0x190 [ 4812.059530] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4812.059542] _raw_spin_lock+0x21/0x30 [ 4812.059550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4812.059561] ? watchdog+0x40/0x40 [ 4812.059566] ? hrtimer_interrupt+0x128/0x2c0 [ 4812.059570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4812.059579] ? apic_timer_interrupt+0xf/0x20 [ 4812.059584] [ 4812.059587] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4812.059592] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4812.059597] do_raw_spin_lock+0x177/0x190 [ 4812.059601] _raw_spin_lock+0x21/0x30 [ 4812.059557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4812.059949] ? do_raw_spin_unlock+0x75/0x190 [ 4812.060032] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4812.060248] ? do_wait_intr+0xf0/0xf0 [ 4812.060260] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4812.060386] Modules linked in: lustre(O) osp(O) [ 4812.060459] mdc_close+0x53b/0xd70 [mdc] [ 4812.060604] lmv_close+0x18d/0x490 [lmv] [ 4812.060711] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 4812.060664] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4812.060752] sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4812.060849] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4812.060855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4812.060823] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4812.060858] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4812.060870] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4812.060874] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4812.060878] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4812.060880] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4812.060882] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4812.060884] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4812.060886] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4812.060889] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4812.060892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4812.060894] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4812.060900] Call Trace: [ 4812.060904] [ 4812.060913] ? show_regs.cold.9+0x22/0x2f [ 4812.060925] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4812.060930] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4812.060933] ? watchdog+0x40/0x40 [ 4812.060937] ? hrtimer_interrupt+0x128/0x2c0 [ 4812.060940] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4812.060946] ? apic_timer_interrupt+0xf/0x20 [ 4812.060951] [ 4812.060954] ? native_safe_halt+0x2e/0x50 [ 4812.060960] kvm_wait+0xcd/0xe0 [ 4812.060968] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4812.060975] do_raw_spin_lock+0x177/0x190 [ 4812.060979] _raw_spin_lock+0x21/0x30 [ 4812.060983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4812.059608] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4812.061878] obd_mod_free+0x46/0x140 [ptlrpc] [ 4812.062084] ? do_raw_spin_unlock+0x75/0x190 [ 4812.062092] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4812.062504] mdc_commit_open+0x91/0xb0 [mdc] [ 4812.060914] ll_md_close+0x274/0x800 [lustre] [ 4812.063514] ll_file_release+0x128/0xc50 [lustre] [ 4812.063595] __fput+0x10d/0x440 [ 4812.063607] ____fput+0x12/0x20 [ 4812.063612] task_work_run+0xc6/0x110 [ 4812.063624] exit_to_usermode_loop+0x1dd/0x1f0 [ 4812.063634] do_syscall_64+0x3d6/0x440 [ 4812.063639] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4812.063650] RIP: 0033:0x7fa749552cd8 [ 4812.063661] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4812.063666] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4812.063672] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4812.063674] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4812.063676] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4812.063691] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4812.063693] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4812.062547] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4812.065808] ? do_wait_intr+0xf0/0xf0 [ 4812.065820] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4812.066314] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4812.066531] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4812.066884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.066999] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4812.067261] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4812.067289] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4812.067432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.067521] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4812.068548] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4812.068584] lmv_intent_open+0x338/0x1150 [lmv] [ 4812.068620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.069435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.069576] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4812.069604] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4812.069804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.069885] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4812.069972] ? do_raw_spin_unlock+0x75/0x190 [ 4812.069982] ? _raw_spin_unlock+0x12/0x30 [ 4812.069988] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4812.070033] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4812.070054] ll_atomic_open+0x36a/0xd70 [lustre] [ 4812.071510] lookup_open+0x356/0xb40 [ 4812.071531] do_last+0x313/0xfc0 [ 4812.071536] ? nd_jump_root+0xe5/0x160 [ 4812.071541] ? path_init+0x437/0x520 [ 4812.071545] path_openat+0xf7/0x500 [ 4812.071550] ? do_raw_spin_unlock+0x75/0x190 [ 4812.071556] do_filp_open+0x99/0x140 [ 4812.071563] ? getname_flags+0x6e/0x330 [ 4812.071568] ? __check_object_size+0xff/0x256 [ 4812.071575] ? do_raw_spin_unlock+0x75/0x190 [ 4812.071579] ? _raw_spin_unlock+0x12/0x30 [ 4812.071586] do_sys_openat2+0x2b4/0x410 [ 4812.071590] do_sys_open+0x73/0xa0 [ 4812.071595] __x64_sys_openat+0x24/0x30 [ 4812.071598] do_syscall_64+0xc1/0x440 [ 4812.071605] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4812.071612] RIP: 0033:0x7fa749552332 [ 4812.071622] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4812.071627] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4812.071644] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4812.071647] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4812.071649] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4812.071651] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4812.071653] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4812.071905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4812.071973] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4812.071979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4812.071984] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4812.072274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4812.072282] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4812.072286] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4812.072290] RAX: 0000000000007e07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4812.072293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4812.072295] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4812.072297] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4812.072300] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4812.072303] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4812.072306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4812.072308] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4812.072318] Call Trace: [ 4812.072322] [ 4812.072325] ? show_regs.cold.9+0x22/0x2f [ 4812.072333] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4812.072339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4812.072343] ? watchdog+0x40/0x40 [ 4812.072347] ? hrtimer_interrupt+0x128/0x2c0 [ 4812.072350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4812.072356] ? apic_timer_interrupt+0xf/0x20 [ 4812.072360] [ 4812.072362] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4812.072367] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4812.072372] do_raw_spin_lock+0x177/0x190 [ 4812.072376] _raw_spin_lock+0x21/0x30 [ 4812.072380] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4812.082731] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4812.083207] after_reply+0xe2c/0x1780 [ptlrpc] [ 4812.083488] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4812.083688] ? __next_timer_interrupt+0x160/0x160 [ 4812.083700] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4812.086433] ? do_wait_intr+0xf0/0xf0 [ 4812.086447] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4812.143322] process_one_work+0x2c8/0x700 [ 4812.143344] worker_thread+0x296/0x6e0 [ 4812.143349] ? rescuer_thread+0x570/0x570 [ 4812.143353] kthread+0x1d1/0x200 [ 4812.143357] ? set_kthread_struct+0x70/0x70 [ 4812.143360] ret_from_fork+0x1f/0x30 [ 4813.430945] kthread+0x1d1/0x200 [ 4813.433030] ? set_kthread_struct+0x70/0x70 [ 4813.438124] ret_from_fork+0x1f/0x30 [ 4817.894369] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2900s! [ 4817.902225] Showing busy workqueues and worker pools: [ 4817.906988] workqueue events: flags=0x0 [ 4817.908337] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4817.917480] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4817.935144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4817.945255] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4817.953023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4817.957279] pending: kfree_rcu_monitor [ 4817.963202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4817.969812] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4817.978960] workqueue events_long: flags=0x0 [ 4817.983740] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4817.986477] pending: lu_objects_flush [obdclass] [ 4817.992704] workqueue events_unbound: flags=0x2 [ 4817.996357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4818.004913] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4818.018500] workqueue events_power_efficient: flags=0x80 [ 4818.022225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4818.028000] pending: check_lifetime [ 4818.035190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4818.039888] pending: neigh_periodic_work [ 4818.044030] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4818.052864] pending: neigh_periodic_work [ 4818.058460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4818.067454] pending: do_cache_clean [ 4818.070156] workqueue mm_percpu_wq: flags=0x8 [ 4818.075087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4818.084157] pending: vmstat_update [ 4818.087284] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4818.095453] pending: vmstat_update [ 4818.102141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4818.104176] pending: vmstat_update [ 4818.107264] workqueue writeback: flags=0x4a [ 4818.108669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4818.116275] pending: wb_workfn, wb_workfn [ 4818.122544] workqueue xprtiod: flags=0xa [ 4818.127581] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4818.134914] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4818.141811] workqueue ipv6_addrconf: flags=0x40008 [ 4818.149940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4818.158910] pending: addrconf_verify_work [ 4818.160306] workqueue knbd0-recv: flags=0x1a [ 4818.164742] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4818.171713] in-flight: 651:recv_work [ 4818.174036] workqueue ptlrpc_pinger: flags=0x2 [ 4818.178971] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4818.181518] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4818.189521] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2901s workers=4 idle: 37 31846 28897 [ 4818.199785] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4808s workers=2 idle: 653 [ 4840.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4840.058042] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4840.059051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 4840.059165] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4840.059170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 4840.059296] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4840.059365] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4840.059370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4840.059377] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4840.059654] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4840.059664] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4840.059668] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4840.059673] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4840.059675] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4840.059677] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4840.059680] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4840.059682] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4840.059686] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4840.059689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4840.059692] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4840.059701] Call Trace: [ 4840.059721] [ 4840.059728] ? show_regs.cold.9+0x22/0x2f [ 4840.059739] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4840.059745] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4840.059750] ? watchdog+0x40/0x40 [ 4840.059755] ? hrtimer_interrupt+0x128/0x2c0 [ 4840.059759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4840.059765] ? apic_timer_interrupt+0xf/0x20 [ 4840.059770] [ 4840.059774] ? native_safe_halt+0x2e/0x50 [ 4840.059778] kvm_wait+0xcd/0xe0 [ 4840.059786] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4840.059793] do_raw_spin_lock+0x177/0x190 [ 4840.059798] _raw_spin_lock+0x21/0x30 [ 4840.059803] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4840.060028] process_one_work+0x2c8/0x700 [ 4840.060036] worker_thread+0x296/0x6e0 [ 4840.060040] ? rescuer_thread+0x570/0x570 [ 4840.060044] kthread+0x1d1/0x200 [ 4840.060049] ? set_kthread_struct+0x70/0x70 [ 4840.060052] ret_from_fork+0x1f/0x30 [ 4840.062594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4840.062668] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4840.062675] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4840.062677] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4840.062689] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4840.062694] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4840.062698] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4840.062701] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4840.062703] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4840.062705] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4840.062707] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4840.062710] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4840.062713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4840.062714] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4840.062723] Call Trace: [ 4840.062728] [ 4840.062733] ? show_regs.cold.9+0x22/0x2f [ 4840.062744] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4840.067457] nfsv4 [ 4840.076630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4840.076646] ? watchdog+0x40/0x40 [ 4840.076652] ? hrtimer_interrupt+0x128/0x2c0 [ 4840.076657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4840.076665] ? apic_timer_interrupt+0xf/0x20 [ 4840.076669] [ 4840.076675] ? native_safe_halt+0x2e/0x50 [ 4840.076679] kvm_wait+0xcd/0xe0 [ 4840.094001] dns_resolver [ 4840.143209] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4840.159113] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4840.159151] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4840.159157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4840.159160] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4840.159174] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4840.159178] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4840.159182] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4840.159184] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4840.159196] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4840.159198] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4840.159201] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4840.159203] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4840.159206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4840.159208] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4840.159215] Call Trace: [ 4840.159221] [ 4840.159227] ? show_regs.cold.9+0x22/0x2f [ 4840.159236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4840.159243] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4840.159247] ? watchdog+0x40/0x40 [ 4840.159251] ? hrtimer_interrupt+0x128/0x2c0 [ 4840.159255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4840.159262] ? apic_timer_interrupt+0xf/0x20 [ 4840.159266] [ 4840.159269] ? native_safe_halt+0x2e/0x50 [ 4840.159273] kvm_wait+0xcd/0xe0 [ 4840.159281] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4840.159291] do_raw_spin_lock+0x177/0x190 [ 4840.159295] _raw_spin_lock+0x21/0x30 [ 4840.159300] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4840.159567] ? do_raw_spin_unlock+0x75/0x190 [ 4840.159573] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4840.159780] ? do_wait_intr+0xf0/0xf0 [ 4840.159788] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4840.159983] mdc_close+0x53b/0xd70 [mdc] [ 4840.160024] lmv_close+0x18d/0x490 [lmv] [ 4840.160054] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4840.160169] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4840.160269] ll_md_close+0x274/0x800 [lustre] [ 4840.160355] ll_file_release+0x128/0xc50 [lustre] [ 4840.160440] __fput+0x10d/0x440 [ 4840.160450] ____fput+0x12/0x20 [ 4840.160454] task_work_run+0xc6/0x110 [ 4840.160462] exit_to_usermode_loop+0x1dd/0x1f0 [ 4840.160470] do_syscall_64+0x3d6/0x440 [ 4840.160474] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4840.160482] RIP: 0033:0x7fa749552cd8 [ 4840.160538] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4840.160543] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4840.160548] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4840.160549] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4840.160551] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4840.160554] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4840.160556] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4840.864432] do_raw_spin_lock+0x177/0x190 [ 4840.868019] _raw_spin_lock+0x21/0x30 [ 4840.870350] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4840.875079] obd_mod_free+0x46/0x140 [ptlrpc] [ 4840.879994] mdc_commit_open+0x91/0xb0 [mdc] [ 4840.883046] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4840.889006] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4840.893916] after_reply+0xe2c/0x1780 [ptlrpc] [ 4840.904419] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4840.907254] ? __next_timer_interrupt+0x160/0x160 [ 4840.911169] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4840.915369] ? do_wait_intr+0xf0/0xf0 [ 4840.917161] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4840.924648] kthread+0x1d1/0x200 [ 4840.925743] ? set_kthread_struct+0x70/0x70 [ 4840.927114] ret_from_fork+0x1f/0x30 [ 4844.056304] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4844.069357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4844.109493] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4844.124408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4844.132334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4844.140253] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4844.158461] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4844.170439] RAX: 0000000000005947 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4844.177822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4844.184791] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4844.191834] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4844.199343] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4844.204310] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4844.208493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4844.213496] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4844.220428] Call Trace: [ 4844.222345] [ 4844.225410] ? show_regs.cold.9+0x22/0x2f [ 4844.226999] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4844.233137] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4844.235278] ? watchdog+0x40/0x40 [ 4844.239939] ? hrtimer_interrupt+0x128/0x2c0 [ 4844.242580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4844.245338] ? apic_timer_interrupt+0xf/0x20 [ 4844.252753] [ 4844.255892] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4844.261058] do_raw_spin_lock+0x177/0x190 [ 4844.263555] _raw_spin_lock+0x21/0x30 [ 4844.267563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4844.273450] ? do_raw_spin_unlock+0x75/0x190 [ 4844.276021] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4844.280078] ? do_wait_intr+0xf0/0xf0 [ 4844.284975] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4844.288413] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4844.295323] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4844.298293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4844.303812] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4844.308181] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4844.311414] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4844.313909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4844.317008] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4844.320370] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4844.323933] lmv_intent_open+0x338/0x1150 [lmv] [ 4844.326345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4844.329111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4844.332760] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4844.335370] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4844.338512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4844.340819] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4844.344016] ? do_raw_spin_unlock+0x75/0x190 [ 4844.346205] ? _raw_spin_unlock+0x12/0x30 [ 4844.348141] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4844.350651] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4844.352406] ll_atomic_open+0x36a/0xd70 [lustre] [ 4844.355013] lookup_open+0x356/0xb40 [ 4844.356771] do_last+0x313/0xfc0 [ 4844.358538] ? nd_jump_root+0xe5/0x160 [ 4844.360394] ? path_init+0x437/0x520 [ 4844.362350] path_openat+0xf7/0x500 [ 4844.363982] ? do_raw_spin_unlock+0x75/0x190 [ 4844.365913] do_filp_open+0x99/0x140 [ 4844.367473] ? getname_flags+0x6e/0x330 [ 4844.369060] ? __check_object_size+0xff/0x256 [ 4844.370765] ? do_raw_spin_unlock+0x75/0x190 [ 4844.372548] ? _raw_spin_unlock+0x12/0x30 [ 4844.374474] do_sys_openat2+0x2b4/0x410 [ 4844.376150] do_sys_open+0x73/0xa0 [ 4844.377918] __x64_sys_openat+0x24/0x30 [ 4844.379715] do_syscall_64+0xc1/0x440 [ 4844.381790] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4844.384405] RIP: 0033:0x7fa749552332 [ 4844.386088] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4844.393525] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4844.396661] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4844.399832] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4844.403051] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4844.406022] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4844.409087] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4848.610813] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2931s! [ 4848.618636] Showing busy workqueues and worker pools: [ 4848.625865] workqueue events: flags=0x0 [ 4848.628022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4848.635471] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4848.665072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4848.666826] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4848.670739] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.673863] pending: kfree_rcu_monitor [ 4848.678284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4848.681060] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4848.685428] workqueue events_long: flags=0x0 [ 4848.687153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.692090] pending: lu_objects_flush [obdclass] [ 4848.696364] workqueue events_unbound: flags=0x2 [ 4848.699012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4848.703215] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4848.707591] workqueue events_power_efficient: flags=0x80 [ 4848.709965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.713023] pending: check_lifetime [ 4848.715882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.719979] pending: neigh_periodic_work [ 4848.722245] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.729149] pending: neigh_periodic_work [ 4848.733321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.736540] pending: do_cache_clean [ 4848.737792] workqueue mm_percpu_wq: flags=0x8 [ 4848.741385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.746076] pending: vmstat_update [ 4848.748035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.753973] pending: vmstat_update [ 4848.755372] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4848.758286] pending: vmstat_update [ 4848.759856] workqueue writeback: flags=0x4a [ 4848.761138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4848.763571] pending: wb_workfn, wb_workfn [ 4848.765738] workqueue xprtiod: flags=0xa [ 4848.769401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4848.772371] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4848.778868] workqueue ipv6_addrconf: flags=0x40008 [ 4848.780876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4848.783242] pending: addrconf_verify_work [ 4848.790539] workqueue knbd0-recv: flags=0x1a [ 4848.796552] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4848.803370] in-flight: 651:recv_work [ 4848.805068] workqueue ptlrpc_pinger: flags=0x2 [ 4848.807124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4848.815394] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4848.818284] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2931s workers=4 idle: 37 31846 28897 [ 4848.823994] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4838s workers=2 idle: 653 [ 4853.753231] rcu: INFO: rcu_sched self-detected stall on CPU [ 4853.755920] rcu: 2-...!: (1601765 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 4853.766195] (t=2940064 jiffies g=258701 q=2429) [ 4853.770984] rcu: rcu_sched kthread starved for 2936859 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4853.777187] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4853.785803] rcu: RCU grace-period kthread stack dump: [ 4853.790266] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4853.800511] Call Trace: [ 4853.801466] __schedule+0x351/0xcb0 [ 4853.802692] schedule+0xc0/0x180 [ 4853.804210] schedule_timeout+0xb4/0x190 [ 4853.807932] ? do_raw_spin_unlock+0x75/0x190 [ 4853.810869] ? __next_timer_interrupt+0x160/0x160 [ 4853.817765] ? prepare_to_swait_event+0xf3/0x140 [ 4853.822417] rcu_gp_kthread+0x8dd/0x15b0 [ 4853.826800] ? force_qs_rnp+0x2c0/0x2c0 [ 4853.830478] kthread+0x1d1/0x200 [ 4853.836409] ? set_kthread_struct+0x70/0x70 [ 4853.841346] ret_from_fork+0x1f/0x30 [ 4853.842563] rcu: Stack dump where RCU GP kthread last ran: [ 4853.851905] NMI backtrace for cpu 2 [ 4853.855389] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4853.867939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4853.877584] Call Trace: [ 4853.878919] [ 4853.879777] ? dump_stack+0xbb/0x10e [ 4853.884281] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4853.890429] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4853.895443] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4853.901509] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4853.909570] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4853.915842] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4853.923984] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 4853.930332] ? update_process_times+0x6d/0xc0 [ 4853.936490] ? tick_sched_handle.isra.18+0x27/0x90 [ 4853.942818] ? tick_sched_timer+0x53/0xb0 [ 4853.947425] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4853.951943] ? tick_sched_handle.isra.18+0x90/0x90 [ 4853.961888] ? hrtimer_interrupt+0x128/0x2c0 [ 4853.963585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4853.972804] ? apic_timer_interrupt+0xf/0x20 [ 4853.975390] [ 4853.976446] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4853.980672] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4853.987321] ? do_raw_spin_lock+0x177/0x190 [ 4853.990557] ? _raw_spin_lock+0x21/0x30 [ 4853.995396] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4854.001538] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4854.006558] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4854.009399] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4854.012967] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4854.019409] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4854.025437] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4854.032413] ? __next_timer_interrupt+0x160/0x160 [ 4854.042091] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4854.043892] ? do_wait_intr+0xf0/0xf0 [ 4854.048201] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4854.054171] ? kthread+0x1d1/0x200 [ 4854.057999] ? set_kthread_struct+0x70/0x70 [ 4854.062480] ? ret_from_fork+0x1f/0x30 [ 4854.066398] NMI backtrace for cpu 2 [ 4854.071801] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4854.086911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4854.097211] Call Trace: [ 4854.098047] [ 4854.098732] ? dump_stack+0xbb/0x10e [ 4854.102262] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4854.108461] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4854.112423] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4854.118192] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4854.124879] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4854.130494] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4854.135752] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4854.140876] ? update_process_times+0x6d/0xc0 [ 4854.144366] ? tick_sched_handle.isra.18+0x27/0x90 [ 4854.149647] ? tick_sched_timer+0x53/0xb0 [ 4854.156257] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4854.161230] ? tick_sched_handle.isra.18+0x90/0x90 [ 4854.168554] ? hrtimer_interrupt+0x128/0x2c0 [ 4854.172390] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4854.179311] ? apic_timer_interrupt+0xf/0x20 [ 4854.184636] [ 4854.187847] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4854.195203] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4854.200011] ? do_raw_spin_lock+0x177/0x190 [ 4854.205300] ? _raw_spin_lock+0x21/0x30 [ 4854.206680] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4854.213794] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4854.220330] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4854.227103] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4854.234242] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4854.240409] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4854.243958] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4854.249920] ? __next_timer_interrupt+0x160/0x160 [ 4854.256298] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4854.263366] ? do_wait_intr+0xf0/0xf0 [ 4854.266805] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4854.275180] ? kthread+0x1d1/0x200 [ 4854.280161] ? set_kthread_struct+0x70/0x70 [ 4854.285959] ? ret_from_fork+0x1f/0x30 [ 4868.057028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4868.059818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4868.062737] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4868.097276] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4868.097288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4868.097294] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4868.100447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4868.100527] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4868.100534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4868.100537] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4868.100549] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4868.100553] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4868.100557] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4868.100559] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4868.100562] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4868.100564] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4868.100566] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4868.100570] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4868.100573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4868.100723] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4868.100733] Call Trace: [ 4868.100738] [ 4868.100743] ? show_regs.cold.9+0x22/0x2f [ 4868.100753] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4868.100758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4868.100762] ? watchdog+0x40/0x40 [ 4868.100767] ? hrtimer_interrupt+0x128/0x2c0 [ 4868.100828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4868.100835] ? apic_timer_interrupt+0xf/0x20 [ 4868.100839] [ 4868.100842] ? native_safe_halt+0x2e/0x50 [ 4868.100845] kvm_wait+0xcd/0xe0 [ 4868.100853] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4868.100861] do_raw_spin_lock+0x177/0x190 [ 4868.100865] _raw_spin_lock+0x21/0x30 [ 4868.100869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4868.101153] ? do_raw_spin_unlock+0x75/0x190 [ 4868.101159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4868.101367] ? do_wait_intr+0xf0/0xf0 [ 4868.101375] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4868.101586] mdc_close+0x53b/0xd70 [mdc] [ 4868.101628] lmv_close+0x18d/0x490 [lmv] [ 4868.105469] [ 4868.105478] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4868.109321] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4868.109445] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4868.109541] ll_md_close+0x274/0x800 [lustre] [ 4868.109628] ll_file_release+0x128/0xc50 [lustre] [ 4868.115927] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4868.115939] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4868.115946] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4868.115948] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4868.115950] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4868.115953] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4868.115956] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4868.115959] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4868.115962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4868.115964] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4868.115972] Call Trace: [ 4868.115978] [ 4868.115983] ? show_regs.cold.9+0x22/0x2f [ 4868.115996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4868.116003] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4868.116008] ? watchdog+0x40/0x40 [ 4868.116013] ? hrtimer_interrupt+0x128/0x2c0 [ 4868.116017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4868.116026] ? apic_timer_interrupt+0xf/0x20 [ 4868.116030] [ 4868.116033] ? native_safe_halt+0x2e/0x50 [ 4868.116039] kvm_wait+0xcd/0xe0 [ 4868.116048] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4868.116059] do_raw_spin_lock+0x177/0x190 [ 4868.116063] _raw_spin_lock+0x21/0x30 [ 4868.116067] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4868.116348] process_one_work+0x2c8/0x700 [ 4868.116362] worker_thread+0x296/0x6e0 [ 4868.116370] ? rescuer_thread+0x570/0x570 [ 4868.116374] kthread+0x1d1/0x200 [ 4868.116379] ? set_kthread_struct+0x70/0x70 [ 4868.116382] ret_from_fork+0x1f/0x30 [ 4868.302519] __fput+0x10d/0x440 [ 4868.303930] ____fput+0x12/0x20 [ 4868.305229] task_work_run+0xc6/0x110 [ 4868.306774] exit_to_usermode_loop+0x1dd/0x1f0 [ 4868.308676] do_syscall_64+0x3d6/0x440 [ 4868.310037] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4868.312240] RIP: 0033:0x7fa749552cd8 [ 4868.313727] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4868.322262] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4868.327649] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4868.333978] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4868.338114] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4868.341319] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4868.349716] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4876.067079] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4876.080359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4876.154998] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4876.177171] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4876.191456] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4876.199247] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4876.231139] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4876.247982] RAX: 0000000000006e83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4876.256092] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4876.261079] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4876.268396] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4876.276639] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4876.288643] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4876.307426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4876.313083] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4876.327591] Call Trace: [ 4876.330449] [ 4876.331343] ? show_regs.cold.9+0x22/0x2f [ 4876.341191] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4876.345293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4876.355139] ? watchdog+0x40/0x40 [ 4876.359748] ? hrtimer_interrupt+0x128/0x2c0 [ 4876.366390] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4876.372234] ? apic_timer_interrupt+0xf/0x20 [ 4876.377754] [ 4876.380033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4876.388981] do_raw_spin_lock+0x177/0x190 [ 4876.392369] _raw_spin_lock+0x21/0x30 [ 4876.405031] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4876.411200] ? do_raw_spin_unlock+0x75/0x190 [ 4876.413191] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4876.418933] ? do_wait_intr+0xf0/0xf0 [ 4876.421981] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4876.431982] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4876.435106] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4876.444631] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4876.451066] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4876.457802] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4876.461004] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4876.464706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4876.474622] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4876.482250] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4876.491159] lmv_intent_open+0x338/0x1150 [lmv] [ 4876.499305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4876.512534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4876.520037] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4876.523075] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4876.533565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4876.542812] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4876.549955] ? do_raw_spin_unlock+0x75/0x190 [ 4876.560053] ? _raw_spin_unlock+0x12/0x30 [ 4876.562995] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4876.567846] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4876.579740] ll_atomic_open+0x36a/0xd70 [lustre] [ 4876.582989] lookup_open+0x356/0xb40 [ 4876.590227] do_last+0x313/0xfc0 [ 4876.594193] ? nd_jump_root+0xe5/0x160 [ 4876.600051] ? path_init+0x437/0x520 [ 4876.606588] path_openat+0xf7/0x500 [ 4876.611970] ? do_raw_spin_unlock+0x75/0x190 [ 4876.623186] do_filp_open+0x99/0x140 [ 4876.624542] ? getname_flags+0x6e/0x330 [ 4876.632481] ? __check_object_size+0xff/0x256 [ 4876.636081] ? do_raw_spin_unlock+0x75/0x190 [ 4876.641562] ? _raw_spin_unlock+0x12/0x30 [ 4876.648706] do_sys_openat2+0x2b4/0x410 [ 4876.651868] do_sys_open+0x73/0xa0 [ 4876.659240] __x64_sys_openat+0x24/0x30 [ 4876.664073] do_syscall_64+0xc1/0x440 [ 4876.668007] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4876.670719] RIP: 0033:0x7fa749552332 [ 4876.676619] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4876.703126] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4876.713390] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4876.719838] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4876.731830] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4876.741735] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4876.750181] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4879.333469] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2962s! [ 4879.336250] Showing busy workqueues and worker pools: [ 4879.348996] workqueue events: flags=0x0 [ 4879.353877] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4879.356429] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4879.370924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4879.376044] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4879.385483] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.398361] pending: kfree_rcu_monitor [ 4879.406815] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4879.417420] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4879.429116] workqueue events_long: flags=0x0 [ 4879.435040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.445233] pending: lu_objects_flush [obdclass] [ 4879.455992] workqueue events_unbound: flags=0x2 [ 4879.463599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4879.465874] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4879.490200] workqueue events_power_efficient: flags=0x80 [ 4879.500143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.504011] pending: check_lifetime [ 4879.507717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.515110] pending: neigh_periodic_work [ 4879.519990] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.523590] pending: neigh_periodic_work [ 4879.527804] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.532185] pending: do_cache_clean [ 4879.536896] workqueue mm_percpu_wq: flags=0x8 [ 4879.542661] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.548097] pending: vmstat_update [ 4879.551876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.557340] pending: vmstat_update [ 4879.560183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4879.568670] pending: vmstat_update [ 4879.571437] workqueue writeback: flags=0x4a [ 4879.574964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4879.581190] pending: wb_workfn, wb_workfn [ 4879.583807] workqueue xprtiod: flags=0xa [ 4879.587334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4879.596888] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4879.603962] workqueue ipv6_addrconf: flags=0x40008 [ 4879.607027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4879.610575] pending: addrconf_verify_work [ 4879.618544] workqueue knbd0-recv: flags=0x1a [ 4879.620206] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4879.632405] in-flight: 651:recv_work [ 4879.636684] workqueue ptlrpc_pinger: flags=0x2 [ 4879.639842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4879.648919] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4879.654108] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2962s workers=4 idle: 37 31846 28897 [ 4879.660729] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4869s workers=2 idle: 653 [ 4880.058091] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4880.062599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4880.095921] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4880.110655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4880.119136] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4880.125882] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4880.144052] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4880.148801] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4880.152408] RDX: 0000000000007386 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4880.155810] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4880.159679] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4880.163292] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4880.168725] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4880.173293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4880.176146] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4880.180638] Call Trace: [ 4880.181942] [ 4880.183111] ? show_regs.cold.9+0x22/0x2f [ 4880.185674] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4880.188396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4880.190636] ? watchdog+0x40/0x40 [ 4880.197802] ? hrtimer_interrupt+0x128/0x2c0 [ 4880.209198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4880.212889] ? apic_timer_interrupt+0xf/0x20 [ 4880.218371] [ 4880.220480] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4880.223452] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4880.226432] do_raw_spin_lock+0x177/0x190 [ 4880.228735] _raw_spin_lock+0x21/0x30 [ 4880.230782] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4880.233972] obd_mod_free+0x46/0x140 [ptlrpc] [ 4880.236580] mdc_commit_open+0x91/0xb0 [mdc] [ 4880.238881] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4880.245543] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4880.253529] after_reply+0xe2c/0x1780 [ptlrpc] [ 4880.260261] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4880.264627] ? __next_timer_interrupt+0x160/0x160 [ 4880.267371] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4880.270075] ? do_wait_intr+0xf0/0xf0 [ 4880.272192] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4880.275307] kthread+0x1d1/0x200 [ 4880.277167] ? set_kthread_struct+0x70/0x70 [ 4880.279520] ret_from_fork+0x1f/0x30 [ 4896.057246] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4896.058643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4896.059645] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4896.083087] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4896.083100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4896.083106] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4896.083497] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4896.083506] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4896.083510] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4896.083520] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4896.083522] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4896.083524] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4896.083527] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4896.083529] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4896.083532] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4896.083535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4896.083538] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4896.083546] Call Trace: [ 4896.083551] [ 4896.083556] ? show_regs.cold.9+0x22/0x2f [ 4896.083565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4896.083572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4896.083576] ? watchdog+0x40/0x40 [ 4896.083581] ? hrtimer_interrupt+0x128/0x2c0 [ 4896.083585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4896.083592] ? apic_timer_interrupt+0xf/0x20 [ 4896.083597] [ 4896.083599] ? native_safe_halt+0x2e/0x50 [ 4896.083603] kvm_wait+0xcd/0xe0 [ 4896.083610] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4896.083618] do_raw_spin_lock+0x177/0x190 [ 4896.083622] _raw_spin_lock+0x21/0x30 [ 4896.083632] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4896.084075] process_one_work+0x2c8/0x700 [ 4896.084082] worker_thread+0x296/0x6e0 [ 4896.084087] ? rescuer_thread+0x570/0x570 [ 4896.084091] kthread+0x1d1/0x200 [ 4896.084096] ? set_kthread_struct+0x70/0x70 [ 4896.084100] ret_from_fork+0x1f/0x30 [ 4896.283928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4896.337441] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4896.343432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4896.349109] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4896.351607] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4896.375366] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4896.377928] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4896.395188] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4896.402539] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4896.409865] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4896.413750] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4896.419567] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4896.423798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4896.430419] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4896.432656] Call Trace: [ 4896.436767] [ 4896.437411] ? show_regs.cold.9+0x22/0x2f [ 4896.440473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4896.443210] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4896.448322] ? watchdog+0x40/0x40 [ 4896.449777] ? hrtimer_interrupt+0x128/0x2c0 [ 4896.451256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4896.454118] ? apic_timer_interrupt+0xf/0x20 [ 4896.455617] [ 4896.460760] ? native_safe_halt+0x2e/0x50 [ 4896.465137] kvm_wait+0xcd/0xe0 [ 4896.466261] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4896.471304] do_raw_spin_lock+0x177/0x190 [ 4896.473561] _raw_spin_lock+0x21/0x30 [ 4896.477536] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4896.479618] ? do_raw_spin_unlock+0x75/0x190 [ 4896.485165] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4896.492427] ? do_wait_intr+0xf0/0xf0 [ 4896.495794] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4896.499506] mdc_close+0x53b/0xd70 [mdc] [ 4896.504886] lmv_close+0x18d/0x490 [lmv] [ 4896.507892] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4896.518525] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4896.523799] ll_md_close+0x274/0x800 [lustre] [ 4896.528010] ll_file_release+0x128/0xc50 [lustre] [ 4896.534906] __fput+0x10d/0x440 [ 4896.536576] ____fput+0x12/0x20 [ 4896.537706] task_work_run+0xc6/0x110 [ 4896.539644] exit_to_usermode_loop+0x1dd/0x1f0 [ 4896.542651] do_syscall_64+0x3d6/0x440 [ 4896.544080] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4896.550030] RIP: 0033:0x7fa749552cd8 [ 4896.552831] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4896.579242] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4896.588006] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4896.595256] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4896.597678] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4896.603156] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4896.609981] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4904.057176] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4904.061573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4904.099606] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4904.109104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4904.121077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4904.125443] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4904.140279] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4904.147465] RAX: 000000000000770a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4904.157592] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4904.167622] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4904.171351] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4904.175369] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4904.179629] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4904.183925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4904.186923] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4904.190233] Call Trace: [ 4904.191337] [ 4904.192274] ? show_regs.cold.9+0x22/0x2f [ 4904.194298] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4904.196300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4904.198246] ? watchdog+0x40/0x40 [ 4904.199861] ? hrtimer_interrupt+0x128/0x2c0 [ 4904.201517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4904.203461] ? apic_timer_interrupt+0xf/0x20 [ 4904.205349] [ 4904.206294] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4904.215036] do_raw_spin_lock+0x177/0x190 [ 4904.217045] _raw_spin_lock+0x21/0x30 [ 4904.221633] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4904.225577] ? do_raw_spin_unlock+0x75/0x190 [ 4904.231646] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4904.236754] ? do_wait_intr+0xf0/0xf0 [ 4904.238977] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4904.252574] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4904.255292] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4904.262283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4904.265613] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4904.269790] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4904.271353] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4904.274922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4904.279307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4904.282290] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4904.285899] lmv_intent_open+0x338/0x1150 [lmv] [ 4904.291303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4904.295971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4904.301734] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4904.306509] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4904.310976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4904.313675] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4904.317251] ? do_raw_spin_unlock+0x75/0x190 [ 4904.320948] ? _raw_spin_unlock+0x12/0x30 [ 4904.322429] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4904.327715] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4904.331392] ll_atomic_open+0x36a/0xd70 [lustre] [ 4904.337399] lookup_open+0x356/0xb40 [ 4904.339483] do_last+0x313/0xfc0 [ 4904.341814] ? nd_jump_root+0xe5/0x160 [ 4904.344282] ? path_init+0x437/0x520 [ 4904.352742] path_openat+0xf7/0x500 [ 4904.358221] ? do_raw_spin_unlock+0x75/0x190 [ 4904.360857] do_filp_open+0x99/0x140 [ 4904.362974] ? getname_flags+0x6e/0x330 [ 4904.370254] ? __check_object_size+0xff/0x256 [ 4904.374673] ? do_raw_spin_unlock+0x75/0x190 [ 4904.380698] ? _raw_spin_unlock+0x12/0x30 [ 4904.388740] do_sys_openat2+0x2b4/0x410 [ 4904.391760] do_sys_open+0x73/0xa0 [ 4904.394425] __x64_sys_openat+0x24/0x30 [ 4904.396482] do_syscall_64+0xc1/0x440 [ 4904.398742] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4904.402453] RIP: 0033:0x7fa749552332 [ 4904.405427] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4904.420976] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4904.437939] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4904.444418] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4904.453130] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4904.458689] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4904.464438] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4908.060030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 4908.064293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4908.114329] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4908.120541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4908.123668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4908.125701] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4908.141416] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4908.145301] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4908.148653] RDX: 00000000000058db RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4908.153537] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4908.157949] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4908.161493] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4908.166426] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4908.170954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4908.174833] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4908.178681] Call Trace: [ 4908.179946] [ 4908.181583] ? show_regs.cold.9+0x22/0x2f [ 4908.184881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4908.186707] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4908.190859] ? watchdog+0x40/0x40 [ 4908.192377] ? hrtimer_interrupt+0x128/0x2c0 [ 4908.193985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4908.198101] ? apic_timer_interrupt+0xf/0x20 [ 4908.201695] [ 4908.203563] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4908.209555] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4908.214526] do_raw_spin_lock+0x177/0x190 [ 4908.215755] _raw_spin_lock+0x21/0x30 [ 4908.216859] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4908.220393] obd_mod_free+0x46/0x140 [ptlrpc] [ 4908.222762] mdc_commit_open+0x91/0xb0 [mdc] [ 4908.226059] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4908.231548] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4908.235016] after_reply+0xe2c/0x1780 [ptlrpc] [ 4908.240690] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4908.243426] ? __next_timer_interrupt+0x160/0x160 [ 4908.245252] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4908.247574] ? do_wait_intr+0xf0/0xf0 [ 4908.248655] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4908.250423] kthread+0x1d1/0x200 [ 4908.251820] ? set_kthread_struct+0x70/0x70 [ 4908.253613] ret_from_fork+0x1f/0x30 [ 4910.052104] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2992s! [ 4910.056305] Showing busy workqueues and worker pools: [ 4910.059051] workqueue events: flags=0x0 [ 4910.061371] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4910.066500] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4910.074861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4910.078207] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4910.084396] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.089488] pending: kfree_rcu_monitor [ 4910.091899] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4910.096166] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4910.102332] workqueue events_long: flags=0x0 [ 4910.105014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.108788] pending: lu_objects_flush [obdclass] [ 4910.112043] workqueue events_unbound: flags=0x2 [ 4910.114852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4910.118204] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4910.123981] workqueue events_power_efficient: flags=0x80 [ 4910.127179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.130830] pending: check_lifetime [ 4910.132927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.137173] pending: neigh_periodic_work [ 4910.139382] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.143297] pending: neigh_periodic_work [ 4910.146040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.148931] pending: do_cache_clean [ 4910.150837] workqueue mm_percpu_wq: flags=0x8 [ 4910.153332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.159086] pending: vmstat_update [ 4910.160499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.164330] pending: vmstat_update [ 4910.165893] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4910.171333] pending: vmstat_update [ 4910.172936] workqueue writeback: flags=0x4a [ 4910.175570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4910.181225] pending: wb_workfn, wb_workfn [ 4910.185549] workqueue xprtiod: flags=0xa [ 4910.189276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4910.193406] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4910.204096] workqueue ipv6_addrconf: flags=0x40008 [ 4910.205731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4910.208871] pending: addrconf_verify_work [ 4910.210990] workqueue knbd0-recv: flags=0x1a [ 4910.213586] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4910.217942] in-flight: 651:recv_work [ 4910.220488] workqueue ptlrpc_pinger: flags=0x2 [ 4910.222628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4910.225638] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4910.229196] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2993s workers=4 idle: 37 31846 28897 [ 4910.235857] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4900s workers=2 idle: 653 [ 4924.057259] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4924.059036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4924.061819] Modules linked in: lustre(O) [ 4924.068091] Modules linked in: lustre(O) [ 4924.070665] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 4924.079923] osp(O) [ 4924.079930] ptlrpc(O) obdclass(O) ksocklnd(O) [ 4924.097842] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4924.097919] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4924.097926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4924.097929] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4924.097941] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4924.097945] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4924.097950] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4924.097952] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4924.098016] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4924.098018] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4924.098021] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4924.098024] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4924.098027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4924.098030] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4924.098038] Call Trace: [ 4924.098043] [ 4924.098048] ? show_regs.cold.9+0x22/0x2f [ 4924.098057] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4924.098064] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4924.098067] ? watchdog+0x40/0x40 [ 4924.098072] ? hrtimer_interrupt+0x128/0x2c0 [ 4924.098076] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4924.098082] ? apic_timer_interrupt+0xf/0x20 [ 4924.098087] [ 4924.098089] ? native_safe_halt+0x2e/0x50 [ 4924.098093] kvm_wait+0xcd/0xe0 [ 4924.098100] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4924.098107] do_raw_spin_lock+0x177/0x190 [ 4924.098111] _raw_spin_lock+0x21/0x30 [ 4924.098115] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4924.098887] ? do_raw_spin_unlock+0x75/0x190 [ 4924.098894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4924.100878] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4924.100917] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4924.100924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4924.100929] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4924.101203] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4924.101224] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4924.101227] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4924.101231] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4924.101233] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4924.101235] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4924.101237] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4924.101239] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4924.101241] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4924.101244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4924.101246] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4924.101253] Call Trace: [ 4924.101259] [ 4924.101262] ? show_regs.cold.9+0x22/0x2f [ 4924.101271] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4924.101276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4924.101280] ? watchdog+0x40/0x40 [ 4924.101283] ? hrtimer_interrupt+0x128/0x2c0 [ 4924.101287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4924.101294] ? apic_timer_interrupt+0xf/0x20 [ 4924.101297] [ 4924.101300] ? native_safe_halt+0x2e/0x50 [ 4924.101303] kvm_wait+0xcd/0xe0 [ 4924.101312] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4924.101318] do_raw_spin_lock+0x177/0x190 [ 4924.101322] _raw_spin_lock+0x21/0x30 [ 4924.101326] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4924.101839] process_one_work+0x2c8/0x700 [ 4924.101846] worker_thread+0x296/0x6e0 [ 4924.101850] ? rescuer_thread+0x570/0x570 [ 4924.101854] kthread+0x1d1/0x200 [ 4924.101858] ? set_kthread_struct+0x70/0x70 [ 4924.101861] ret_from_fork+0x1f/0x30 [ 4924.382832] ? do_wait_intr+0xf0/0xf0 [ 4924.388325] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4924.391557] mdc_close+0x53b/0xd70 [mdc] [ 4924.393286] lmv_close+0x18d/0x490 [lmv] [ 4924.395561] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4924.398323] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4924.400568] ll_md_close+0x274/0x800 [lustre] [ 4924.403277] ll_file_release+0x128/0xc50 [lustre] [ 4924.405668] __fput+0x10d/0x440 [ 4924.407276] ____fput+0x12/0x20 [ 4924.408842] task_work_run+0xc6/0x110 [ 4924.410761] exit_to_usermode_loop+0x1dd/0x1f0 [ 4924.416967] do_syscall_64+0x3d6/0x440 [ 4924.425383] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4924.427466] RIP: 0033:0x7fa749552cd8 [ 4924.431600] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4924.444021] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4924.447753] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4924.450584] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4924.453733] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4924.456590] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4924.458720] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4936.056652] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 4936.060788] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4936.065284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4936.065431] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4936.065441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4936.065443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.065458] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4936.065462] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4936.065467] RAX: 0000000000007f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4936.065469] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4936.065472] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4936.065474] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4936.065477] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4936.065480] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4936.065482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4936.065485] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4936.065494] Call Trace: [ 4936.065499] [ 4936.065505] ? show_regs.cold.9+0x22/0x2f [ 4936.065545] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4936.065552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4936.065556] ? watchdog+0x40/0x40 [ 4936.065561] ? hrtimer_interrupt+0x128/0x2c0 [ 4936.065565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4936.065574] ? apic_timer_interrupt+0xf/0x20 [ 4936.065579] [ 4936.065583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.065591] do_raw_spin_lock+0x177/0x190 [ 4936.065596] _raw_spin_lock+0x21/0x30 [ 4936.065601] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4936.066480] ? do_raw_spin_unlock+0x75/0x190 [ 4936.066487] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4936.066713] ? do_wait_intr+0xf0/0xf0 [ 4936.066722] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4936.068031] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4936.068326] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4936.075437] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4936.075526] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4936.075534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4936.075536] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4936.075550] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4936.075555] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4936.075559] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4936.075561] RDX: 000000000000638b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4936.075564] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4936.075566] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4936.075569] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4936.075572] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4936.075574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4936.075577] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4936.075585] Call Trace: [ 4936.075591] [ 4936.075596] ? show_regs.cold.9+0x22/0x2f [ 4936.075605] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4936.075612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4936.075617] ? watchdog+0x40/0x40 [ 4936.075621] ? hrtimer_interrupt+0x128/0x2c0 [ 4936.075625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4936.075634] ? apic_timer_interrupt+0xf/0x20 [ 4936.075639] [ 4936.075642] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4936.075647] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4936.075652] do_raw_spin_lock+0x177/0x190 [ 4936.075657] _raw_spin_lock+0x21/0x30 [ 4936.075663] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4936.075938] obd_mod_free+0x46/0x140 [ptlrpc] [ 4936.143116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.156033] mdc_commit_open+0x91/0xb0 [mdc] [ 4936.156079] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4936.156380] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4936.156754] after_reply+0xe2c/0x1780 [ptlrpc] [ 4936.157090] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4936.157570] ? __next_timer_interrupt+0x160/0x160 [ 4936.157583] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4936.157816] ? do_wait_intr+0xf0/0xf0 [ 4936.157826] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4936.159796] kthread+0x1d1/0x200 [ 4936.159807] ? set_kthread_struct+0x70/0x70 [ 4936.159812] ret_from_fork+0x1f/0x30 [ 4936.416775] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4936.419073] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4936.421135] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4936.423014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.425209] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4936.428009] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4936.430207] lmv_intent_open+0x338/0x1150 [lmv] [ 4936.432150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.434463] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.437021] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4936.438838] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4936.440928] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.443485] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4936.445767] ? do_raw_spin_unlock+0x75/0x190 [ 4936.447527] ? _raw_spin_unlock+0x12/0x30 [ 4936.449080] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4936.450777] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4936.452209] ll_atomic_open+0x36a/0xd70 [lustre] [ 4936.454399] lookup_open+0x356/0xb40 [ 4936.455969] do_last+0x313/0xfc0 [ 4936.457407] ? nd_jump_root+0xe5/0x160 [ 4936.458861] ? path_init+0x437/0x520 [ 4936.460295] path_openat+0xf7/0x500 [ 4936.461778] ? do_raw_spin_unlock+0x75/0x190 [ 4936.463518] do_filp_open+0x99/0x140 [ 4936.465165] ? getname_flags+0x6e/0x330 [ 4936.466688] ? __check_object_size+0xff/0x256 [ 4936.468494] ? do_raw_spin_unlock+0x75/0x190 [ 4936.470248] ? _raw_spin_unlock+0x12/0x30 [ 4936.471758] do_sys_openat2+0x2b4/0x410 [ 4936.473052] do_sys_open+0x73/0xa0 [ 4936.474829] __x64_sys_openat+0x24/0x30 [ 4936.476465] do_syscall_64+0xc1/0x440 [ 4936.478025] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4936.480105] RIP: 0033:0x7fa749552332 [ 4936.481691] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4936.489424] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4936.492455] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4936.495799] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4936.499294] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4936.502043] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4936.504950] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4940.266674] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3023s! [ 4940.277367] Showing busy workqueues and worker pools: [ 4940.286490] workqueue events: flags=0x0 [ 4940.289981] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4940.294696] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4940.308150] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4940.316740] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4940.327010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.330718] pending: kfree_rcu_monitor [ 4940.335593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4940.345728] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4940.359455] workqueue events_long: flags=0x0 [ 4940.361759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.367160] pending: lu_objects_flush [obdclass] [ 4940.373122] workqueue events_unbound: flags=0x2 [ 4940.380805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4940.394124] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4940.403203] workqueue events_power_efficient: flags=0x80 [ 4940.407833] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.416720] pending: check_lifetime [ 4940.421934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.431683] pending: neigh_periodic_work [ 4940.433305] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.442059] pending: neigh_periodic_work [ 4940.447071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.453877] pending: do_cache_clean [ 4940.456930] workqueue mm_percpu_wq: flags=0x8 [ 4940.459888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.466592] pending: vmstat_update [ 4940.474958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.477391] pending: vmstat_update [ 4940.481755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4940.488717] pending: vmstat_update [ 4940.492094] workqueue writeback: flags=0x4a [ 4940.498269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4940.502713] pending: wb_workfn, wb_workfn [ 4940.507782] workqueue xprtiod: flags=0xa [ 4940.511572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4940.516964] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4940.523871] workqueue ipv6_addrconf: flags=0x40008 [ 4940.526342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4940.529595] pending: addrconf_verify_work [ 4940.532386] workqueue knbd0-recv: flags=0x1a [ 4940.534705] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4940.538980] in-flight: 651:recv_work [ 4940.541473] workqueue ptlrpc_pinger: flags=0x2 [ 4940.544368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4940.547379] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4940.549754] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3023s workers=4 idle: 37 31846 28897 [ 4940.554875] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4930s workers=2 idle: 653 [ 4952.057017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 4952.060246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4952.062606] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 4952.077135] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4952.077149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4952.077155] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4952.077453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4952.077475] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4952.077479] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4952.077484] RAX: 00000000000052f4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4952.077487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4952.077489] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4952.077491] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4952.077493] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4952.077496] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4952.077499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4952.077502] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4952.077509] Call Trace: [ 4952.077513] [ 4952.077517] ? show_regs.cold.9+0x22/0x2f [ 4952.077526] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4952.077531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4952.077535] ? watchdog+0x40/0x40 [ 4952.077539] ? hrtimer_interrupt+0x128/0x2c0 [ 4952.077543] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4952.077551] ? apic_timer_interrupt+0xf/0x20 [ 4952.077555] [ 4952.077558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4952.077562] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4952.077567] do_raw_spin_lock+0x177/0x190 [ 4952.077571] _raw_spin_lock+0x21/0x30 [ 4952.077577] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4952.077776] process_one_work+0x2c8/0x700 [ 4952.077784] worker_thread+0x296/0x6e0 [ 4952.077788] ? rescuer_thread+0x570/0x570 [ 4952.077792] kthread+0x1d1/0x200 [ 4952.077796] ? set_kthread_struct+0x70/0x70 [ 4952.077800] ret_from_fork+0x1f/0x30 [ 4952.165813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4952.184391] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4952.188254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4952.191390] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4952.193347] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4952.200318] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4952.203221] RAX: 0000000000005331 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4952.206083] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4952.209036] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4952.211839] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4952.214783] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4952.217759] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4952.221136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4952.223247] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4952.225925] Call Trace: [ 4952.227411] [ 4952.228256] ? show_regs.cold.9+0x22/0x2f [ 4952.229757] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4952.231782] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4952.234095] ? watchdog+0x40/0x40 [ 4952.235367] ? hrtimer_interrupt+0x128/0x2c0 [ 4952.237206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4952.239112] ? apic_timer_interrupt+0xf/0x20 [ 4952.241089] [ 4952.242019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4952.244112] do_raw_spin_lock+0x177/0x190 [ 4952.245636] _raw_spin_lock+0x21/0x30 [ 4952.247253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4952.249332] ? do_raw_spin_unlock+0x75/0x190 [ 4952.250999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4952.252769] ? do_wait_intr+0xf0/0xf0 [ 4952.254025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4952.256099] mdc_close+0x53b/0xd70 [mdc] [ 4952.257646] lmv_close+0x18d/0x490 [lmv] [ 4952.259249] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4952.261422] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4952.263400] ll_md_close+0x274/0x800 [lustre] [ 4952.265311] ll_file_release+0x128/0xc50 [lustre] [ 4952.268079] __fput+0x10d/0x440 [ 4952.269292] ____fput+0x12/0x20 [ 4952.270331] task_work_run+0xc6/0x110 [ 4952.271752] exit_to_usermode_loop+0x1dd/0x1f0 [ 4952.273431] do_syscall_64+0x3d6/0x440 [ 4952.275354] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4952.277642] RIP: 0033:0x7fa749552cd8 [ 4952.279123] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4952.286334] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4952.289302] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4952.292875] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4952.296314] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4952.298916] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4952.301579] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4964.058444] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4964.064232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4964.095684] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4964.102563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4964.109203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4964.116700] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4964.130812] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4964.135460] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4964.139305] RDX: 000000000000176b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4964.145954] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4964.149114] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4964.157238] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4964.160590] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4964.165378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4964.169528] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4964.173106] Call Trace: [ 4964.174100] [ 4964.174838] ? show_regs.cold.9+0x22/0x2f [ 4964.176568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4964.179839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4964.181846] ? watchdog+0x40/0x40 [ 4964.183378] ? hrtimer_interrupt+0x128/0x2c0 [ 4964.184673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4964.189404] ? apic_timer_interrupt+0xf/0x20 [ 4964.191392] [ 4964.192006] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4964.197538] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4964.199562] do_raw_spin_lock+0x177/0x190 [ 4964.203161] _raw_spin_lock+0x21/0x30 [ 4964.204535] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4964.206573] obd_mod_free+0x46/0x140 [ptlrpc] [ 4964.208197] mdc_commit_open+0x91/0xb0 [mdc] [ 4964.210316] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4964.212129] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4964.215427] after_reply+0xe2c/0x1780 [ptlrpc] [ 4964.218488] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4964.223418] ? __next_timer_interrupt+0x160/0x160 [ 4964.226171] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4964.228376] ? do_wait_intr+0xf0/0xf0 [ 4964.230587] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4964.233680] kthread+0x1d1/0x200 [ 4964.236865] ? set_kthread_struct+0x70/0x70 [ 4964.238430] ret_from_fork+0x1f/0x30 [ 4968.054066] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4968.056219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4968.151105] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4968.161200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4968.169793] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4968.173289] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4968.190223] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4968.194787] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4968.201557] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4968.208416] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4968.212627] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4968.221652] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4968.227546] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4968.238290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4968.241936] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4968.246193] Call Trace: [ 4968.247562] [ 4968.248972] ? show_regs.cold.9+0x22/0x2f [ 4968.251516] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4968.254486] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4968.257567] ? watchdog+0x40/0x40 [ 4968.260002] ? hrtimer_interrupt+0x128/0x2c0 [ 4968.263566] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4968.266049] ? apic_timer_interrupt+0xf/0x20 [ 4968.268848] [ 4968.270244] ? native_safe_halt+0x2e/0x50 [ 4968.272790] kvm_wait+0xcd/0xe0 [ 4968.275040] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4968.280768] do_raw_spin_lock+0x177/0x190 [ 4968.286625] _raw_spin_lock+0x21/0x30 [ 4968.290181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4968.294010] ? do_raw_spin_unlock+0x75/0x190 [ 4968.296614] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4968.299875] ? do_wait_intr+0xf0/0xf0 [ 4968.301729] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4968.305006] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4968.308057] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4968.311606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4968.314123] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4968.318812] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4968.321637] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4968.325632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4968.331568] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4968.335342] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4968.338959] lmv_intent_open+0x338/0x1150 [lmv] [ 4968.345352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4968.348653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4968.352296] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4968.354456] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4968.357847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4968.361648] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4968.365497] ? do_raw_spin_unlock+0x75/0x190 [ 4968.367912] ? _raw_spin_unlock+0x12/0x30 [ 4968.372559] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4968.375511] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4968.382644] ll_atomic_open+0x36a/0xd70 [lustre] [ 4968.385271] lookup_open+0x356/0xb40 [ 4968.387966] do_last+0x313/0xfc0 [ 4968.389603] ? nd_jump_root+0xe5/0x160 [ 4968.391908] ? path_init+0x437/0x520 [ 4968.393890] path_openat+0xf7/0x500 [ 4968.396172] ? do_raw_spin_unlock+0x75/0x190 [ 4968.398889] do_filp_open+0x99/0x140 [ 4968.400921] ? getname_flags+0x6e/0x330 [ 4968.403189] ? __check_object_size+0xff/0x256 [ 4968.405965] ? do_raw_spin_unlock+0x75/0x190 [ 4968.408676] ? _raw_spin_unlock+0x12/0x30 [ 4968.413824] do_sys_openat2+0x2b4/0x410 [ 4968.419050] do_sys_open+0x73/0xa0 [ 4968.428432] __x64_sys_openat+0x24/0x30 [ 4968.431202] do_syscall_64+0xc1/0x440 [ 4968.440860] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4968.455232] RIP: 0033:0x7fa749552332 [ 4968.464229] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4968.490212] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4968.494781] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4968.501521] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4968.506300] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4968.510641] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4968.520518] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 4970.976635] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3053s! [ 4970.981558] Showing busy workqueues and worker pools: [ 4970.986765] workqueue events: flags=0x0 [ 4970.990512] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4970.996817] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 4971.007747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4971.020786] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4971.028556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.037269] pending: kfree_rcu_monitor [ 4971.042296] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4971.051422] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 4971.059238] workqueue events_long: flags=0x0 [ 4971.064537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.069254] pending: lu_objects_flush [obdclass] [ 4971.072505] workqueue events_unbound: flags=0x2 [ 4971.078133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 4971.082221] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4971.091429] workqueue events_power_efficient: flags=0x80 [ 4971.096503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.102475] pending: check_lifetime [ 4971.103728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.108088] pending: neigh_periodic_work [ 4971.109578] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.117149] pending: neigh_periodic_work [ 4971.120814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.127199] pending: do_cache_clean [ 4971.130405] workqueue mm_percpu_wq: flags=0x8 [ 4971.132851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.136586] pending: vmstat_update [ 4971.139942] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.142499] pending: vmstat_update [ 4971.143586] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4971.148128] pending: vmstat_update [ 4971.151207] workqueue writeback: flags=0x4a [ 4971.152742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4971.155529] pending: wb_workfn, wb_workfn [ 4971.156791] workqueue xprtiod: flags=0xa [ 4971.157723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4971.167388] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4971.175578] workqueue ipv6_addrconf: flags=0x40008 [ 4971.179362] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4971.184326] pending: addrconf_verify_work [ 4971.186446] workqueue knbd0-recv: flags=0x1a [ 4971.189956] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4971.197756] in-flight: 651:recv_work [ 4971.199480] workqueue ptlrpc_pinger: flags=0x2 [ 4971.202505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4971.205935] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4971.210170] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3054s workers=4 idle: 37 31846 28897 [ 4971.215090] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4961s workers=2 idle: 653 [ 4980.057213] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 4980.060954] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4980.061214] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 4980.080115] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4980.080127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4980.080134] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4980.080597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4980.080610] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4980.080613] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4980.080617] RAX: 000000000000300f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4980.080620] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 4980.080622] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4980.080623] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 4980.080626] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 4980.080630] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 4980.080632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4980.080635] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 4980.080644] Call Trace: [ 4980.080650] [ 4980.080655] ? show_regs.cold.9+0x22/0x2f [ 4980.080664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4980.080670] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4980.080675] ? watchdog+0x40/0x40 [ 4980.080679] ? hrtimer_interrupt+0x128/0x2c0 [ 4980.080683] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4980.080691] ? apic_timer_interrupt+0xf/0x20 [ 4980.080696] [ 4980.080699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4980.080703] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4980.080709] do_raw_spin_lock+0x177/0x190 [ 4980.080714] _raw_spin_lock+0x21/0x30 [ 4980.080719] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4980.080932] process_one_work+0x2c8/0x700 [ 4980.080939] worker_thread+0x296/0x6e0 [ 4980.080943] ? rescuer_thread+0x570/0x570 [ 4980.080946] kthread+0x1d1/0x200 [ 4980.080951] ? set_kthread_struct+0x70/0x70 [ 4980.080955] ret_from_fork+0x1f/0x30 [ 4980.171912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4980.191120] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4980.195837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4980.199503] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4980.202160] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4980.211887] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4980.215568] RAX: 0000000000002185 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4980.218155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 4980.221212] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4980.225138] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 4980.228234] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 4980.231723] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 4980.235530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4980.238021] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 4980.241090] Call Trace: [ 4980.242185] [ 4980.243182] ? show_regs.cold.9+0x22/0x2f [ 4980.244903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4980.247062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4980.249614] ? watchdog+0x40/0x40 [ 4980.251086] ? hrtimer_interrupt+0x128/0x2c0 [ 4980.253212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4980.255105] ? apic_timer_interrupt+0xf/0x20 [ 4980.257147] [ 4980.257955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4980.260328] do_raw_spin_lock+0x177/0x190 [ 4980.262388] _raw_spin_lock+0x21/0x30 [ 4980.264104] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4980.266945] ? do_raw_spin_unlock+0x75/0x190 [ 4980.268949] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4980.271545] ? do_wait_intr+0xf0/0xf0 [ 4980.273262] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4980.275924] mdc_close+0x53b/0xd70 [mdc] [ 4980.278075] lmv_close+0x18d/0x490 [lmv] [ 4980.280116] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4980.282380] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4980.284804] ll_md_close+0x274/0x800 [lustre] [ 4980.286630] ll_file_release+0x128/0xc50 [lustre] [ 4980.288884] __fput+0x10d/0x440 [ 4980.290390] ____fput+0x12/0x20 [ 4980.291986] task_work_run+0xc6/0x110 [ 4980.293413] exit_to_usermode_loop+0x1dd/0x1f0 [ 4980.295829] do_syscall_64+0x3d6/0x440 [ 4980.297542] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4980.299881] RIP: 0033:0x7fa749552cd8 [ 4980.301538] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4980.309950] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4980.313322] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 4980.317074] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 4980.320470] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 4980.323556] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 4980.327045] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4992.060069] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 4992.067125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4992.114326] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4992.138041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4992.152499] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4992.170446] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4992.189177] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4992.196014] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 4992.211787] RDX: 00000000000009f7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 4992.229159] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 4992.231110] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 4992.240094] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4992.257320] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 4992.265730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4992.271068] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 4992.276936] Call Trace: [ 4992.278143] [ 4992.280016] ? show_regs.cold.9+0x22/0x2f [ 4992.281479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4992.289373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4992.292638] ? watchdog+0x40/0x40 [ 4992.297450] ? hrtimer_interrupt+0x128/0x2c0 [ 4992.306542] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4992.309282] ? apic_timer_interrupt+0xf/0x20 [ 4992.315514] [ 4992.316293] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4992.321682] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4992.334671] do_raw_spin_lock+0x177/0x190 [ 4992.336142] _raw_spin_lock+0x21/0x30 [ 4992.337433] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4992.346138] obd_mod_free+0x46/0x140 [ptlrpc] [ 4992.352200] mdc_commit_open+0x91/0xb0 [mdc] [ 4992.360134] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4992.362197] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4992.367212] after_reply+0xe2c/0x1780 [ptlrpc] [ 4992.370351] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4992.381918] ? __next_timer_interrupt+0x160/0x160 [ 4992.386343] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4992.391926] ? do_wait_intr+0xf0/0xf0 [ 4992.397868] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4992.400067] kthread+0x1d1/0x200 [ 4992.400979] ? set_kthread_struct+0x70/0x70 [ 4992.405123] ret_from_fork+0x1f/0x30 [ 4996.058315] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 4996.070364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4996.117785] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4996.131649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4996.136529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4996.147799] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4996.168638] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4996.178409] RAX: 0000000000006595 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 4996.187941] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 4996.195108] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 4996.199800] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 4996.207789] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 4996.213141] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 4996.216987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4996.219537] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 4996.225988] Call Trace: [ 4996.231036] [ 4996.232364] ? show_regs.cold.9+0x22/0x2f [ 4996.235745] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4996.238051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4996.239637] ? watchdog+0x40/0x40 [ 4996.243892] ? hrtimer_interrupt+0x128/0x2c0 [ 4996.250689] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4996.259846] ? apic_timer_interrupt+0xf/0x20 [ 4996.264328] [ 4996.268728] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4996.276010] do_raw_spin_lock+0x177/0x190 [ 4996.280112] _raw_spin_lock+0x21/0x30 [ 4996.284805] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4996.288121] ? do_raw_spin_unlock+0x75/0x190 [ 4996.293437] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4996.298406] ? do_wait_intr+0xf0/0xf0 [ 4996.303747] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4996.306438] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4996.309292] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4996.315246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4996.321112] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4996.327239] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4996.330025] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4996.335740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4996.338968] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4996.341815] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4996.344837] lmv_intent_open+0x338/0x1150 [lmv] [ 4996.347329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4996.349811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4996.352758] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4996.355046] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4996.357272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4996.360051] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4996.362423] ? do_raw_spin_unlock+0x75/0x190 [ 4996.364139] ? _raw_spin_unlock+0x12/0x30 [ 4996.365859] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4996.368249] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4996.371940] ll_atomic_open+0x36a/0xd70 [lustre] [ 4996.374628] lookup_open+0x356/0xb40 [ 4996.375727] do_last+0x313/0xfc0 [ 4996.376970] ? nd_jump_root+0xe5/0x160 [ 4996.378647] ? path_init+0x437/0x520 [ 4996.380072] path_openat+0xf7/0x500 [ 4996.381691] ? do_raw_spin_unlock+0x75/0x190 [ 4996.384328] do_filp_open+0x99/0x140 [ 4996.387045] ? getname_flags+0x6e/0x330 [ 4996.389894] ? __check_object_size+0xff/0x256 [ 4996.392084] ? do_raw_spin_unlock+0x75/0x190 [ 4996.393862] ? _raw_spin_unlock+0x12/0x30 [ 4996.395398] do_sys_openat2+0x2b4/0x410 [ 4996.397000] do_sys_open+0x73/0xa0 [ 4996.398346] __x64_sys_openat+0x24/0x30 [ 4996.399742] do_syscall_64+0xc1/0x440 [ 4996.401898] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4996.403237] RIP: 0033:0x7fa749552332 [ 4996.409408] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4996.421813] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4996.429157] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 4996.434407] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 4996.438312] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 4996.442328] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 4996.445585] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5001.698500] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3084s! [ 5001.704446] Showing busy workqueues and worker pools: [ 5001.711045] workqueue events: flags=0x0 [ 5001.713385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5001.717501] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5001.725128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5001.733333] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5001.746019] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.756606] pending: kfree_rcu_monitor [ 5001.758847] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5001.762260] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5001.772843] workqueue events_long: flags=0x0 [ 5001.775244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.778750] pending: lu_objects_flush [obdclass] [ 5001.787955] workqueue events_unbound: flags=0x2 [ 5001.790953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5001.801963] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5001.810372] workqueue events_power_efficient: flags=0x80 [ 5001.814409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.818928] pending: check_lifetime [ 5001.821512] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.827871] pending: neigh_periodic_work [ 5001.830757] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.835168] pending: neigh_periodic_work [ 5001.837846] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.840328] pending: do_cache_clean [ 5001.842412] workqueue mm_percpu_wq: flags=0x8 [ 5001.844382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.846344] pending: vmstat_update [ 5001.849306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.852213] pending: vmstat_update [ 5001.855022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5001.858337] pending: vmstat_update [ 5001.860012] workqueue writeback: flags=0x4a [ 5001.863397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5001.867031] pending: wb_workfn, wb_workfn [ 5001.870065] workqueue xprtiod: flags=0xa [ 5001.872285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5001.876453] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5001.881237] workqueue ipv6_addrconf: flags=0x40008 [ 5001.884392] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5001.887825] pending: addrconf_verify_work [ 5001.889260] workqueue knbd0-recv: flags=0x1a [ 5001.892421] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5001.899197] in-flight: 651:recv_work [ 5001.901760] workqueue ptlrpc_pinger: flags=0x2 [ 5001.908406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5001.912887] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5001.917089] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3084s workers=4 idle: 37 31846 28897 [ 5001.923434] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4991s workers=2 idle: 653 [ 5008.061279] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5008.070794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul [ 5008.070963] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5008.071113] crc32_pclmul [ 5008.102198] Modules linked in: [ 5008.109973] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5008.110001] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5008.110008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5008.110016] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5008.111111] lustre(O) [ 5008.112529] [ 5008.112535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5008.112553] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5008.112557] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5008.112562] RAX: 0000000000007107 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5008.112565] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5008.112567] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5008.112569] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5008.112571] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5008.112574] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5008.112577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5008.112580] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5008.112588] Call Trace: [ 5008.112593] [ 5008.112599] ? show_regs.cold.9+0x22/0x2f [ 5008.112607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5008.112613] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5008.112618] ? watchdog+0x40/0x40 [ 5008.112621] ? hrtimer_interrupt+0x128/0x2c0 [ 5008.112625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5008.112633] ? apic_timer_interrupt+0xf/0x20 [ 5008.112637] [ 5008.112640] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5008.112644] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5008.112652] do_raw_spin_lock+0x177/0x190 [ 5008.112657] _raw_spin_lock+0x21/0x30 [ 5008.112663] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5008.117809] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [ 5008.123570] process_one_work+0x2c8/0x700 [ 5008.126550] dm_mod sha512_ssse3 sha512_generic [ 5008.126565] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5008.126572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5008.126575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5008.126590] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5008.126594] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5008.126598] RAX: 0000000000007a89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5008.126600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5008.126602] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5008.126604] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5008.126607] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5008.126610] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5008.126613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5008.126615] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5008.126623] Call Trace: [ 5008.126629] [ 5008.126633] ? show_regs.cold.9+0x22/0x2f [ 5008.126644] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5008.126650] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5008.126654] ? watchdog+0x40/0x40 [ 5008.126658] ? hrtimer_interrupt+0x128/0x2c0 [ 5008.126661] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5008.126670] ? apic_timer_interrupt+0xf/0x20 [ 5008.126674] [ 5008.126677] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5008.126685] do_raw_spin_lock+0x177/0x190 [ 5008.126690] _raw_spin_lock+0x21/0x30 [ 5008.126695] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5008.129270] worker_thread+0x296/0x6e0 [ 5008.130196] ? do_raw_spin_unlock+0x75/0x190 [ 5008.130208] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5008.131259] ? rescuer_thread+0x570/0x570 [ 5008.131268] kthread+0x1d1/0x200 [ 5008.131273] ? set_kthread_struct+0x70/0x70 [ 5008.131276] ret_from_fork+0x1f/0x30 [ 5008.471299] ? do_wait_intr+0xf0/0xf0 [ 5008.472679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5008.475417] mdc_close+0x53b/0xd70 [mdc] [ 5008.479633] lmv_close+0x18d/0x490 [lmv] [ 5008.482119] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5008.483676] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5008.487122] ll_md_close+0x274/0x800 [lustre] [ 5008.489696] ll_file_release+0x128/0xc50 [lustre] [ 5008.492494] __fput+0x10d/0x440 [ 5008.496380] ____fput+0x12/0x20 [ 5008.498391] task_work_run+0xc6/0x110 [ 5008.501506] exit_to_usermode_loop+0x1dd/0x1f0 [ 5008.507661] do_syscall_64+0x3d6/0x440 [ 5008.509674] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5008.514096] RIP: 0033:0x7fa749552cd8 [ 5008.516734] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5008.529187] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5008.531806] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5008.534554] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5008.539953] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5008.543819] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5008.547838] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5020.058960] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5020.063445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5020.087579] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5020.095416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5020.105519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5020.112517] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5020.158433] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5020.169745] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5020.173113] RDX: 0000000000001448 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5020.180460] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5020.186487] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5020.189920] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5020.193696] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5020.200001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5020.204159] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5020.208201] Call Trace: [ 5020.209208] [ 5020.209941] ? show_regs.cold.9+0x22/0x2f [ 5020.212474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5020.214465] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5020.216964] ? watchdog+0x40/0x40 [ 5020.218165] ? hrtimer_interrupt+0x128/0x2c0 [ 5020.219884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5020.222061] ? apic_timer_interrupt+0xf/0x20 [ 5020.223670] [ 5020.224224] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5020.225747] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5020.228259] do_raw_spin_lock+0x177/0x190 [ 5020.229988] _raw_spin_lock+0x21/0x30 [ 5020.231553] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5020.234316] obd_mod_free+0x46/0x140 [ptlrpc] [ 5020.236964] mdc_commit_open+0x91/0xb0 [mdc] [ 5020.238646] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5020.241212] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5020.243786] after_reply+0xe2c/0x1780 [ptlrpc] [ 5020.246332] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5020.248797] ? __next_timer_interrupt+0x160/0x160 [ 5020.250805] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5020.252549] ? do_wait_intr+0xf0/0xf0 [ 5020.253902] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5020.256633] kthread+0x1d1/0x200 [ 5020.258114] ? set_kthread_struct+0x70/0x70 [ 5020.259935] ret_from_fork+0x1f/0x30 [ 5028.054965] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5028.061238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5028.091456] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5028.097168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5028.106840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5028.111146] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5028.135264] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5028.142290] RAX: 0000000000003b5d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5028.157414] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5028.162934] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5028.172105] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5028.175803] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5028.184799] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5028.190422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5028.197289] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5028.204203] Call Trace: [ 5028.205418] [ 5028.207189] ? show_regs.cold.9+0x22/0x2f [ 5028.212247] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5028.220814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5028.224151] ? watchdog+0x40/0x40 [ 5028.229363] ? hrtimer_interrupt+0x128/0x2c0 [ 5028.232144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5028.240401] ? apic_timer_interrupt+0xf/0x20 [ 5028.247297] [ 5028.250566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5028.261114] do_raw_spin_lock+0x177/0x190 [ 5028.265662] _raw_spin_lock+0x21/0x30 [ 5028.266951] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5028.275358] ? do_raw_spin_unlock+0x75/0x190 [ 5028.278109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5028.281205] ? do_wait_intr+0xf0/0xf0 [ 5028.283340] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5028.286824] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5028.291633] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5028.298213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.304297] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5028.310419] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5028.314922] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5028.320279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.325152] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5028.330984] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5028.339838] lmv_intent_open+0x338/0x1150 [lmv] [ 5028.344132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.347393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.352921] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5028.356121] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5028.359332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.365531] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5028.369394] ? do_raw_spin_unlock+0x75/0x190 [ 5028.371667] ? _raw_spin_unlock+0x12/0x30 [ 5028.373103] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5028.374395] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5028.379908] ll_atomic_open+0x36a/0xd70 [lustre] [ 5028.385202] lookup_open+0x356/0xb40 [ 5028.389117] do_last+0x313/0xfc0 [ 5028.391119] ? nd_jump_root+0xe5/0x160 [ 5028.392803] ? path_init+0x437/0x520 [ 5028.394303] path_openat+0xf7/0x500 [ 5028.395554] ? do_raw_spin_unlock+0x75/0x190 [ 5028.398873] do_filp_open+0x99/0x140 [ 5028.400832] ? getname_flags+0x6e/0x330 [ 5028.402546] ? __check_object_size+0xff/0x256 [ 5028.405713] ? do_raw_spin_unlock+0x75/0x190 [ 5028.407907] ? _raw_spin_unlock+0x12/0x30 [ 5028.410005] do_sys_openat2+0x2b4/0x410 [ 5028.412213] do_sys_open+0x73/0xa0 [ 5028.414045] __x64_sys_openat+0x24/0x30 [ 5028.415785] do_syscall_64+0xc1/0x440 [ 5028.417961] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5028.420142] RIP: 0033:0x7fa749552332 [ 5028.422126] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5028.442131] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5028.444662] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5028.449094] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5028.463911] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5028.475573] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5028.482785] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5032.416409] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3115s! [ 5032.421288] Showing busy workqueues and worker pools: [ 5032.428526] workqueue events: flags=0x0 [ 5032.433369] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5032.440813] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5032.449274] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5032.453043] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5032.458404] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.461377] pending: kfree_rcu_monitor [ 5032.463128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5032.466707] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5032.470808] workqueue events_long: flags=0x0 [ 5032.473267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.476232] pending: lu_objects_flush [obdclass] [ 5032.479264] workqueue events_unbound: flags=0x2 [ 5032.481713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5032.483862] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5032.487613] workqueue events_power_efficient: flags=0x80 [ 5032.490205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.493367] pending: check_lifetime [ 5032.495504] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.498002] pending: neigh_periodic_work [ 5032.499984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.503175] pending: neigh_periodic_work [ 5032.505130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.509032] pending: do_cache_clean [ 5032.510652] workqueue mm_percpu_wq: flags=0x8 [ 5032.512142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.514934] pending: vmstat_update [ 5032.516750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.519601] pending: vmstat_update [ 5032.521041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5032.522976] pending: vmstat_update [ 5032.524095] workqueue writeback: flags=0x4a [ 5032.525480] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5032.528938] pending: wb_workfn, wb_workfn [ 5032.530797] workqueue xprtiod: flags=0xa [ 5032.542298] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5032.549129] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5032.558001] workqueue ipv6_addrconf: flags=0x40008 [ 5032.563437] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5032.568050] pending: addrconf_verify_work [ 5032.570690] workqueue knbd0-recv: flags=0x1a [ 5032.577075] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5032.584887] in-flight: 651:recv_work [ 5032.588332] workqueue ptlrpc_pinger: flags=0x2 [ 5032.590678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5032.593734] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5032.597072] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3115s workers=4 idle: 37 31846 28897 [ 5032.601247] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5022s workers=2 idle: 653 [ 5033.756023] rcu: INFO: rcu_sched self-detected stall on CPU [ 5033.768430] rcu: 2-...!: (1696051 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 5033.777880] (t=3120067 jiffies g=258701 q=2477) [ 5033.780531] rcu: rcu_sched kthread starved for 3116862 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 5033.787727] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5033.795908] rcu: RCU grace-period kthread stack dump: [ 5033.797834] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5033.805082] Call Trace: [ 5033.806151] __schedule+0x351/0xcb0 [ 5033.809377] schedule+0xc0/0x180 [ 5033.811873] schedule_timeout+0xb4/0x190 [ 5033.815724] ? do_raw_spin_unlock+0x75/0x190 [ 5033.817115] ? __next_timer_interrupt+0x160/0x160 [ 5033.820385] ? prepare_to_swait_event+0xf3/0x140 [ 5033.825101] rcu_gp_kthread+0x8dd/0x15b0 [ 5033.833364] ? force_qs_rnp+0x2c0/0x2c0 [ 5033.835517] kthread+0x1d1/0x200 [ 5033.837482] ? set_kthread_struct+0x70/0x70 [ 5033.839995] ret_from_fork+0x1f/0x30 [ 5033.841356] rcu: Stack dump where RCU GP kthread last ran: [ 5033.843032] Sending NMI from CPU 2 to CPUs 3: [ 5033.847524] NMI backtrace for cpu 3 [ 5033.847531] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5033.847534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5033.847536] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5033.847537] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5033.847539] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 [ 5033.847542] RAX: 0000000000007f09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5033.847544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5033.847545] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5033.847546] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5033.847547] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5033.847548] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5033.847549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5033.847551] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5033.847552] Call Trace: [ 5033.847552] [ 5033.847553] ? show_regs.cold.9+0x22/0x2f [ 5033.847555] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5033.847556] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5033.847557] ? nmi_handle+0xbd/0x210 [ 5033.847557] ? default_do_nmi+0x4b/0x1b0 [ 5033.847559] ? do_nmi+0x280/0x380 [ 5033.847559] ? end_repeat_nmi+0x16/0x1f [ 5033.847560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5033.847562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5033.847563] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5033.847563] [ 5033.847564] do_raw_spin_lock+0x177/0x190 [ 5033.847565] _raw_spin_lock+0x21/0x30 [ 5033.847567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5033.847568] ? do_raw_spin_unlock+0x75/0x190 [ 5033.847569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5033.847570] ? do_wait_intr+0xf0/0xf0 [ 5033.847571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5033.847572] mdc_close+0x53b/0xd70 [mdc] [ 5033.847573] lmv_close+0x18d/0x490 [lmv] [ 5033.847574] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5033.847576] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5033.847577] ll_md_close+0x274/0x800 [lustre] [ 5033.847578] ll_file_release+0x128/0xc50 [lustre] [ 5033.847579] __fput+0x10d/0x440 [ 5033.847580] ____fput+0x12/0x20 [ 5033.847581] task_work_run+0xc6/0x110 [ 5033.847582] exit_to_usermode_loop+0x1dd/0x1f0 [ 5033.847583] do_syscall_64+0x3d6/0x440 [ 5033.847584] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5033.847585] RIP: 0033:0x7fa749552cd8 [ 5033.847587] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5033.847601] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5033.847604] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5033.847605] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5033.847607] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5033.847609] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5033.847610] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5033.847915] NMI backtrace for cpu 2 [ 5034.094707] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5034.099272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5034.109960] Call Trace: [ 5034.115997] [ 5034.117748] ? dump_stack+0xbb/0x10e [ 5034.121099] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5034.126626] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5034.128116] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5034.134897] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5034.136662] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5034.156224] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5034.164248] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5034.170354] ? update_process_times+0x6d/0xc0 [ 5034.173986] ? tick_sched_handle.isra.18+0x27/0x90 [ 5034.179748] ? tick_sched_timer+0x53/0xb0 [ 5034.181597] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5034.184786] ? tick_sched_handle.isra.18+0x90/0x90 [ 5034.187779] ? hrtimer_interrupt+0x128/0x2c0 [ 5034.189866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5034.192963] ? apic_timer_interrupt+0xf/0x20 [ 5034.197325] [ 5034.198377] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5034.205897] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5034.211843] ? do_raw_spin_lock+0x177/0x190 [ 5034.215232] ? _raw_spin_lock+0x21/0x30 [ 5034.216950] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5034.219765] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5034.223107] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5034.225552] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5034.227673] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5034.238090] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5034.241133] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5034.247465] ? __next_timer_interrupt+0x160/0x160 [ 5034.249972] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5034.256341] ? do_wait_intr+0xf0/0xf0 [ 5034.261650] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5034.267016] ? kthread+0x1d1/0x200 [ 5034.269625] ? set_kthread_struct+0x70/0x70 [ 5034.274784] ? ret_from_fork+0x1f/0x30 [ 5036.058562] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5036.073204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5036.153619] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5036.171141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5036.178506] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5036.182415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5036.185667] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5036.197263] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5036.202581] RAX: 0000000000004c6f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5036.206319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5036.214003] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5036.217571] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5036.223682] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5036.227277] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5036.238018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5036.241457] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5036.247125] Call Trace: [ 5036.250137] [ 5036.250843] ? show_regs.cold.9+0x22/0x2f [ 5036.252843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5036.257877] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5036.263721] ? watchdog+0x40/0x40 [ 5036.265439] ? hrtimer_interrupt+0x128/0x2c0 [ 5036.268066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5036.272223] ? apic_timer_interrupt+0xf/0x20 [ 5036.275435] [ 5036.277337] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5036.280376] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5036.283429] do_raw_spin_lock+0x177/0x190 [ 5036.285804] _raw_spin_lock+0x21/0x30 [ 5036.290224] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5036.292217] process_one_work+0x2c8/0x700 [ 5036.293652] worker_thread+0x296/0x6e0 [ 5036.296893] ? rescuer_thread+0x570/0x570 [ 5036.299227] kthread+0x1d1/0x200 [ 5036.303614] ? set_kthread_struct+0x70/0x70 [ 5036.310895] ret_from_fork+0x1f/0x30 [ 5060.054063] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5060.057182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.058022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5060.060195] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5060.060207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.060277] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.060282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.060284] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.060298] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5060.060302] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5060.060307] RAX: 0000000000005022 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5060.060309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5060.060312] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5060.060314] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5060.060316] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5060.060319] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5060.060322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5060.060325] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5060.060333] Call Trace: [ 5060.060348] [ 5060.060353] ? show_regs.cold.9+0x22/0x2f [ 5060.060363] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5060.060369] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5060.060374] ? watchdog+0x40/0x40 [ 5060.060378] ? hrtimer_interrupt+0x128/0x2c0 [ 5060.060382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5060.060391] ? apic_timer_interrupt+0xf/0x20 [ 5060.060396] [ 5060.060398] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.060405] do_raw_spin_lock+0x177/0x190 [ 5060.060410] _raw_spin_lock+0x21/0x30 [ 5060.060415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5060.060736] ? do_raw_spin_unlock+0x75/0x190 [ 5060.060744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5060.060944] ? do_wait_intr+0xf0/0xf0 [ 5060.060951] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5060.061222] mdc_close+0x53b/0xd70 [mdc] [ 5060.061265] lmv_close+0x18d/0x490 [lmv] [ 5060.061294] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5060.061431] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5060.061511] ll_md_close+0x274/0x800 [lustre] [ 5060.061594] ll_file_release+0x128/0xc50 [lustre] [ 5060.061675] __fput+0x10d/0x440 [ 5060.061684] ____fput+0x12/0x20 [ 5060.061688] task_work_run+0xc6/0x110 [ 5060.061696] exit_to_usermode_loop+0x1dd/0x1f0 [ 5060.061703] do_syscall_64+0x3d6/0x440 [ 5060.061707] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5060.061714] RIP: 0033:0x7fa749552cd8 [ 5060.061721] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5060.061725] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5060.061729] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5060.061731] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5060.061733] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5060.061735] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5060.061737] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5060.078381] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.078395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.078398] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.078412] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5060.078417] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5060.078422] RAX: 0000000000002799 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5060.078424] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5060.078426] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5060.078428] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5060.078431] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5060.078434] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5060.078436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5060.078439] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5060.078447] Call Trace: [ 5060.078452] [ 5060.078456] ? show_regs.cold.9+0x22/0x2f [ 5060.078465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5060.078470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5060.078474] ? watchdog+0x40/0x40 [ 5060.078478] ? hrtimer_interrupt+0x128/0x2c0 [ 5060.078483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5060.078491] ? apic_timer_interrupt+0xf/0x20 [ 5060.078495] [ 5060.078498] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.078506] do_raw_spin_lock+0x177/0x190 [ 5060.078510] _raw_spin_lock+0x21/0x30 [ 5060.078515] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5060.078792] ? do_raw_spin_unlock+0x75/0x190 [ 5060.078798] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5060.078991] ? do_wait_intr+0xf0/0xf0 [ 5060.079000] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5060.079462] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5060.079682] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5060.079888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.079998] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5060.080037] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5060.080202] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5060.080233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.080318] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5060.080555] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5060.080583] lmv_intent_open+0x338/0x1150 [lmv] [ 5060.080612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.080694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.080779] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5060.080802] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5060.080886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.080970] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5060.081270] ? do_raw_spin_unlock+0x75/0x190 [ 5060.081277] ? _raw_spin_unlock+0x12/0x30 [ 5060.081284] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5060.081315] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5060.081346] ll_atomic_open+0x36a/0xd70 [lustre] [ 5060.081432] lookup_open+0x356/0xb40 [ 5060.081441] do_last+0x313/0xfc0 [ 5060.081446] ? nd_jump_root+0xe5/0x160 [ 5060.081451] ? path_init+0x437/0x520 [ 5060.081456] path_openat+0xf7/0x500 [ 5060.081461] ? do_raw_spin_unlock+0x75/0x190 [ 5060.081465] do_filp_open+0x99/0x140 [ 5060.081472] ? getname_flags+0x6e/0x330 [ 5060.081477] ? __check_object_size+0xff/0x256 [ 5060.081484] ? do_raw_spin_unlock+0x75/0x190 [ 5060.081488] ? _raw_spin_unlock+0x12/0x30 [ 5060.081492] do_sys_openat2+0x2b4/0x410 [ 5060.081496] do_sys_open+0x73/0xa0 [ 5060.081499] __x64_sys_openat+0x24/0x30 [ 5060.081503] do_syscall_64+0xc1/0x440 [ 5060.081510] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5060.081516] RIP: 0033:0x7fa749552332 [ 5060.081560] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5060.081565] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5060.081571] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5060.081573] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5060.081575] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5060.081578] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5060.081579] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5060.914889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.951896] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.961006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.968030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5060.973866] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5060.987636] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5060.991290] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5060.997383] RDX: 0000000000001b95 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5061.003809] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5061.006822] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5061.009873] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5061.012895] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5061.016255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5061.020180] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5061.024451] Call Trace: [ 5061.025333] [ 5061.026116] ? show_regs.cold.9+0x22/0x2f [ 5061.029279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5061.034981] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5061.037796] ? watchdog+0x40/0x40 [ 5061.043995] ? hrtimer_interrupt+0x128/0x2c0 [ 5061.045476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5061.047062] ? apic_timer_interrupt+0xf/0x20 [ 5061.051682] [ 5061.053605] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5061.057727] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5061.060715] do_raw_spin_lock+0x177/0x190 [ 5061.062777] _raw_spin_lock+0x21/0x30 [ 5061.063837] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5061.069204] obd_mod_free+0x46/0x140 [ptlrpc] [ 5061.073296] mdc_commit_open+0x91/0xb0 [mdc] [ 5061.076567] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5061.079682] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5061.083884] after_reply+0xe2c/0x1780 [ptlrpc] [ 5061.085957] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5061.092575] ? __next_timer_interrupt+0x160/0x160 [ 5061.094470] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5061.096361] ? do_wait_intr+0xf0/0xf0 [ 5061.097675] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5061.100654] kthread+0x1d1/0x200 [ 5061.102128] ? set_kthread_struct+0x70/0x70 [ 5061.104189] ret_from_fork+0x1f/0x30 [ 5062.624898] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3145s! [ 5062.628742] Showing busy workqueues and worker pools: [ 5062.634256] workqueue events: flags=0x0 [ 5062.639221] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5062.643985] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5062.654714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5062.659555] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5062.665029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.669950] pending: kfree_rcu_monitor [ 5062.674697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5062.679744] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5062.683784] workqueue events_long: flags=0x0 [ 5062.685802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.688750] pending: lu_objects_flush [obdclass] [ 5062.691244] workqueue events_unbound: flags=0x2 [ 5062.694432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5062.697382] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5062.701837] workqueue events_power_efficient: flags=0x80 [ 5062.706270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.708877] pending: check_lifetime [ 5062.711438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.716364] pending: neigh_periodic_work [ 5062.718506] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.722606] pending: neigh_periodic_work [ 5062.724693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.730867] pending: do_cache_clean [ 5062.732765] workqueue mm_percpu_wq: flags=0x8 [ 5062.737860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.741510] pending: vmstat_update [ 5062.745241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.749243] pending: vmstat_update [ 5062.750825] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5062.757189] pending: vmstat_update [ 5062.759930] workqueue writeback: flags=0x4a [ 5062.761864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5062.769364] pending: wb_workfn, wb_workfn [ 5062.774650] workqueue xprtiod: flags=0xa [ 5062.776717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5062.781203] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5062.787205] workqueue ipv6_addrconf: flags=0x40008 [ 5062.791073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5062.795472] pending: addrconf_verify_work [ 5062.798309] workqueue knbd0-recv: flags=0x1a [ 5062.800458] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5062.803284] in-flight: 651:recv_work [ 5062.804542] workqueue ptlrpc_pinger: flags=0x2 [ 5062.808300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5062.812720] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5062.815539] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3145s workers=4 idle: 37 31846 28897 [ 5062.821802] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5052s workers=2 idle: 653 [ 5064.058038] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5064.065295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5064.120253] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5064.136733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5064.149593] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5064.157946] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5064.161993] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5064.182695] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5064.191550] RAX: 000000000000600c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5064.202790] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5064.208791] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5064.218314] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5064.228245] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5064.238221] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5064.248415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5064.261276] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5064.269663] Call Trace: [ 5064.271576] [ 5064.278788] ? show_regs.cold.9+0x22/0x2f [ 5064.286486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5064.293036] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5064.296048] ? watchdog+0x40/0x40 [ 5064.298676] ? hrtimer_interrupt+0x128/0x2c0 [ 5064.302666] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5064.311312] ? apic_timer_interrupt+0xf/0x20 [ 5064.316283] [ 5064.318402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5064.323945] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5064.331930] do_raw_spin_lock+0x177/0x190 [ 5064.333514] _raw_spin_lock+0x21/0x30 [ 5064.334432] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5064.341933] process_one_work+0x2c8/0x700 [ 5064.348058] worker_thread+0x296/0x6e0 [ 5064.351716] ? rescuer_thread+0x570/0x570 [ 5064.358156] kthread+0x1d1/0x200 [ 5064.362515] ? set_kthread_struct+0x70/0x70 [ 5064.365076] ret_from_fork+0x1f/0x30 [ 5088.054117] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5088.057883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.058486] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5088.059075] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5088.059081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.059144] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.059149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.059151] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5088.059164] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5088.059169] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5088.059174] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5088.059176] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5088.059178] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5088.059181] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5088.059183] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5088.059186] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5088.059189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.059192] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5088.059200] Call Trace: [ 5088.059205] [ 5088.059209] ? show_regs.cold.9+0x22/0x2f [ 5088.059217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.059221] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.059224] ? watchdog+0x40/0x40 [ 5088.059227] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.059230] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.059235] ? apic_timer_interrupt+0xf/0x20 [ 5088.059238] [ 5088.059240] ? native_safe_halt+0x2e/0x50 [ 5088.059242] kvm_wait+0xcd/0xe0 [ 5088.059248] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5088.059253] do_raw_spin_lock+0x177/0x190 [ 5088.059256] _raw_spin_lock+0x21/0x30 [ 5088.059259] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5088.059449] ? do_raw_spin_unlock+0x75/0x190 [ 5088.059453] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5088.059593] ? do_wait_intr+0xf0/0xf0 [ 5088.059600] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5088.059869] mdc_close+0x53b/0xd70 [mdc] [ 5088.059906] lmv_close+0x18d/0x490 [lmv] [ 5088.059932] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5088.060039] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5088.060118] ll_md_close+0x274/0x800 [lustre] [ 5088.060183] ll_file_release+0x128/0xc50 [lustre] [ 5088.060236] __fput+0x10d/0x440 [ 5088.060243] ____fput+0x12/0x20 [ 5088.060246] task_work_run+0xc6/0x110 [ 5088.060252] exit_to_usermode_loop+0x1dd/0x1f0 [ 5088.060257] do_syscall_64+0x3d6/0x440 [ 5088.060260] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5088.060264] RIP: 0033:0x7fa749552cd8 [ 5088.060269] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5088.060272] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5088.060275] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5088.060277] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5088.060278] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5088.060280] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5088.060281] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5088.090924] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.090943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.090946] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5088.090963] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5088.090970] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5088.090975] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5088.090978] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5088.090985] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5088.090987] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5088.090989] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5088.090995] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5088.091049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.091052] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5088.091060] Call Trace: [ 5088.091066] [ 5088.091072] ? show_regs.cold.9+0x22/0x2f [ 5088.091082] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.091088] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.091092] ? watchdog+0x40/0x40 [ 5088.091096] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.091100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.091106] ? apic_timer_interrupt+0xf/0x20 [ 5088.091111] [ 5088.091119] ? native_safe_halt+0x2e/0x50 [ 5088.091124] kvm_wait+0xcd/0xe0 [ 5088.091135] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5088.091149] do_raw_spin_lock+0x177/0x190 [ 5088.091154] _raw_spin_lock+0x21/0x30 [ 5088.091160] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5088.099769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.099858] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.099866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.099869] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5088.099882] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5088.099887] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5088.099892] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5088.099894] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5088.099896] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5088.099899] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5088.099901] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5088.099904] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5088.099907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.099909] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5088.099918] Call Trace: [ 5088.099923] [ 5088.099929] ? show_regs.cold.9+0x22/0x2f [ 5088.099938] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.099943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.099948] ? watchdog+0x40/0x40 [ 5088.099952] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.099956] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.099963] ? apic_timer_interrupt+0xf/0x20 [ 5088.099967] [ 5088.099970] ? native_safe_halt+0x2e/0x50 [ 5088.099974] kvm_wait+0xcd/0xe0 [ 5088.099981] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5088.099988] do_raw_spin_lock+0x177/0x190 [ 5088.099993] _raw_spin_lock+0x21/0x30 [ 5088.100112] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5088.100397] obd_mod_free+0x46/0x140 [ptlrpc] [ 5088.101127] mdc_commit_open+0x91/0xb0 [mdc] [ 5088.101164] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5088.101372] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5088.102151] after_reply+0xe2c/0x1780 [ptlrpc] [ 5088.102362] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5088.110183] ? do_raw_spin_unlock+0x75/0x190 [ 5088.110203] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5088.110476] ? do_wait_intr+0xf0/0xf0 [ 5088.110485] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5088.110711] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5088.110935] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5088.111204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.111322] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5088.111359] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5088.111390] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5088.111415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.111499] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5088.111817] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5088.111861] lmv_intent_open+0x338/0x1150 [lmv] [ 5088.111894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.111981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.112115] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5088.112136] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5088.112229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.112319] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5088.112412] ? do_raw_spin_unlock+0x75/0x190 [ 5088.112419] ? _raw_spin_unlock+0x12/0x30 [ 5088.112427] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5088.112455] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5088.112621] ll_atomic_open+0x36a/0xd70 [lustre] [ 5088.112733] lookup_open+0x356/0xb40 [ 5088.112743] do_last+0x313/0xfc0 [ 5088.112748] ? nd_jump_root+0xe5/0x160 [ 5088.112754] ? path_init+0x437/0x520 [ 5088.112758] path_openat+0xf7/0x500 [ 5088.112763] ? do_raw_spin_unlock+0x75/0x190 [ 5088.112768] do_filp_open+0x99/0x140 [ 5088.112776] ? getname_flags+0x6e/0x330 [ 5088.112781] ? __check_object_size+0xff/0x256 [ 5088.112788] ? do_raw_spin_unlock+0x75/0x190 [ 5088.112793] ? _raw_spin_unlock+0x12/0x30 [ 5088.112797] do_sys_openat2+0x2b4/0x410 [ 5088.112802] do_sys_open+0x73/0xa0 [ 5088.112806] __x64_sys_openat+0x24/0x30 [ 5088.112810] do_syscall_64+0xc1/0x440 [ 5088.112822] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5088.112830] RIP: 0033:0x7fa749552332 [ 5088.112837] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5088.112840] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5088.112856] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5088.112858] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5088.112860] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5088.112862] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5088.112864] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5088.620656] ? __next_timer_interrupt+0x160/0x160 [ 5088.622918] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5088.625588] ? do_wait_intr+0xf0/0xf0 [ 5088.627334] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5088.630376] kthread+0x1d1/0x200 [ 5088.632584] ? set_kthread_struct+0x70/0x70 [ 5088.635191] ret_from_fork+0x1f/0x30 [ 5092.064813] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5092.071626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5092.119097] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5092.137666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5092.146408] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5092.162630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5092.176537] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5092.197583] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5092.204645] RAX: 0000000000007d7d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5092.212632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5092.229651] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5092.231270] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5092.247866] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5092.259435] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5092.267740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5092.285468] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5092.298233] Call Trace: [ 5092.302892] [ 5092.308184] ? show_regs.cold.9+0x22/0x2f [ 5092.309610] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5092.324496] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5092.336224] ? watchdog+0x40/0x40 [ 5092.347635] ? hrtimer_interrupt+0x128/0x2c0 [ 5092.349113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5092.363589] ? apic_timer_interrupt+0xf/0x20 [ 5092.365050] [ 5092.375337] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5092.377304] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5092.385348] do_raw_spin_lock+0x177/0x190 [ 5092.386787] _raw_spin_lock+0x21/0x30 [ 5092.390874] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5092.399054] process_one_work+0x2c8/0x700 [ 5092.404304] worker_thread+0x296/0x6e0 [ 5092.405455] ? rescuer_thread+0x570/0x570 [ 5092.409598] kthread+0x1d1/0x200 [ 5092.411342] ? set_kthread_struct+0x70/0x70 [ 5092.424142] ret_from_fork+0x1f/0x30 [ 5092.832377] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3175s! [ 5092.836723] Showing busy workqueues and worker pools: [ 5092.843460] workqueue events: flags=0x0 [ 5092.845928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5092.854857] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5092.869811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5092.873139] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5092.877000] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.885348] pending: kfree_rcu_monitor [ 5092.889115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5092.899293] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5092.907708] workqueue events_long: flags=0x0 [ 5092.909385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.911209] pending: lu_objects_flush [obdclass] [ 5092.917385] workqueue events_unbound: flags=0x2 [ 5092.918830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5092.923396] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5092.927860] workqueue events_power_efficient: flags=0x80 [ 5092.931263] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.936781] pending: check_lifetime [ 5092.938750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.943832] pending: neigh_periodic_work [ 5092.952619] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.959581] pending: neigh_periodic_work [ 5092.963640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.968557] pending: do_cache_clean [ 5092.970169] workqueue mm_percpu_wq: flags=0x8 [ 5092.972441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.975585] pending: vmstat_update [ 5092.977388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.980556] pending: vmstat_update [ 5092.982289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5092.985381] pending: vmstat_update [ 5092.986975] workqueue writeback: flags=0x4a [ 5092.989176] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5092.996978] pending: wb_workfn, wb_workfn [ 5093.002897] workqueue xprtiod: flags=0xa [ 5093.006865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5093.012861] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5093.019393] workqueue ipv6_addrconf: flags=0x40008 [ 5093.023696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5093.026832] pending: addrconf_verify_work [ 5093.030962] workqueue knbd0-recv: flags=0x1a [ 5093.032557] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5093.038020] in-flight: 651:recv_work [ 5093.041808] workqueue ptlrpc_pinger: flags=0x2 [ 5093.047333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5093.057201] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5093.063109] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3175s workers=4 idle: 37 31846 28897 [ 5093.071186] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5082s workers=2 idle: 653 [ 5116.058986] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5116.064128] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5116.069004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5116.069095] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5116.069102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5116.069105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5116.069117] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5116.069121] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5116.069125] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5116.069127] RDX: 0000000000001b5b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5116.069129] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5116.069131] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5116.069134] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5116.069137] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5116.069140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5116.069142] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5116.069151] Call Trace: [ 5116.069155] [ 5116.069161] ? show_regs.cold.9+0x22/0x2f [ 5116.069170] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5116.069177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5116.069180] ? watchdog+0x40/0x40 [ 5116.069185] ? hrtimer_interrupt+0x128/0x2c0 [ 5116.069189] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5116.069198] ? apic_timer_interrupt+0xf/0x20 [ 5116.069203] [ 5116.069206] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5116.069211] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5116.069215] do_raw_spin_lock+0x177/0x190 [ 5116.069219] _raw_spin_lock+0x21/0x30 [ 5116.069225] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5116.076259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5116.076337] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5116.076345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5116.076349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5116.077315] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5116.077320] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5116.077325] RAX: 000000000000798f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5116.077327] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5116.077330] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5116.077331] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5116.077334] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5116.077337] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5116.077340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5116.077342] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5116.077352] Call Trace: [ 5116.077369] [ 5116.077374] ? show_regs.cold.9+0x22/0x2f [ 5116.077383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5116.077390] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5116.077394] ? watchdog+0x40/0x40 [ 5116.077399] ? hrtimer_interrupt+0x128/0x2c0 [ 5116.077403] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5116.077410] ? apic_timer_interrupt+0xf/0x20 [ 5116.077415] [ 5116.077417] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5116.077425] do_raw_spin_lock+0x177/0x190 [ 5116.077429] _raw_spin_lock+0x21/0x30 [ 5116.077434] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5116.114425] obd_mod_free+0x46/0x140 [ptlrpc] [ 5116.114881] mdc_commit_open+0x91/0xb0 [mdc] [ 5116.114919] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5116.115203] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5116.115874] after_reply+0xe2c/0x1780 [ptlrpc] [ 5116.116145] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5116.116351] ? __next_timer_interrupt+0x160/0x160 [ 5116.116373] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5116.116826] ? do_wait_intr+0xf0/0xf0 [ 5116.116836] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5116.117130] kthread+0x1d1/0x200 [ 5116.117142] ? set_kthread_struct+0x70/0x70 [ 5116.117146] ret_from_fork+0x1f/0x30 [ 5116.687654] ? do_raw_spin_unlock+0x75/0x190 [ 5116.691392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5116.695431] ? do_wait_intr+0xf0/0xf0 [ 5116.699912] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5116.705387] mdc_close+0x53b/0xd70 [mdc] [ 5116.708400] lmv_close+0x18d/0x490 [lmv] [ 5116.714205] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5116.719850] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5116.726573] ll_md_close+0x274/0x800 [lustre] [ 5116.731978] ll_file_release+0x128/0xc50 [lustre] [ 5116.744966] __fput+0x10d/0x440 [ 5116.754752] ____fput+0x12/0x20 [ 5116.759757] task_work_run+0xc6/0x110 [ 5116.766428] exit_to_usermode_loop+0x1dd/0x1f0 [ 5116.770878] do_syscall_64+0x3d6/0x440 [ 5116.779846] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5116.784471] RIP: 0033:0x7fa749552cd8 [ 5116.786495] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5116.808482] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5116.818403] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5116.824440] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5116.831203] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5116.833523] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5116.841366] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5120.057204] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5120.057698] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5120.061803] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5120.061875] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5120.061881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5120.061887] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5120.062167] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5120.062176] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5120.062180] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5120.062185] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5120.062187] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5120.062189] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5120.062191] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5120.062194] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5120.062197] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5120.062199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5120.062201] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5120.062210] Call Trace: [ 5120.062215] [ 5120.062221] ? show_regs.cold.9+0x22/0x2f [ 5120.062230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5120.062235] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5120.062239] ? watchdog+0x40/0x40 [ 5120.062244] ? hrtimer_interrupt+0x128/0x2c0 [ 5120.062248] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5120.062254] ? apic_timer_interrupt+0xf/0x20 [ 5120.062259] [ 5120.062262] ? native_safe_halt+0x2e/0x50 [ 5120.062266] kvm_wait+0xcd/0xe0 [ 5120.062272] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5120.062280] do_raw_spin_lock+0x177/0x190 [ 5120.062297] _raw_spin_lock+0x21/0x30 [ 5120.062302] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5120.062512] process_one_work+0x2c8/0x700 [ 5120.062520] worker_thread+0x296/0x6e0 [ 5120.062525] ? rescuer_thread+0x570/0x570 [ 5120.062528] kthread+0x1d1/0x200 [ 5120.062533] ? set_kthread_struct+0x70/0x70 [ 5120.062537] ret_from_fork+0x1f/0x30 [ 5120.417654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5120.447773] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5120.461092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5120.474841] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5120.480616] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5120.502246] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5120.517941] RAX: 000000000000780c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5120.525515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5120.534254] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5120.541985] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5120.548136] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5120.553126] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5120.563297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5120.566682] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5120.571575] Call Trace: [ 5120.577262] [ 5120.578261] ? show_regs.cold.9+0x22/0x2f [ 5120.584894] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5120.588669] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5120.595645] ? watchdog+0x40/0x40 [ 5120.598665] ? hrtimer_interrupt+0x128/0x2c0 [ 5120.602013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5120.607109] ? apic_timer_interrupt+0xf/0x20 [ 5120.612215] [ 5120.614586] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5120.619787] do_raw_spin_lock+0x177/0x190 [ 5120.623428] _raw_spin_lock+0x21/0x30 [ 5120.626101] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5120.632818] ? do_raw_spin_unlock+0x75/0x190 [ 5120.641072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5120.646835] ? do_wait_intr+0xf0/0xf0 [ 5120.652326] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5120.661595] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5120.664567] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5120.667493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5120.674768] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5120.683729] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5120.693655] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5120.697268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5120.702937] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5120.714811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5120.720005] lmv_intent_open+0x338/0x1150 [lmv] [ 5120.727835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5120.735186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5120.743618] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5120.746885] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5120.749490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5120.752280] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5120.757911] ? do_raw_spin_unlock+0x75/0x190 [ 5120.761429] ? _raw_spin_unlock+0x12/0x30 [ 5120.765713] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5120.776438] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5120.782282] ll_atomic_open+0x36a/0xd70 [lustre] [ 5120.786844] lookup_open+0x356/0xb40 [ 5120.794137] do_last+0x313/0xfc0 [ 5120.798817] ? nd_jump_root+0xe5/0x160 [ 5120.803411] ? path_init+0x437/0x520 [ 5120.806520] path_openat+0xf7/0x500 [ 5120.808677] ? do_raw_spin_unlock+0x75/0x190 [ 5120.811083] do_filp_open+0x99/0x140 [ 5120.815772] ? getname_flags+0x6e/0x330 [ 5120.817942] ? __check_object_size+0xff/0x256 [ 5120.819247] ? do_raw_spin_unlock+0x75/0x190 [ 5120.822665] ? _raw_spin_unlock+0x12/0x30 [ 5120.827657] do_sys_openat2+0x2b4/0x410 [ 5120.829716] do_sys_open+0x73/0xa0 [ 5120.831453] __x64_sys_openat+0x24/0x30 [ 5120.832907] do_syscall_64+0xc1/0x440 [ 5120.834724] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5120.836300] RIP: 0033:0x7fa749552332 [ 5120.837516] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5120.849541] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5120.853542] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5120.857140] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5120.874020] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5120.883694] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5120.887884] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5123.555987] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3206s! [ 5123.561422] Showing busy workqueues and worker pools: [ 5123.570626] workqueue events: flags=0x0 [ 5123.575553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5123.585815] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5123.604028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5123.612023] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5123.617562] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.620621] pending: kfree_rcu_monitor [ 5123.621837] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5123.625152] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5123.636891] workqueue events_long: flags=0x0 [ 5123.644746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.651519] pending: lu_objects_flush [obdclass] [ 5123.658067] workqueue events_unbound: flags=0x2 [ 5123.661209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5123.669767] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5123.682054] workqueue events_power_efficient: flags=0x80 [ 5123.687922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.701335] pending: check_lifetime [ 5123.705054] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.712797] pending: neigh_periodic_work [ 5123.718541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.727787] pending: neigh_periodic_work [ 5123.730901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.738177] pending: do_cache_clean [ 5123.744477] workqueue mm_percpu_wq: flags=0x8 [ 5123.750031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.756849] pending: vmstat_update [ 5123.758083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.765082] pending: vmstat_update [ 5123.767169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5123.774036] pending: vmstat_update [ 5123.780053] workqueue writeback: flags=0x4a [ 5123.784264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5123.791955] pending: wb_workfn, wb_workfn [ 5123.793575] workqueue xprtiod: flags=0xa [ 5123.802323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5123.811194] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5123.815695] workqueue ipv6_addrconf: flags=0x40008 [ 5123.818790] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5123.826794] pending: addrconf_verify_work [ 5123.832011] workqueue knbd0-recv: flags=0x1a [ 5123.834204] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5123.839987] in-flight: 651:recv_work [ 5123.847213] workqueue ptlrpc_pinger: flags=0x2 [ 5123.856207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5123.865380] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5123.871890] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3206s workers=4 idle: 37 31846 28897 [ 5123.880392] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5113s workers=2 idle: 653 [ 5144.058009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5144.061433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.065998] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5144.089792] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.093063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.093140] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.093386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5144.093388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.093405] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5144.093409] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5144.093414] RAX: 0000000000007cae RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5144.093416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5144.093418] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5144.093421] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5144.093423] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5144.093427] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5144.093429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.093432] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5144.093441] Call Trace: [ 5144.093447] [ 5144.093453] ? show_regs.cold.9+0x22/0x2f [ 5144.093463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.093469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.093473] ? watchdog+0x40/0x40 [ 5144.093477] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.093481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.093490] ? apic_timer_interrupt+0xf/0x20 [ 5144.093495] [ 5144.093498] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.093504] do_raw_spin_lock+0x177/0x190 [ 5144.093508] _raw_spin_lock+0x21/0x30 [ 5144.093513] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5144.093788] ? do_raw_spin_unlock+0x75/0x190 [ 5144.093794] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5144.094083] ? do_wait_intr+0xf0/0xf0 [ 5144.094091] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5144.094409] mdc_close+0x53b/0xd70 [mdc] [ 5144.094451] lmv_close+0x18d/0x490 [lmv] [ 5144.094482] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5144.094598] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5144.094679] ll_md_close+0x274/0x800 [lustre] [ 5144.094767] ll_file_release+0x128/0xc50 [lustre] [ 5144.094852] __fput+0x10d/0x440 [ 5144.094862] ____fput+0x12/0x20 [ 5144.094876] task_work_run+0xc6/0x110 [ 5144.094884] exit_to_usermode_loop+0x1dd/0x1f0 [ 5144.094890] do_syscall_64+0x3d6/0x440 [ 5144.094894] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5144.094901] RIP: 0033:0x7fa749552cd8 [ 5144.094909] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5144.094912] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5144.094918] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5144.094920] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5144.094923] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5144.094924] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5144.094926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5144.343376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5144.352213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5144.358777] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5144.374981] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5144.381307] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5144.384891] RDX: 000000000000689e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5144.390174] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5144.399768] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5144.410072] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5144.414441] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5144.421537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.426502] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5144.430813] Call Trace: [ 5144.432067] [ 5144.432800] ? show_regs.cold.9+0x22/0x2f [ 5144.434898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.437375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.439866] ? watchdog+0x40/0x40 [ 5144.440984] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.442999] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.445089] ? apic_timer_interrupt+0xf/0x20 [ 5144.446930] [ 5144.447917] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5144.449903] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5144.453689] do_raw_spin_lock+0x177/0x190 [ 5144.455258] _raw_spin_lock+0x21/0x30 [ 5144.457222] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5144.461065] obd_mod_free+0x46/0x140 [ptlrpc] [ 5144.463426] mdc_commit_open+0x91/0xb0 [mdc] [ 5144.465689] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5144.469708] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5144.472121] after_reply+0xe2c/0x1780 [ptlrpc] [ 5144.475021] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5144.479201] ? __next_timer_interrupt+0x160/0x160 [ 5144.481446] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5144.484850] ? do_wait_intr+0xf0/0xf0 [ 5144.487186] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5144.489821] kthread+0x1d1/0x200 [ 5144.491001] ? set_kthread_struct+0x70/0x70 [ 5144.493093] ret_from_fork+0x1f/0x30 [ 5148.054087] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5148.057027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5148.064126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5148.064206] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5148.064215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5148.064219] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5148.064231] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5148.064235] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5148.064240] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5148.064243] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5148.064245] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5148.064247] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5148.064249] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5148.064252] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5148.064255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5148.064257] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5148.064267] Call Trace: [ 5148.064273] [ 5148.064278] ? show_regs.cold.9+0x22/0x2f [ 5148.064287] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5148.064294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5148.064298] ? watchdog+0x40/0x40 [ 5148.064302] ? hrtimer_interrupt+0x128/0x2c0 [ 5148.064306] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5148.064313] ? apic_timer_interrupt+0xf/0x20 [ 5148.064317] [ 5148.064319] ? native_safe_halt+0x2e/0x50 [ 5148.064323] kvm_wait+0xcd/0xe0 [ 5148.064331] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5148.064340] do_raw_spin_lock+0x177/0x190 [ 5148.064344] _raw_spin_lock+0x21/0x30 [ 5148.064348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5148.068423] Modules linked in: [ 5148.129248] ? do_raw_spin_unlock+0x75/0x190 [ 5148.129275] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5148.129749] ? do_wait_intr+0xf0/0xf0 [ 5148.129759] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5148.129995] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5148.130327] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5148.130584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5148.130927] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5148.130980] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5148.131016] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5148.131045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5148.131205] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5148.131470] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5148.131573] lmv_intent_open+0x338/0x1150 [lmv] [ 5148.131688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5148.131779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5148.131960] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5148.132039] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5148.132123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5148.132238] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5148.132363] ? do_raw_spin_unlock+0x75/0x190 [ 5148.132370] ? _raw_spin_unlock+0x12/0x30 [ 5148.132377] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5148.132416] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5148.132440] ll_atomic_open+0x36a/0xd70 [lustre] [ 5148.132727] lookup_open+0x356/0xb40 [ 5148.132737] do_last+0x313/0xfc0 [ 5148.132742] ? nd_jump_root+0xe5/0x160 [ 5148.132747] ? path_init+0x437/0x520 [ 5148.132751] path_openat+0xf7/0x500 [ 5148.132756] ? do_raw_spin_unlock+0x75/0x190 [ 5148.132760] do_filp_open+0x99/0x140 [ 5148.132766] ? getname_flags+0x6e/0x330 [ 5148.132770] ? __check_object_size+0xff/0x256 [ 5148.132777] ? do_raw_spin_unlock+0x75/0x190 [ 5148.132780] ? _raw_spin_unlock+0x12/0x30 [ 5148.132785] do_sys_openat2+0x2b4/0x410 [ 5148.132790] do_sys_open+0x73/0xa0 [ 5148.132807] __x64_sys_openat+0x24/0x30 [ 5148.132812] do_syscall_64+0xc1/0x440 [ 5148.132818] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5148.132824] RIP: 0033:0x7fa749552332 [ 5148.132831] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5148.132835] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5148.132847] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5148.132850] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5148.132855] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5148.132858] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5148.132859] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5148.356387] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5148.376183] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5148.385612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5148.394177] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5148.398539] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5148.401193] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5148.413179] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5148.417429] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5148.420765] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5148.423956] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5148.428043] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5148.432798] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5148.438303] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5148.442301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5148.445703] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5148.449158] Call Trace: [ 5148.450809] [ 5148.452299] ? show_regs.cold.9+0x22/0x2f [ 5148.454259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5148.456949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5148.459738] ? watchdog+0x40/0x40 [ 5148.461815] ? hrtimer_interrupt+0x128/0x2c0 [ 5148.463920] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5148.466585] ? apic_timer_interrupt+0xf/0x20 [ 5148.468891] [ 5148.470302] ? native_safe_halt+0x2e/0x50 [ 5148.472562] kvm_wait+0xcd/0xe0 [ 5148.474324] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5148.476210] do_raw_spin_lock+0x177/0x190 [ 5148.478861] _raw_spin_lock+0x21/0x30 [ 5148.481412] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5148.484331] process_one_work+0x2c8/0x700 [ 5148.486883] worker_thread+0x296/0x6e0 [ 5148.489274] ? rescuer_thread+0x570/0x570 [ 5148.491688] kthread+0x1d1/0x200 [ 5148.493758] ? set_kthread_struct+0x70/0x70 [ 5148.496146] ret_from_fork+0x1f/0x30 [ 5154.272083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3237s! [ 5154.276834] Showing busy workqueues and worker pools: [ 5154.279623] workqueue events: flags=0x0 [ 5154.282585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5154.286058] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5154.292422] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5154.295619] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5154.313240] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.315205] pending: kfree_rcu_monitor [ 5154.318700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5154.326320] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5154.330320] workqueue events_long: flags=0x0 [ 5154.331793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.340726] pending: lu_objects_flush [obdclass] [ 5154.342320] workqueue events_unbound: flags=0x2 [ 5154.345845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5154.353221] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5154.357014] workqueue events_power_efficient: flags=0x80 [ 5154.361547] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.365885] pending: check_lifetime [ 5154.368607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.371252] pending: neigh_periodic_work [ 5154.373154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.374883] pending: neigh_periodic_work [ 5154.378142] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.381212] pending: do_cache_clean [ 5154.383754] workqueue mm_percpu_wq: flags=0x8 [ 5154.387219] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.393722] pending: vmstat_update [ 5154.394881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.400009] pending: vmstat_update [ 5154.402721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5154.407842] pending: vmstat_update [ 5154.411667] workqueue writeback: flags=0x4a [ 5154.414643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5154.419881] pending: wb_workfn, wb_workfn [ 5154.421416] workqueue xprtiod: flags=0xa [ 5154.422502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5154.425463] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5154.429654] workqueue ipv6_addrconf: flags=0x40008 [ 5154.432814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5154.436382] pending: addrconf_verify_work [ 5154.444570] workqueue knbd0-recv: flags=0x1a [ 5154.447614] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5154.451272] in-flight: 651:recv_work [ 5154.453345] workqueue ptlrpc_pinger: flags=0x2 [ 5154.456372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5154.463480] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5154.465160] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3237s workers=4 idle: 37 31846 28897 [ 5154.471469] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5144s workers=2 idle: 653 [ 5172.059056] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5172.059312] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5172.064016] Modules linked in: [ 5172.069340] Modules linked in: [ 5172.069343] lustre(O) osp(O) [ 5172.071236] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.071310] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.071317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5172.071320] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5172.071333] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5172.071337] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5172.071341] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5172.071344] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5172.071346] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5172.071348] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5172.071351] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5172.071354] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5172.071358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.071360] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5172.071369] Call Trace: [ 5172.071387] [ 5172.071393] ? show_regs.cold.9+0x22/0x2f [ 5172.071403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.071410] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.071414] ? watchdog+0x40/0x40 [ 5172.071418] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.071423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.071430] ? apic_timer_interrupt+0xf/0x20 [ 5172.071435] [ 5172.071437] ? native_safe_halt+0x2e/0x50 [ 5172.071441] kvm_wait+0xcd/0xe0 [ 5172.071448] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5172.071456] do_raw_spin_lock+0x177/0x190 [ 5172.071460] _raw_spin_lock+0x21/0x30 [ 5172.071464] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5172.072075] ? do_raw_spin_unlock+0x75/0x190 [ 5172.072081] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5172.072285] ? do_wait_intr+0xf0/0xf0 [ 5172.072293] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5172.072503] mdc_close+0x53b/0xd70 [mdc] [ 5172.072542] lmv_close+0x18d/0x490 [lmv] [ 5172.077982] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.078061] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.078068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5172.078071] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5172.078084] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5172.078088] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5172.078093] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5172.078095] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5172.078096] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5172.078099] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5172.078101] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5172.078104] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5172.078107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.078109] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5172.078119] Call Trace: [ 5172.078125] [ 5172.078130] ? show_regs.cold.9+0x22/0x2f [ 5172.078140] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.078148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.078152] ? watchdog+0x40/0x40 [ 5172.078156] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.078160] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.078166] ? apic_timer_interrupt+0xf/0x20 [ 5172.078171] [ 5172.078174] ? native_safe_halt+0x2e/0x50 [ 5172.078178] kvm_wait+0xcd/0xe0 [ 5172.078185] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5172.078192] do_raw_spin_lock+0x177/0x190 [ 5172.078197] _raw_spin_lock+0x21/0x30 [ 5172.078201] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5172.078482] obd_mod_free+0x46/0x140 [ptlrpc] [ 5172.078676] mdc_commit_open+0x91/0xb0 [mdc] [ 5172.078711] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5172.082140] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5172.119329] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5172.128324] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5172.131141] after_reply+0xe2c/0x1780 [ptlrpc] [ 5172.131356] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5172.131531] ? __next_timer_interrupt+0x160/0x160 [ 5172.131540] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5172.131694] ? do_wait_intr+0xf0/0xf0 [ 5172.131702] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5172.131832] kthread+0x1d1/0x200 [ 5172.131836] ? set_kthread_struct+0x70/0x70 [ 5172.131838] ret_from_fork+0x1f/0x30 [ 5172.280869] ll_md_close+0x274/0x800 [lustre] [ 5172.282840] ll_file_release+0x128/0xc50 [lustre] [ 5172.285219] __fput+0x10d/0x440 [ 5172.286224] ____fput+0x12/0x20 [ 5172.287234] task_work_run+0xc6/0x110 [ 5172.288423] exit_to_usermode_loop+0x1dd/0x1f0 [ 5172.291533] do_syscall_64+0x3d6/0x440 [ 5172.292857] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5172.295553] RIP: 0033:0x7fa749552cd8 [ 5172.296991] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5172.305379] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5172.308292] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5172.310762] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5172.313394] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5172.316300] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5172.319328] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5176.061634] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5176.072097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5176.131420] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5176.144171] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5176.148962] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5176.151846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5176.154946] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5176.168638] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5176.172993] RAX: 000000000000010b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5176.178995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5176.187047] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5176.190654] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5176.194132] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5176.200652] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5176.211432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5176.216453] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5176.222610] Call Trace: [ 5176.223649] [ 5176.225363] ? show_regs.cold.9+0x22/0x2f [ 5176.226639] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5176.227805] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5176.233640] ? watchdog+0x40/0x40 [ 5176.237532] ? hrtimer_interrupt+0x128/0x2c0 [ 5176.245594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5176.248272] ? apic_timer_interrupt+0xf/0x20 [ 5176.250065] [ 5176.250850] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5176.253397] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5176.256358] do_raw_spin_lock+0x177/0x190 [ 5176.261556] _raw_spin_lock+0x21/0x30 [ 5176.263554] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5176.267336] process_one_work+0x2c8/0x700 [ 5176.269152] worker_thread+0x296/0x6e0 [ 5176.271287] ? rescuer_thread+0x570/0x570 [ 5176.272755] kthread+0x1d1/0x200 [ 5176.276452] ? set_kthread_struct+0x70/0x70 [ 5176.279361] ret_from_fork+0x1f/0x30 [ 5180.054809] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5180.060707] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5180.109582] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5180.115804] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5180.123204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5180.127417] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5180.135430] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5180.139139] RAX: 00000000000074d2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5180.142830] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5180.149161] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5180.153757] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5180.157606] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5180.162695] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5180.167005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5180.170436] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5180.175187] Call Trace: [ 5180.177754] [ 5180.179396] ? show_regs.cold.9+0x22/0x2f [ 5180.181725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5180.185513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5180.187533] ? watchdog+0x40/0x40 [ 5180.189418] ? hrtimer_interrupt+0x128/0x2c0 [ 5180.191565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5180.194212] ? apic_timer_interrupt+0xf/0x20 [ 5180.196458] [ 5180.197673] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5180.201192] do_raw_spin_lock+0x177/0x190 [ 5180.203207] _raw_spin_lock+0x21/0x30 [ 5180.205412] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5180.209979] ? do_raw_spin_unlock+0x75/0x190 [ 5180.212382] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5180.216449] ? do_wait_intr+0xf0/0xf0 [ 5180.218257] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5180.222287] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5180.225473] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5180.228630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5180.231132] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5180.233839] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5180.236309] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5180.237924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5180.240344] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5180.242802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5180.245483] lmv_intent_open+0x338/0x1150 [lmv] [ 5180.247705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5180.250412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5180.254670] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5180.259757] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5180.263354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5180.266405] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5180.270430] ? do_raw_spin_unlock+0x75/0x190 [ 5180.274107] ? _raw_spin_unlock+0x12/0x30 [ 5180.275483] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5180.280957] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5180.285171] ll_atomic_open+0x36a/0xd70 [lustre] [ 5180.286458] lookup_open+0x356/0xb40 [ 5180.288179] do_last+0x313/0xfc0 [ 5180.290301] ? nd_jump_root+0xe5/0x160 [ 5180.292355] ? path_init+0x437/0x520 [ 5180.293782] path_openat+0xf7/0x500 [ 5180.297177] ? do_raw_spin_unlock+0x75/0x190 [ 5180.304670] do_filp_open+0x99/0x140 [ 5180.307460] ? getname_flags+0x6e/0x330 [ 5180.310944] ? __check_object_size+0xff/0x256 [ 5180.316822] ? do_raw_spin_unlock+0x75/0x190 [ 5180.319959] ? _raw_spin_unlock+0x12/0x30 [ 5180.321444] do_sys_openat2+0x2b4/0x410 [ 5180.324154] do_sys_open+0x73/0xa0 [ 5180.327722] __x64_sys_openat+0x24/0x30 [ 5180.328958] do_syscall_64+0xc1/0x440 [ 5180.345661] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5180.351272] RIP: 0033:0x7fa749552332 [ 5180.360193] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5180.377931] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5180.387164] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5180.394035] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5180.400771] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5180.404086] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5180.410730] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5184.485758] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3267s! [ 5184.495325] Showing busy workqueues and worker pools: [ 5184.496906] workqueue events: flags=0x0 [ 5184.498090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5184.504731] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5184.515009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5184.528002] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5184.539060] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.543322] pending: kfree_rcu_monitor [ 5184.554860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5184.565497] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5184.578138] workqueue events_long: flags=0x0 [ 5184.579523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.591761] pending: lu_objects_flush [obdclass] [ 5184.595738] workqueue events_unbound: flags=0x2 [ 5184.598316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5184.608355] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5184.622480] workqueue events_power_efficient: flags=0x80 [ 5184.627503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.636087] pending: check_lifetime [ 5184.643763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.647904] pending: neigh_periodic_work [ 5184.656073] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.668145] pending: neigh_periodic_work [ 5184.672617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.680831] pending: do_cache_clean [ 5184.684065] workqueue mm_percpu_wq: flags=0x8 [ 5184.686995] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.691565] pending: vmstat_update [ 5184.697546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.702135] pending: vmstat_update [ 5184.705405] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5184.709758] pending: vmstat_update [ 5184.715888] workqueue writeback: flags=0x4a [ 5184.720831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5184.736174] pending: wb_workfn, wb_workfn [ 5184.750915] workqueue xprtiod: flags=0xa [ 5184.752344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5184.763396] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5184.770734] workqueue ipv6_addrconf: flags=0x40008 [ 5184.774175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5184.782112] pending: addrconf_verify_work [ 5184.783749] workqueue knbd0-recv: flags=0x1a [ 5184.786146] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5184.789517] in-flight: 651:recv_work [ 5184.791462] workqueue ptlrpc_pinger: flags=0x2 [ 5184.793501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5184.796532] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5184.805088] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3267s workers=4 idle: 37 31846 28897 [ 5184.812505] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5174s workers=2 idle: 653 [ 5200.058016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5200.060625] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5200.063963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.064052] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.064060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.064063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5200.064077] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5200.064081] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5200.064085] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5200.064088] RDX: 00000000000015d6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5200.064090] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5200.064093] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5200.064095] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5200.064132] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5200.064136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.064139] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5200.064147] Call Trace: [ 5200.064153] [ 5200.064159] ? show_regs.cold.9+0x22/0x2f [ 5200.064168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.064175] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.064179] ? watchdog+0x40/0x40 [ 5200.064183] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.064192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.064200] ? apic_timer_interrupt+0xf/0x20 [ 5200.064205] [ 5200.064208] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5200.064213] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5200.064217] do_raw_spin_lock+0x177/0x190 [ 5200.064222] _raw_spin_lock+0x21/0x30 [ 5200.064227] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5200.065100] obd_mod_free+0x46/0x140 [ptlrpc] [ 5200.065456] mdc_commit_open+0x91/0xb0 [mdc] [ 5200.066017] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5200.066343] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5200.066574] after_reply+0xe2c/0x1780 [ptlrpc] [ 5200.067763] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5200.068180] ? __next_timer_interrupt+0x160/0x160 [ 5200.068190] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5200.068408] ? do_wait_intr+0xf0/0xf0 [ 5200.068417] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5200.068640] kthread+0x1d1/0x200 [ 5200.068648] ? set_kthread_struct+0x70/0x70 [ 5200.068652] ret_from_fork+0x1f/0x30 [ 5200.277861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.316782] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.321238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.325449] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5200.328481] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5200.344127] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5200.347333] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5200.350442] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5200.353338] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5200.356548] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5200.361534] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5200.363631] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5200.371376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.373950] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5200.383288] Call Trace: [ 5200.385484] [ 5200.386313] ? show_regs.cold.9+0x22/0x2f [ 5200.391831] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.400219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.402764] ? watchdog+0x40/0x40 [ 5200.408085] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.412114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.418062] ? apic_timer_interrupt+0xf/0x20 [ 5200.421756] [ 5200.422364] ? native_safe_halt+0x2e/0x50 [ 5200.426439] kvm_wait+0xcd/0xe0 [ 5200.428481] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5200.433145] do_raw_spin_lock+0x177/0x190 [ 5200.434214] _raw_spin_lock+0x21/0x30 [ 5200.435243] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5200.444183] ? do_raw_spin_unlock+0x75/0x190 [ 5200.446855] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5200.453272] ? do_wait_intr+0xf0/0xf0 [ 5200.454280] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5200.455759] mdc_close+0x53b/0xd70 [mdc] [ 5200.469295] lmv_close+0x18d/0x490 [lmv] [ 5200.473944] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5200.479566] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5200.483580] ll_md_close+0x274/0x800 [lustre] [ 5200.486107] ll_file_release+0x128/0xc50 [lustre] [ 5200.493517] __fput+0x10d/0x440 [ 5200.494451] ____fput+0x12/0x20 [ 5200.495441] task_work_run+0xc6/0x110 [ 5200.501157] exit_to_usermode_loop+0x1dd/0x1f0 [ 5200.505340] do_syscall_64+0x3d6/0x440 [ 5200.511569] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5200.515422] RIP: 0033:0x7fa749552cd8 [ 5200.520750] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5200.532222] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5200.534207] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5200.541292] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5200.555178] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5200.557103] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5200.566952] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5204.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5204.060039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5204.081124] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5204.085636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5204.089576] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5204.092774] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5204.094726] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5204.103188] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5204.108921] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5204.113617] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5204.117007] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5204.120669] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5204.123907] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5204.127763] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5204.131055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5204.133648] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5204.137715] Call Trace: [ 5204.138738] [ 5204.139913] ? show_regs.cold.9+0x22/0x2f [ 5204.141701] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5204.143931] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5204.145775] ? watchdog+0x40/0x40 [ 5204.147278] ? hrtimer_interrupt+0x128/0x2c0 [ 5204.149191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5204.151337] ? apic_timer_interrupt+0xf/0x20 [ 5204.153149] [ 5204.154423] ? native_safe_halt+0x2e/0x50 [ 5204.156433] kvm_wait+0xcd/0xe0 [ 5204.157993] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5204.160697] do_raw_spin_lock+0x177/0x190 [ 5204.162762] _raw_spin_lock+0x21/0x30 [ 5204.164184] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5204.167106] process_one_work+0x2c8/0x700 [ 5204.169250] worker_thread+0x296/0x6e0 [ 5204.170943] ? rescuer_thread+0x570/0x570 [ 5204.172759] kthread+0x1d1/0x200 [ 5204.174154] ? set_kthread_struct+0x70/0x70 [ 5204.176084] ret_from_fork+0x1f/0x30 [ 5212.058399] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5212.069710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5212.115067] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5212.121585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5212.130789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5212.136098] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5212.150591] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5212.154115] RAX: 0000000000006695 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5212.157600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5212.161522] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5212.166185] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5212.169887] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5212.172459] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5212.175821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5212.177997] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5212.180885] Call Trace: [ 5212.181828] [ 5212.182635] ? show_regs.cold.9+0x22/0x2f [ 5212.184599] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5212.186334] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5212.188556] ? watchdog+0x40/0x40 [ 5212.189930] ? hrtimer_interrupt+0x128/0x2c0 [ 5212.191445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5212.193634] ? apic_timer_interrupt+0xf/0x20 [ 5212.195869] [ 5212.197530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5212.206950] do_raw_spin_lock+0x177/0x190 [ 5212.210102] _raw_spin_lock+0x21/0x30 [ 5212.215699] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5212.218795] ? do_raw_spin_unlock+0x75/0x190 [ 5212.222699] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5212.225104] ? do_wait_intr+0xf0/0xf0 [ 5212.226922] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5212.236457] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5212.243233] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5212.247700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5212.251389] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5212.255172] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5212.260594] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5212.263150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5212.268468] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5212.272380] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5212.278049] lmv_intent_open+0x338/0x1150 [lmv] [ 5212.281280] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5212.284036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5212.286779] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5212.288389] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5212.291567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5212.302020] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5212.303864] ? do_raw_spin_unlock+0x75/0x190 [ 5212.311692] ? _raw_spin_unlock+0x12/0x30 [ 5212.315688] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5212.318808] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5212.323619] ll_atomic_open+0x36a/0xd70 [lustre] [ 5212.328069] lookup_open+0x356/0xb40 [ 5212.337195] do_last+0x313/0xfc0 [ 5212.341373] ? nd_jump_root+0xe5/0x160 [ 5212.345689] ? path_init+0x437/0x520 [ 5212.347822] path_openat+0xf7/0x500 [ 5212.354049] ? do_raw_spin_unlock+0x75/0x190 [ 5212.357037] do_filp_open+0x99/0x140 [ 5212.361967] ? getname_flags+0x6e/0x330 [ 5212.365248] ? __check_object_size+0xff/0x256 [ 5212.368774] ? do_raw_spin_unlock+0x75/0x190 [ 5212.369809] ? _raw_spin_unlock+0x12/0x30 [ 5212.370850] do_sys_openat2+0x2b4/0x410 [ 5212.375587] do_sys_open+0x73/0xa0 [ 5212.377263] __x64_sys_openat+0x24/0x30 [ 5212.379902] do_syscall_64+0xc1/0x440 [ 5212.381131] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5212.389123] RIP: 0033:0x7fa749552332 [ 5212.390563] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5212.412337] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5212.422039] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5212.424167] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5212.426733] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5212.437299] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5212.442372] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5213.759141] rcu: INFO: rcu_sched self-detected stall on CPU [ 5213.763690] rcu: 2-...!: (1792612 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 5213.776090] (t=3300070 jiffies g=258701 q=2512) [ 5213.779566] rcu: rcu_sched kthread starved for 3296865 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5213.796579] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5213.818351] rcu: RCU grace-period kthread stack dump: [ 5213.820787] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5213.826886] Call Trace: [ 5213.828980] __schedule+0x351/0xcb0 [ 5213.835288] schedule+0xc0/0x180 [ 5213.838539] schedule_timeout+0xb4/0x190 [ 5213.843343] ? do_raw_spin_unlock+0x75/0x190 [ 5213.846163] ? __next_timer_interrupt+0x160/0x160 [ 5213.847754] ? prepare_to_swait_event+0xf3/0x140 [ 5213.854516] rcu_gp_kthread+0x8dd/0x15b0 [ 5213.858181] ? force_qs_rnp+0x2c0/0x2c0 [ 5213.862595] kthread+0x1d1/0x200 [ 5213.866596] ? set_kthread_struct+0x70/0x70 [ 5213.870807] ret_from_fork+0x1f/0x30 [ 5213.872152] rcu: Stack dump where RCU GP kthread last ran: [ 5213.882913] Sending NMI from CPU 2 to CPUs 1: [ 5213.887305] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 5213.890107] NMI backtrace for cpu 2 [ 5213.896249] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5213.909935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5213.917671] Call Trace: [ 5213.919756] [ 5213.920346] ? dump_stack+0xbb/0x10e [ 5213.921131] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5213.922785] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5213.925668] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5213.928074] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5213.930884] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5213.933556] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5213.935599] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5213.938265] ? update_process_times+0x6d/0xc0 [ 5213.941215] ? tick_sched_handle.isra.18+0x27/0x90 [ 5213.943469] ? tick_sched_timer+0x53/0xb0 [ 5213.945755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5213.947355] ? tick_sched_handle.isra.18+0x90/0x90 [ 5213.949648] ? hrtimer_interrupt+0x128/0x2c0 [ 5213.950925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5213.953171] ? apic_timer_interrupt+0xf/0x20 [ 5213.954497] [ 5213.957253] ? native_safe_halt+0x2e/0x50 [ 5213.961429] ? kvm_wait+0xcd/0xe0 [ 5213.962982] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5213.965716] ? do_raw_spin_lock+0x177/0x190 [ 5213.969793] ? _raw_spin_lock+0x21/0x30 [ 5213.971215] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5213.979656] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5213.983586] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5213.985611] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5213.998212] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5214.000217] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5214.009032] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5214.013981] ? __next_timer_interrupt+0x160/0x160 [ 5214.019703] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5214.025369] ? do_wait_intr+0xf0/0xf0 [ 5214.026823] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5214.038842] ? kthread+0x1d1/0x200 [ 5214.042534] ? set_kthread_struct+0x70/0x70 [ 5214.044636] ? ret_from_fork+0x1f/0x30 [ 5215.202302] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3298s! [ 5215.209264] Showing busy workqueues and worker pools: [ 5215.213491] workqueue events: flags=0x0 [ 5215.217039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5215.224016] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5215.231996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5215.239958] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5215.249950] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.261641] pending: kfree_rcu_monitor [ 5215.267440] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5215.273704] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5215.278481] workqueue events_long: flags=0x0 [ 5215.280708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.288759] pending: lu_objects_flush [obdclass] [ 5215.295200] workqueue events_unbound: flags=0x2 [ 5215.299502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5215.308746] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5215.318405] workqueue events_power_efficient: flags=0x80 [ 5215.327220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.335579] pending: check_lifetime [ 5215.337804] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.347509] pending: neigh_periodic_work [ 5215.354061] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.358881] pending: neigh_periodic_work [ 5215.361123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.364650] pending: do_cache_clean [ 5215.366837] workqueue mm_percpu_wq: flags=0x8 [ 5215.374300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.379048] pending: vmstat_update [ 5215.382435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.387601] pending: vmstat_update [ 5215.391783] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5215.395071] pending: vmstat_update [ 5215.396043] workqueue writeback: flags=0x4a [ 5215.402046] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5215.405043] pending: wb_workfn, wb_workfn [ 5215.409760] workqueue xprtiod: flags=0xa [ 5215.410926] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5215.414592] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5215.419015] workqueue ipv6_addrconf: flags=0x40008 [ 5215.421499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5215.424784] pending: addrconf_verify_work [ 5215.427779] workqueue knbd0-recv: flags=0x1a [ 5215.429701] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5215.432888] in-flight: 651:recv_work [ 5215.434592] workqueue ptlrpc_pinger: flags=0x2 [ 5215.436453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5215.439074] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5215.443567] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3298s workers=4 idle: 37 31846 28897 [ 5215.450103] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5205s workers=2 idle: 653 [ 5228.059202] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5228.064961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5228.090753] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5228.096847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5228.102073] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.105876] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5228.117092] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5228.121055] RAX: 0000000000002583 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5228.126947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5228.129447] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5228.133772] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5228.138859] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5228.144711] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5228.149389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5228.154886] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5228.159766] Call Trace: [ 5228.160681] [ 5228.161357] ? show_regs.cold.9+0x22/0x2f [ 5228.165997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5228.171836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5228.174072] ? watchdog+0x40/0x40 [ 5228.177797] ? hrtimer_interrupt+0x128/0x2c0 [ 5228.179893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5228.183891] ? apic_timer_interrupt+0xf/0x20 [ 5228.185374] [ 5228.186106] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.199103] do_raw_spin_lock+0x177/0x190 [ 5228.204251] _raw_spin_lock+0x21/0x30 [ 5228.207577] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5228.213180] ? do_raw_spin_unlock+0x75/0x190 [ 5228.219974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5228.223827] ? do_wait_intr+0xf0/0xf0 [ 5228.225216] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5228.228956] mdc_close+0x53b/0xd70 [mdc] [ 5228.231220] lmv_close+0x18d/0x490 [lmv] [ 5228.233645] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5228.236753] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5228.239794] ll_md_close+0x274/0x800 [lustre] [ 5228.242882] ll_file_release+0x128/0xc50 [lustre] [ 5228.245778] __fput+0x10d/0x440 [ 5228.247310] ____fput+0x12/0x20 [ 5228.249238] task_work_run+0xc6/0x110 [ 5228.251158] exit_to_usermode_loop+0x1dd/0x1f0 [ 5228.252731] do_syscall_64+0x3d6/0x440 [ 5228.254302] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5228.258204] RIP: 0033:0x7fa749552cd8 [ 5228.260225] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5228.271101] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5228.275942] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5228.280934] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5228.285979] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5228.292315] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5228.297619] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5240.054060] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5240.057132] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5240.058438] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5240.058450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.058526] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.058532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.058534] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5240.058548] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5240.058554] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5240.058559] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5240.058561] RDX: 0000000000005ac8 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5240.058563] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5240.058588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.058643] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.058649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.058652] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5240.058660] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5240.058664] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5240.058668] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5240.058671] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5240.058673] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5240.058675] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5240.058678] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5240.058680] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5240.058684] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5240.058688] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5240.058691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.058693] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5240.058701] Call Trace: [ 5240.058707] [ 5240.058714] ? show_regs.cold.9+0x22/0x2f [ 5240.058724] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.058731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.058736] ? watchdog+0x40/0x40 [ 5240.058740] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.058745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.058752] ? apic_timer_interrupt+0xf/0x20 [ 5240.058757] [ 5240.058760] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5240.058765] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5240.058770] do_raw_spin_lock+0x177/0x190 [ 5240.058775] _raw_spin_lock+0x21/0x30 [ 5240.058804] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5240.058808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.058879] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5240.058887] Call Trace: [ 5240.058890] [ 5240.058894] ? show_regs.cold.9+0x22/0x2f [ 5240.058901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.058906] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.058910] ? watchdog+0x40/0x40 [ 5240.058914] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.058919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.058925] ? apic_timer_interrupt+0xf/0x20 [ 5240.058929] [ 5240.058932] ? native_safe_halt+0x2e/0x50 [ 5240.058936] kvm_wait+0xcd/0xe0 [ 5240.058782] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5240.058944] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5240.058954] do_raw_spin_lock+0x177/0x190 [ 5240.058958] _raw_spin_lock+0x21/0x30 [ 5240.058963] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5240.059483] ? do_raw_spin_unlock+0x75/0x190 [ 5240.059490] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5240.059837] ? do_wait_intr+0xf0/0xf0 [ 5240.059847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5240.060578] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5240.060928] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5240.061261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.061445] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5240.061482] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5240.061513] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5240.061541] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.061624] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5240.062055] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5240.062086] lmv_intent_open+0x338/0x1150 [lmv] [ 5240.062195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.062353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.062439] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5240.062463] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5240.062548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.062637] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5240.062931] ? do_raw_spin_unlock+0x75/0x190 [ 5240.062939] ? _raw_spin_unlock+0x12/0x30 [ 5240.062946] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5240.062975] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5240.062996] ll_atomic_open+0x36a/0xd70 [lustre] [ 5240.063092] lookup_open+0x356/0xb40 [ 5240.063100] do_last+0x313/0xfc0 [ 5240.063105] ? nd_jump_root+0xe5/0x160 [ 5240.063110] ? path_init+0x437/0x520 [ 5240.063202] path_openat+0xf7/0x500 [ 5240.063208] ? do_raw_spin_unlock+0x75/0x190 [ 5240.063213] do_filp_open+0x99/0x140 [ 5240.063219] ? getname_flags+0x6e/0x330 [ 5240.063223] ? __check_object_size+0xff/0x256 [ 5240.063230] ? do_raw_spin_unlock+0x75/0x190 [ 5240.063234] ? _raw_spin_unlock+0x12/0x30 [ 5240.063239] do_sys_openat2+0x2b4/0x410 [ 5240.063243] do_sys_open+0x73/0xa0 [ 5240.063248] __x64_sys_openat+0x24/0x30 [ 5240.063251] do_syscall_64+0xc1/0x440 [ 5240.063257] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5240.063264] RIP: 0033:0x7fa749552332 [ 5240.063272] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5240.063276] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5240.063455] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5240.063459] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5240.063462] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5240.063464] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5240.063467] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5240.059175] obd_mod_free+0x46/0x140 [ptlrpc] [ 5240.063729] mdc_commit_open+0x91/0xb0 [mdc] [ 5240.063763] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5240.063971] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5240.064173] after_reply+0xe2c/0x1780 [ptlrpc] [ 5240.073412] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.074204] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.074212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.074219] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5240.082559] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5240.083289] ? __next_timer_interrupt+0x160/0x160 [ 5240.083300] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5240.131886] [ 5240.131959] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5240.131976] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5240.131981] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5240.131985] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5240.132004] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5240.132007] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5240.132009] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5240.132011] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5240.132014] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5240.132017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.132021] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5240.132029] Call Trace: [ 5240.132035] [ 5240.132097] ? show_regs.cold.9+0x22/0x2f [ 5240.132110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.132117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.132123] ? watchdog+0x40/0x40 [ 5240.132127] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.132131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.132137] ? apic_timer_interrupt+0xf/0x20 [ 5240.132141] [ 5240.132144] ? native_safe_halt+0x2e/0x50 [ 5240.132148] kvm_wait+0xcd/0xe0 [ 5240.132155] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5240.132164] do_raw_spin_lock+0x177/0x190 [ 5240.132168] _raw_spin_lock+0x21/0x30 [ 5240.132172] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5240.132766] process_one_work+0x2c8/0x700 [ 5240.132780] worker_thread+0x296/0x6e0 [ 5240.132785] ? rescuer_thread+0x570/0x570 [ 5240.132788] kthread+0x1d1/0x200 [ 5240.132793] ? set_kthread_struct+0x70/0x70 [ 5240.132796] ret_from_fork+0x1f/0x30 [ 5240.937575] ? do_wait_intr+0xf0/0xf0 [ 5240.943258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5240.952217] kthread+0x1d1/0x200 [ 5240.953438] ? set_kthread_struct+0x70/0x70 [ 5240.966695] ret_from_fork+0x1f/0x30 [ 5245.921069] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3328s! [ 5245.925095] Showing busy workqueues and worker pools: [ 5245.928402] workqueue events: flags=0x0 [ 5245.930671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5245.935617] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5245.941763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5245.945618] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5245.952488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5245.957544] pending: kfree_rcu_monitor [ 5245.965187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5245.974756] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5245.987024] workqueue events_long: flags=0x0 [ 5245.990367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5245.992740] pending: lu_objects_flush [obdclass] [ 5245.996434] workqueue events_unbound: flags=0x2 [ 5245.998651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5246.001710] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5246.006609] workqueue events_power_efficient: flags=0x80 [ 5246.009137] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5246.012517] pending: check_lifetime [ 5246.014551] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5246.017988] pending: neigh_periodic_work [ 5246.020716] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5246.025178] pending: neigh_periodic_work [ 5246.028719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5246.035745] pending: do_cache_clean [ 5246.040589] workqueue mm_percpu_wq: flags=0x8 [ 5246.044758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5246.052738] pending: vmstat_update [ 5246.057356] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5246.065184] pending: vmstat_update [ 5246.070715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5246.079409] pending: vmstat_update [ 5246.082665] workqueue writeback: flags=0x4a [ 5246.089234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5246.096708] pending: wb_workfn, wb_workfn [ 5246.101611] workqueue xprtiod: flags=0xa [ 5246.104733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5246.112650] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5246.124868] workqueue ipv6_addrconf: flags=0x40008 [ 5246.131285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5246.143580] pending: addrconf_verify_work [ 5246.149650] workqueue knbd0-recv: flags=0x1a [ 5246.154693] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5246.161812] in-flight: 651:recv_work [ 5246.166529] workqueue ptlrpc_pinger: flags=0x2 [ 5246.169755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5246.176785] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5246.184504] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3329s workers=4 idle: 37 31846 28897 [ 5246.191018] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5236s workers=2 idle: 653 [ 5256.059618] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5256.069087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5256.107874] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5256.113608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5256.130340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.139100] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5256.156459] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5256.165455] RAX: 0000000000002f4a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5256.172833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5256.181814] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5256.191730] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5256.195758] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5256.203297] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5256.214310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5256.225611] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5256.236315] Call Trace: [ 5256.237421] [ 5256.238674] ? show_regs.cold.9+0x22/0x2f [ 5256.244241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5256.249726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5256.255495] ? watchdog+0x40/0x40 [ 5256.256864] ? hrtimer_interrupt+0x128/0x2c0 [ 5256.260424] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5256.266669] ? apic_timer_interrupt+0xf/0x20 [ 5256.270430] [ 5256.271842] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.274404] do_raw_spin_lock+0x177/0x190 [ 5256.276695] _raw_spin_lock+0x21/0x30 [ 5256.280312] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5256.288859] ? do_raw_spin_unlock+0x75/0x190 [ 5256.290395] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5256.294384] ? do_wait_intr+0xf0/0xf0 [ 5256.302380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5256.304699] mdc_close+0x53b/0xd70 [mdc] [ 5256.306078] lmv_close+0x18d/0x490 [lmv] [ 5256.310978] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5256.313576] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5256.318233] ll_md_close+0x274/0x800 [lustre] [ 5256.320522] ll_file_release+0x128/0xc50 [lustre] [ 5256.324357] __fput+0x10d/0x440 [ 5256.326786] ____fput+0x12/0x20 [ 5256.327929] task_work_run+0xc6/0x110 [ 5256.329236] exit_to_usermode_loop+0x1dd/0x1f0 [ 5256.333578] do_syscall_64+0x3d6/0x440 [ 5256.335955] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5256.338139] RIP: 0033:0x7fa749552cd8 [ 5256.339227] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5256.347983] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5256.350520] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5256.356735] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5256.359477] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5256.363846] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5256.366894] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5268.057058] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5268.058037] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5268.059940] Modules linked in: [ 5268.063216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5268.063297] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5268.063307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5268.063311] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5268.063324] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5268.063329] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5268.063333] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5268.063335] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5268.063338] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5268.063340] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5268.063343] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5268.063346] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5268.063349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5268.063351] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5268.063361] Call Trace: [ 5268.063367] [ 5268.063374] ? show_regs.cold.9+0x22/0x2f [ 5268.063384] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5268.063391] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5268.063395] ? watchdog+0x40/0x40 [ 5268.063399] ? hrtimer_interrupt+0x128/0x2c0 [ 5268.063403] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5268.063412] ? apic_timer_interrupt+0xf/0x20 [ 5268.063417] [ 5268.063419] ? native_safe_halt+0x2e/0x50 [ 5268.063423] kvm_wait+0xcd/0xe0 [ 5268.063431] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5268.063439] do_raw_spin_lock+0x177/0x190 [ 5268.063444] _raw_spin_lock+0x21/0x30 [ 5268.063450] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5268.063740] obd_mod_free+0x46/0x140 [ptlrpc] [ 5268.064963] lustre(O) [ 5268.082849] mdc_commit_open+0x91/0xb0 [mdc] [ 5268.082902] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5268.083171] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5268.083363] after_reply+0xe2c/0x1780 [ptlrpc] [ 5268.088244] osp(O) [ 5268.092002] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5268.092285] ? __next_timer_interrupt+0x160/0x160 [ 5268.092297] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5268.092500] ? do_wait_intr+0xf0/0xf0 [ 5268.092509] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5268.092725] kthread+0x1d1/0x200 [ 5268.097641] ofd(O) [ 5268.104797] ? set_kthread_struct+0x70/0x70 [ 5268.104815] ret_from_fork+0x1f/0x30 [ 5268.197709] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5268.219116] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5268.224201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5268.228354] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5268.231678] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5268.233612] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5268.241756] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5268.245161] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5268.247952] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5268.251385] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5268.254519] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5268.257773] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5268.260890] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5268.264372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5268.267761] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5268.271754] Call Trace: [ 5268.272636] [ 5268.273390] ? show_regs.cold.9+0x22/0x2f [ 5268.275786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5268.278069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5268.280154] ? watchdog+0x40/0x40 [ 5268.281493] ? hrtimer_interrupt+0x128/0x2c0 [ 5268.283335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5268.285487] ? apic_timer_interrupt+0xf/0x20 [ 5268.287038] [ 5268.287886] ? native_safe_halt+0x2e/0x50 [ 5268.289141] kvm_wait+0xcd/0xe0 [ 5268.290955] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5268.293092] do_raw_spin_lock+0x177/0x190 [ 5268.294390] _raw_spin_lock+0x21/0x30 [ 5268.297805] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5268.300560] process_one_work+0x2c8/0x700 [ 5268.302727] worker_thread+0x296/0x6e0 [ 5268.304619] ? rescuer_thread+0x570/0x570 [ 5268.305754] kthread+0x1d1/0x200 [ 5268.306865] ? set_kthread_struct+0x70/0x70 [ 5268.308511] ret_from_fork+0x1f/0x30 [ 5272.061026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5272.066243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5272.119654] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5272.148695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5272.160659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5272.166758] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5272.185091] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5272.196968] RAX: 0000000000007d87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5272.206927] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5272.221961] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5272.233059] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5272.255181] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5272.262831] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5272.272213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5272.281881] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5272.293133] Call Trace: [ 5272.298932] [ 5272.302523] ? show_regs.cold.9+0x22/0x2f [ 5272.308047] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5272.312142] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5272.319386] ? watchdog+0x40/0x40 [ 5272.322911] ? hrtimer_interrupt+0x128/0x2c0 [ 5272.331441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5272.334857] ? apic_timer_interrupt+0xf/0x20 [ 5272.337009] [ 5272.337958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5272.346922] do_raw_spin_lock+0x177/0x190 [ 5272.348962] _raw_spin_lock+0x21/0x30 [ 5272.350007] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5272.355805] ? do_raw_spin_unlock+0x75/0x190 [ 5272.357547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5272.362224] ? do_wait_intr+0xf0/0xf0 [ 5272.363863] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5272.366690] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5272.369468] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5272.371846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5272.374860] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5272.377864] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5272.385098] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5272.386446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5272.392314] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5272.400689] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5272.405902] lmv_intent_open+0x338/0x1150 [lmv] [ 5272.410151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5272.413265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5272.415802] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5272.419045] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5272.423510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5272.427255] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5272.433657] ? do_raw_spin_unlock+0x75/0x190 [ 5272.436956] ? _raw_spin_unlock+0x12/0x30 [ 5272.443795] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5272.449224] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5272.455243] ll_atomic_open+0x36a/0xd70 [lustre] [ 5272.460155] lookup_open+0x356/0xb40 [ 5272.464465] do_last+0x313/0xfc0 [ 5272.468236] ? nd_jump_root+0xe5/0x160 [ 5272.470932] ? path_init+0x437/0x520 [ 5272.474411] path_openat+0xf7/0x500 [ 5272.480049] ? do_raw_spin_unlock+0x75/0x190 [ 5272.482341] do_filp_open+0x99/0x140 [ 5272.483525] ? getname_flags+0x6e/0x330 [ 5272.486650] ? __check_object_size+0xff/0x256 [ 5272.490238] ? do_raw_spin_unlock+0x75/0x190 [ 5272.494432] ? _raw_spin_unlock+0x12/0x30 [ 5272.495792] do_sys_openat2+0x2b4/0x410 [ 5272.504834] do_sys_open+0x73/0xa0 [ 5272.505829] __x64_sys_openat+0x24/0x30 [ 5272.506743] do_syscall_64+0xc1/0x440 [ 5272.522746] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5272.530820] RIP: 0033:0x7fa749552332 [ 5272.542231] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5272.561544] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5272.567056] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5272.570204] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5272.573693] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5272.577966] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5272.582220] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5276.640149] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3359s! [ 5276.644104] Showing busy workqueues and worker pools: [ 5276.646414] workqueue events: flags=0x0 [ 5276.648202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5276.651444] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5276.656839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5276.660134] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5276.663792] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.666932] pending: kfree_rcu_monitor [ 5276.668597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5276.671513] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5276.675718] workqueue events_long: flags=0x0 [ 5276.677324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.680336] pending: lu_objects_flush [obdclass] [ 5276.682803] workqueue events_unbound: flags=0x2 [ 5276.684792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5276.687655] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5276.691700] workqueue events_power_efficient: flags=0x80 [ 5276.693713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.696914] pending: check_lifetime [ 5276.698348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.701343] pending: neigh_periodic_work [ 5276.703392] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.705981] pending: neigh_periodic_work [ 5276.708212] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.711203] pending: do_cache_clean [ 5276.712936] workqueue mm_percpu_wq: flags=0x8 [ 5276.714947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.717992] pending: vmstat_update [ 5276.719616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.722479] pending: vmstat_update [ 5276.724121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5276.726926] pending: vmstat_update [ 5276.728476] workqueue writeback: flags=0x4a [ 5276.730385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5276.733211] pending: wb_workfn, wb_workfn [ 5276.735201] workqueue xprtiod: flags=0xa [ 5276.736958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5276.739559] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5276.742554] workqueue ipv6_addrconf: flags=0x40008 [ 5276.744514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5276.747401] pending: addrconf_verify_work [ 5276.749371] workqueue knbd0-recv: flags=0x1a [ 5276.751250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5276.754067] in-flight: 651:recv_work [ 5276.756010] workqueue ptlrpc_pinger: flags=0x2 [ 5276.757634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5276.760099] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5276.762698] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3359s workers=4 idle: 37 31846 28897 [ 5276.765893] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5266s workers=2 idle: 653 [ 5284.059009] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5284.060936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5284.081229] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5284.086240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5284.090300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.093071] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5284.101671] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5284.104901] RAX: 00000000000047e6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5284.108040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5284.111371] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5284.114516] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5284.117654] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5284.120831] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5284.124434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5284.127284] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5284.130317] Call Trace: [ 5284.131388] [ 5284.132390] ? show_regs.cold.9+0x22/0x2f [ 5284.134297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5284.136512] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5284.138645] ? watchdog+0x40/0x40 [ 5284.140175] ? hrtimer_interrupt+0x128/0x2c0 [ 5284.142000] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5284.144246] ? apic_timer_interrupt+0xf/0x20 [ 5284.146256] [ 5284.147128] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.149582] do_raw_spin_lock+0x177/0x190 [ 5284.151240] _raw_spin_lock+0x21/0x30 [ 5284.152934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5284.155394] ? do_raw_spin_unlock+0x75/0x190 [ 5284.157475] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5284.160100] ? do_wait_intr+0xf0/0xf0 [ 5284.161840] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5284.164341] mdc_close+0x53b/0xd70 [mdc] [ 5284.166360] lmv_close+0x18d/0x490 [lmv] [ 5284.168211] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5284.170944] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5284.172982] ll_md_close+0x274/0x800 [lustre] [ 5284.175299] ll_file_release+0x128/0xc50 [lustre] [ 5284.178255] __fput+0x10d/0x440 [ 5284.179760] ____fput+0x12/0x20 [ 5284.181565] task_work_run+0xc6/0x110 [ 5284.183376] exit_to_usermode_loop+0x1dd/0x1f0 [ 5284.185969] do_syscall_64+0x3d6/0x440 [ 5284.187733] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5284.189913] RIP: 0033:0x7fa749552cd8 [ 5284.191633] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5284.200969] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5284.204929] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5284.207926] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5284.211069] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5284.214855] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5284.217438] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5296.058063] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5296.059321] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5296.060998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5296.061074] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5296.061082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5296.061085] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5296.061099] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5296.061103] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5296.061107] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5296.061110] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5296.061112] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5296.061114] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5296.061117] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5296.061120] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5296.061123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5296.061126] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5296.061249] Call Trace: [ 5296.061256] [ 5296.061262] ? show_regs.cold.9+0x22/0x2f [ 5296.061272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5296.061279] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5296.061283] ? watchdog+0x40/0x40 [ 5296.061287] ? hrtimer_interrupt+0x128/0x2c0 [ 5296.061292] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5296.061298] ? apic_timer_interrupt+0xf/0x20 [ 5296.061303] [ 5296.061306] ? native_safe_halt+0x2e/0x50 [ 5296.061310] kvm_wait+0xcd/0xe0 [ 5296.061317] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5296.061324] do_raw_spin_lock+0x177/0x190 [ 5296.061328] _raw_spin_lock+0x21/0x30 [ 5296.061333] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5296.061621] obd_mod_free+0x46/0x140 [ptlrpc] [ 5296.068084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 5296.086392] mdc_commit_open+0x91/0xb0 [mdc] [ 5296.086449] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5296.086747] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5296.086958] after_reply+0xe2c/0x1780 [ptlrpc] [ 5296.087290] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5296.103597] obdclass(O) [ 5296.107890] ? __next_timer_interrupt+0x160/0x160 [ 5296.116467] ksocklnd(O) [ 5296.133271] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5296.133765] ? do_wait_intr+0xf0/0xf0 [ 5296.133777] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5296.134168] kthread+0x1d1/0x200 [ 5296.134180] ? set_kthread_struct+0x70/0x70 [ 5296.134183] ret_from_fork+0x1f/0x30 [ 5296.388036] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5296.413316] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5296.424875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5296.437493] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5296.444554] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5296.449365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5296.464380] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5296.468855] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5296.475101] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5296.481598] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5296.488909] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5296.495284] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5296.497510] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5296.503703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5296.509112] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5296.513018] Call Trace: [ 5296.514899] [ 5296.517580] ? show_regs.cold.9+0x22/0x2f [ 5296.520727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5296.525429] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5296.528162] ? watchdog+0x40/0x40 [ 5296.534217] ? hrtimer_interrupt+0x128/0x2c0 [ 5296.535902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5296.541041] ? apic_timer_interrupt+0xf/0x20 [ 5296.542348] [ 5296.542898] ? native_safe_halt+0x2e/0x50 [ 5296.548129] kvm_wait+0xcd/0xe0 [ 5296.549169] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5296.555721] do_raw_spin_lock+0x177/0x190 [ 5296.557024] _raw_spin_lock+0x21/0x30 [ 5296.558487] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5296.560664] process_one_work+0x2c8/0x700 [ 5296.562423] worker_thread+0x296/0x6e0 [ 5296.563735] ? rescuer_thread+0x570/0x570 [ 5296.565275] kthread+0x1d1/0x200 [ 5296.566538] ? set_kthread_struct+0x70/0x70 [ 5296.568096] ret_from_fork+0x1f/0x30 [ 5304.054028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5304.058176] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5304.080667] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5304.087076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5304.090965] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5304.094141] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5304.103344] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5304.107177] RAX: 0000000000002a10 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5304.111014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5304.114230] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5304.117836] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5304.121121] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5304.125078] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5304.129568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5304.132591] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5304.136114] Call Trace: [ 5304.137497] [ 5304.138424] ? show_regs.cold.9+0x22/0x2f [ 5304.140415] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5304.142114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5304.144334] ? watchdog+0x40/0x40 [ 5304.145627] ? hrtimer_interrupt+0x128/0x2c0 [ 5304.147787] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5304.150391] ? apic_timer_interrupt+0xf/0x20 [ 5304.152222] [ 5304.153167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5304.155775] do_raw_spin_lock+0x177/0x190 [ 5304.157840] _raw_spin_lock+0x21/0x30 [ 5304.158925] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5304.162508] ? do_raw_spin_unlock+0x75/0x190 [ 5304.166030] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5304.168664] ? do_wait_intr+0xf0/0xf0 [ 5304.170498] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5304.173521] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5304.176056] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5304.179046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.181848] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5304.184665] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5304.186539] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5304.188876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.191492] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5304.194661] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5304.197440] lmv_intent_open+0x338/0x1150 [lmv] [ 5304.199622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.202339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.204896] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5304.207131] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5304.209539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.212443] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5304.215025] ? do_raw_spin_unlock+0x75/0x190 [ 5304.217183] ? _raw_spin_unlock+0x12/0x30 [ 5304.219209] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5304.221722] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5304.225191] ll_atomic_open+0x36a/0xd70 [lustre] [ 5304.227886] lookup_open+0x356/0xb40 [ 5304.229686] do_last+0x313/0xfc0 [ 5304.231055] ? nd_jump_root+0xe5/0x160 [ 5304.232789] ? path_init+0x437/0x520 [ 5304.234415] path_openat+0xf7/0x500 [ 5304.236004] ? do_raw_spin_unlock+0x75/0x190 [ 5304.237703] do_filp_open+0x99/0x140 [ 5304.239418] ? getname_flags+0x6e/0x330 [ 5304.241385] ? __check_object_size+0xff/0x256 [ 5304.243412] ? do_raw_spin_unlock+0x75/0x190 [ 5304.245597] ? _raw_spin_unlock+0x12/0x30 [ 5304.247849] do_sys_openat2+0x2b4/0x410 [ 5304.249842] do_sys_open+0x73/0xa0 [ 5304.251946] __x64_sys_openat+0x24/0x30 [ 5304.254020] do_syscall_64+0xc1/0x440 [ 5304.255931] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5304.257646] RIP: 0033:0x7fa749552332 [ 5304.259412] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5304.267865] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5304.271526] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5304.275445] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5304.278598] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5304.282201] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5304.285710] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5306.854262] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3389s! [ 5306.862438] Showing busy workqueues and worker pools: [ 5306.867532] workqueue events: flags=0x0 [ 5306.870604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5306.880092] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5306.889377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5306.895317] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5306.902422] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5306.910312] pending: kfree_rcu_monitor [ 5306.913635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5306.916720] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5306.930997] workqueue events_long: flags=0x0 [ 5306.937768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5306.941660] pending: lu_objects_flush [obdclass] [ 5306.944600] workqueue events_unbound: flags=0x2 [ 5306.952964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5306.958874] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5306.989706] workqueue events_power_efficient: flags=0x80 [ 5306.991589] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5307.009101] pending: check_lifetime [ 5307.015706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5307.025774] pending: neigh_periodic_work [ 5307.029463] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5307.041483] pending: neigh_periodic_work [ 5307.050250] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5307.060684] pending: do_cache_clean [ 5307.065512] workqueue mm_percpu_wq: flags=0x8 [ 5307.073457] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5307.078376] pending: vmstat_update [ 5307.094265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5307.106715] pending: vmstat_update [ 5307.112081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5307.116744] pending: vmstat_update [ 5307.127654] workqueue writeback: flags=0x4a [ 5307.135036] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5307.142952] pending: wb_workfn, wb_workfn [ 5307.144508] workqueue xprtiod: flags=0xa [ 5307.149050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5307.162049] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5307.168432] workqueue ipv6_addrconf: flags=0x40008 [ 5307.169940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5307.174090] pending: addrconf_verify_work [ 5307.176989] workqueue knbd0-recv: flags=0x1a [ 5307.181924] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5307.184007] in-flight: 651:recv_work [ 5307.186078] workqueue ptlrpc_pinger: flags=0x2 [ 5307.187256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5307.191469] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5307.200965] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3390s workers=4 idle: 37 31846 28897 [ 5307.207266] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5297s workers=2 idle: 653 [ 5312.059133] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5312.064336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5312.092723] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5312.101081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5312.105861] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5312.108611] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5312.117761] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5312.122007] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5312.126299] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5312.129776] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5312.132931] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5312.136462] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5312.140160] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5312.144350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5312.146360] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5312.148734] Call Trace: [ 5312.149752] [ 5312.150591] ? show_regs.cold.9+0x22/0x2f [ 5312.152255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5312.154422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5312.156516] ? watchdog+0x40/0x40 [ 5312.158149] ? hrtimer_interrupt+0x128/0x2c0 [ 5312.160181] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5312.162405] ? apic_timer_interrupt+0xf/0x20 [ 5312.164378] [ 5312.165422] ? native_safe_halt+0x2e/0x50 [ 5312.167432] kvm_wait+0xcd/0xe0 [ 5312.169117] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5312.171374] do_raw_spin_lock+0x177/0x190 [ 5312.173106] _raw_spin_lock+0x21/0x30 [ 5312.174320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5312.177279] ? do_raw_spin_unlock+0x75/0x190 [ 5312.179474] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5312.182743] ? do_wait_intr+0xf0/0xf0 [ 5312.184762] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5312.187505] mdc_close+0x53b/0xd70 [mdc] [ 5312.189280] lmv_close+0x18d/0x490 [lmv] [ 5312.190973] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5312.193315] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5312.195626] ll_md_close+0x274/0x800 [lustre] [ 5312.197596] ll_file_release+0x128/0xc50 [lustre] [ 5312.200073] __fput+0x10d/0x440 [ 5312.201500] ____fput+0x12/0x20 [ 5312.203242] task_work_run+0xc6/0x110 [ 5312.204833] exit_to_usermode_loop+0x1dd/0x1f0 [ 5312.207304] do_syscall_64+0x3d6/0x440 [ 5312.209129] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5312.211225] RIP: 0033:0x7fa749552cd8 [ 5312.212852] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5312.221716] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5312.225222] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5312.228464] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5312.231548] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5312.234736] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5312.237812] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5324.060921] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5324.063374] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5324.071621] Modules linked in: [ 5324.075014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.075093] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.075101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5324.075104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5324.075119] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5324.075123] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5324.075128] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5324.075131] RDX: 0000000000006032 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5324.075133] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5324.075135] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5324.075138] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5324.075141] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5324.075144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.075147] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5324.075156] Call Trace: [ 5324.075162] [ 5324.075167] ? show_regs.cold.9+0x22/0x2f [ 5324.075176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.075182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.075186] ? watchdog+0x40/0x40 [ 5324.075191] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.075195] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.075203] ? apic_timer_interrupt+0xf/0x20 [ 5324.075207] [ 5324.075210] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5324.075215] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5324.075220] do_raw_spin_lock+0x177/0x190 [ 5324.075225] _raw_spin_lock+0x21/0x30 [ 5324.075230] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5324.075524] obd_mod_free+0x46/0x140 [ptlrpc] [ 5324.076124] mdc_commit_open+0x91/0xb0 [mdc] [ 5324.076162] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5324.076357] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5324.079315] lustre(O) [ 5324.132720] after_reply+0xe2c/0x1780 [ptlrpc] [ 5324.133400] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5324.147018] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.147094] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.147102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5324.147108] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5324.147641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5324.147654] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5324.147659] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5324.147664] RAX: 0000000000007187 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5324.147666] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5324.147669] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5324.147671] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5324.147674] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5324.147677] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5324.147680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.147683] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5324.147692] Call Trace: [ 5324.147697] [ 5324.147703] ? show_regs.cold.9+0x22/0x2f [ 5324.147727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.147733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.147739] ? watchdog+0x40/0x40 [ 5324.147743] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.147747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.147758] ? apic_timer_interrupt+0xf/0x20 [ 5324.147762] [ 5324.147765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5324.147769] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5324.147775] do_raw_spin_lock+0x177/0x190 [ 5324.147780] _raw_spin_lock+0x21/0x30 [ 5324.147786] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5324.148560] process_one_work+0x2c8/0x700 [ 5324.148568] worker_thread+0x296/0x6e0 [ 5324.148572] ? rescuer_thread+0x570/0x570 [ 5324.148575] kthread+0x1d1/0x200 [ 5324.148579] ? set_kthread_struct+0x70/0x70 [ 5324.148582] ret_from_fork+0x1f/0x30 [ 5324.507005] ? __next_timer_interrupt+0x160/0x160 [ 5324.512627] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5324.518336] ? do_wait_intr+0xf0/0xf0 [ 5324.520110] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5324.528093] kthread+0x1d1/0x200 [ 5324.531083] ? set_kthread_struct+0x70/0x70 [ 5324.538335] ret_from_fork+0x1f/0x30 [ 5332.054150] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5332.058086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5332.077654] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5332.081776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5332.092092] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5332.097443] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5332.109377] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5332.116416] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5332.122119] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5332.126171] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5332.131881] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5332.135407] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5332.141476] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5332.147170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5332.150829] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5332.158137] Call Trace: [ 5332.159011] [ 5332.159651] ? show_regs.cold.9+0x22/0x2f [ 5332.162835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5332.166713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5332.169495] ? watchdog+0x40/0x40 [ 5332.170814] ? hrtimer_interrupt+0x128/0x2c0 [ 5332.173144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5332.177586] ? apic_timer_interrupt+0xf/0x20 [ 5332.179712] [ 5332.183876] ? native_safe_halt+0x2e/0x50 [ 5332.189762] kvm_wait+0xcd/0xe0 [ 5332.193742] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5332.197687] do_raw_spin_lock+0x177/0x190 [ 5332.205413] _raw_spin_lock+0x21/0x30 [ 5332.209603] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5332.215971] ? do_raw_spin_unlock+0x75/0x190 [ 5332.218813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5332.222784] ? do_wait_intr+0xf0/0xf0 [ 5332.225598] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5332.228482] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5332.231337] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5332.235429] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5332.237540] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5332.243263] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5332.245654] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5332.248142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5332.254397] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5332.260727] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5332.268061] lmv_intent_open+0x338/0x1150 [lmv] [ 5332.270667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5332.277036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5332.287040] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5332.292440] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5332.295617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5332.301749] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5332.306560] ? do_raw_spin_unlock+0x75/0x190 [ 5332.309090] ? _raw_spin_unlock+0x12/0x30 [ 5332.313106] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5332.319210] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5332.324612] ll_atomic_open+0x36a/0xd70 [lustre] [ 5332.326466] lookup_open+0x356/0xb40 [ 5332.330946] do_last+0x313/0xfc0 [ 5332.335100] ? nd_jump_root+0xe5/0x160 [ 5332.340108] ? path_init+0x437/0x520 [ 5332.343400] path_openat+0xf7/0x500 [ 5332.344848] ? do_raw_spin_unlock+0x75/0x190 [ 5332.346782] do_filp_open+0x99/0x140 [ 5332.349345] ? getname_flags+0x6e/0x330 [ 5332.350942] ? __check_object_size+0xff/0x256 [ 5332.353724] ? do_raw_spin_unlock+0x75/0x190 [ 5332.358515] ? _raw_spin_unlock+0x12/0x30 [ 5332.359656] do_sys_openat2+0x2b4/0x410 [ 5332.362615] do_sys_open+0x73/0xa0 [ 5332.363806] __x64_sys_openat+0x24/0x30 [ 5332.364759] do_syscall_64+0xc1/0x440 [ 5332.366684] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5332.368149] RIP: 0033:0x7fa749552332 [ 5332.369578] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5332.382032] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5332.389198] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5332.394646] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5332.398663] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5332.402474] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5332.405925] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5337.571038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3420s! [ 5337.574997] Showing busy workqueues and worker pools: [ 5337.577884] workqueue events: flags=0x0 [ 5337.583944] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5337.587723] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5337.597385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5337.604654] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5337.609460] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.612998] pending: kfree_rcu_monitor [ 5337.615867] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5337.619248] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5337.625671] workqueue events_long: flags=0x0 [ 5337.629367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.633861] pending: lu_objects_flush [obdclass] [ 5337.637626] workqueue events_unbound: flags=0x2 [ 5337.639420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5337.643416] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5337.649366] workqueue events_power_efficient: flags=0x80 [ 5337.652639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.656737] pending: check_lifetime [ 5337.658901] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.662986] pending: neigh_periodic_work [ 5337.665413] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.669838] pending: neigh_periodic_work [ 5337.672384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.676926] pending: do_cache_clean [ 5337.680330] workqueue mm_percpu_wq: flags=0x8 [ 5337.683856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.687674] pending: vmstat_update [ 5337.690160] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.694235] pending: vmstat_update [ 5337.696260] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5337.700636] pending: vmstat_update [ 5337.702564] workqueue writeback: flags=0x4a [ 5337.704759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5337.707989] pending: wb_workfn, wb_workfn [ 5337.709559] workqueue xprtiod: flags=0xa [ 5337.710959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5337.713576] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5337.723500] workqueue ipv6_addrconf: flags=0x40008 [ 5337.733523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5337.736981] pending: addrconf_verify_work [ 5337.742875] workqueue knbd0-recv: flags=0x1a [ 5337.745834] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5337.751489] in-flight: 651:recv_work [ 5337.754850] workqueue ptlrpc_pinger: flags=0x2 [ 5337.759675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5337.766584] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5337.771664] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3420s workers=4 idle: 37 31846 28897 [ 5337.777342] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5327s workers=2 idle: 653 [ 5340.062207] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5340.066919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5340.103032] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5340.108266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5340.112706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5340.116034] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5340.133344] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5340.140844] RAX: 0000000000007d91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5340.145999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5340.150753] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5340.155040] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5340.158480] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5340.162840] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5340.169901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5340.178521] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5340.185292] Call Trace: [ 5340.186523] [ 5340.187466] ? show_regs.cold.9+0x22/0x2f [ 5340.190141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5340.192706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5340.194832] ? watchdog+0x40/0x40 [ 5340.196838] ? hrtimer_interrupt+0x128/0x2c0 [ 5340.198936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5340.201084] ? apic_timer_interrupt+0xf/0x20 [ 5340.203802] [ 5340.205338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5340.207089] do_raw_spin_lock+0x177/0x190 [ 5340.208749] _raw_spin_lock+0x21/0x30 [ 5340.209936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5340.212346] ? do_raw_spin_unlock+0x75/0x190 [ 5340.214118] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5340.216119] ? do_wait_intr+0xf0/0xf0 [ 5340.217873] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5340.220514] mdc_close+0x53b/0xd70 [mdc] [ 5340.222273] lmv_close+0x18d/0x490 [lmv] [ 5340.224009] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5340.226342] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5340.228306] ll_md_close+0x274/0x800 [lustre] [ 5340.230079] ll_file_release+0x128/0xc50 [lustre] [ 5340.231378] __fput+0x10d/0x440 [ 5340.232915] ____fput+0x12/0x20 [ 5340.234076] task_work_run+0xc6/0x110 [ 5340.239088] exit_to_usermode_loop+0x1dd/0x1f0 [ 5340.243454] do_syscall_64+0x3d6/0x440 [ 5340.245679] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5340.248439] RIP: 0033:0x7fa749552cd8 [ 5340.255048] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5340.269545] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5340.276758] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5340.279120] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5340.288823] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5340.294419] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5340.299993] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5352.058021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5352.058625] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5352.064313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.064391] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.064398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.064401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5352.064414] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5352.064418] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5352.064423] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5352.064425] RDX: 0000000000001a29 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5352.064427] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5352.064429] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5352.064431] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5352.064434] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5352.064437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.064440] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5352.064449] Call Trace: [ 5352.064455] [ 5352.064461] ? show_regs.cold.9+0x22/0x2f [ 5352.064471] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.064479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.064483] ? watchdog+0x40/0x40 [ 5352.064487] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.064492] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.064500] ? apic_timer_interrupt+0xf/0x20 [ 5352.064505] [ 5352.064509] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5352.064513] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5352.064519] do_raw_spin_lock+0x177/0x190 [ 5352.064524] _raw_spin_lock+0x21/0x30 [ 5352.064530] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5352.064970] obd_mod_free+0x46/0x140 [ptlrpc] [ 5352.065321] mdc_commit_open+0x91/0xb0 [mdc] [ 5352.065358] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5352.065560] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5352.065886] after_reply+0xe2c/0x1780 [ptlrpc] [ 5352.066156] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5352.066377] ? __next_timer_interrupt+0x160/0x160 [ 5352.066386] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5352.070488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.070566] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.070573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.070580] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5352.071085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5352.071097] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5352.071102] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5352.071106] RAX: 0000000000005293 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5352.071109] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5352.071111] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5352.071113] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5352.071115] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5352.071118] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5352.071121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.071123] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5352.071133] Call Trace: [ 5352.071138] [ 5352.071144] ? show_regs.cold.9+0x22/0x2f [ 5352.071154] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.071160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.071164] ? watchdog+0x40/0x40 [ 5352.071169] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.071173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.071180] ? apic_timer_interrupt+0xf/0x20 [ 5352.071185] [ 5352.071188] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5352.071193] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5352.071199] do_raw_spin_lock+0x177/0x190 [ 5352.071205] _raw_spin_lock+0x21/0x30 [ 5352.071210] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5352.071477] process_one_work+0x2c8/0x700 [ 5352.071484] worker_thread+0x296/0x6e0 [ 5352.071489] ? rescuer_thread+0x570/0x570 [ 5352.071493] kthread+0x1d1/0x200 [ 5352.071497] ? set_kthread_struct+0x70/0x70 [ 5352.071501] ret_from_fork+0x1f/0x30 [ 5352.405210] ? do_wait_intr+0xf0/0xf0 [ 5352.407114] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5352.414758] kthread+0x1d1/0x200 [ 5352.415907] ? set_kthread_struct+0x70/0x70 [ 5352.420880] ret_from_fork+0x1f/0x30 [ 5364.056269] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5364.067507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5364.107990] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5364.113555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5364.117972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5364.120866] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5364.128358] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5364.132347] RAX: 0000000000007e26 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5364.136377] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5364.139487] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5364.142477] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5364.145476] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5364.154886] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5364.159860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5364.164377] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5364.170908] Call Trace: [ 5364.172418] [ 5364.173235] ? show_regs.cold.9+0x22/0x2f [ 5364.174992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5364.176965] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5364.178671] ? watchdog+0x40/0x40 [ 5364.182430] ? hrtimer_interrupt+0x128/0x2c0 [ 5364.185029] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5364.189231] ? apic_timer_interrupt+0xf/0x20 [ 5364.190842] [ 5364.191611] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5364.195958] do_raw_spin_lock+0x177/0x190 [ 5364.199266] _raw_spin_lock+0x21/0x30 [ 5364.201464] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5364.206116] ? do_raw_spin_unlock+0x75/0x190 [ 5364.208546] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5364.211552] ? do_wait_intr+0xf0/0xf0 [ 5364.216126] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5364.227412] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5364.232970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5364.241358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5364.248159] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5364.251697] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5364.267576] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5364.274083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5364.279380] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5364.283658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5364.294135] lmv_intent_open+0x338/0x1150 [lmv] [ 5364.295778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5364.303436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5364.314255] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5364.315918] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5364.331158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5364.337358] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5364.340597] ? do_raw_spin_unlock+0x75/0x190 [ 5364.345537] ? _raw_spin_unlock+0x12/0x30 [ 5364.348614] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5364.356503] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5364.364689] ll_atomic_open+0x36a/0xd70 [lustre] [ 5364.370128] lookup_open+0x356/0xb40 [ 5364.373991] do_last+0x313/0xfc0 [ 5364.375617] ? nd_jump_root+0xe5/0x160 [ 5364.379393] ? path_init+0x437/0x520 [ 5364.382190] path_openat+0xf7/0x500 [ 5364.383493] ? do_raw_spin_unlock+0x75/0x190 [ 5364.388118] do_filp_open+0x99/0x140 [ 5364.390179] ? getname_flags+0x6e/0x330 [ 5364.392129] ? __check_object_size+0xff/0x256 [ 5364.394315] ? do_raw_spin_unlock+0x75/0x190 [ 5364.396936] ? _raw_spin_unlock+0x12/0x30 [ 5364.398421] do_sys_openat2+0x2b4/0x410 [ 5364.399785] do_sys_open+0x73/0xa0 [ 5364.403139] __x64_sys_openat+0x24/0x30 [ 5364.408498] do_syscall_64+0xc1/0x440 [ 5364.411878] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5364.415605] RIP: 0033:0x7fa749552332 [ 5364.420984] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5364.433382] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5364.441516] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5364.449765] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5364.455483] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5364.462468] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5364.465528] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5368.062596] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5368.071794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5368.114105] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5368.126286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5368.133139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5368.139797] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5368.153608] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5368.160557] RAX: 00000000000047a5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5368.167793] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5368.175234] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5368.183638] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5368.190626] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5368.193911] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5368.197592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5368.200548] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5368.203497] Call Trace: [ 5368.204440] [ 5368.205416] ? show_regs.cold.9+0x22/0x2f [ 5368.207430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5368.208950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5368.210881] ? watchdog+0x40/0x40 [ 5368.212348] ? hrtimer_interrupt+0x128/0x2c0 [ 5368.214411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5368.216367] ? apic_timer_interrupt+0xf/0x20 [ 5368.217983] [ 5368.218808] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5368.220848] do_raw_spin_lock+0x177/0x190 [ 5368.222268] _raw_spin_lock+0x21/0x30 [ 5368.223901] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5368.226135] ? do_raw_spin_unlock+0x75/0x190 [ 5368.227535] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5368.229686] ? do_wait_intr+0xf0/0xf0 [ 5368.231825] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5368.238757] mdc_close+0x53b/0xd70 [mdc] [ 5368.242313] lmv_close+0x18d/0x490 [lmv] [ 5368.247506] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5368.249960] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5368.255980] ll_md_close+0x274/0x800 [lustre] [ 5368.257532] ll_file_release+0x128/0xc50 [lustre] [ 5368.263935] __fput+0x10d/0x440 [ 5368.266489] ____fput+0x12/0x20 [ 5368.271644] task_work_run+0xc6/0x110 [ 5368.273385] exit_to_usermode_loop+0x1dd/0x1f0 [ 5368.277064] do_syscall_64+0x3d6/0x440 [ 5368.278969] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5368.282186] RIP: 0033:0x7fa749552cd8 [ 5368.286380] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5368.291772] BUG: workqueue lockup - pool [ 5368.310482] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5368.310499] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5368.310501] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5368.312265] cpus=0-3 flags=0x4 nice=0 stuck for 3451s! [ 5368.312283] Showing busy workqueues and worker pools: [ 5368.312285] workqueue events: flags=0x0 [ 5368.312287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5368.312295] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5368.312632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5368.312641] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5368.313316] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.313326] pending: kfree_rcu_monitor [ 5368.313335] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5368.313342] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5368.313734] workqueue events_long: flags=0x0 [ 5368.313737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.313745] pending: lu_objects_flush [obdclass] [ 5368.314155] workqueue events_unbound: flags=0x2 [ 5368.314158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5368.314166] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5368.314185] workqueue events_power_efficient: flags=0x80 [ 5368.314187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.314195] pending: check_lifetime [ 5368.314202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.314209] pending: neigh_periodic_work [ 5368.314216] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.314223] pending: neigh_periodic_work [ 5368.314228] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.314235] pending: do_cache_clean [ 5368.314244] workqueue mm_percpu_wq: flags=0x8 [ 5368.314246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.314252] pending: vmstat_update [ 5368.314257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.314263] pending: vmstat_update [ 5368.314267] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5368.314273] pending: vmstat_update [ 5368.314279] workqueue writeback: flags=0x4a [ 5368.314281] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5368.314287] pending: wb_workfn, wb_workfn [ 5368.314298] workqueue xprtiod: flags=0xa [ 5368.314300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5368.314305] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5368.314322] workqueue ipv6_addrconf: flags=0x40008 [ 5368.314325] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5368.314377] pending: addrconf_verify_work [ 5368.314386] workqueue knbd0-recv: flags=0x1a [ 5368.314388] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5368.314395] in-flight: 651:recv_work [ 5368.314405] workqueue ptlrpc_pinger: flags=0x2 [ 5368.314407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5368.314412] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5368.324165] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5368.324175] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5368.324178] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5368.462576] [ 5368.463442] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3451s workers=4 idle: 37 31846 28897 [ 5368.466914] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5358s workers=2 idle: 653 [ 5380.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5380.058048] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5380.059788] Modules linked in: [ 5380.063014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.063110] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.063120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.063124] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5380.063143] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5380.063148] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5380.063154] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5380.063157] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5380.063159] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5380.063162] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5380.063165] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5380.063169] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5380.063173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.063175] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5380.063184] Call Trace: [ 5380.063191] [ 5380.063199] ? show_regs.cold.9+0x22/0x2f [ 5380.063211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.063218] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.063222] ? watchdog+0x40/0x40 [ 5380.063226] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.063230] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.063236] ? apic_timer_interrupt+0xf/0x20 [ 5380.063241] [ 5380.063244] ? native_safe_halt+0x2e/0x50 [ 5380.063247] kvm_wait+0xcd/0xe0 [ 5380.063256] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5380.063266] do_raw_spin_lock+0x177/0x190 [ 5380.063271] _raw_spin_lock+0x21/0x30 [ 5380.063277] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5380.063579] obd_mod_free+0x46/0x140 [ptlrpc] [ 5380.063817] mdc_commit_open+0x91/0xb0 [mdc] [ 5380.063862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5380.064058] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5380.064259] after_reply+0xe2c/0x1780 [ptlrpc] [ 5380.064461] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5380.064660] ? __next_timer_interrupt+0x160/0x160 [ 5380.064673] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5380.064902] ? do_wait_intr+0xf0/0xf0 [ 5380.064914] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5380.065116] kthread+0x1d1/0x200 [ 5380.065122] ? set_kthread_struct+0x70/0x70 [ 5380.065127] ret_from_fork+0x1f/0x30 [ 5380.326322] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.368606] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.382081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.388581] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5380.395257] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5380.398197] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5380.414304] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5380.418310] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5380.420924] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5380.426682] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5380.430193] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5380.435079] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5380.439501] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5380.442531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.444929] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5380.447892] Call Trace: [ 5380.449375] [ 5380.450360] ? show_regs.cold.9+0x22/0x2f [ 5380.451946] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.453865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.455310] ? watchdog+0x40/0x40 [ 5380.456915] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.458692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.460647] ? apic_timer_interrupt+0xf/0x20 [ 5380.462142] [ 5380.463046] ? native_safe_halt+0x2e/0x50 [ 5380.464773] kvm_wait+0xcd/0xe0 [ 5380.466084] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5380.469925] do_raw_spin_lock+0x177/0x190 [ 5380.472038] _raw_spin_lock+0x21/0x30 [ 5380.474018] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5380.476258] process_one_work+0x2c8/0x700 [ 5380.478197] worker_thread+0x296/0x6e0 [ 5380.479870] ? rescuer_thread+0x570/0x570 [ 5380.482867] kthread+0x1d1/0x200 [ 5380.484133] ? set_kthread_struct+0x70/0x70 [ 5380.486081] ret_from_fork+0x1f/0x30 [ 5393.762039] rcu: INFO: rcu_sched self-detected stall on CPU [ 5393.767758] rcu: 2-...!: (1896910 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 5393.776806] (t=3480073 jiffies g=258701 q=2512) [ 5393.780906] rcu: rcu_sched kthread starved for 3476868 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 5393.788725] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5393.793393] rcu: RCU grace-period kthread stack dump: [ 5393.795844] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5393.800911] Call Trace: [ 5393.802352] __schedule+0x351/0xcb0 [ 5393.803917] schedule+0xc0/0x180 [ 5393.805608] schedule_timeout+0xb4/0x190 [ 5393.807601] ? do_raw_spin_unlock+0x75/0x190 [ 5393.810168] ? __next_timer_interrupt+0x160/0x160 [ 5393.812385] ? prepare_to_swait_event+0xf3/0x140 [ 5393.814635] rcu_gp_kthread+0x8dd/0x15b0 [ 5393.816656] ? force_qs_rnp+0x2c0/0x2c0 [ 5393.818395] kthread+0x1d1/0x200 [ 5393.820038] ? set_kthread_struct+0x70/0x70 [ 5393.821913] ret_from_fork+0x1f/0x30 [ 5393.823082] rcu: Stack dump where RCU GP kthread last ran: [ 5393.824334] Sending NMI from CPU 2 to CPUs 3: [ 5393.827605] NMI backtrace for cpu 3 [ 5393.827612] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5393.827614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5393.827616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5393.827617] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5393.827619] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 [ 5393.827622] RAX: 00000000000052ff RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5393.827623] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5393.827625] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5393.827626] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5393.827628] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5393.827629] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5393.827630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5393.827631] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5393.827632] Call Trace: [ 5393.827633] [ 5393.827634] ? show_regs.cold.9+0x22/0x2f [ 5393.827635] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5393.827637] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5393.827638] ? nmi_handle+0xbd/0x210 [ 5393.827639] ? default_do_nmi+0x4b/0x1b0 [ 5393.827640] ? do_nmi+0x280/0x380 [ 5393.827641] ? end_repeat_nmi+0x16/0x1f [ 5393.827642] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5393.827644] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5393.827645] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5393.827646] [ 5393.827648] do_raw_spin_lock+0x177/0x190 [ 5393.827649] _raw_spin_lock+0x21/0x30 [ 5393.827650] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5393.827651] ? do_raw_spin_unlock+0x75/0x190 [ 5393.827652] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5393.827653] ? do_wait_intr+0xf0/0xf0 [ 5393.827654] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5393.827655] mdc_close+0x53b/0xd70 [mdc] [ 5393.827656] lmv_close+0x18d/0x490 [lmv] [ 5393.827657] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5393.827658] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5393.827660] ll_md_close+0x274/0x800 [lustre] [ 5393.827661] ll_file_release+0x128/0xc50 [lustre] [ 5393.827662] __fput+0x10d/0x440 [ 5393.827663] ____fput+0x12/0x20 [ 5393.827664] task_work_run+0xc6/0x110 [ 5393.827665] exit_to_usermode_loop+0x1dd/0x1f0 [ 5393.827666] do_syscall_64+0x3d6/0x440 [ 5393.827667] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5393.827668] RIP: 0033:0x7fa749552cd8 [ 5393.827670] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5393.827672] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5393.827675] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5393.827676] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5393.827678] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5393.827679] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5393.827681] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5393.827971] NMI backtrace for cpu 2 [ 5394.001048] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5394.006439] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5394.010338] Call Trace: [ 5394.011437] [ 5394.012886] ? dump_stack+0xbb/0x10e [ 5394.014153] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5394.016414] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5394.018608] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5394.021344] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5394.024197] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5394.027235] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5394.029668] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5394.032818] ? update_process_times+0x6d/0xc0 [ 5394.035495] ? tick_sched_handle.isra.18+0x27/0x90 [ 5394.037573] ? tick_sched_timer+0x53/0xb0 [ 5394.039477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5394.042081] ? tick_sched_handle.isra.18+0x90/0x90 [ 5394.049651] ? hrtimer_interrupt+0x128/0x2c0 [ 5394.054794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5394.058584] ? apic_timer_interrupt+0xf/0x20 [ 5394.060771] [ 5394.062106] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5394.065578] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5394.070632] ? do_raw_spin_lock+0x177/0x190 [ 5394.074630] ? _raw_spin_lock+0x21/0x30 [ 5394.076918] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5394.080441] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5394.082850] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5394.085277] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5394.088503] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5394.092137] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5394.095917] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5394.100294] ? __next_timer_interrupt+0x160/0x160 [ 5394.103269] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5394.106665] ? do_wait_intr+0xf0/0xf0 [ 5394.109569] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5394.113210] ? kthread+0x1d1/0x200 [ 5394.114940] ? set_kthread_struct+0x70/0x70 [ 5394.117454] ? ret_from_fork+0x1f/0x30 [ 5396.062308] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5396.074474] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5396.130523] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5396.140326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5396.147027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5396.149037] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5396.157578] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5396.160633] RAX: 000000000000240b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5396.163908] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5396.166951] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5396.169609] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5396.172517] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5396.175767] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5396.182764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5396.191850] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5396.202758] Call Trace: [ 5396.205494] [ 5396.207489] ? show_regs.cold.9+0x22/0x2f [ 5396.209279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5396.217383] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5396.219804] ? watchdog+0x40/0x40 [ 5396.222447] ? hrtimer_interrupt+0x128/0x2c0 [ 5396.224010] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5396.229951] ? apic_timer_interrupt+0xf/0x20 [ 5396.231373] [ 5396.232071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5396.237621] do_raw_spin_lock+0x177/0x190 [ 5396.242693] _raw_spin_lock+0x21/0x30 [ 5396.244108] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5396.247021] ? do_raw_spin_unlock+0x75/0x190 [ 5396.251094] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5396.256899] ? do_wait_intr+0xf0/0xf0 [ 5396.258287] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5396.263643] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5396.269138] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5396.272142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.279140] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5396.284606] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5396.286433] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5396.291677] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.295747] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5396.299264] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5396.306568] lmv_intent_open+0x338/0x1150 [lmv] [ 5396.308023] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.313866] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.317926] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5396.322361] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5396.325038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.329062] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5396.336105] ? do_raw_spin_unlock+0x75/0x190 [ 5396.337556] ? _raw_spin_unlock+0x12/0x30 [ 5396.338987] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5396.344578] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5396.349346] ll_atomic_open+0x36a/0xd70 [lustre] [ 5396.351660] lookup_open+0x356/0xb40 [ 5396.355425] do_last+0x313/0xfc0 [ 5396.356488] ? nd_jump_root+0xe5/0x160 [ 5396.359512] ? path_init+0x437/0x520 [ 5396.365805] path_openat+0xf7/0x500 [ 5396.368315] ? do_raw_spin_unlock+0x75/0x190 [ 5396.370945] do_filp_open+0x99/0x140 [ 5396.374882] ? getname_flags+0x6e/0x330 [ 5396.379753] ? __check_object_size+0xff/0x256 [ 5396.384748] ? do_raw_spin_unlock+0x75/0x190 [ 5396.386271] ? _raw_spin_unlock+0x12/0x30 [ 5396.387815] do_sys_openat2+0x2b4/0x410 [ 5396.390519] do_sys_open+0x73/0xa0 [ 5396.394213] __x64_sys_openat+0x24/0x30 [ 5396.395518] do_syscall_64+0xc1/0x440 [ 5396.399577] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5396.401912] RIP: 0033:0x7fa749552332 [ 5396.406956] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5396.421303] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5396.426877] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5396.430943] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5396.434970] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5396.442579] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5396.448928] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5398.499022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3481s! [ 5398.502948] Showing busy workqueues and worker pools: [ 5398.505285] workqueue events: flags=0x0 [ 5398.507288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5398.510504] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5398.515431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5398.518158] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5398.522972] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.526476] pending: kfree_rcu_monitor [ 5398.528226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5398.531724] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5398.535573] workqueue events_long: flags=0x0 [ 5398.537526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.540981] pending: lu_objects_flush [obdclass] [ 5398.543605] workqueue events_unbound: flags=0x2 [ 5398.545918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5398.549365] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5398.554027] workqueue events_power_efficient: flags=0x80 [ 5398.556690] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.559943] pending: check_lifetime [ 5398.561684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.565221] pending: neigh_periodic_work [ 5398.567722] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.571031] pending: neigh_periodic_work [ 5398.573253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.577240] pending: do_cache_clean [ 5398.578942] workqueue mm_percpu_wq: flags=0x8 [ 5398.581819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.585624] pending: vmstat_update [ 5398.587669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.591255] pending: vmstat_update [ 5398.593160] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5398.595807] pending: vmstat_update [ 5398.597958] workqueue writeback: flags=0x4a [ 5398.600345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5398.603872] pending: wb_workfn, wb_workfn [ 5398.605807] workqueue xprtiod: flags=0xa [ 5398.607527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5398.610636] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5398.614575] workqueue ipv6_addrconf: flags=0x40008 [ 5398.616696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5398.620020] pending: addrconf_verify_work [ 5398.622384] workqueue knbd0-recv: flags=0x1a [ 5398.624974] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5398.628524] in-flight: 651:recv_work [ 5398.630643] workqueue ptlrpc_pinger: flags=0x2 [ 5398.632737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5398.635794] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5398.638418] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3481s workers=4 idle: 37 31846 28897 [ 5398.641606] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5388s workers=2 idle: 653 [ 5408.057063] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5408.069425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5408.128092] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5408.135031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5408.139158] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5408.146062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.150351] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5408.165300] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5408.167780] RAX: 000000000000525b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5408.170411] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5408.174171] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5408.177150] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5408.183824] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5408.186061] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5408.192723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5408.196223] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5408.201866] Call Trace: [ 5408.204688] [ 5408.206619] ? show_regs.cold.9+0x22/0x2f [ 5408.209674] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5408.212428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5408.214319] ? watchdog+0x40/0x40 [ 5408.215264] ? hrtimer_interrupt+0x128/0x2c0 [ 5408.218482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5408.220388] ? apic_timer_interrupt+0xf/0x20 [ 5408.222219] [ 5408.223051] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.224750] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5408.227114] do_raw_spin_lock+0x177/0x190 [ 5408.228916] _raw_spin_lock+0x21/0x30 [ 5408.230119] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5408.241417] process_one_work+0x2c8/0x700 [ 5408.243762] worker_thread+0x296/0x6e0 [ 5408.245116] ? rescuer_thread+0x570/0x570 [ 5408.247373] kthread+0x1d1/0x200 [ 5408.248880] ? set_kthread_struct+0x70/0x70 [ 5408.254520] ret_from_fork+0x1f/0x30 [ 5420.058029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5420.061873] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5420.062617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5420.062693] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5420.062700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5420.062703] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5420.062717] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5420.062722] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5420.062726] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5420.062729] RDX: 0000000000000a66 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5420.062731] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5420.062734] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5420.062737] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5420.062740] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5420.062743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5420.062746] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5420.062753] Call Trace: [ 5420.062763] [ 5420.062768] ? show_regs.cold.9+0x22/0x2f [ 5420.062778] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5420.062784] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5420.062789] ? watchdog+0x40/0x40 [ 5420.062793] ? hrtimer_interrupt+0x128/0x2c0 [ 5420.062797] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5420.062805] ? apic_timer_interrupt+0xf/0x20 [ 5420.062810] [ 5420.062816] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5420.062821] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5420.062826] do_raw_spin_lock+0x177/0x190 [ 5420.062831] _raw_spin_lock+0x21/0x30 [ 5420.062836] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5420.063182] obd_mod_free+0x46/0x140 [ptlrpc] [ 5420.063456] mdc_commit_open+0x91/0xb0 [mdc] [ 5420.063502] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5420.063699] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5420.064363] after_reply+0xe2c/0x1780 [ptlrpc] [ 5420.064552] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5420.066441] ? __next_timer_interrupt+0x160/0x160 [ 5420.066458] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5420.067135] ? do_wait_intr+0xf0/0xf0 [ 5420.067145] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5420.067546] kthread+0x1d1/0x200 [ 5420.067553] ? set_kthread_struct+0x70/0x70 [ 5420.067557] ret_from_fork+0x1f/0x30 [ 5420.222268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5420.255483] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5420.263679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5420.268811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5420.271996] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5420.288071] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5420.291650] RAX: 0000000000007fb8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5420.294687] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5420.300597] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5420.304984] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5420.309601] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5420.312719] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5420.316721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5420.325122] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5420.328928] Call Trace: [ 5420.330255] [ 5420.331180] ? show_regs.cold.9+0x22/0x2f [ 5420.332983] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5420.338014] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5420.339422] ? watchdog+0x40/0x40 [ 5420.342732] ? hrtimer_interrupt+0x128/0x2c0 [ 5420.344319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5420.346221] ? apic_timer_interrupt+0xf/0x20 [ 5420.348270] [ 5420.349479] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5420.358079] do_raw_spin_lock+0x177/0x190 [ 5420.359313] _raw_spin_lock+0x21/0x30 [ 5420.363480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5420.366114] ? do_raw_spin_unlock+0x75/0x190 [ 5420.367532] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5420.371947] ? do_wait_intr+0xf0/0xf0 [ 5420.373860] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5420.376268] mdc_close+0x53b/0xd70 [mdc] [ 5420.379189] lmv_close+0x18d/0x490 [lmv] [ 5420.381459] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5420.384284] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5420.386463] ll_md_close+0x274/0x800 [lustre] [ 5420.388691] ll_file_release+0x128/0xc50 [lustre] [ 5420.390518] __fput+0x10d/0x440 [ 5420.392156] ____fput+0x12/0x20 [ 5420.393522] task_work_run+0xc6/0x110 [ 5420.398205] exit_to_usermode_loop+0x1dd/0x1f0 [ 5420.400118] do_syscall_64+0x3d6/0x440 [ 5420.401793] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5420.404068] RIP: 0033:0x7fa749552cd8 [ 5420.405604] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5420.416498] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5420.419613] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5420.422685] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5420.428268] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5420.431412] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5420.435774] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5424.055493] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5424.064516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5424.108138] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5424.115991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5424.122452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5424.128579] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5424.145929] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5424.153283] RAX: 000000000000271c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5424.157300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5424.165137] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5424.169848] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5424.172501] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5424.174764] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5424.181303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5424.190576] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5424.199085] Call Trace: [ 5424.200128] [ 5424.200928] ? show_regs.cold.9+0x22/0x2f [ 5424.208259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5424.211344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5424.216477] ? watchdog+0x40/0x40 [ 5424.219042] ? hrtimer_interrupt+0x128/0x2c0 [ 5424.223855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5424.227331] ? apic_timer_interrupt+0xf/0x20 [ 5424.232098] [ 5424.233007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5424.238035] do_raw_spin_lock+0x177/0x190 [ 5424.239393] _raw_spin_lock+0x21/0x30 [ 5424.244039] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5424.245895] ? do_raw_spin_unlock+0x75/0x190 [ 5424.253418] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5424.257865] ? do_wait_intr+0xf0/0xf0 [ 5424.264129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5424.270187] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5424.277523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5424.283542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.285967] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5424.294192] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5424.300014] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5424.301737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.303925] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5424.306384] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5424.308609] lmv_intent_open+0x338/0x1150 [lmv] [ 5424.310330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.314512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.319207] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5424.323079] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5424.326483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.332019] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5424.335057] ? do_raw_spin_unlock+0x75/0x190 [ 5424.338545] ? _raw_spin_unlock+0x12/0x30 [ 5424.341268] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5424.345375] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5424.351044] ll_atomic_open+0x36a/0xd70 [lustre] [ 5424.355102] lookup_open+0x356/0xb40 [ 5424.357419] do_last+0x313/0xfc0 [ 5424.360934] ? nd_jump_root+0xe5/0x160 [ 5424.362768] ? path_init+0x437/0x520 [ 5424.365813] path_openat+0xf7/0x500 [ 5424.366901] ? do_raw_spin_unlock+0x75/0x190 [ 5424.373373] do_filp_open+0x99/0x140 [ 5424.378147] ? getname_flags+0x6e/0x330 [ 5424.383983] ? __check_object_size+0xff/0x256 [ 5424.385478] ? do_raw_spin_unlock+0x75/0x190 [ 5424.386920] ? _raw_spin_unlock+0x12/0x30 [ 5424.394157] do_sys_openat2+0x2b4/0x410 [ 5424.399167] do_sys_open+0x73/0xa0 [ 5424.401585] __x64_sys_openat+0x24/0x30 [ 5424.412201] do_syscall_64+0xc1/0x440 [ 5424.418424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5424.422711] RIP: 0033:0x7fa749552332 [ 5424.424914] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5424.451531] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5424.456572] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5424.464608] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5424.468973] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5424.473986] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5424.480639] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5428.708048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3511s! [ 5428.713844] Showing busy workqueues and worker pools: [ 5428.716908] workqueue events: flags=0x0 [ 5428.718688] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5428.722080] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5428.736836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5428.745548] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5428.754855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.759226] pending: kfree_rcu_monitor [ 5428.762565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5428.769004] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5428.773786] workqueue events_long: flags=0x0 [ 5428.776557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.780564] pending: lu_objects_flush [obdclass] [ 5428.787157] workqueue events_unbound: flags=0x2 [ 5428.794804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5428.802785] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5428.809114] workqueue events_power_efficient: flags=0x80 [ 5428.812261] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.818769] pending: check_lifetime [ 5428.820989] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.823693] pending: neigh_periodic_work [ 5428.826112] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.838612] pending: neigh_periodic_work [ 5428.840499] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.844782] pending: do_cache_clean [ 5428.847815] workqueue mm_percpu_wq: flags=0x8 [ 5428.849815] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.868915] pending: vmstat_update [ 5428.870102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.877976] pending: vmstat_update [ 5428.879163] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5428.887788] pending: vmstat_update [ 5428.890552] workqueue writeback: flags=0x4a [ 5428.891895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5428.900491] pending: wb_workfn, wb_workfn [ 5428.903045] workqueue xprtiod: flags=0xa [ 5428.905505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5428.909156] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5428.913085] workqueue ipv6_addrconf: flags=0x40008 [ 5428.915520] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5428.918523] pending: addrconf_verify_work [ 5428.921980] workqueue knbd0-recv: flags=0x1a [ 5428.924773] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5428.928471] in-flight: 651:recv_work [ 5428.930193] workqueue ptlrpc_pinger: flags=0x2 [ 5428.932175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5428.934973] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5428.937965] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3511s workers=4 idle: 37 31846 28897 [ 5428.946198] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5418s workers=2 idle: 653 [ 5436.058623] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5436.064390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.102602] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.106954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5436.113183] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5436.117061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5436.121366] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5436.138746] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5436.144627] RAX: 00000000000049ad RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5436.147353] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5436.152656] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5436.156415] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5436.160914] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5436.164356] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5436.168970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.174117] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5436.178964] Call Trace: [ 5436.179881] [ 5436.180552] ? show_regs.cold.9+0x22/0x2f [ 5436.182042] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.186892] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.190959] ? watchdog+0x40/0x40 [ 5436.192510] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.196627] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.198640] ? apic_timer_interrupt+0xf/0x20 [ 5436.200930] [ 5436.202937] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5436.209928] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5436.213330] do_raw_spin_lock+0x177/0x190 [ 5436.217527] _raw_spin_lock+0x21/0x30 [ 5436.221221] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5436.226851] process_one_work+0x2c8/0x700 [ 5436.230255] worker_thread+0x296/0x6e0 [ 5436.232267] ? rescuer_thread+0x570/0x570 [ 5436.235340] kthread+0x1d1/0x200 [ 5436.241038] ? set_kthread_struct+0x70/0x70 [ 5436.243057] ret_from_fork+0x1f/0x30 [ 5448.059006] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5448.059733] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5448.069078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5448.069161] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5448.069169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5448.069172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5448.069191] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5448.069195] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5448.069200] RAX: 0000000000007c9e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5448.069202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5448.069356] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5448.069358] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5448.069361] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5448.069364] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5448.069366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5448.069369] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5448.069377] Call Trace: [ 5448.069383] [ 5448.069387] ? show_regs.cold.9+0x22/0x2f [ 5448.069397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5448.069403] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5448.069413] ? watchdog+0x40/0x40 [ 5448.069417] ? hrtimer_interrupt+0x128/0x2c0 [ 5448.069421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5448.069428] ? apic_timer_interrupt+0xf/0x20 [ 5448.069432] [ 5448.069434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5448.069441] do_raw_spin_lock+0x177/0x190 [ 5448.069446] _raw_spin_lock+0x21/0x30 [ 5448.069450] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5448.069957] ? do_raw_spin_unlock+0x75/0x190 [ 5448.069963] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5448.070178] ? do_wait_intr+0xf0/0xf0 [ 5448.070186] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5448.070728] mdc_close+0x53b/0xd70 [mdc] [ 5448.070766] lmv_close+0x18d/0x490 [lmv] [ 5448.070800] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5448.081717] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5448.081804] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5448.081812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5448.081815] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5448.081831] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5448.081835] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5448.081839] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5448.081841] RDX: 000000000000253a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5448.081843] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5448.081845] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5448.081847] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5448.081850] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5448.081853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5448.081855] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5448.081937] Call Trace: [ 5448.081944] [ 5448.081949] ? show_regs.cold.9+0x22/0x2f [ 5448.081960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5448.081967] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5448.081971] ? watchdog+0x40/0x40 [ 5448.081976] ? hrtimer_interrupt+0x128/0x2c0 [ 5448.081980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5448.081988] ? apic_timer_interrupt+0xf/0x20 [ 5448.081993] [ 5448.081996] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5448.082001] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5448.082006] do_raw_spin_lock+0x177/0x190 [ 5448.082011] _raw_spin_lock+0x21/0x30 [ 5448.082017] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5448.118657] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5448.118800] ll_md_close+0x274/0x800 [lustre] [ 5448.119025] ll_file_release+0x128/0xc50 [lustre] [ 5448.119140] __fput+0x10d/0x440 [ 5448.119149] ____fput+0x12/0x20 [ 5448.119157] task_work_run+0xc6/0x110 [ 5448.119165] exit_to_usermode_loop+0x1dd/0x1f0 [ 5448.119175] do_syscall_64+0x3d6/0x440 [ 5448.119179] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5448.119187] RIP: 0033:0x7fa749552cd8 [ 5448.119194] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5448.119199] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5448.119207] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5448.119210] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5448.119212] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5448.119214] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5448.119395] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5448.543441] obd_mod_free+0x46/0x140 [ptlrpc] [ 5448.547086] mdc_commit_open+0x91/0xb0 [mdc] [ 5448.552393] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5448.561675] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5448.569002] after_reply+0xe2c/0x1780 [ptlrpc] [ 5448.575045] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5448.579396] ? __next_timer_interrupt+0x160/0x160 [ 5448.585451] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5448.591731] ? do_wait_intr+0xf0/0xf0 [ 5448.596014] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5448.603755] kthread+0x1d1/0x200 [ 5448.604987] ? set_kthread_struct+0x70/0x70 [ 5448.610200] ret_from_fork+0x1f/0x30 [ 5456.056334] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5456.062312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5456.104440] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5456.114685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5456.122311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5456.127762] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5456.144323] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5456.150779] RAX: 0000000000006122 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5456.157636] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5456.164356] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5456.170412] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5456.174200] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5456.182636] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5456.190042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5456.195615] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5456.201721] Call Trace: [ 5456.204857] [ 5456.206031] ? show_regs.cold.9+0x22/0x2f [ 5456.209310] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5456.210924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5456.217709] ? watchdog+0x40/0x40 [ 5456.221282] ? hrtimer_interrupt+0x128/0x2c0 [ 5456.225097] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5456.229988] ? apic_timer_interrupt+0xf/0x20 [ 5456.233943] [ 5456.236585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5456.241439] do_raw_spin_lock+0x177/0x190 [ 5456.249982] _raw_spin_lock+0x21/0x30 [ 5456.253854] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5456.258074] ? do_raw_spin_unlock+0x75/0x190 [ 5456.262233] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5456.266630] ? do_wait_intr+0xf0/0xf0 [ 5456.268916] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5456.271552] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5456.276238] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5456.278161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.283932] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5456.288644] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5456.292736] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5456.297600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.302538] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5456.308844] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5456.314585] lmv_intent_open+0x338/0x1150 [lmv] [ 5456.317822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.320051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.322091] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5456.328617] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5456.336303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.345966] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5456.351009] ? do_raw_spin_unlock+0x75/0x190 [ 5456.360651] ? _raw_spin_unlock+0x12/0x30 [ 5456.370533] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5456.378729] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5456.385747] ll_atomic_open+0x36a/0xd70 [lustre] [ 5456.391622] lookup_open+0x356/0xb40 [ 5456.394707] do_last+0x313/0xfc0 [ 5456.395746] ? nd_jump_root+0xe5/0x160 [ 5456.404128] ? path_init+0x437/0x520 [ 5456.409002] path_openat+0xf7/0x500 [ 5456.415268] ? do_raw_spin_unlock+0x75/0x190 [ 5456.420014] do_filp_open+0x99/0x140 [ 5456.423916] ? getname_flags+0x6e/0x330 [ 5456.427972] ? __check_object_size+0xff/0x256 [ 5456.432948] ? do_raw_spin_unlock+0x75/0x190 [ 5456.437747] ? _raw_spin_unlock+0x12/0x30 [ 5456.442088] do_sys_openat2+0x2b4/0x410 [ 5456.446875] do_sys_open+0x73/0xa0 [ 5456.451215] __x64_sys_openat+0x24/0x30 [ 5456.455666] do_syscall_64+0xc1/0x440 [ 5456.459996] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5456.464294] RIP: 0033:0x7fa749552332 [ 5456.468860] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5456.484154] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5456.492195] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5456.495285] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5456.499001] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5456.508319] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5456.511231] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5459.427261] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3542s! [ 5459.430796] Showing busy workqueues and worker pools: [ 5459.433660] workqueue events: flags=0x0 [ 5459.434865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5459.438095] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5459.443840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5459.447999] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5459.460328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.470574] pending: kfree_rcu_monitor [ 5459.471997] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5459.478396] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5459.485240] workqueue events_long: flags=0x0 [ 5459.487408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.493982] pending: lu_objects_flush [obdclass] [ 5459.496321] workqueue events_unbound: flags=0x2 [ 5459.498663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5459.502901] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5459.506977] workqueue events_power_efficient: flags=0x80 [ 5459.509291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.513940] pending: check_lifetime [ 5459.516278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.518738] pending: neigh_periodic_work [ 5459.520305] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.522495] pending: neigh_periodic_work [ 5459.524373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.527471] pending: do_cache_clean [ 5459.528948] workqueue mm_percpu_wq: flags=0x8 [ 5459.530691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.532670] pending: vmstat_update [ 5459.534711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.537608] pending: vmstat_update [ 5459.538930] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5459.542670] pending: vmstat_update [ 5459.543845] workqueue writeback: flags=0x4a [ 5459.545415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5459.547943] pending: wb_workfn, wb_workfn [ 5459.552805] workqueue xprtiod: flags=0xa [ 5459.557251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5459.559948] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5459.564729] workqueue ipv6_addrconf: flags=0x40008 [ 5459.570552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5459.573623] pending: addrconf_verify_work [ 5459.578746] workqueue knbd0-recv: flags=0x1a [ 5459.580058] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5459.587593] in-flight: 651:recv_work [ 5459.589351] workqueue ptlrpc_pinger: flags=0x2 [ 5459.591248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5459.594429] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5459.599186] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3542s workers=4 idle: 37 31846 28897 [ 5459.603121] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5449s workers=2 idle: 653 [ 5464.057437] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5464.063166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5464.090125] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5464.095561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5464.099688] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5464.102361] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5464.103909] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5464.114628] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5464.118414] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5464.121197] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5464.125148] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5464.128065] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5464.130897] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5464.133835] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5464.140328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5464.144837] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5464.152445] Call Trace: [ 5464.154837] [ 5464.157632] ? show_regs.cold.9+0x22/0x2f [ 5464.162323] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5464.169685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5464.176685] ? watchdog+0x40/0x40 [ 5464.177705] ? hrtimer_interrupt+0x128/0x2c0 [ 5464.179143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5464.188300] ? apic_timer_interrupt+0xf/0x20 [ 5464.190598] [ 5464.192189] ? native_safe_halt+0x2e/0x50 [ 5464.193691] kvm_wait+0xcd/0xe0 [ 5464.194916] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5464.198182] do_raw_spin_lock+0x177/0x190 [ 5464.200339] _raw_spin_lock+0x21/0x30 [ 5464.202933] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5464.204977] process_one_work+0x2c8/0x700 [ 5464.209181] worker_thread+0x296/0x6e0 [ 5464.210801] ? rescuer_thread+0x570/0x570 [ 5464.214374] kthread+0x1d1/0x200 [ 5464.215957] ? set_kthread_struct+0x70/0x70 [ 5464.219124] ret_from_fork+0x1f/0x30 [ 5476.058007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5476.059011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5476.064126] Modules linked in: [ 5476.067388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5476.067577] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5476.067585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5476.067588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5476.067614] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5476.067619] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5476.067624] RAX: 0000000000000630 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5476.067627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5476.067629] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5476.067631] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5476.067633] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5476.067637] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5476.067640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5476.067643] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5476.067652] Call Trace: [ 5476.067669] [ 5476.067677] ? show_regs.cold.9+0x22/0x2f [ 5476.067687] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5476.067694] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5476.067699] ? watchdog+0x40/0x40 [ 5476.067703] ? hrtimer_interrupt+0x128/0x2c0 [ 5476.067708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5476.067718] ? apic_timer_interrupt+0xf/0x20 [ 5476.067723] [ 5476.067727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5476.067733] do_raw_spin_lock+0x177/0x190 [ 5476.067739] _raw_spin_lock+0x21/0x30 [ 5476.067744] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5476.068119] ? do_raw_spin_unlock+0x75/0x190 [ 5476.068127] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5476.068537] ? do_wait_intr+0xf0/0xf0 [ 5476.068546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5476.068741] mdc_close+0x53b/0xd70 [mdc] [ 5476.068870] lmv_close+0x18d/0x490 [lmv] [ 5476.068901] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5476.069017] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5476.069110] ll_md_close+0x274/0x800 [lustre] [ 5476.069205] ll_file_release+0x128/0xc50 [lustre] [ 5476.069560] __fput+0x10d/0x440 [ 5476.069570] ____fput+0x12/0x20 [ 5476.069575] task_work_run+0xc6/0x110 [ 5476.069583] exit_to_usermode_loop+0x1dd/0x1f0 [ 5476.069590] do_syscall_64+0x3d6/0x440 [ 5476.069594] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5476.069600] RIP: 0033:0x7fa749552cd8 [ 5476.069627] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5476.069632] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5476.069637] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5476.069639] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5476.069642] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5476.069644] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5476.069646] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5476.311826] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5476.360451] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5476.371817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5476.376746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5476.379603] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5476.406433] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5476.412619] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5476.417327] RDX: 0000000000002b0b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5476.420717] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5476.423389] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5476.426172] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5476.431053] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5476.441966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5476.450125] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5476.458662] Call Trace: [ 5476.463325] [ 5476.465553] ? show_regs.cold.9+0x22/0x2f [ 5476.470692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5476.473287] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5476.476189] ? watchdog+0x40/0x40 [ 5476.478353] ? hrtimer_interrupt+0x128/0x2c0 [ 5476.484577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5476.488261] ? apic_timer_interrupt+0xf/0x20 [ 5476.493441] [ 5476.496706] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5476.498675] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5476.502924] do_raw_spin_lock+0x177/0x190 [ 5476.508412] _raw_spin_lock+0x21/0x30 [ 5476.511564] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5476.515548] obd_mod_free+0x46/0x140 [ptlrpc] [ 5476.523088] mdc_commit_open+0x91/0xb0 [mdc] [ 5476.526941] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5476.541456] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5476.552412] after_reply+0xe2c/0x1780 [ptlrpc] [ 5476.556387] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5476.558586] ? __next_timer_interrupt+0x160/0x160 [ 5476.560963] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5476.562716] ? do_wait_intr+0xf0/0xf0 [ 5476.565121] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5476.568925] kthread+0x1d1/0x200 [ 5476.569926] ? set_kthread_struct+0x70/0x70 [ 5476.571366] ret_from_fork+0x1f/0x30 [ 5484.056278] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5484.060936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5484.090144] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5484.098064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5484.105212] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5484.109546] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5484.117761] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5484.121468] RAX: 0000000000005427 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5484.124226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5484.127232] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5484.130291] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5484.133726] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5484.136769] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5484.139859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5484.144214] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5484.148125] Call Trace: [ 5484.148852] [ 5484.149542] ? show_regs.cold.9+0x22/0x2f [ 5484.151449] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5484.153210] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5484.156510] ? watchdog+0x40/0x40 [ 5484.159907] ? hrtimer_interrupt+0x128/0x2c0 [ 5484.161557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5484.165509] ? apic_timer_interrupt+0xf/0x20 [ 5484.167987] [ 5484.170514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5484.172487] do_raw_spin_lock+0x177/0x190 [ 5484.174030] _raw_spin_lock+0x21/0x30 [ 5484.177144] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5484.180984] ? do_raw_spin_unlock+0x75/0x190 [ 5484.185565] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5484.190570] ? do_wait_intr+0xf0/0xf0 [ 5484.194842] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5484.199262] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5484.203870] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5484.211294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.215757] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5484.220404] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5484.222398] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5484.226706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.231725] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5484.235344] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5484.241176] lmv_intent_open+0x338/0x1150 [lmv] [ 5484.244399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.246838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.248851] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5484.254431] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5484.257566] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.260706] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5484.265752] ? do_raw_spin_unlock+0x75/0x190 [ 5484.268104] ? _raw_spin_unlock+0x12/0x30 [ 5484.270195] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5484.273497] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5484.275440] ll_atomic_open+0x36a/0xd70 [lustre] [ 5484.277491] lookup_open+0x356/0xb40 [ 5484.279178] do_last+0x313/0xfc0 [ 5484.280795] ? nd_jump_root+0xe5/0x160 [ 5484.282395] ? path_init+0x437/0x520 [ 5484.284253] path_openat+0xf7/0x500 [ 5484.285897] ? do_raw_spin_unlock+0x75/0x190 [ 5484.287895] do_filp_open+0x99/0x140 [ 5484.291297] ? getname_flags+0x6e/0x330 [ 5484.296101] ? __check_object_size+0xff/0x256 [ 5484.298091] ? do_raw_spin_unlock+0x75/0x190 [ 5484.300243] ? _raw_spin_unlock+0x12/0x30 [ 5484.303192] do_sys_openat2+0x2b4/0x410 [ 5484.305105] do_sys_open+0x73/0xa0 [ 5484.309603] __x64_sys_openat+0x24/0x30 [ 5484.315029] do_syscall_64+0xc1/0x440 [ 5484.316982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5484.318963] RIP: 0033:0x7fa749552332 [ 5484.320195] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5484.330581] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5484.337475] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5484.347969] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5484.356325] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5484.361691] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5484.366158] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5489.633990] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3572s! [ 5489.641218] Showing busy workqueues and worker pools: [ 5489.643125] workqueue events: flags=0x0 [ 5489.647777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5489.651944] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5489.657014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5489.660134] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5489.664088] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.666297] pending: kfree_rcu_monitor [ 5489.668329] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5489.670924] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5489.675034] workqueue events_long: flags=0x0 [ 5489.679480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.684204] pending: lu_objects_flush [obdclass] [ 5489.687490] workqueue events_unbound: flags=0x2 [ 5489.689250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5489.691299] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5489.695581] workqueue events_power_efficient: flags=0x80 [ 5489.699041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.701972] pending: check_lifetime [ 5489.703413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.706088] pending: neigh_periodic_work [ 5489.708013] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.713348] pending: neigh_periodic_work [ 5489.714941] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.717091] pending: do_cache_clean [ 5489.718268] workqueue mm_percpu_wq: flags=0x8 [ 5489.719914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.722341] pending: vmstat_update [ 5489.726232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.731349] pending: vmstat_update [ 5489.738126] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5489.743007] pending: vmstat_update [ 5489.745319] workqueue writeback: flags=0x4a [ 5489.748925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5489.754454] pending: wb_workfn, wb_workfn [ 5489.755718] workqueue xprtiod: flags=0xa [ 5489.756726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5489.761756] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5489.771972] workqueue ipv6_addrconf: flags=0x40008 [ 5489.776591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5489.778737] pending: addrconf_verify_work [ 5489.780318] workqueue knbd0-recv: flags=0x1a [ 5489.781779] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5489.783992] in-flight: 651:recv_work [ 5489.785314] workqueue ptlrpc_pinger: flags=0x2 [ 5489.786968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5489.789030] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5489.791279] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3572s workers=4 idle: 37 31846 28897 [ 5489.794171] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5479s workers=2 idle: 653 [ 5492.057054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5492.059502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5492.074215] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5492.077886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5492.080713] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5492.083134] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5492.084462] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5492.090407] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5492.093150] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5492.095364] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5492.097872] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5492.100251] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5492.102447] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5492.105114] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5492.108136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5492.110291] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5492.112598] Call Trace: [ 5492.113573] [ 5492.114746] ? show_regs.cold.9+0x22/0x2f [ 5492.116260] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5492.118307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5492.119809] ? watchdog+0x40/0x40 [ 5492.120881] ? hrtimer_interrupt+0x128/0x2c0 [ 5492.122236] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5492.123923] ? apic_timer_interrupt+0xf/0x20 [ 5492.125617] [ 5492.126419] ? native_safe_halt+0x2e/0x50 [ 5492.127948] kvm_wait+0xcd/0xe0 [ 5492.129197] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5492.131433] do_raw_spin_lock+0x177/0x190 [ 5492.133137] _raw_spin_lock+0x21/0x30 [ 5492.134322] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5492.136443] process_one_work+0x2c8/0x700 [ 5492.138097] worker_thread+0x296/0x6e0 [ 5492.139252] ? rescuer_thread+0x570/0x570 [ 5492.140410] kthread+0x1d1/0x200 [ 5492.141921] ? set_kthread_struct+0x70/0x70 [ 5492.143333] ret_from_fork+0x1f/0x30 [ 5504.058234] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5504.060618] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5504.061901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5504.061983] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5504.062169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5504.062172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5504.062186] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5504.062190] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5504.062194] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5504.062197] RDX: 0000000000001172 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5504.062199] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5504.062202] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5504.062204] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5504.062206] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5504.062209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5504.062211] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5504.062219] Call Trace: [ 5504.062225] [ 5504.062230] ? show_regs.cold.9+0x22/0x2f [ 5504.062239] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5504.062245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5504.062249] ? watchdog+0x40/0x40 [ 5504.062253] ? hrtimer_interrupt+0x128/0x2c0 [ 5504.062256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5504.062265] ? apic_timer_interrupt+0xf/0x20 [ 5504.062280] [ 5504.062284] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5504.062288] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5504.062293] do_raw_spin_lock+0x177/0x190 [ 5504.062297] _raw_spin_lock+0x21/0x30 [ 5504.062302] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5504.062749] obd_mod_free+0x46/0x140 [ptlrpc] [ 5504.062949] mdc_commit_open+0x91/0xb0 [mdc] [ 5504.062989] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5504.063184] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5504.063393] after_reply+0xe2c/0x1780 [ptlrpc] [ 5504.063760] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5504.063971] ? __next_timer_interrupt+0x160/0x160 [ 5504.063986] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5504.064190] ? do_wait_intr+0xf0/0xf0 [ 5504.064198] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5504.064479] kthread+0x1d1/0x200 [ 5504.064485] ? set_kthread_struct+0x70/0x70 [ 5504.064489] ret_from_fork+0x1f/0x30 [ 5504.184730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5504.202870] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5504.205742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5504.208444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5504.210184] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5504.215252] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5504.217250] RAX: 0000000000001a94 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5504.219255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5504.221145] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5504.223325] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5504.225147] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5504.227042] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5504.229006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5504.230489] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5504.232862] Call Trace: [ 5504.233507] [ 5504.234037] ? show_regs.cold.9+0x22/0x2f [ 5504.235141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5504.236386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5504.237664] ? watchdog+0x40/0x40 [ 5504.239394] ? hrtimer_interrupt+0x128/0x2c0 [ 5504.240900] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5504.242622] ? apic_timer_interrupt+0xf/0x20 [ 5504.244255] [ 5504.244903] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5504.246461] do_raw_spin_lock+0x177/0x190 [ 5504.247734] _raw_spin_lock+0x21/0x30 [ 5504.248870] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5504.250916] ? do_raw_spin_unlock+0x75/0x190 [ 5504.252422] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5504.254484] ? do_wait_intr+0xf0/0xf0 [ 5504.255752] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5504.258203] mdc_close+0x53b/0xd70 [mdc] [ 5504.259629] lmv_close+0x18d/0x490 [lmv] [ 5504.260904] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5504.263516] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5504.265736] ll_md_close+0x274/0x800 [lustre] [ 5504.267329] ll_file_release+0x128/0xc50 [lustre] [ 5504.269114] __fput+0x10d/0x440 [ 5504.270099] ____fput+0x12/0x20 [ 5504.271053] task_work_run+0xc6/0x110 [ 5504.272418] exit_to_usermode_loop+0x1dd/0x1f0 [ 5504.273916] do_syscall_64+0x3d6/0x440 [ 5504.275263] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5504.277002] RIP: 0033:0x7fa749552cd8 [ 5504.278037] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5504.283514] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5504.285360] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5504.287263] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5504.289530] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5504.291618] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5504.293857] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5516.055321] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5516.057442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5516.089205] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5516.093113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5516.097577] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5516.099740] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5516.107831] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5516.116695] RAX: 0000000000006b29 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5516.119577] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5516.128839] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5516.132419] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5516.136316] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5516.139337] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5516.143409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5516.146133] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5516.149224] Call Trace: [ 5516.156482] [ 5516.157290] ? show_regs.cold.9+0x22/0x2f [ 5516.159364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5516.163508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5516.169037] ? watchdog+0x40/0x40 [ 5516.170102] ? hrtimer_interrupt+0x128/0x2c0 [ 5516.172052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5516.173976] ? apic_timer_interrupt+0xf/0x20 [ 5516.177441] [ 5516.178131] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5516.181283] do_raw_spin_lock+0x177/0x190 [ 5516.182685] _raw_spin_lock+0x21/0x30 [ 5516.184231] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5516.187811] ? do_raw_spin_unlock+0x75/0x190 [ 5516.189175] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5516.193517] ? do_wait_intr+0xf0/0xf0 [ 5516.195919] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5516.198387] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5516.201774] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5516.205760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.209266] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5516.211041] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5516.213676] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5516.215431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.218926] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5516.222578] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5516.224863] lmv_intent_open+0x338/0x1150 [lmv] [ 5516.228881] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.237974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.242040] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5516.244561] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5516.246658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.250708] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5516.254926] ? do_raw_spin_unlock+0x75/0x190 [ 5516.258229] ? _raw_spin_unlock+0x12/0x30 [ 5516.260228] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5516.264088] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5516.265718] ll_atomic_open+0x36a/0xd70 [lustre] [ 5516.268244] lookup_open+0x356/0xb40 [ 5516.272879] do_last+0x313/0xfc0 [ 5516.275216] ? nd_jump_root+0xe5/0x160 [ 5516.276844] ? path_init+0x437/0x520 [ 5516.278585] path_openat+0xf7/0x500 [ 5516.279813] ? do_raw_spin_unlock+0x75/0x190 [ 5516.283124] do_filp_open+0x99/0x140 [ 5516.284672] ? getname_flags+0x6e/0x330 [ 5516.289419] ? __check_object_size+0xff/0x256 [ 5516.296694] ? do_raw_spin_unlock+0x75/0x190 [ 5516.298565] ? _raw_spin_unlock+0x12/0x30 [ 5516.300445] do_sys_openat2+0x2b4/0x410 [ 5516.301862] do_sys_open+0x73/0xa0 [ 5516.303291] __x64_sys_openat+0x24/0x30 [ 5516.304781] do_syscall_64+0xc1/0x440 [ 5516.306259] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5516.307977] RIP: 0033:0x7fa749552332 [ 5516.309007] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5516.317421] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5516.320400] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5516.323166] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5516.326287] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5516.329428] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5516.333338] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5519.843035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3602s! [ 5519.847642] Showing busy workqueues and worker pools: [ 5519.850243] workqueue events: flags=0x0 [ 5519.852276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5519.856164] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5519.862841] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5519.865854] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5519.869675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.873554] pending: kfree_rcu_monitor [ 5519.875844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5519.879572] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5519.886068] workqueue events_long: flags=0x0 [ 5519.888072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.891971] pending: lu_objects_flush [obdclass] [ 5519.894867] workqueue events_unbound: flags=0x2 [ 5519.897258] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5519.900296] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5519.905418] workqueue events_power_efficient: flags=0x80 [ 5519.908333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.911689] pending: check_lifetime [ 5519.914125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.917099] pending: neigh_periodic_work [ 5519.919357] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.923313] pending: neigh_periodic_work [ 5519.925554] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.929343] pending: do_cache_clean [ 5519.933462] workqueue mm_percpu_wq: flags=0x8 [ 5519.936203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.939766] pending: vmstat_update [ 5519.942200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.946178] pending: vmstat_update [ 5519.948217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5519.951458] pending: vmstat_update [ 5519.953820] workqueue writeback: flags=0x4a [ 5519.955867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5519.959257] pending: wb_workfn, wb_workfn [ 5519.961548] workqueue xprtiod: flags=0xa [ 5519.963295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5519.966792] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5519.970813] workqueue ipv6_addrconf: flags=0x40008 [ 5519.973842] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5519.978197] pending: addrconf_verify_work [ 5519.981078] workqueue knbd0-recv: flags=0x1a [ 5519.983543] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5519.987749] in-flight: 651:recv_work [ 5519.990367] workqueue ptlrpc_pinger: flags=0x2 [ 5519.993333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5519.997273] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5520.000611] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3602s workers=4 idle: 37 31846 28897 [ 5520.006100] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5509s workers=2 idle: 653 [ 5520.057491] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5520.062540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5520.091272] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5520.097786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5520.104052] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5520.108316] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5520.112873] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5520.124640] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5520.129446] RAX: 000000000000284d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5520.134484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5520.138436] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5520.142322] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5520.145908] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5520.149861] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5520.154307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5520.157192] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5520.161260] Call Trace: [ 5520.162446] [ 5520.163668] ? show_regs.cold.9+0x22/0x2f [ 5520.165859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5520.168763] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5520.171363] ? watchdog+0x40/0x40 [ 5520.173384] ? hrtimer_interrupt+0x128/0x2c0 [ 5520.175953] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5520.179840] ? apic_timer_interrupt+0xf/0x20 [ 5520.181894] [ 5520.183406] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5520.187849] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5520.190719] do_raw_spin_lock+0x177/0x190 [ 5520.192222] _raw_spin_lock+0x21/0x30 [ 5520.193565] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5520.197868] process_one_work+0x2c8/0x700 [ 5520.201239] worker_thread+0x296/0x6e0 [ 5520.203622] ? rescuer_thread+0x570/0x570 [ 5520.207090] kthread+0x1d1/0x200 [ 5520.209084] ? set_kthread_struct+0x70/0x70 [ 5520.211755] ret_from_fork+0x1f/0x30 [ 5532.058580] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5532.060113] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5532.064723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.064814] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.064822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5532.064825] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5532.064839] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5532.064844] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5532.064848] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5532.064851] RDX: 0000000000005284 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5532.064854] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5532.064856] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5532.064858] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5532.064861] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5532.064864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.064866] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5532.064874] Call Trace: [ 5532.064880] [ 5532.064886] ? show_regs.cold.9+0x22/0x2f [ 5532.064895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.064901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.064906] ? watchdog+0x40/0x40 [ 5532.064910] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.064915] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.064923] ? apic_timer_interrupt+0xf/0x20 [ 5532.064927] [ 5532.064930] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5532.064935] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5532.064940] do_raw_spin_lock+0x177/0x190 [ 5532.064944] _raw_spin_lock+0x21/0x30 [ 5532.064951] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5532.066417] obd_mod_free+0x46/0x140 [ptlrpc] [ 5532.066689] mdc_commit_open+0x91/0xb0 [mdc] [ 5532.066724] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5532.066979] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5532.067723] after_reply+0xe2c/0x1780 [ptlrpc] [ 5532.067990] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5532.068707] ? __next_timer_interrupt+0x160/0x160 [ 5532.068719] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5532.069364] ? do_wait_intr+0xf0/0xf0 [ 5532.069373] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5532.084928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.085003] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.085010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5532.085012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5532.085026] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5532.085029] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5532.085033] RAX: 0000000000003c84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5532.085035] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5532.085038] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5532.085040] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5532.085042] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5532.085045] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5532.085049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.085051] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5532.085059] Call Trace: [ 5532.085065] [ 5532.085083] ? show_regs.cold.9+0x22/0x2f [ 5532.085093] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.085100] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.085105] ? watchdog+0x40/0x40 [ 5532.085110] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.085114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.085122] ? apic_timer_interrupt+0xf/0x20 [ 5532.085127] [ 5532.085129] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5532.085136] do_raw_spin_lock+0x177/0x190 [ 5532.085140] _raw_spin_lock+0x21/0x30 [ 5532.085145] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5532.085395] ? do_raw_spin_unlock+0x75/0x190 [ 5532.085401] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5532.085589] ? do_wait_intr+0xf0/0xf0 [ 5532.085596] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5532.085791] mdc_close+0x53b/0xd70 [mdc] [ 5532.085828] lmv_close+0x18d/0x490 [lmv] [ 5532.085853] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5532.085963] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5532.086040] ll_md_close+0x274/0x800 [lustre] [ 5532.086134] ll_file_release+0x128/0xc50 [lustre] [ 5532.086216] __fput+0x10d/0x440 [ 5532.086224] ____fput+0x12/0x20 [ 5532.086229] task_work_run+0xc6/0x110 [ 5532.086235] exit_to_usermode_loop+0x1dd/0x1f0 [ 5532.086244] do_syscall_64+0x3d6/0x440 [ 5532.086249] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5532.086255] RIP: 0033:0x7fa749552cd8 [ 5532.086261] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5532.086265] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5532.086269] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5532.086271] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5532.086273] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5532.086275] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5532.086277] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5532.564377] kthread+0x1d1/0x200 [ 5532.567331] ? set_kthread_struct+0x70/0x70 [ 5532.571631] ret_from_fork+0x1f/0x30 [ 5544.054012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5544.057321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5544.099709] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5544.107557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5544.114764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5544.117250] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5544.137485] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5544.141916] RAX: 00000000000062ea RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5544.145068] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5544.148755] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5544.152077] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5544.155305] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5544.160308] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5544.164922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5544.167541] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5544.170501] Call Trace: [ 5544.175865] [ 5544.176756] ? show_regs.cold.9+0x22/0x2f [ 5544.179394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5544.182306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5544.184425] ? watchdog+0x40/0x40 [ 5544.186284] ? hrtimer_interrupt+0x128/0x2c0 [ 5544.188196] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5544.190310] ? apic_timer_interrupt+0xf/0x20 [ 5544.192251] [ 5544.193192] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5544.195789] do_raw_spin_lock+0x177/0x190 [ 5544.198212] _raw_spin_lock+0x21/0x30 [ 5544.200157] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5544.206771] ? do_raw_spin_unlock+0x75/0x190 [ 5544.208367] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5544.216519] ? do_wait_intr+0xf0/0xf0 [ 5544.219635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5544.228559] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5544.231737] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5544.235957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.242489] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5544.247378] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5544.252437] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5544.256289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.258774] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5544.264511] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5544.270196] lmv_intent_open+0x338/0x1150 [lmv] [ 5544.275973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.281047] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.283424] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5544.288309] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5544.290213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.294052] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5544.299620] ? do_raw_spin_unlock+0x75/0x190 [ 5544.301296] ? _raw_spin_unlock+0x12/0x30 [ 5544.305770] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5544.307778] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5544.311841] ll_atomic_open+0x36a/0xd70 [lustre] [ 5544.314168] lookup_open+0x356/0xb40 [ 5544.317289] do_last+0x313/0xfc0 [ 5544.318673] ? nd_jump_root+0xe5/0x160 [ 5544.320310] ? path_init+0x437/0x520 [ 5544.321900] path_openat+0xf7/0x500 [ 5544.323151] ? do_raw_spin_unlock+0x75/0x190 [ 5544.325008] do_filp_open+0x99/0x140 [ 5544.326680] ? getname_flags+0x6e/0x330 [ 5544.328205] ? __check_object_size+0xff/0x256 [ 5544.329780] ? do_raw_spin_unlock+0x75/0x190 [ 5544.331494] ? _raw_spin_unlock+0x12/0x30 [ 5544.333224] do_sys_openat2+0x2b4/0x410 [ 5544.338117] do_sys_open+0x73/0xa0 [ 5544.339399] __x64_sys_openat+0x24/0x30 [ 5544.340821] do_syscall_64+0xc1/0x440 [ 5544.346532] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5544.349141] RIP: 0033:0x7fa749552332 [ 5544.351004] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5544.362427] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5544.369400] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5544.374027] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5544.381719] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5544.390711] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5544.396627] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5548.057027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5548.060011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5548.081281] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5548.087422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5548.092312] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5548.101413] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5548.103386] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5548.112206] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5548.115628] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5548.119704] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5548.123964] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5548.130700] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5548.137724] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5548.149580] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5548.160082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5548.164246] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5548.170369] Call Trace: [ 5548.174487] [ 5548.177825] ? show_regs.cold.9+0x22/0x2f [ 5548.179562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5548.181120] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5548.190160] ? watchdog+0x40/0x40 [ 5548.191384] ? hrtimer_interrupt+0x128/0x2c0 [ 5548.196637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5548.199679] ? apic_timer_interrupt+0xf/0x20 [ 5548.202310] [ 5548.203175] ? native_safe_halt+0x2e/0x50 [ 5548.212143] kvm_wait+0xcd/0xe0 [ 5548.213290] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5548.215966] do_raw_spin_lock+0x177/0x190 [ 5548.218090] _raw_spin_lock+0x21/0x30 [ 5548.220923] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5548.222874] process_one_work+0x2c8/0x700 [ 5548.225816] worker_thread+0x296/0x6e0 [ 5548.233832] ? rescuer_thread+0x570/0x570 [ 5548.238634] kthread+0x1d1/0x200 [ 5548.243029] ? set_kthread_struct+0x70/0x70 [ 5548.246081] ret_from_fork+0x1f/0x30 [ 5550.049173] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3632s! [ 5550.055322] Showing busy workqueues and worker pools: [ 5550.058909] workqueue events: flags=0x0 [ 5550.062862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5550.067718] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5550.072259] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5550.074939] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5550.079340] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.081947] pending: kfree_rcu_monitor [ 5550.084187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5550.087220] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5550.091427] workqueue events_long: flags=0x0 [ 5550.093225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.096717] pending: lu_objects_flush [obdclass] [ 5550.099174] workqueue events_unbound: flags=0x2 [ 5550.100913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5550.103243] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5550.107388] workqueue events_power_efficient: flags=0x80 [ 5550.109686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.112202] pending: check_lifetime [ 5550.113848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.116303] pending: neigh_periodic_work [ 5550.118337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.122159] pending: neigh_periodic_work [ 5550.124011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.127052] pending: do_cache_clean [ 5550.128381] workqueue mm_percpu_wq: flags=0x8 [ 5550.130029] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.133064] pending: vmstat_update [ 5550.135134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.137717] pending: vmstat_update [ 5550.139356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5550.142515] pending: vmstat_update [ 5550.143994] workqueue writeback: flags=0x4a [ 5550.145311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5550.147730] pending: wb_workfn, wb_workfn [ 5550.149899] workqueue xprtiod: flags=0xa [ 5550.152125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5550.154687] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5550.158229] workqueue ipv6_addrconf: flags=0x40008 [ 5550.160343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5550.162918] pending: addrconf_verify_work [ 5550.166725] workqueue knbd0-recv: flags=0x1a [ 5550.168719] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5550.171700] in-flight: 651:recv_work [ 5550.174216] workqueue ptlrpc_pinger: flags=0x2 [ 5550.176535] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5550.182078] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5550.184401] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3633s workers=4 idle: 37 31846 28897 [ 5550.189196] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5540s workers=2 idle: 653 [ 5560.058504] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5560.061052] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5560.064641] Modules linked in: lustre(O) [ 5560.071726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5560.071814] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5560.071822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5560.071824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5560.071839] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5560.071844] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5560.071848] RAX: 0000000000007f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5560.071851] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5560.071853] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5560.071855] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5560.071858] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5560.071861] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5560.071864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5560.071867] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5560.071876] Call Trace: [ 5560.071881] [ 5560.071886] ? show_regs.cold.9+0x22/0x2f [ 5560.071895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5560.071902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5560.071913] ? watchdog+0x40/0x40 [ 5560.071918] ? hrtimer_interrupt+0x128/0x2c0 [ 5560.071922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5560.071931] ? apic_timer_interrupt+0xf/0x20 [ 5560.071936] [ 5560.071940] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5560.071947] do_raw_spin_lock+0x177/0x190 [ 5560.071952] _raw_spin_lock+0x21/0x30 [ 5560.071957] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5560.072253] ? do_raw_spin_unlock+0x75/0x190 [ 5560.072260] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5560.072603] ? do_wait_intr+0xf0/0xf0 [ 5560.072611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5560.072807] mdc_close+0x53b/0xd70 [mdc] [ 5560.072846] lmv_close+0x18d/0x490 [lmv] [ 5560.072888] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5560.073001] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5560.073159] ll_md_close+0x274/0x800 [lustre] [ 5560.073245] ll_file_release+0x128/0xc50 [lustre] [ 5560.074683] __fput+0x10d/0x440 [ 5560.074693] ____fput+0x12/0x20 [ 5560.074698] task_work_run+0xc6/0x110 [ 5560.074705] exit_to_usermode_loop+0x1dd/0x1f0 [ 5560.074713] do_syscall_64+0x3d6/0x440 [ 5560.074718] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5560.074728] RIP: 0033:0x7fa749552cd8 [ 5560.074736] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5560.074740] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5560.074745] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5560.074747] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5560.074749] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5560.074751] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5560.074753] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5560.394828] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5560.423715] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5560.437953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5560.445864] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5560.453099] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5560.472902] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5560.480894] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5560.488054] RDX: 0000000000007e1a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5560.495964] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5560.503031] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5560.510088] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5560.518177] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5560.525515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5560.534052] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5560.542514] Call Trace: [ 5560.543366] [ 5560.544046] ? show_regs.cold.9+0x22/0x2f [ 5560.554258] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5560.560079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5560.569977] ? watchdog+0x40/0x40 [ 5560.576209] ? hrtimer_interrupt+0x128/0x2c0 [ 5560.581678] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5560.594978] ? apic_timer_interrupt+0xf/0x20 [ 5560.601855] [ 5560.602912] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5560.619824] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5560.632516] do_raw_spin_lock+0x177/0x190 [ 5560.639445] _raw_spin_lock+0x21/0x30 [ 5560.649978] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5560.651882] obd_mod_free+0x46/0x140 [ptlrpc] [ 5560.658489] mdc_commit_open+0x91/0xb0 [mdc] [ 5560.668379] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5560.674126] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5560.680856] after_reply+0xe2c/0x1780 [ptlrpc] [ 5560.686436] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5560.692477] ? __next_timer_interrupt+0x160/0x160 [ 5560.698864] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5560.701505] ? do_wait_intr+0xf0/0xf0 [ 5560.705969] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5560.710486] kthread+0x1d1/0x200 [ 5560.711494] ? set_kthread_struct+0x70/0x70 [ 5560.714505] ret_from_fork+0x1f/0x30 [ 5573.765026] rcu: INFO: rcu_sched self-detected stall on CPU [ 5573.767649] rcu: 2-...!: (2001299 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 5573.773251] (t=3660076 jiffies g=258701 q=2531) [ 5573.775409] rcu: rcu_sched kthread starved for 3656871 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5573.785870] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5573.789867] rcu: RCU grace-period kthread stack dump: [ 5573.792584] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5573.797156] Call Trace: [ 5573.798240] __schedule+0x351/0xcb0 [ 5573.799668] schedule+0xc0/0x180 [ 5573.801285] schedule_timeout+0xb4/0x190 [ 5573.802808] ? do_raw_spin_unlock+0x75/0x190 [ 5573.804168] ? __next_timer_interrupt+0x160/0x160 [ 5573.806163] ? prepare_to_swait_event+0xf3/0x140 [ 5573.808258] rcu_gp_kthread+0x8dd/0x15b0 [ 5573.810242] ? force_qs_rnp+0x2c0/0x2c0 [ 5573.817914] kthread+0x1d1/0x200 [ 5573.821965] ? set_kthread_struct+0x70/0x70 [ 5573.830761] ret_from_fork+0x1f/0x30 [ 5573.838796] rcu: Stack dump where RCU GP kthread last ran: [ 5573.847789] NMI backtrace for cpu 2 [ 5573.854226] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5573.868088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5573.880996] Call Trace: [ 5573.883122] [ 5573.889293] ? dump_stack+0xbb/0x10e [ 5573.890671] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5573.894754] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5573.901746] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5573.909005] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5573.920226] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5573.925003] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5573.933156] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 5573.937107] ? update_process_times+0x6d/0xc0 [ 5573.941737] ? tick_sched_handle.isra.18+0x27/0x90 [ 5573.947894] ? tick_sched_timer+0x53/0xb0 [ 5573.948835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5573.951932] ? tick_sched_handle.isra.18+0x90/0x90 [ 5573.958237] ? hrtimer_interrupt+0x128/0x2c0 [ 5573.963784] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5573.969864] ? apic_timer_interrupt+0xf/0x20 [ 5573.974619] [ 5573.981815] ? native_safe_halt+0x2e/0x50 [ 5573.983248] ? kvm_wait+0xcd/0xe0 [ 5573.987527] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5573.991969] ? do_raw_spin_lock+0x177/0x190 [ 5573.994667] ? _raw_spin_lock+0x21/0x30 [ 5573.998357] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5574.002748] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5574.008356] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5574.010505] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5574.016373] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5574.021809] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5574.027251] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5574.031654] ? __next_timer_interrupt+0x160/0x160 [ 5574.036856] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5574.041861] ? do_wait_intr+0xf0/0xf0 [ 5574.045171] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5574.049089] ? kthread+0x1d1/0x200 [ 5574.050002] ? set_kthread_struct+0x70/0x70 [ 5574.051542] ? ret_from_fork+0x1f/0x30 [ 5574.052930] NMI backtrace for cpu 2 [ 5574.054193] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5574.060907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5574.073951] Call Trace: [ 5574.076291] [ 5574.079208] ? dump_stack+0xbb/0x10e [ 5574.082781] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5574.093792] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5574.098225] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5574.107983] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5574.109624] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5574.118642] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5574.120159] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5574.124021] ? update_process_times+0x6d/0xc0 [ 5574.128218] ? tick_sched_handle.isra.18+0x27/0x90 [ 5574.130806] ? tick_sched_timer+0x53/0xb0 [ 5574.132990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5574.135012] ? tick_sched_handle.isra.18+0x90/0x90 [ 5574.137895] ? hrtimer_interrupt+0x128/0x2c0 [ 5574.140637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5574.146392] ? apic_timer_interrupt+0xf/0x20 [ 5574.147871] [ 5574.150139] ? native_safe_halt+0x2e/0x50 [ 5574.151956] ? kvm_wait+0xcd/0xe0 [ 5574.156915] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5574.160921] ? do_raw_spin_lock+0x177/0x190 [ 5574.164501] ? _raw_spin_lock+0x21/0x30 [ 5574.167446] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5574.169317] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5574.177125] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5574.178929] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5574.189085] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5574.197297] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5574.203501] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5574.206709] ? __next_timer_interrupt+0x160/0x160 [ 5574.210311] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5574.212864] ? do_wait_intr+0xf0/0xf0 [ 5574.215385] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5574.221973] ? kthread+0x1d1/0x200 [ 5574.224904] ? set_kthread_struct+0x70/0x70 [ 5574.229431] ? ret_from_fork+0x1f/0x30 [ 5576.054830] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5576.057985] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5576.062963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5576.063061] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5576.063069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5576.063072] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.063086] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5576.063091] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5576.063096] RAX: 0000000000004785 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5576.063099] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5576.063101] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5576.063108] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5576.063111] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5576.063115] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5576.063118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5576.063120] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5576.063129] Call Trace: [ 5576.063134] [ 5576.063140] ? show_regs.cold.9+0x22/0x2f [ 5576.063154] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5576.063161] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5576.063165] ? watchdog+0x40/0x40 [ 5576.063170] ? hrtimer_interrupt+0x128/0x2c0 [ 5576.063175] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5576.063182] ? apic_timer_interrupt+0xf/0x20 [ 5576.063192] [ 5576.063195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.063202] do_raw_spin_lock+0x177/0x190 [ 5576.063207] _raw_spin_lock+0x21/0x30 [ 5576.063212] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5576.063618] ? do_raw_spin_unlock+0x75/0x190 [ 5576.063625] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5576.070651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5576.070723] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5576.070729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5576.070736] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5576.071007] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.071017] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5576.071021] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5576.071026] RAX: 000000000000780d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5576.071028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5576.107520] ? do_wait_intr+0xf0/0xf0 [ 5576.107542] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5576.108148] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5576.108381] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5576.109828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5576.109954] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5576.110002] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5576.112267] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5576.112321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5576.112543] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5576.112828] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5576.112859] lmv_intent_open+0x338/0x1150 [lmv] [ 5576.112897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5576.112993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5576.113099] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5576.113124] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5576.113214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5576.113296] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5576.113395] ? do_raw_spin_unlock+0x75/0x190 [ 5576.113403] ? _raw_spin_unlock+0x12/0x30 [ 5576.113410] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5576.132132] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5576.132142] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5576.132145] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5576.132148] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5576.132151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5576.132153] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5576.132162] Call Trace: [ 5576.132168] [ 5576.132174] ? show_regs.cold.9+0x22/0x2f [ 5576.132187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5576.132194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5576.132199] ? watchdog+0x40/0x40 [ 5576.132203] ? hrtimer_interrupt+0x128/0x2c0 [ 5576.132207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5576.132216] ? apic_timer_interrupt+0xf/0x20 [ 5576.132221] [ 5576.132224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.132230] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5576.132238] do_raw_spin_lock+0x177/0x190 [ 5576.132243] _raw_spin_lock+0x21/0x30 [ 5576.132249] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5576.132722] process_one_work+0x2c8/0x700 [ 5576.132730] worker_thread+0x296/0x6e0 [ 5576.132734] ? rescuer_thread+0x570/0x570 [ 5576.132738] kthread+0x1d1/0x200 [ 5576.132743] ? set_kthread_struct+0x70/0x70 [ 5576.132746] ret_from_fork+0x1f/0x30 [ 5576.496353] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5576.498069] ll_atomic_open+0x36a/0xd70 [lustre] [ 5576.499912] lookup_open+0x356/0xb40 [ 5576.501370] do_last+0x313/0xfc0 [ 5576.502640] ? nd_jump_root+0xe5/0x160 [ 5576.503879] ? path_init+0x437/0x520 [ 5576.505357] path_openat+0xf7/0x500 [ 5576.506657] ? do_raw_spin_unlock+0x75/0x190 [ 5576.508138] do_filp_open+0x99/0x140 [ 5576.509548] ? getname_flags+0x6e/0x330 [ 5576.510959] ? __check_object_size+0xff/0x256 [ 5576.512943] ? do_raw_spin_unlock+0x75/0x190 [ 5576.524603] ? _raw_spin_unlock+0x12/0x30 [ 5576.528437] do_sys_openat2+0x2b4/0x410 [ 5576.531581] do_sys_open+0x73/0xa0 [ 5576.534710] __x64_sys_openat+0x24/0x30 [ 5576.544493] do_syscall_64+0xc1/0x440 [ 5576.547804] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5576.550719] RIP: 0033:0x7fa749552332 [ 5576.552384] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5576.560551] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5576.563874] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5576.568133] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5576.573800] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5576.583895] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5576.589038] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5580.256035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3663s! [ 5580.262633] Showing busy workqueues and worker pools: [ 5580.265593] workqueue events: flags=0x0 [ 5580.275508] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5580.281197] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5580.294792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5580.298366] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5580.302475] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.305208] pending: kfree_rcu_monitor [ 5580.306707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5580.310036] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5580.314807] workqueue events_long: flags=0x0 [ 5580.317000] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.320162] pending: lu_objects_flush [obdclass] [ 5580.323298] workqueue events_unbound: flags=0x2 [ 5580.325539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5580.329004] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5580.333655] workqueue events_power_efficient: flags=0x80 [ 5580.336788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.339743] pending: check_lifetime [ 5580.341561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.346956] pending: neigh_periodic_work [ 5580.348530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.351660] pending: neigh_periodic_work [ 5580.353643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.358079] pending: do_cache_clean [ 5580.360078] workqueue mm_percpu_wq: flags=0x8 [ 5580.361919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.366040] pending: vmstat_update [ 5580.367794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.370729] pending: vmstat_update [ 5580.372839] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5580.376324] pending: vmstat_update [ 5580.378053] workqueue writeback: flags=0x4a [ 5580.380110] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5580.383014] pending: wb_workfn, wb_workfn [ 5580.385079] workqueue xprtiod: flags=0xa [ 5580.387307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5580.390894] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5580.394731] workqueue ipv6_addrconf: flags=0x40008 [ 5580.396934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5580.399696] pending: addrconf_verify_work [ 5580.401792] workqueue knbd0-recv: flags=0x1a [ 5580.404633] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5580.407831] in-flight: 651:recv_work [ 5580.410034] workqueue ptlrpc_pinger: flags=0x2 [ 5580.412171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5580.414924] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5580.417483] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3663s workers=4 idle: 37 31846 28897 [ 5580.420494] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5570s workers=2 idle: 653 [ 5588.059767] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5588.067227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5588.133555] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5588.151753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5588.165085] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5588.168994] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5588.200486] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5588.217449] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5588.226399] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5588.233368] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5588.238392] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5588.242959] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5588.248719] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5588.257080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5588.263293] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5588.275001] Call Trace: [ 5588.279154] [ 5588.280779] ? show_regs.cold.9+0x22/0x2f [ 5588.286004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5588.288886] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5588.291276] ? watchdog+0x40/0x40 [ 5588.293170] ? hrtimer_interrupt+0x128/0x2c0 [ 5588.296363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5588.301105] ? apic_timer_interrupt+0xf/0x20 [ 5588.303773] [ 5588.305390] ? native_safe_halt+0x2e/0x50 [ 5588.309505] kvm_wait+0xcd/0xe0 [ 5588.312586] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5588.319542] do_raw_spin_lock+0x177/0x190 [ 5588.321405] _raw_spin_lock+0x21/0x30 [ 5588.322980] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5588.334143] ? do_raw_spin_unlock+0x75/0x190 [ 5588.335709] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5588.342977] ? do_wait_intr+0xf0/0xf0 [ 5588.348477] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5588.352956] mdc_close+0x53b/0xd70 [mdc] [ 5588.356460] lmv_close+0x18d/0x490 [lmv] [ 5588.364969] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5588.373333] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5588.377673] ll_md_close+0x274/0x800 [lustre] [ 5588.379943] ll_file_release+0x128/0xc50 [lustre] [ 5588.393415] __fput+0x10d/0x440 [ 5588.394541] ____fput+0x12/0x20 [ 5588.399853] task_work_run+0xc6/0x110 [ 5588.404102] exit_to_usermode_loop+0x1dd/0x1f0 [ 5588.412156] do_syscall_64+0x3d6/0x440 [ 5588.418586] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5588.427718] RIP: 0033:0x7fa749552cd8 [ 5588.431381] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5588.458932] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5588.467991] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5588.474099] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5588.491789] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5588.498498] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5588.508377] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5600.060789] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5600.071869] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5600.113931] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5600.124310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5600.136767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5600.138840] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5600.161473] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5600.169802] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5600.176246] RDX: 0000000000002a32 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5600.180759] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5600.190343] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5600.195484] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5600.198451] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5600.208370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5600.215572] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5600.220367] Call Trace: [ 5600.222174] [ 5600.224372] ? show_regs.cold.9+0x22/0x2f [ 5600.227180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5600.229895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5600.238792] ? watchdog+0x40/0x40 [ 5600.241142] ? hrtimer_interrupt+0x128/0x2c0 [ 5600.242635] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5600.253357] ? apic_timer_interrupt+0xf/0x20 [ 5600.255894] [ 5600.257485] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5600.267631] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5600.269575] do_raw_spin_lock+0x177/0x190 [ 5600.275316] _raw_spin_lock+0x21/0x30 [ 5600.278242] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5600.280438] obd_mod_free+0x46/0x140 [ptlrpc] [ 5600.291084] mdc_commit_open+0x91/0xb0 [mdc] [ 5600.292658] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5600.302097] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5600.308310] after_reply+0xe2c/0x1780 [ptlrpc] [ 5600.312375] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5600.320109] ? __next_timer_interrupt+0x160/0x160 [ 5600.321835] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5600.326453] ? do_wait_intr+0xf0/0xf0 [ 5600.329581] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5600.335307] kthread+0x1d1/0x200 [ 5600.336846] ? set_kthread_struct+0x70/0x70 [ 5600.338812] ret_from_fork+0x1f/0x30 [ 5604.058178] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5604.065093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5604.091487] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5604.095848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5604.099489] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5604.102772] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5604.105136] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5604.113044] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5604.116141] RAX: 0000000000004108 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5604.119011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5604.121928] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5604.124849] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5604.127870] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5604.130824] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5604.134171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5604.136451] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5604.139699] Call Trace: [ 5604.140626] [ 5604.142829] ? show_regs.cold.9+0x22/0x2f [ 5604.146092] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5604.147347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5604.148794] ? watchdog+0x40/0x40 [ 5604.150817] ? hrtimer_interrupt+0x128/0x2c0 [ 5604.152882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5604.155162] ? apic_timer_interrupt+0xf/0x20 [ 5604.157003] [ 5604.158184] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5604.160475] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5604.163274] do_raw_spin_lock+0x177/0x190 [ 5604.164757] _raw_spin_lock+0x21/0x30 [ 5604.166358] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5604.168750] process_one_work+0x2c8/0x700 [ 5604.170443] worker_thread+0x296/0x6e0 [ 5604.172391] ? rescuer_thread+0x570/0x570 [ 5604.174148] kthread+0x1d1/0x200 [ 5604.175408] ? set_kthread_struct+0x70/0x70 [ 5604.177257] ret_from_fork+0x1f/0x30 [ 5608.054977] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5608.062548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5608.084813] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5608.090014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5608.094095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5608.096442] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5608.105704] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5608.109762] RAX: 0000000000002ee3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5608.113014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5608.115837] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5608.119209] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5608.122417] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5608.125290] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5608.129140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5608.131354] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5608.134674] Call Trace: [ 5608.135818] [ 5608.136830] ? show_regs.cold.9+0x22/0x2f [ 5608.138708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5608.141024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5608.143206] ? watchdog+0x40/0x40 [ 5608.144613] ? hrtimer_interrupt+0x128/0x2c0 [ 5608.146435] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5608.148423] ? apic_timer_interrupt+0xf/0x20 [ 5608.150410] [ 5608.151271] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5608.153109] do_raw_spin_lock+0x177/0x190 [ 5608.154322] _raw_spin_lock+0x21/0x30 [ 5608.155649] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5608.157731] ? do_raw_spin_unlock+0x75/0x190 [ 5608.159045] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5608.161094] ? do_wait_intr+0xf0/0xf0 [ 5608.162380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5608.164450] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5608.166437] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5608.168886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5608.170963] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5608.173039] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5608.174797] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5608.176322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5608.178260] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5608.180596] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5608.182359] lmv_intent_open+0x338/0x1150 [lmv] [ 5608.183949] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5608.185956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5608.187686] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5608.189332] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5608.191226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5608.193657] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5608.195824] ? do_raw_spin_unlock+0x75/0x190 [ 5608.197665] ? _raw_spin_unlock+0x12/0x30 [ 5608.199356] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5608.201446] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5608.203808] ll_atomic_open+0x36a/0xd70 [lustre] [ 5608.205910] lookup_open+0x356/0xb40 [ 5608.207296] do_last+0x313/0xfc0 [ 5608.208791] ? nd_jump_root+0xe5/0x160 [ 5608.210321] ? path_init+0x437/0x520 [ 5608.211911] path_openat+0xf7/0x500 [ 5608.213326] ? do_raw_spin_unlock+0x75/0x190 [ 5608.214998] do_filp_open+0x99/0x140 [ 5608.216604] ? getname_flags+0x6e/0x330 [ 5608.218196] ? __check_object_size+0xff/0x256 [ 5608.220094] ? do_raw_spin_unlock+0x75/0x190 [ 5608.221960] ? _raw_spin_unlock+0x12/0x30 [ 5608.223668] do_sys_openat2+0x2b4/0x410 [ 5608.225308] do_sys_open+0x73/0xa0 [ 5608.226798] __x64_sys_openat+0x24/0x30 [ 5608.228223] do_syscall_64+0xc1/0x440 [ 5608.229812] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5608.231990] RIP: 0033:0x7fa749552332 [ 5608.233674] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5608.241263] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5608.244404] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5608.247589] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5608.250591] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5608.254728] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5608.263922] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5610.467052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3693s! [ 5610.471137] Showing busy workqueues and worker pools: [ 5610.474056] workqueue events: flags=0x0 [ 5610.476156] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5610.479637] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5610.487023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5610.490832] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5610.495572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.499366] pending: kfree_rcu_monitor [ 5610.501348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5610.504885] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5610.509355] workqueue events_long: flags=0x0 [ 5610.510766] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.513518] pending: lu_objects_flush [obdclass] [ 5610.516202] workqueue events_unbound: flags=0x2 [ 5610.518586] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5610.521364] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5610.529151] workqueue events_power_efficient: flags=0x80 [ 5610.534024] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.540067] pending: check_lifetime [ 5610.542517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.546047] pending: neigh_periodic_work [ 5610.548959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.553114] pending: neigh_periodic_work [ 5610.555923] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.559531] pending: do_cache_clean [ 5610.561865] workqueue mm_percpu_wq: flags=0x8 [ 5610.564161] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.567737] pending: vmstat_update [ 5610.570157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.574977] pending: vmstat_update [ 5610.580065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5610.587691] pending: vmstat_update [ 5610.590133] workqueue writeback: flags=0x4a [ 5610.596314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5610.604066] pending: wb_workfn, wb_workfn [ 5610.607080] workqueue xprtiod: flags=0xa [ 5610.612626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5610.618640] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5610.628198] workqueue ipv6_addrconf: flags=0x40008 [ 5610.630216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5610.641211] pending: addrconf_verify_work [ 5610.654520] workqueue knbd0-recv: flags=0x1a [ 5610.664160] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5610.671925] in-flight: 651:recv_work [ 5610.674107] workqueue ptlrpc_pinger: flags=0x2 [ 5610.681056] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5610.688375] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5610.694945] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3693s workers=4 idle: 37 31846 28897 [ 5610.702862] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5600s workers=2 idle: 653 [ 5616.060581] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5616.072285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5616.109774] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5616.118067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5616.134248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5616.143579] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5616.160676] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5616.177831] RAX: 0000000000007f83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5616.180815] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5616.184049] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5616.190754] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5616.194214] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5616.198949] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5616.202980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5616.206509] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5616.210440] Call Trace: [ 5616.211867] [ 5616.215776] ? show_regs.cold.9+0x22/0x2f [ 5616.217338] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5616.221863] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5616.226004] ? watchdog+0x40/0x40 [ 5616.232052] ? hrtimer_interrupt+0x128/0x2c0 [ 5616.235991] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5616.248145] ? apic_timer_interrupt+0xf/0x20 [ 5616.249641] [ 5616.250371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5616.268480] do_raw_spin_lock+0x177/0x190 [ 5616.277418] _raw_spin_lock+0x21/0x30 [ 5616.278531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5616.280412] ? do_raw_spin_unlock+0x75/0x190 [ 5616.287916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5616.297375] ? do_wait_intr+0xf0/0xf0 [ 5616.302809] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5616.315049] mdc_close+0x53b/0xd70 [mdc] [ 5616.319946] lmv_close+0x18d/0x490 [lmv] [ 5616.325155] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5616.328377] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5616.330121] ll_md_close+0x274/0x800 [lustre] [ 5616.343292] ll_file_release+0x128/0xc50 [lustre] [ 5616.351782] __fput+0x10d/0x440 [ 5616.355516] ____fput+0x12/0x20 [ 5616.360714] task_work_run+0xc6/0x110 [ 5616.362060] exit_to_usermode_loop+0x1dd/0x1f0 [ 5616.368677] do_syscall_64+0x3d6/0x440 [ 5616.370223] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5616.375445] RIP: 0033:0x7fa749552cd8 [ 5616.376633] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5616.388286] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5616.394184] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5616.401710] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5616.410671] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5616.417486] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5616.425157] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5628.058035] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5628.061833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5628.087578] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5628.093273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5628.098942] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5628.100903] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5628.115028] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5628.119176] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5628.127000] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5628.138427] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5628.143472] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5628.146202] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5628.149093] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5628.152896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5628.155454] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5628.159711] Call Trace: [ 5628.160807] [ 5628.161640] ? show_regs.cold.9+0x22/0x2f [ 5628.163870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5628.166824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5628.169039] ? watchdog+0x40/0x40 [ 5628.171275] ? hrtimer_interrupt+0x128/0x2c0 [ 5628.174088] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5628.176799] ? apic_timer_interrupt+0xf/0x20 [ 5628.178623] [ 5628.180133] ? native_safe_halt+0x2e/0x50 [ 5628.181782] kvm_wait+0xcd/0xe0 [ 5628.184048] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5628.188981] do_raw_spin_lock+0x177/0x190 [ 5628.192731] _raw_spin_lock+0x21/0x30 [ 5628.196907] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5628.198725] obd_mod_free+0x46/0x140 [ptlrpc] [ 5628.200712] mdc_commit_open+0x91/0xb0 [mdc] [ 5628.202020] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5628.209261] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5628.213326] after_reply+0xe2c/0x1780 [ptlrpc] [ 5628.215372] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5628.222819] ? __next_timer_interrupt+0x160/0x160 [ 5628.226516] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5628.228299] ? do_wait_intr+0xf0/0xf0 [ 5628.229791] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5628.232292] kthread+0x1d1/0x200 [ 5628.233241] ? set_kthread_struct+0x70/0x70 [ 5628.240245] ret_from_fork+0x1f/0x30 [ 5632.058703] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5632.069772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5632.096398] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5632.099689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5632.102484] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5632.105513] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.107710] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5632.114414] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5632.116984] RAX: 0000000000006f8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5632.120514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5632.125146] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5632.128729] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5632.138755] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5632.145126] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5632.150647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5632.153654] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5632.156992] Call Trace: [ 5632.158386] [ 5632.159246] ? show_regs.cold.9+0x22/0x2f [ 5632.161615] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5632.163914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5632.166152] ? watchdog+0x40/0x40 [ 5632.167904] ? hrtimer_interrupt+0x128/0x2c0 [ 5632.169988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5632.172155] ? apic_timer_interrupt+0xf/0x20 [ 5632.175158] [ 5632.176109] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.179615] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5632.182934] do_raw_spin_lock+0x177/0x190 [ 5632.185306] _raw_spin_lock+0x21/0x30 [ 5632.187194] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5632.189925] process_one_work+0x2c8/0x700 [ 5632.191769] worker_thread+0x296/0x6e0 [ 5632.193788] ? rescuer_thread+0x570/0x570 [ 5632.195640] kthread+0x1d1/0x200 [ 5632.197273] ? set_kthread_struct+0x70/0x70 [ 5632.199728] ret_from_fork+0x1f/0x30 [ 5636.054370] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5636.058416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5636.075689] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5636.079643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5636.083064] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5636.084711] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5636.090938] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5636.093481] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5636.095826] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5636.098212] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5636.100638] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5636.103186] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5636.105569] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5636.108318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5636.110363] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5636.112804] Call Trace: [ 5636.113700] [ 5636.114404] ? show_regs.cold.9+0x22/0x2f [ 5636.115847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5636.117769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5636.119499] ? watchdog+0x40/0x40 [ 5636.120724] ? hrtimer_interrupt+0x128/0x2c0 [ 5636.122205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5636.123791] ? apic_timer_interrupt+0xf/0x20 [ 5636.125359] [ 5636.126338] ? native_safe_halt+0x2e/0x50 [ 5636.127762] kvm_wait+0xcd/0xe0 [ 5636.128731] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5636.130472] do_raw_spin_lock+0x177/0x190 [ 5636.131745] _raw_spin_lock+0x21/0x30 [ 5636.132820] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5636.134621] ? do_raw_spin_unlock+0x75/0x190 [ 5636.136132] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5636.137875] ? do_wait_intr+0xf0/0xf0 [ 5636.139143] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5636.141067] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5636.143327] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5636.145622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5636.148104] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5636.150310] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5636.152293] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5636.153775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5636.155624] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5636.157645] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5636.159717] lmv_intent_open+0x338/0x1150 [lmv] [ 5636.161481] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5636.163798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5636.165913] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5636.168129] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5636.170124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5636.172086] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5636.174363] ? do_raw_spin_unlock+0x75/0x190 [ 5636.176041] ? _raw_spin_unlock+0x12/0x30 [ 5636.177316] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5636.179256] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5636.184056] ll_atomic_open+0x36a/0xd70 [lustre] [ 5636.185828] lookup_open+0x356/0xb40 [ 5636.186743] do_last+0x313/0xfc0 [ 5636.188214] ? nd_jump_root+0xe5/0x160 [ 5636.189331] ? path_init+0x437/0x520 [ 5636.190665] path_openat+0xf7/0x500 [ 5636.191843] ? do_raw_spin_unlock+0x75/0x190 [ 5636.193426] do_filp_open+0x99/0x140 [ 5636.195224] ? getname_flags+0x6e/0x330 [ 5636.199995] ? __check_object_size+0xff/0x256 [ 5636.201212] ? do_raw_spin_unlock+0x75/0x190 [ 5636.203205] ? _raw_spin_unlock+0x12/0x30 [ 5636.204719] do_sys_openat2+0x2b4/0x410 [ 5636.206028] do_sys_open+0x73/0xa0 [ 5636.206953] __x64_sys_openat+0x24/0x30 [ 5636.208124] do_syscall_64+0xc1/0x440 [ 5636.209244] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5636.213911] RIP: 0033:0x7fa749552332 [ 5636.215100] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5636.223369] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5636.229016] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5636.231929] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5636.234966] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5636.237345] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5636.239445] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5641.185902] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3724s! [ 5641.204802] Showing busy workqueues and worker pools: [ 5641.211328] workqueue events: flags=0x0 [ 5641.215520] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5641.224260] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5641.235542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5641.244214] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5641.250655] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.253398] pending: kfree_rcu_monitor [ 5641.255238] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5641.258223] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5641.261837] workqueue events_long: flags=0x0 [ 5641.264174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.266940] pending: lu_objects_flush [obdclass] [ 5641.272599] workqueue events_unbound: flags=0x2 [ 5641.275216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5641.279629] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5641.289515] workqueue events_power_efficient: flags=0x80 [ 5641.291754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.294664] pending: check_lifetime [ 5641.300220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.303450] pending: neigh_periodic_work [ 5641.309401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.315234] pending: neigh_periodic_work [ 5641.317565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.321475] pending: do_cache_clean [ 5641.323883] workqueue mm_percpu_wq: flags=0x8 [ 5641.326397] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.330313] pending: vmstat_update [ 5641.332236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.337975] pending: vmstat_update [ 5641.343291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5641.350782] pending: vmstat_update [ 5641.352882] workqueue writeback: flags=0x4a [ 5641.355837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5641.359589] pending: wb_workfn, wb_workfn [ 5641.368836] workqueue xprtiod: flags=0xa [ 5641.374402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5641.380427] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5641.389478] workqueue ipv6_addrconf: flags=0x40008 [ 5641.390855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5641.397549] pending: addrconf_verify_work [ 5641.399549] workqueue knbd0-recv: flags=0x1a [ 5641.404126] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5641.409605] in-flight: 651:recv_work [ 5641.412261] workqueue ptlrpc_pinger: flags=0x2 [ 5641.418688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5641.427880] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5641.433246] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3724s workers=4 idle: 37 31846 28897 [ 5641.441339] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5631s workers=2 idle: 653 [ 5644.059541] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5644.067788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5644.098308] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5644.103090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5644.108020] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5644.110276] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5644.118907] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5644.123753] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5644.127919] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5644.132337] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5644.136132] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5644.140263] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5644.144146] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5644.146737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5644.150404] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5644.153391] Call Trace: [ 5644.155149] [ 5644.155955] ? show_regs.cold.9+0x22/0x2f [ 5644.157384] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5644.160691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5644.162615] ? watchdog+0x40/0x40 [ 5644.165163] ? hrtimer_interrupt+0x128/0x2c0 [ 5644.166763] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5644.171079] ? apic_timer_interrupt+0xf/0x20 [ 5644.175294] [ 5644.176379] ? native_safe_halt+0x2e/0x50 [ 5644.178380] kvm_wait+0xcd/0xe0 [ 5644.179830] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5644.183081] do_raw_spin_lock+0x177/0x190 [ 5644.184486] _raw_spin_lock+0x21/0x30 [ 5644.187127] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5644.190975] ? do_raw_spin_unlock+0x75/0x190 [ 5644.193770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5644.196593] ? do_wait_intr+0xf0/0xf0 [ 5644.198084] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5644.200770] mdc_close+0x53b/0xd70 [mdc] [ 5644.202142] lmv_close+0x18d/0x490 [lmv] [ 5644.203504] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5644.205908] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5644.207594] ll_md_close+0x274/0x800 [lustre] [ 5644.211355] ll_file_release+0x128/0xc50 [lustre] [ 5644.214151] __fput+0x10d/0x440 [ 5644.215576] ____fput+0x12/0x20 [ 5644.217197] task_work_run+0xc6/0x110 [ 5644.218708] exit_to_usermode_loop+0x1dd/0x1f0 [ 5644.220448] do_syscall_64+0x3d6/0x440 [ 5644.221713] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5644.225145] RIP: 0033:0x7fa749552cd8 [ 5644.227620] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5644.241017] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5644.245323] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5644.248757] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5644.252376] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5644.255642] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5644.259149] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5656.063324] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5656.066429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5656.091263] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5656.097134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5656.101867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5656.105836] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5656.115052] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5656.118605] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5656.122047] RDX: 000000000000561c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5656.125565] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5656.129360] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5656.132835] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5656.136786] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5656.141088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5656.143967] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5656.147421] Call Trace: [ 5656.149119] [ 5656.150284] ? show_regs.cold.9+0x22/0x2f [ 5656.153369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5656.157437] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5656.159855] ? watchdog+0x40/0x40 [ 5656.161283] ? hrtimer_interrupt+0x128/0x2c0 [ 5656.163430] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5656.165863] ? apic_timer_interrupt+0xf/0x20 [ 5656.168187] [ 5656.169292] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5656.172252] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5656.174986] do_raw_spin_lock+0x177/0x190 [ 5656.176980] _raw_spin_lock+0x21/0x30 [ 5656.179058] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5656.181479] obd_mod_free+0x46/0x140 [ptlrpc] [ 5656.184152] mdc_commit_open+0x91/0xb0 [mdc] [ 5656.186121] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5656.188688] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5656.191732] after_reply+0xe2c/0x1780 [ptlrpc] [ 5656.194728] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5656.197785] ? __next_timer_interrupt+0x160/0x160 [ 5656.200053] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5656.203277] ? do_wait_intr+0xf0/0xf0 [ 5656.205079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5656.208330] kthread+0x1d1/0x200 [ 5656.210046] ? set_kthread_struct+0x70/0x70 [ 5656.212069] ret_from_fork+0x1f/0x30 [ 5660.057578] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5660.063237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5660.123709] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5660.136461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5660.147839] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5660.166748] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5660.185820] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5660.215431] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5660.217796] RAX: 0000000000001612 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5660.242002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5660.245294] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5660.250597] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5660.259349] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5660.268766] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5660.279662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5660.284879] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5660.302512] Call Trace: [ 5660.303440] [ 5660.304243] ? show_regs.cold.9+0x22/0x2f [ 5660.314981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5660.316643] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5660.321960] ? watchdog+0x40/0x40 [ 5660.323092] ? hrtimer_interrupt+0x128/0x2c0 [ 5660.329528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5660.334337] ? apic_timer_interrupt+0xf/0x20 [ 5660.338962] [ 5660.343487] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5660.355248] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5660.363429] do_raw_spin_lock+0x177/0x190 [ 5660.370277] _raw_spin_lock+0x21/0x30 [ 5660.377525] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5660.381083] process_one_work+0x2c8/0x700 [ 5660.387446] worker_thread+0x296/0x6e0 [ 5660.393414] ? rescuer_thread+0x570/0x570 [ 5660.401407] kthread+0x1d1/0x200 [ 5660.405971] ? set_kthread_struct+0x70/0x70 [ 5660.414676] ret_from_fork+0x1f/0x30 [ 5668.054030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5668.058030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5668.101789] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5668.109321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5668.116169] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5668.119238] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5668.134808] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5668.137373] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5668.140874] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5668.144673] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5668.148065] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5668.151667] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5668.154624] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5668.157393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5668.159553] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5668.165227] Call Trace: [ 5668.166867] [ 5668.168073] ? show_regs.cold.9+0x22/0x2f [ 5668.170376] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5668.173206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5668.174504] ? watchdog+0x40/0x40 [ 5668.175372] ? hrtimer_interrupt+0x128/0x2c0 [ 5668.177074] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5668.179518] ? apic_timer_interrupt+0xf/0x20 [ 5668.181484] [ 5668.182218] ? native_safe_halt+0x2e/0x50 [ 5668.184983] kvm_wait+0xcd/0xe0 [ 5668.186242] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5668.190554] do_raw_spin_lock+0x177/0x190 [ 5668.192039] _raw_spin_lock+0x21/0x30 [ 5668.193065] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5668.194667] ? do_raw_spin_unlock+0x75/0x190 [ 5668.199652] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5668.203624] ? do_wait_intr+0xf0/0xf0 [ 5668.206295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5668.212361] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5668.214904] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5668.217494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5668.220328] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5668.224099] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5668.226142] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5668.228800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5668.230562] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5668.233728] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5668.236127] lmv_intent_open+0x338/0x1150 [lmv] [ 5668.238242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5668.243300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5668.245338] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5668.247187] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5668.250431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5668.255601] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5668.258411] ? do_raw_spin_unlock+0x75/0x190 [ 5668.261445] ? _raw_spin_unlock+0x12/0x30 [ 5668.262673] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5668.264289] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5668.267300] ll_atomic_open+0x36a/0xd70 [lustre] [ 5668.273293] lookup_open+0x356/0xb40 [ 5668.277818] do_last+0x313/0xfc0 [ 5668.283492] ? nd_jump_root+0xe5/0x160 [ 5668.286929] ? path_init+0x437/0x520 [ 5668.291244] path_openat+0xf7/0x500 [ 5668.292101] ? do_raw_spin_unlock+0x75/0x190 [ 5668.294017] do_filp_open+0x99/0x140 [ 5668.295340] ? getname_flags+0x6e/0x330 [ 5668.298656] ? __check_object_size+0xff/0x256 [ 5668.302212] ? do_raw_spin_unlock+0x75/0x190 [ 5668.311312] ? _raw_spin_unlock+0x12/0x30 [ 5668.314449] do_sys_openat2+0x2b4/0x410 [ 5668.316091] do_sys_open+0x73/0xa0 [ 5668.317652] __x64_sys_openat+0x24/0x30 [ 5668.319554] do_syscall_64+0xc1/0x440 [ 5668.324855] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5668.330493] RIP: 0033:0x7fa749552332 [ 5668.334983] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5668.352032] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5668.355242] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5668.360691] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5668.364869] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5668.367907] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5668.371670] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5671.917307] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3754s! [ 5671.927962] Showing busy workqueues and worker pools: [ 5671.933498] workqueue events: flags=0x0 [ 5671.939858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5671.947999] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5671.960496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5671.965398] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5671.971285] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5671.974952] pending: kfree_rcu_monitor [ 5671.977555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5671.981292] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5671.986108] workqueue events_long: flags=0x0 [ 5671.987656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5671.993110] pending: lu_objects_flush [obdclass] [ 5671.995901] workqueue events_unbound: flags=0x2 [ 5671.998607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5672.004599] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5672.014028] workqueue events_power_efficient: flags=0x80 [ 5672.021257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5672.024785] pending: check_lifetime [ 5672.026941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5672.036156] pending: neigh_periodic_work [ 5672.042045] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5672.048556] pending: neigh_periodic_work [ 5672.050713] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5672.057752] pending: do_cache_clean [ 5672.062276] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5672.062442] workqueue mm_percpu_wq: flags=0x8 [ 5672.062448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5672.062460] pending: vmstat_update [ 5672.062471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5672.062478] pending: vmstat_update [ 5672.062483] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5672.062490] pending: vmstat_update [ 5672.062497] workqueue writeback: flags=0x4a [ 5672.062499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5672.062506] pending: wb_workfn, wb_workfn [ 5672.062569] workqueue xprtiod: flags=0xa [ 5672.062573] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5672.062579] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5672.062599] workqueue ipv6_addrconf: flags=0x40008 [ 5672.062602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5672.062608] pending: addrconf_verify_work [ 5672.062648] workqueue knbd0-recv: flags=0x1a [ 5672.062652] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5672.062659] in-flight: 651:recv_work [ 5672.062669] workqueue ptlrpc_pinger: flags=0x2 [ 5672.062671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5672.062676] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5672.063125] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3754s workers=4 idle: 37 31846 28897 [ 5672.063140] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5661s workers=2 idle: 653 [ 5672.179534] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5672.219678] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5672.228924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5672.232930] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5672.236307] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5672.244468] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5672.247223] RAX: 0000000000007a87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5672.252574] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5672.257409] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5672.261902] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5672.267910] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5672.272679] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5672.279281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5672.286049] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5672.292280] Call Trace: [ 5672.293729] [ 5672.295241] ? show_regs.cold.9+0x22/0x2f [ 5672.297601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5672.300194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5672.305165] ? watchdog+0x40/0x40 [ 5672.309047] ? hrtimer_interrupt+0x128/0x2c0 [ 5672.314111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5672.317097] ? apic_timer_interrupt+0xf/0x20 [ 5672.320888] [ 5672.323309] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5672.327273] do_raw_spin_lock+0x177/0x190 [ 5672.329705] _raw_spin_lock+0x21/0x30 [ 5672.331789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5672.334484] ? do_raw_spin_unlock+0x75/0x190 [ 5672.336320] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5672.339476] ? do_wait_intr+0xf0/0xf0 [ 5672.340817] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5672.343307] mdc_close+0x53b/0xd70 [mdc] [ 5672.345349] lmv_close+0x18d/0x490 [lmv] [ 5672.346835] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5672.350050] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5672.351851] ll_md_close+0x274/0x800 [lustre] [ 5672.353300] ll_file_release+0x128/0xc50 [lustre] [ 5672.355806] __fput+0x10d/0x440 [ 5672.356878] ____fput+0x12/0x20 [ 5672.360198] task_work_run+0xc6/0x110 [ 5672.362649] exit_to_usermode_loop+0x1dd/0x1f0 [ 5672.368052] do_syscall_64+0x3d6/0x440 [ 5672.369803] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5672.374424] RIP: 0033:0x7fa749552cd8 [ 5672.375710] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5672.388214] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5672.393101] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5672.395559] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5672.401209] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5672.406781] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5672.409518] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5684.060284] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5684.066116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.085120] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.092177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5684.096359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5684.099367] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5684.110847] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5684.118959] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5684.123226] RDX: 0000000000004e4b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5684.126581] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5684.132090] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5684.139288] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5684.144446] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5684.149501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.156578] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5684.160175] Call Trace: [ 5684.161070] [ 5684.162357] ? show_regs.cold.9+0x22/0x2f [ 5684.164457] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.166883] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.168867] ? watchdog+0x40/0x40 [ 5684.175434] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.181755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.183902] ? apic_timer_interrupt+0xf/0x20 [ 5684.185635] [ 5684.186431] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5684.189288] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5684.191776] do_raw_spin_lock+0x177/0x190 [ 5684.193936] _raw_spin_lock+0x21/0x30 [ 5684.199856] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5684.203811] obd_mod_free+0x46/0x140 [ptlrpc] [ 5684.206629] mdc_commit_open+0x91/0xb0 [mdc] [ 5684.208190] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5684.218210] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5684.223463] after_reply+0xe2c/0x1780 [ptlrpc] [ 5684.226437] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5684.229921] ? __next_timer_interrupt+0x160/0x160 [ 5684.232377] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5684.239059] ? do_wait_intr+0xf0/0xf0 [ 5684.240862] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5684.246972] kthread+0x1d1/0x200 [ 5684.253320] ? set_kthread_struct+0x70/0x70 [ 5684.257010] ret_from_fork+0x1f/0x30 [ 5688.065929] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5688.076646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5688.129034] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5688.144075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5688.154401] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5688.166441] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5688.168890] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5688.179580] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5688.186999] RAX: 000000000000738d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5688.192581] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5688.196525] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5688.201648] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5688.211489] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5688.217735] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5688.221914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5688.230255] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5688.236661] Call Trace: [ 5688.237782] [ 5688.238880] ? show_regs.cold.9+0x22/0x2f [ 5688.241124] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5688.244933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5688.246343] ? watchdog+0x40/0x40 [ 5688.249464] ? hrtimer_interrupt+0x128/0x2c0 [ 5688.251526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5688.258154] ? apic_timer_interrupt+0xf/0x20 [ 5688.262474] [ 5688.266041] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5688.275979] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5688.279661] do_raw_spin_lock+0x177/0x190 [ 5688.281864] _raw_spin_lock+0x21/0x30 [ 5688.283710] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5688.298033] process_one_work+0x2c8/0x700 [ 5688.299420] worker_thread+0x296/0x6e0 [ 5688.302662] ? rescuer_thread+0x570/0x570 [ 5688.305240] kthread+0x1d1/0x200 [ 5688.313541] ? set_kthread_struct+0x70/0x70 [ 5688.314999] ret_from_fork+0x1f/0x30 [ 5700.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5700.056919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5700.059036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5700.078472] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5700.078487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5700.078489] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5700.078502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5700.078507] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5700.078512] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5700.078514] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5700.078517] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5700.078518] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5700.078521] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5700.078524] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5700.078526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5700.078529] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5700.078537] Call Trace: [ 5700.078543] [ 5700.078548] ? show_regs.cold.9+0x22/0x2f [ 5700.078558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5700.078564] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5700.078568] ? watchdog+0x40/0x40 [ 5700.078572] ? hrtimer_interrupt+0x128/0x2c0 [ 5700.078575] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5700.078581] ? apic_timer_interrupt+0xf/0x20 [ 5700.078586] [ 5700.078589] ? native_safe_halt+0x2e/0x50 [ 5700.078593] kvm_wait+0xcd/0xe0 [ 5700.078602] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5700.078612] do_raw_spin_lock+0x177/0x190 [ 5700.078617] _raw_spin_lock+0x21/0x30 [ 5700.078621] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5700.078947] ? do_raw_spin_unlock+0x75/0x190 [ 5700.078953] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5700.079156] ? do_wait_intr+0xf0/0xf0 [ 5700.079164] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5700.079357] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5700.079571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5700.079839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5700.079948] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5700.079985] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5700.080014] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5700.080041] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5700.080127] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5700.080350] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5700.080379] lmv_intent_open+0x338/0x1150 [lmv] [ 5700.080410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5700.080495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5700.080582] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5700.080605] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5700.080753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5700.080841] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5700.080922] ? do_raw_spin_unlock+0x75/0x190 [ 5700.080929] ? _raw_spin_unlock+0x12/0x30 [ 5700.080935] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5700.080962] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5700.080982] ll_atomic_open+0x36a/0xd70 [lustre] [ 5700.081074] lookup_open+0x356/0xb40 [ 5700.081084] do_last+0x313/0xfc0 [ 5700.081089] ? nd_jump_root+0xe5/0x160 [ 5700.081094] ? path_init+0x437/0x520 [ 5700.081099] path_openat+0xf7/0x500 [ 5700.081105] ? do_raw_spin_unlock+0x75/0x190 [ 5700.081110] do_filp_open+0x99/0x140 [ 5700.081117] ? getname_flags+0x6e/0x330 [ 5700.081122] ? __check_object_size+0xff/0x256 [ 5700.081140] ? do_raw_spin_unlock+0x75/0x190 [ 5700.081145] ? _raw_spin_unlock+0x12/0x30 [ 5700.081149] do_sys_openat2+0x2b4/0x410 [ 5700.081154] do_sys_open+0x73/0xa0 [ 5700.081158] __x64_sys_openat+0x24/0x30 [ 5700.081162] do_syscall_64+0xc1/0x440 [ 5700.081168] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5700.081175] RIP: 0033:0x7fa749552332 [ 5700.081183] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5700.081188] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5700.081192] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5700.081194] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5700.081196] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5700.081198] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5700.081201] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5700.500723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5700.533052] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5700.549925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5700.557823] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5700.560879] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5700.578785] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5700.582630] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5700.585842] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5700.588733] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5700.593589] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5700.599033] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5700.603860] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5700.610389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5700.612464] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5700.621130] Call Trace: [ 5700.623287] [ 5700.623967] ? show_regs.cold.9+0x22/0x2f [ 5700.626110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5700.629692] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5700.631991] ? watchdog+0x40/0x40 [ 5700.633465] ? hrtimer_interrupt+0x128/0x2c0 [ 5700.635949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5700.637529] ? apic_timer_interrupt+0xf/0x20 [ 5700.641388] [ 5700.642673] ? native_safe_halt+0x2e/0x50 [ 5700.644608] kvm_wait+0xcd/0xe0 [ 5700.648461] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5700.650759] do_raw_spin_lock+0x177/0x190 [ 5700.652584] _raw_spin_lock+0x21/0x30 [ 5700.658023] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5700.660682] ? do_raw_spin_unlock+0x75/0x190 [ 5700.662253] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5700.669303] ? do_wait_intr+0xf0/0xf0 [ 5700.671160] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5700.674155] mdc_close+0x53b/0xd70 [mdc] [ 5700.678698] lmv_close+0x18d/0x490 [lmv] [ 5700.682412] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5700.685944] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5700.687770] ll_md_close+0x274/0x800 [lustre] [ 5700.690768] ll_file_release+0x128/0xc50 [lustre] [ 5700.693267] __fput+0x10d/0x440 [ 5700.694465] ____fput+0x12/0x20 [ 5700.699159] task_work_run+0xc6/0x110 [ 5700.701079] exit_to_usermode_loop+0x1dd/0x1f0 [ 5700.705167] do_syscall_64+0x3d6/0x440 [ 5700.708643] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5700.714733] RIP: 0033:0x7fa749552cd8 [ 5700.719691] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5700.742277] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5700.744780] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5700.748663] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5700.751829] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5700.760308] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5700.763165] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5702.113506] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3784s! [ 5702.118440] Showing busy workqueues and worker pools: [ 5702.122470] workqueue events: flags=0x0 [ 5702.127015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5702.133290] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5702.143327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5702.150984] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5702.161767] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.167439] pending: kfree_rcu_monitor [ 5702.170635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5702.175166] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5702.178697] workqueue events_long: flags=0x0 [ 5702.181777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.188240] pending: lu_objects_flush [obdclass] [ 5702.196679] workqueue events_unbound: flags=0x2 [ 5702.201557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5702.205113] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5702.208433] workqueue events_power_efficient: flags=0x80 [ 5702.210153] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.214071] pending: check_lifetime [ 5702.215765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.219523] pending: neigh_periodic_work [ 5702.221089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.224264] pending: neigh_periodic_work [ 5702.225669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.228081] pending: do_cache_clean [ 5702.230340] workqueue mm_percpu_wq: flags=0x8 [ 5702.231477] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.234649] pending: vmstat_update [ 5702.236600] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.238987] pending: vmstat_update [ 5702.240340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5702.242853] pending: vmstat_update [ 5702.244656] workqueue writeback: flags=0x4a [ 5702.248083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5702.251612] pending: wb_workfn, wb_workfn [ 5702.253439] workqueue xprtiod: flags=0xa [ 5702.256014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5702.258478] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5702.262449] workqueue ipv6_addrconf: flags=0x40008 [ 5702.265160] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5702.267794] pending: addrconf_verify_work [ 5702.269660] workqueue knbd0-recv: flags=0x1a [ 5702.271671] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5702.274319] in-flight: 651:recv_work [ 5702.276293] workqueue ptlrpc_pinger: flags=0x2 [ 5702.278514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5702.282236] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5702.284733] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3785s workers=4 idle: 37 31846 28897 [ 5702.289604] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5692s workers=2 idle: 653 [ 5712.062075] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5712.064446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.096804] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.104476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5712.111262] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5712.113609] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5712.127250] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5712.133567] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5712.140431] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5712.146539] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5712.153314] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5712.162940] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5712.168831] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5712.172883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.179359] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5712.181045] Call Trace: [ 5712.181829] [ 5712.184483] ? show_regs.cold.9+0x22/0x2f [ 5712.187565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.189880] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.191843] ? watchdog+0x40/0x40 [ 5712.193438] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.196558] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.208492] ? apic_timer_interrupt+0xf/0x20 [ 5712.216617] [ 5712.218979] ? native_safe_halt+0x2e/0x50 [ 5712.220672] kvm_wait+0xcd/0xe0 [ 5712.222152] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5712.225095] do_raw_spin_lock+0x177/0x190 [ 5712.227100] _raw_spin_lock+0x21/0x30 [ 5712.228740] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5712.231202] obd_mod_free+0x46/0x140 [ptlrpc] [ 5712.233183] mdc_commit_open+0x91/0xb0 [mdc] [ 5712.235094] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5712.237565] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5712.239792] after_reply+0xe2c/0x1780 [ptlrpc] [ 5712.241957] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5712.244855] ? __next_timer_interrupt+0x160/0x160 [ 5712.247440] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5712.250359] ? do_wait_intr+0xf0/0xf0 [ 5712.251962] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5712.254617] kthread+0x1d1/0x200 [ 5712.256073] ? set_kthread_struct+0x70/0x70 [ 5712.257983] ret_from_fork+0x1f/0x30 [ 5716.057830] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5716.063208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5716.105740] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5716.110537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5716.115138] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5716.122662] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5716.127453] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5716.139164] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5716.143155] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5716.146838] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5716.150196] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5716.153209] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5716.155504] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5716.158401] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5716.163925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5716.167104] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5716.170737] Call Trace: [ 5716.171807] [ 5716.172782] ? show_regs.cold.9+0x22/0x2f [ 5716.175267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5716.177676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5716.179785] ? watchdog+0x40/0x40 [ 5716.181206] ? hrtimer_interrupt+0x128/0x2c0 [ 5716.183719] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5716.185710] ? apic_timer_interrupt+0xf/0x20 [ 5716.187800] [ 5716.188695] ? native_safe_halt+0x2e/0x50 [ 5716.190250] kvm_wait+0xcd/0xe0 [ 5716.191694] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5716.194235] do_raw_spin_lock+0x177/0x190 [ 5716.195999] _raw_spin_lock+0x21/0x30 [ 5716.197410] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5716.200670] process_one_work+0x2c8/0x700 [ 5716.202644] worker_thread+0x296/0x6e0 [ 5716.204160] ? rescuer_thread+0x570/0x570 [ 5716.206277] kthread+0x1d1/0x200 [ 5716.207801] ? set_kthread_struct+0x70/0x70 [ 5716.209306] ret_from_fork+0x1f/0x30 [ 5728.055832] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5728.059012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5728.059843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5728.059918] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5728.059925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5728.059928] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5728.059944] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5728.059948] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5728.059953] RAX: 0000000000007f92 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5728.059955] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5728.059957] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5728.059959] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5728.059961] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5728.059964] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5728.059968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5728.059970] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5728.059978] Call Trace: [ 5728.059983] [ 5728.059989] ? show_regs.cold.9+0x22/0x2f [ 5728.059998] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5728.060004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5728.060009] ? watchdog+0x40/0x40 [ 5728.060013] ? hrtimer_interrupt+0x128/0x2c0 [ 5728.060017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5728.060025] ? apic_timer_interrupt+0xf/0x20 [ 5728.060030] [ 5728.060033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5728.060040] do_raw_spin_lock+0x177/0x190 [ 5728.060045] _raw_spin_lock+0x21/0x30 [ 5728.060050] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5728.060391] ? do_raw_spin_unlock+0x75/0x190 [ 5728.060397] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5728.068353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5728.068417] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5728.068422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5728.068424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5728.068434] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5728.068438] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5728.068441] RAX: 0000000000003254 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5728.068442] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5728.068444] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5728.068445] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5728.068447] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5728.068450] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5728.068452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5728.068453] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5728.068459] Call Trace: [ 5728.068464] [ 5728.068468] ? show_regs.cold.9+0x22/0x2f [ 5728.068475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5728.068480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5728.068483] ? watchdog+0x40/0x40 [ 5728.068486] ? hrtimer_interrupt+0x128/0x2c0 [ 5728.068488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5728.068494] ? apic_timer_interrupt+0xf/0x20 [ 5728.068497] [ 5728.068498] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5728.068502] do_raw_spin_lock+0x177/0x190 [ 5728.068506] _raw_spin_lock+0x21/0x30 [ 5728.068509] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5728.068783] ? do_raw_spin_unlock+0x75/0x190 [ 5728.068789] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5728.068987] ? do_wait_intr+0xf0/0xf0 [ 5728.068995] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5728.069186] mdc_close+0x53b/0xd70 [mdc] [ 5728.069223] lmv_close+0x18d/0x490 [lmv] [ 5728.069249] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5728.069351] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5728.069429] ll_md_close+0x274/0x800 [lustre] [ 5728.069511] ll_file_release+0x128/0xc50 [lustre] [ 5728.069605] __fput+0x10d/0x440 [ 5728.069614] ____fput+0x12/0x20 [ 5728.069618] task_work_run+0xc6/0x110 [ 5728.069625] exit_to_usermode_loop+0x1dd/0x1f0 [ 5728.069632] do_syscall_64+0x3d6/0x440 [ 5728.069636] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5728.069651] RIP: 0033:0x7fa749552cd8 [ 5728.069656] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5728.069660] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5728.069664] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5728.069666] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5728.069669] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5728.069671] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5728.069673] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5728.488147] ? do_wait_intr+0xf0/0xf0 [ 5728.489664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5728.492134] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5728.494872] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5728.497322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5728.499717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5728.502060] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5728.504241] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5728.506651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5728.509001] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5728.511831] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5728.514292] lmv_intent_open+0x338/0x1150 [lmv] [ 5728.516167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5728.518526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5728.520298] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5728.522141] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5728.524615] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5728.526894] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5728.528460] ? do_raw_spin_unlock+0x75/0x190 [ 5728.530119] ? _raw_spin_unlock+0x12/0x30 [ 5728.532082] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5728.534262] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5728.536814] ll_atomic_open+0x36a/0xd70 [lustre] [ 5728.538341] lookup_open+0x356/0xb40 [ 5728.539391] do_last+0x313/0xfc0 [ 5728.540340] ? nd_jump_root+0xe5/0x160 [ 5728.541772] ? path_init+0x437/0x520 [ 5728.543476] path_openat+0xf7/0x500 [ 5728.544865] ? do_raw_spin_unlock+0x75/0x190 [ 5728.546704] do_filp_open+0x99/0x140 [ 5728.548208] ? getname_flags+0x6e/0x330 [ 5728.550027] ? __check_object_size+0xff/0x256 [ 5728.552062] ? do_raw_spin_unlock+0x75/0x190 [ 5728.554110] ? _raw_spin_unlock+0x12/0x30 [ 5728.556135] do_sys_openat2+0x2b4/0x410 [ 5728.557991] do_sys_open+0x73/0xa0 [ 5728.559663] __x64_sys_openat+0x24/0x30 [ 5728.561568] do_syscall_64+0xc1/0x440 [ 5728.563194] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5728.565230] RIP: 0033:0x7fa749552332 [ 5728.567076] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5728.575848] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5728.579514] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5728.582546] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5728.585051] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5728.588158] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5728.591653] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5732.320625] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3815s! [ 5732.328848] Showing busy workqueues and worker pools: [ 5732.332078] workqueue events: flags=0x0 [ 5732.334049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5732.338686] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5732.345969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5732.349645] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5732.355481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.360023] pending: kfree_rcu_monitor [ 5732.362272] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5732.366120] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5732.376774] workqueue events_long: flags=0x0 [ 5732.383949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.388405] pending: lu_objects_flush [obdclass] [ 5732.393464] workqueue events_unbound: flags=0x2 [ 5732.398471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5732.405676] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5732.412841] workqueue events_power_efficient: flags=0x80 [ 5732.416124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.422487] pending: check_lifetime [ 5732.424389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.427963] pending: neigh_periodic_work [ 5732.430405] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.437379] pending: neigh_periodic_work [ 5732.448431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.452876] pending: do_cache_clean [ 5732.459814] workqueue mm_percpu_wq: flags=0x8 [ 5732.464933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.475037] pending: vmstat_update [ 5732.476293] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.483426] pending: vmstat_update [ 5732.486741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5732.491127] pending: vmstat_update [ 5732.497521] workqueue writeback: flags=0x4a [ 5732.499577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5732.504057] pending: wb_workfn, wb_workfn [ 5732.505814] workqueue xprtiod: flags=0xa [ 5732.514014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5732.522521] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5732.529154] workqueue ipv6_addrconf: flags=0x40008 [ 5732.534933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5732.537298] pending: addrconf_verify_work [ 5732.543858] workqueue knbd0-recv: flags=0x1a [ 5732.546617] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5732.549734] in-flight: 651:recv_work [ 5732.551575] workqueue ptlrpc_pinger: flags=0x2 [ 5732.553299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5732.556136] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5732.558549] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3815s workers=4 idle: 37 31846 28897 [ 5732.561793] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5722s workers=2 idle: 653 [ 5740.062401] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5740.070759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.110253] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.128114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5740.137540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5740.148751] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5740.165686] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5740.175908] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5740.186986] RDX: 00000000000011ee RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5740.198240] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5740.207032] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5740.216703] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5740.221304] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5740.228289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.232924] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5740.236695] Call Trace: [ 5740.239134] [ 5740.240233] ? show_regs.cold.9+0x22/0x2f [ 5740.242902] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.245545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.249137] ? watchdog+0x40/0x40 [ 5740.250457] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.253719] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.255729] ? apic_timer_interrupt+0xf/0x20 [ 5740.259778] [ 5740.261521] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5740.264504] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5740.268095] do_raw_spin_lock+0x177/0x190 [ 5740.269917] _raw_spin_lock+0x21/0x30 [ 5740.272876] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5740.278834] obd_mod_free+0x46/0x140 [ptlrpc] [ 5740.281269] mdc_commit_open+0x91/0xb0 [mdc] [ 5740.282682] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5740.286280] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5740.290845] after_reply+0xe2c/0x1780 [ptlrpc] [ 5740.293764] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5740.301560] ? __next_timer_interrupt+0x160/0x160 [ 5740.305221] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5740.309368] ? do_wait_intr+0xf0/0xf0 [ 5740.313944] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5740.322039] kthread+0x1d1/0x200 [ 5740.326464] ? set_kthread_struct+0x70/0x70 [ 5740.330946] ret_from_fork+0x1f/0x30 [ 5744.057072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5744.059980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5744.107218] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5744.113049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5744.119128] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5744.126951] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5744.129876] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5744.141332] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5744.146329] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5744.149957] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5744.153189] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5744.159119] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5744.163140] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5744.168007] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5744.176792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5744.179659] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5744.191675] Call Trace: [ 5744.192486] [ 5744.193940] ? show_regs.cold.9+0x22/0x2f [ 5744.198709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5744.202875] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5744.206931] ? watchdog+0x40/0x40 [ 5744.209308] ? hrtimer_interrupt+0x128/0x2c0 [ 5744.213977] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5744.217709] ? apic_timer_interrupt+0xf/0x20 [ 5744.225437] [ 5744.226136] ? native_safe_halt+0x2e/0x50 [ 5744.230695] kvm_wait+0xcd/0xe0 [ 5744.232750] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5744.238780] do_raw_spin_lock+0x177/0x190 [ 5744.242017] _raw_spin_lock+0x21/0x30 [ 5744.246923] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5744.253780] process_one_work+0x2c8/0x700 [ 5744.256774] worker_thread+0x296/0x6e0 [ 5744.261015] ? rescuer_thread+0x570/0x570 [ 5744.264271] kthread+0x1d1/0x200 [ 5744.268879] ? set_kthread_struct+0x70/0x70 [ 5744.270859] ret_from_fork+0x1f/0x30 [ 5753.769491] rcu: INFO: rcu_sched self-detected stall on CPU [ 5753.775845] rcu: 2-...!: (2111545 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 5753.787219] (t=3840080 jiffies g=258701 q=2590) [ 5753.791493] rcu: rcu_sched kthread starved for 3836875 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5753.796491] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5753.803479] rcu: RCU grace-period kthread stack dump: [ 5753.808351] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5753.816461] Call Trace: [ 5753.817541] __schedule+0x351/0xcb0 [ 5753.818889] schedule+0xc0/0x180 [ 5753.822492] schedule_timeout+0xb4/0x190 [ 5753.826007] ? do_raw_spin_unlock+0x75/0x190 [ 5753.829809] ? __next_timer_interrupt+0x160/0x160 [ 5753.832642] ? prepare_to_swait_event+0xf3/0x140 [ 5753.835731] rcu_gp_kthread+0x8dd/0x15b0 [ 5753.838270] ? force_qs_rnp+0x2c0/0x2c0 [ 5753.840278] kthread+0x1d1/0x200 [ 5753.845773] ? set_kthread_struct+0x70/0x70 [ 5753.846957] ret_from_fork+0x1f/0x30 [ 5753.850462] rcu: Stack dump where RCU GP kthread last ran: [ 5753.851800] Sending NMI from CPU 2 to CPUs 0: [ 5753.857642] NMI backtrace for cpu 0 [ 5753.857651] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5753.857653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5753.857654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5753.857656] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5753.857658] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 [ 5753.857661] RAX: 0000000000007f3e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5753.857662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5753.857664] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5753.857665] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5753.857666] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5753.857667] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5753.857669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5753.857670] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5753.857671] Call Trace: [ 5753.857672] [ 5753.857673] ? show_regs.cold.9+0x22/0x2f [ 5753.857674] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5753.857675] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5753.857676] ? nmi_handle+0xbd/0x210 [ 5753.857676] ? default_do_nmi+0x4b/0x1b0 [ 5753.857677] ? do_nmi+0x280/0x380 [ 5753.857679] ? end_repeat_nmi+0x16/0x1f [ 5753.857680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5753.857681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5753.857683] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5753.857684] [ 5753.857685] do_raw_spin_lock+0x177/0x190 [ 5753.857686] _raw_spin_lock+0x21/0x30 [ 5753.857687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5753.857688] ? do_raw_spin_unlock+0x75/0x190 [ 5753.857689] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5753.857691] ? do_wait_intr+0xf0/0xf0 [ 5753.857692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5753.857693] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5753.857694] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5753.857696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5753.857697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5753.857698] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5753.857699] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5753.857700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5753.857702] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5753.857703] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5753.857704] lmv_intent_open+0x338/0x1150 [lmv] [ 5753.857705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5753.857706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5753.857707] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5753.857708] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5753.857709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5753.857710] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5753.857711] ? do_raw_spin_unlock+0x75/0x190 [ 5753.857712] ? _raw_spin_unlock+0x12/0x30 [ 5753.857713] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5753.857714] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5753.857715] ll_atomic_open+0x36a/0xd70 [lustre] [ 5753.857716] lookup_open+0x356/0xb40 [ 5753.857717] do_last+0x313/0xfc0 [ 5753.857718] ? nd_jump_root+0xe5/0x160 [ 5753.857719] ? path_init+0x437/0x520 [ 5753.857719] path_openat+0xf7/0x500 [ 5753.857720] ? do_raw_spin_unlock+0x75/0x190 [ 5753.857721] do_filp_open+0x99/0x140 [ 5753.857722] ? getname_flags+0x6e/0x330 [ 5753.857723] ? __check_object_size+0xff/0x256 [ 5753.857724] ? do_raw_spin_unlock+0x75/0x190 [ 5753.857726] ? _raw_spin_unlock+0x12/0x30 [ 5753.857726] do_sys_openat2+0x2b4/0x410 [ 5753.857727] do_sys_open+0x73/0xa0 [ 5753.857728] __x64_sys_openat+0x24/0x30 [ 5753.857729] do_syscall_64+0xc1/0x440 [ 5753.857730] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5753.857731] RIP: 0033:0x7fa749552332 [ 5753.857733] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5753.857735] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5753.857737] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5753.857738] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5753.857740] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5753.857741] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5753.857743] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5753.857971] NMI backtrace for cpu 2 [ 5754.291051] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5754.297191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5754.309764] Call Trace: [ 5754.310829] [ 5754.311692] ? dump_stack+0xbb/0x10e [ 5754.313336] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5754.315650] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5754.318313] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5754.321411] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5754.324989] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5754.326738] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5754.329681] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5754.333831] ? update_process_times+0x6d/0xc0 [ 5754.338167] ? tick_sched_handle.isra.18+0x27/0x90 [ 5754.342971] ? tick_sched_timer+0x53/0xb0 [ 5754.344130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5754.345677] ? tick_sched_handle.isra.18+0x90/0x90 [ 5754.352198] ? hrtimer_interrupt+0x128/0x2c0 [ 5754.356419] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5754.363292] ? apic_timer_interrupt+0xf/0x20 [ 5754.367895] [ 5754.368708] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5754.373722] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5754.375937] ? do_raw_spin_lock+0x177/0x190 [ 5754.377990] ? _raw_spin_lock+0x21/0x30 [ 5754.379475] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5754.381872] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5754.384174] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5754.385848] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5754.388474] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5754.391017] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5754.393700] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5754.395879] ? __next_timer_interrupt+0x160/0x160 [ 5754.398202] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5754.400816] ? do_wait_intr+0xf0/0xf0 [ 5754.402366] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5754.405675] ? kthread+0x1d1/0x200 [ 5754.407221] ? set_kthread_struct+0x70/0x70 [ 5754.409803] ? ret_from_fork+0x1f/0x30 [ 5756.061045] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5756.073441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5756.104307] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5756.112213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5756.121407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5756.125069] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5756.143215] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5756.145524] RAX: 0000000000005c85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5756.151492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5756.160366] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5756.165522] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5756.173333] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5756.180950] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5756.191972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5756.198040] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5756.203180] Call Trace: [ 5756.204528] [ 5756.205452] ? show_regs.cold.9+0x22/0x2f [ 5756.207392] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5756.214505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5756.220387] ? watchdog+0x40/0x40 [ 5756.224010] ? hrtimer_interrupt+0x128/0x2c0 [ 5756.226575] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5756.229319] ? apic_timer_interrupt+0xf/0x20 [ 5756.231188] [ 5756.232701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5756.236027] do_raw_spin_lock+0x177/0x190 [ 5756.240319] _raw_spin_lock+0x21/0x30 [ 5756.244714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5756.251249] ? do_raw_spin_unlock+0x75/0x190 [ 5756.252773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5756.258269] ? do_wait_intr+0xf0/0xf0 [ 5756.259721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5756.267059] mdc_close+0x53b/0xd70 [mdc] [ 5756.270075] lmv_close+0x18d/0x490 [lmv] [ 5756.274066] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5756.280629] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5756.291445] ll_md_close+0x274/0x800 [lustre] [ 5756.297694] ll_file_release+0x128/0xc50 [lustre] [ 5756.301661] __fput+0x10d/0x440 [ 5756.303730] ____fput+0x12/0x20 [ 5756.307197] task_work_run+0xc6/0x110 [ 5756.311501] exit_to_usermode_loop+0x1dd/0x1f0 [ 5756.317611] do_syscall_64+0x3d6/0x440 [ 5756.321524] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5756.322849] RIP: 0033:0x7fa749552cd8 [ 5756.323740] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5756.338823] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5756.345006] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5756.350766] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5756.353063] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5756.363125] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5756.375352] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5763.040114] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3845s! [ 5763.057769] Showing busy workqueues and worker pools: [ 5763.060113] workqueue events: flags=0x0 [ 5763.061669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5763.065437] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5763.073148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5763.079356] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5763.090446] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.102435] pending: kfree_rcu_monitor [ 5763.108660] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5763.120159] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5763.127442] workqueue events_long: flags=0x0 [ 5763.131888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.138959] pending: lu_objects_flush [obdclass] [ 5763.142920] workqueue events_unbound: flags=0x2 [ 5763.151642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5763.157959] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5763.170572] workqueue events_power_efficient: flags=0x80 [ 5763.174914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.187706] pending: check_lifetime [ 5763.197856] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.210524] pending: neigh_periodic_work [ 5763.215641] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.220801] pending: neigh_periodic_work [ 5763.229772] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.247877] pending: do_cache_clean [ 5763.249336] workqueue mm_percpu_wq: flags=0x8 [ 5763.250890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.259076] pending: vmstat_update [ 5763.267499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.284674] pending: vmstat_update [ 5763.286162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5763.302692] pending: vmstat_update [ 5763.310302] workqueue writeback: flags=0x4a [ 5763.315726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5763.326190] pending: wb_workfn, wb_workfn [ 5763.330531] workqueue xprtiod: flags=0xa [ 5763.334854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5763.336983] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5763.345656] workqueue ipv6_addrconf: flags=0x40008 [ 5763.348924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5763.357246] pending: addrconf_verify_work [ 5763.364341] workqueue knbd0-recv: flags=0x1a [ 5763.373070] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5763.380939] in-flight: 651:recv_work [ 5763.382337] workqueue ptlrpc_pinger: flags=0x2 [ 5763.385404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5763.395625] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5763.400254] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3846s workers=4 idle: 37 31846 28897 [ 5763.410636] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5753s workers=2 idle: 653 [ 5772.057024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5772.061475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5772.084880] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5772.091644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5772.099184] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5772.103268] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5772.106927] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5772.117770] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5772.128169] RAX: 00000000000019c3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5772.134306] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5772.138522] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5772.142777] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5772.147479] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5772.154324] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5772.159737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5772.162567] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5772.166406] Call Trace: [ 5772.167832] [ 5772.168898] ? show_regs.cold.9+0x22/0x2f [ 5772.171764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5772.174412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5772.177288] ? watchdog+0x40/0x40 [ 5772.179366] ? hrtimer_interrupt+0x128/0x2c0 [ 5772.181363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5772.183285] ? apic_timer_interrupt+0xf/0x20 [ 5772.185023] [ 5772.186222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5772.189527] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5772.193279] do_raw_spin_lock+0x177/0x190 [ 5772.195391] _raw_spin_lock+0x21/0x30 [ 5772.197362] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5772.203485] process_one_work+0x2c8/0x700 [ 5772.205434] worker_thread+0x296/0x6e0 [ 5772.209410] ? rescuer_thread+0x570/0x570 [ 5772.212315] kthread+0x1d1/0x200 [ 5772.214228] ? set_kthread_struct+0x70/0x70 [ 5772.216422] ret_from_fork+0x1f/0x30 [ 5780.058155] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5780.062349] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5780.121410] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5780.133516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5780.145813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5780.153894] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5780.174934] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5780.185483] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5780.191359] RDX: 0000000000000865 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5780.200608] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5780.205601] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5780.213553] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5780.220344] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5780.225662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5780.232843] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5780.237937] Call Trace: [ 5780.239844] [ 5780.241187] ? show_regs.cold.9+0x22/0x2f [ 5780.244761] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5780.249847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5780.255896] ? watchdog+0x40/0x40 [ 5780.261187] ? hrtimer_interrupt+0x128/0x2c0 [ 5780.263269] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5780.270305] ? apic_timer_interrupt+0xf/0x20 [ 5780.271633] [ 5780.274641] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5780.280881] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5780.293797] do_raw_spin_lock+0x177/0x190 [ 5780.298348] _raw_spin_lock+0x21/0x30 [ 5780.305531] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5780.316116] obd_mod_free+0x46/0x140 [ptlrpc] [ 5780.318489] mdc_commit_open+0x91/0xb0 [mdc] [ 5780.321114] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5780.324063] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5780.329329] after_reply+0xe2c/0x1780 [ptlrpc] [ 5780.333545] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5780.336737] ? __next_timer_interrupt+0x160/0x160 [ 5780.340566] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5780.352287] ? do_wait_intr+0xf0/0xf0 [ 5780.356128] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5780.359923] kthread+0x1d1/0x200 [ 5780.364547] ? set_kthread_struct+0x70/0x70 [ 5780.375012] ret_from_fork+0x1f/0x30 [ 5784.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5784.063494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5784.084588] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5784.094944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5784.108054] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5784.119174] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5784.136646] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5784.148460] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5784.159083] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5784.173134] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5784.181809] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5784.194297] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5784.203142] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5784.207923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5784.211483] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5784.215131] Call Trace: [ 5784.216500] [ 5784.217635] ? show_regs.cold.9+0x22/0x2f [ 5784.222180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5784.228336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5784.235649] ? watchdog+0x40/0x40 [ 5784.238342] ? hrtimer_interrupt+0x128/0x2c0 [ 5784.242311] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5784.245295] ? apic_timer_interrupt+0xf/0x20 [ 5784.249170] [ 5784.250239] ? native_safe_halt+0x2e/0x50 [ 5784.256798] kvm_wait+0xcd/0xe0 [ 5784.257668] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5784.263308] do_raw_spin_lock+0x177/0x190 [ 5784.271135] _raw_spin_lock+0x21/0x30 [ 5784.275998] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5784.282671] ? do_raw_spin_unlock+0x75/0x190 [ 5784.285365] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5784.294123] ? do_wait_intr+0xf0/0xf0 [ 5784.301602] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5784.308554] mdc_close+0x53b/0xd70 [mdc] [ 5784.314014] lmv_close+0x18d/0x490 [lmv] [ 5784.316709] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5784.323714] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5784.335539] ll_md_close+0x274/0x800 [lustre] [ 5784.340364] ll_file_release+0x128/0xc50 [lustre] [ 5784.344917] __fput+0x10d/0x440 [ 5784.354395] ____fput+0x12/0x20 [ 5784.355578] task_work_run+0xc6/0x110 [ 5784.360017] exit_to_usermode_loop+0x1dd/0x1f0 [ 5784.370141] do_syscall_64+0x3d6/0x440 [ 5784.371521] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5784.373961] RIP: 0033:0x7fa749552cd8 [ 5784.376043] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5784.402274] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5784.405079] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5784.421242] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5784.436560] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5784.442587] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5784.453292] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5788.056192] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5788.060786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5788.082053] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5788.087085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5788.091215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5788.094938] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5788.104869] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5788.108737] RAX: 0000000000006618 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5788.112444] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5788.116060] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5788.119501] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5788.122385] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5788.125947] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5788.129470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5788.132355] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5788.135516] Call Trace: [ 5788.137347] [ 5788.138296] ? show_regs.cold.9+0x22/0x2f [ 5788.140830] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5788.143086] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5788.145115] ? watchdog+0x40/0x40 [ 5788.146510] ? hrtimer_interrupt+0x128/0x2c0 [ 5788.148418] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5788.150739] ? apic_timer_interrupt+0xf/0x20 [ 5788.152766] [ 5788.153558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5788.156151] do_raw_spin_lock+0x177/0x190 [ 5788.157984] _raw_spin_lock+0x21/0x30 [ 5788.159531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5788.162002] ? do_raw_spin_unlock+0x75/0x190 [ 5788.164177] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5788.166522] ? do_wait_intr+0xf0/0xf0 [ 5788.167998] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5788.170452] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5788.172342] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5788.174921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5788.177326] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5788.179971] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5788.181524] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5788.183504] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5788.186182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5788.189560] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5788.192430] lmv_intent_open+0x338/0x1150 [lmv] [ 5788.194581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5788.198162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5788.201971] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5788.204777] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5788.207478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5788.211043] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5788.213199] ? do_raw_spin_unlock+0x75/0x190 [ 5788.214905] ? _raw_spin_unlock+0x12/0x30 [ 5788.216514] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5788.218742] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5788.221067] ll_atomic_open+0x36a/0xd70 [lustre] [ 5788.223743] lookup_open+0x356/0xb40 [ 5788.225353] do_last+0x313/0xfc0 [ 5788.226880] ? nd_jump_root+0xe5/0x160 [ 5788.228679] ? path_init+0x437/0x520 [ 5788.230269] path_openat+0xf7/0x500 [ 5788.232461] ? do_raw_spin_unlock+0x75/0x190 [ 5788.234131] do_filp_open+0x99/0x140 [ 5788.237008] ? getname_flags+0x6e/0x330 [ 5788.238392] ? __check_object_size+0xff/0x256 [ 5788.240559] ? do_raw_spin_unlock+0x75/0x190 [ 5788.242726] ? _raw_spin_unlock+0x12/0x30 [ 5788.244770] do_sys_openat2+0x2b4/0x410 [ 5788.246468] do_sys_open+0x73/0xa0 [ 5788.248032] __x64_sys_openat+0x24/0x30 [ 5788.249878] do_syscall_64+0xc1/0x440 [ 5788.251468] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5788.253708] RIP: 0033:0x7fa749552332 [ 5788.255245] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5788.267503] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5788.273659] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5788.282330] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5788.288287] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5788.297015] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5788.303526] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5793.760778] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3876s! [ 5793.763797] Showing busy workqueues and worker pools: [ 5793.765795] workqueue events: flags=0x0 [ 5793.766697] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5793.768139] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5793.774107] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5793.776581] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5793.780526] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.783791] pending: kfree_rcu_monitor [ 5793.786279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5793.790428] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5793.795616] workqueue events_long: flags=0x0 [ 5793.796579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.798029] pending: lu_objects_flush [obdclass] [ 5793.800316] workqueue events_unbound: flags=0x2 [ 5793.801346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5793.805393] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5793.807879] workqueue events_power_efficient: flags=0x80 [ 5793.810931] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.819102] pending: check_lifetime [ 5793.820188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.826815] pending: neigh_periodic_work [ 5793.832434] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.835950] pending: neigh_periodic_work [ 5793.838116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.843500] pending: do_cache_clean [ 5793.845450] workqueue mm_percpu_wq: flags=0x8 [ 5793.847862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.852319] pending: vmstat_update [ 5793.853968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.868552] pending: vmstat_update [ 5793.869875] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5793.875075] pending: vmstat_update [ 5793.876402] workqueue writeback: flags=0x4a [ 5793.881854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5793.889578] pending: wb_workfn, wb_workfn [ 5793.897359] workqueue xprtiod: flags=0xa [ 5793.899425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5793.906533] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5793.915907] workqueue ipv6_addrconf: flags=0x40008 [ 5793.921476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5793.923970] pending: addrconf_verify_work [ 5793.926400] workqueue knbd0-recv: flags=0x1a [ 5793.929818] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5793.933536] in-flight: 651:recv_work [ 5793.936067] workqueue ptlrpc_pinger: flags=0x2 [ 5793.938588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5793.941448] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5793.944459] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3876s workers=4 idle: 37 31846 28897 [ 5793.952132] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5783s workers=2 idle: 653 [ 5800.058089] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5800.062675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5800.089490] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5800.093943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5800.097204] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5800.099559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5800.101458] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5800.107915] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5800.110296] RAX: 0000000000007e84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5800.112646] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5800.114984] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5800.117333] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5800.119914] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5800.122569] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5800.125613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5800.127743] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5800.132920] Call Trace: [ 5800.135210] [ 5800.136528] ? show_regs.cold.9+0x22/0x2f [ 5800.138165] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5800.141993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5800.146456] ? watchdog+0x40/0x40 [ 5800.153615] ? hrtimer_interrupt+0x128/0x2c0 [ 5800.156842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5800.160011] ? apic_timer_interrupt+0xf/0x20 [ 5800.166657] [ 5800.167927] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5800.174059] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5800.180852] do_raw_spin_lock+0x177/0x190 [ 5800.183106] _raw_spin_lock+0x21/0x30 [ 5800.186990] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5800.192535] process_one_work+0x2c8/0x700 [ 5800.194591] worker_thread+0x296/0x6e0 [ 5800.197020] ? rescuer_thread+0x570/0x570 [ 5800.199016] kthread+0x1d1/0x200 [ 5800.201224] ? set_kthread_struct+0x70/0x70 [ 5800.203007] ret_from_fork+0x1f/0x30 [ 5808.058111] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5808.061705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.075218] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.078275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.080606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5808.082186] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5808.088295] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5808.090622] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5808.093050] RDX: 0000000000005a59 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5808.095349] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5808.098075] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5808.100332] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5808.102682] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5808.105701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.107874] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5808.110545] Call Trace: [ 5808.111539] [ 5808.112366] ? show_regs.cold.9+0x22/0x2f [ 5808.113903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.115741] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.117412] ? watchdog+0x40/0x40 [ 5808.118680] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.120592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.122392] ? apic_timer_interrupt+0xf/0x20 [ 5808.123772] [ 5808.124708] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5808.126761] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5808.129203] do_raw_spin_lock+0x177/0x190 [ 5808.130739] _raw_spin_lock+0x21/0x30 [ 5808.132183] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5808.134331] obd_mod_free+0x46/0x140 [ptlrpc] [ 5808.136282] mdc_commit_open+0x91/0xb0 [mdc] [ 5808.137801] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5808.139883] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5808.142235] after_reply+0xe2c/0x1780 [ptlrpc] [ 5808.144175] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5808.146703] ? __next_timer_interrupt+0x160/0x160 [ 5808.148567] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5808.150457] ? do_wait_intr+0xf0/0xf0 [ 5808.151820] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5808.154055] kthread+0x1d1/0x200 [ 5808.155288] ? set_kthread_struct+0x70/0x70 [ 5808.156816] ret_from_fork+0x1f/0x30 [ 5812.059253] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5812.063026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5812.084047] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5812.087531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5812.090851] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5812.092737] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5812.098522] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5812.101223] RAX: 00000000000071a7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5812.103362] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5812.105495] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5812.107739] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5812.110141] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5812.112443] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5812.115707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5812.118309] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5812.122122] Call Trace: [ 5812.123500] [ 5812.124542] ? show_regs.cold.9+0x22/0x2f [ 5812.126722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5812.129427] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5812.132250] ? watchdog+0x40/0x40 [ 5812.133712] ? hrtimer_interrupt+0x128/0x2c0 [ 5812.135677] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5812.137932] ? apic_timer_interrupt+0xf/0x20 [ 5812.139698] [ 5812.141213] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5812.143151] do_raw_spin_lock+0x177/0x190 [ 5812.144782] _raw_spin_lock+0x21/0x30 [ 5812.146426] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5812.148973] ? do_raw_spin_unlock+0x75/0x190 [ 5812.150775] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5812.153227] ? do_wait_intr+0xf0/0xf0 [ 5812.155556] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5812.158937] mdc_close+0x53b/0xd70 [mdc] [ 5812.161090] lmv_close+0x18d/0x490 [lmv] [ 5812.163322] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5812.166544] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5812.169203] ll_md_close+0x274/0x800 [lustre] [ 5812.171023] ll_file_release+0x128/0xc50 [lustre] [ 5812.174087] __fput+0x10d/0x440 [ 5812.175122] ____fput+0x12/0x20 [ 5812.176314] task_work_run+0xc6/0x110 [ 5812.177865] exit_to_usermode_loop+0x1dd/0x1f0 [ 5812.178958] do_syscall_64+0x3d6/0x440 [ 5812.181255] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5812.183506] RIP: 0033:0x7fa749552cd8 [ 5812.186042] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5812.198628] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5812.203136] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5812.207764] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5812.211113] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5812.217288] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5812.221718] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5820.054036] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5820.057249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5820.077622] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5820.082670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5820.086759] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5820.088926] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5820.096911] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5820.100425] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5820.103491] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5820.106591] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5820.108701] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5820.111644] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5820.114883] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5820.117981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5820.120180] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5820.122888] Call Trace: [ 5820.123574] [ 5820.124322] ? show_regs.cold.9+0x22/0x2f [ 5820.125615] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5820.127943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5820.129901] ? watchdog+0x40/0x40 [ 5820.131349] ? hrtimer_interrupt+0x128/0x2c0 [ 5820.133183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5820.135147] ? apic_timer_interrupt+0xf/0x20 [ 5820.136944] [ 5820.137854] ? native_safe_halt+0x2e/0x50 [ 5820.139695] kvm_wait+0xcd/0xe0 [ 5820.141111] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5820.143011] do_raw_spin_lock+0x177/0x190 [ 5820.144493] _raw_spin_lock+0x21/0x30 [ 5820.145789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5820.147704] ? do_raw_spin_unlock+0x75/0x190 [ 5820.149176] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5820.151105] ? do_wait_intr+0xf0/0xf0 [ 5820.152274] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5820.154135] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5820.156493] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5820.158607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.160895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5820.162723] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5820.164390] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5820.166713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.168264] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5820.170430] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5820.174117] lmv_intent_open+0x338/0x1150 [lmv] [ 5820.176673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.179794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.181846] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5820.183785] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5820.185701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.188329] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5820.190851] ? do_raw_spin_unlock+0x75/0x190 [ 5820.193059] ? _raw_spin_unlock+0x12/0x30 [ 5820.195368] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5820.198695] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5820.201418] ll_atomic_open+0x36a/0xd70 [lustre] [ 5820.203788] lookup_open+0x356/0xb40 [ 5820.205129] do_last+0x313/0xfc0 [ 5820.208308] ? nd_jump_root+0xe5/0x160 [ 5820.210097] ? path_init+0x437/0x520 [ 5820.212284] path_openat+0xf7/0x500 [ 5820.216461] ? do_raw_spin_unlock+0x75/0x190 [ 5820.219735] do_filp_open+0x99/0x140 [ 5820.222152] ? getname_flags+0x6e/0x330 [ 5820.224429] ? __check_object_size+0xff/0x256 [ 5820.227249] ? do_raw_spin_unlock+0x75/0x190 [ 5820.229132] ? _raw_spin_unlock+0x12/0x30 [ 5820.230972] do_sys_openat2+0x2b4/0x410 [ 5820.232609] do_sys_open+0x73/0xa0 [ 5820.233921] __x64_sys_openat+0x24/0x30 [ 5820.235702] do_syscall_64+0xc1/0x440 [ 5820.237132] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5820.239191] RIP: 0033:0x7fa749552332 [ 5820.241134] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5820.251593] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5820.255854] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5820.259073] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5820.262948] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5820.267157] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5820.271407] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5823.968108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3906s! [ 5823.973019] Showing busy workqueues and worker pools: [ 5823.975289] workqueue events: flags=0x0 [ 5823.977067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5823.980838] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5823.987493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5823.991522] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5823.997411] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.001632] pending: kfree_rcu_monitor [ 5824.003847] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5824.008055] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5824.013736] workqueue events_long: flags=0x0 [ 5824.016561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.020930] pending: lu_objects_flush [obdclass] [ 5824.025828] workqueue events_unbound: flags=0x2 [ 5824.029606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5824.033586] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5824.038776] workqueue events_power_efficient: flags=0x80 [ 5824.041966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.045558] pending: check_lifetime [ 5824.047655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.051425] pending: neigh_periodic_work [ 5824.054400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.059435] pending: neigh_periodic_work [ 5824.062199] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.066018] pending: do_cache_clean [ 5824.068429] workqueue mm_percpu_wq: flags=0x8 [ 5824.070803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.074636] pending: vmstat_update [ 5824.076907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.081102] pending: vmstat_update [ 5824.083514] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5824.087505] pending: vmstat_update [ 5824.089738] workqueue writeback: flags=0x4a [ 5824.092546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5824.096070] pending: wb_workfn, wb_workfn [ 5824.099000] workqueue xprtiod: flags=0xa [ 5824.101452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5824.105253] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5824.110455] workqueue ipv6_addrconf: flags=0x40008 [ 5824.113357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5824.117063] pending: addrconf_verify_work [ 5824.120052] workqueue knbd0-recv: flags=0x1a [ 5824.122279] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5824.124716] in-flight: 651:recv_work [ 5824.127015] workqueue ptlrpc_pinger: flags=0x2 [ 5824.129596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5824.133378] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5824.136559] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3906s workers=4 idle: 37 31846 28897 [ 5824.141501] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5814s workers=2 idle: 653 [ 5828.057074] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5828.060123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5828.080137] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5828.085174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5828.096209] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5828.101409] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5828.104294] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5828.115527] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5828.118478] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5828.123653] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5828.129350] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5828.132466] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5828.136584] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5828.141978] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5828.145194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5828.147695] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5828.150951] Call Trace: [ 5828.152005] [ 5828.153078] ? show_regs.cold.9+0x22/0x2f [ 5828.155008] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5828.157847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5828.160089] ? watchdog+0x40/0x40 [ 5828.161758] ? hrtimer_interrupt+0x128/0x2c0 [ 5828.163203] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5828.165379] ? apic_timer_interrupt+0xf/0x20 [ 5828.167259] [ 5828.168160] ? native_safe_halt+0x2e/0x50 [ 5828.169894] kvm_wait+0xcd/0xe0 [ 5828.172358] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5828.174745] do_raw_spin_lock+0x177/0x190 [ 5828.177165] _raw_spin_lock+0x21/0x30 [ 5828.179710] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5828.182374] process_one_work+0x2c8/0x700 [ 5828.184295] worker_thread+0x296/0x6e0 [ 5828.185841] ? rescuer_thread+0x570/0x570 [ 5828.187584] kthread+0x1d1/0x200 [ 5828.188813] ? set_kthread_struct+0x70/0x70 [ 5828.190337] ret_from_fork+0x1f/0x30 [ 5836.060077] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5836.062776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.104597] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.111949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.114960] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5836.118326] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5836.131195] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5836.134561] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5836.137551] RDX: 0000000000000af8 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5836.140398] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5836.147016] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5836.150277] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5836.152673] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5836.154686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.159268] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5836.163854] Call Trace: [ 5836.167205] [ 5836.168399] ? show_regs.cold.9+0x22/0x2f [ 5836.172265] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.175797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.178324] ? watchdog+0x40/0x40 [ 5836.180162] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.181463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.183449] ? apic_timer_interrupt+0xf/0x20 [ 5836.185249] [ 5836.186028] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5836.187922] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5836.190169] do_raw_spin_lock+0x177/0x190 [ 5836.191963] _raw_spin_lock+0x21/0x30 [ 5836.193951] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5836.197974] obd_mod_free+0x46/0x140 [ptlrpc] [ 5836.200410] mdc_commit_open+0x91/0xb0 [mdc] [ 5836.203700] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5836.205537] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5836.208875] after_reply+0xe2c/0x1780 [ptlrpc] [ 5836.218453] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5836.221014] ? __next_timer_interrupt+0x160/0x160 [ 5836.222514] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5836.229627] ? do_wait_intr+0xf0/0xf0 [ 5836.231810] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5836.234091] kthread+0x1d1/0x200 [ 5836.236571] ? set_kthread_struct+0x70/0x70 [ 5836.237631] ret_from_fork+0x1f/0x30 [ 5840.059161] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5840.062059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5840.088168] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5840.094965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5840.099493] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5840.102298] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5840.114461] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5840.118350] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5840.122930] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5840.127137] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5840.131350] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5840.135943] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5840.139606] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5840.145579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5840.148623] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5840.153317] Call Trace: [ 5840.155023] [ 5840.156173] ? show_regs.cold.9+0x22/0x2f [ 5840.158747] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5840.162285] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5840.164477] ? watchdog+0x40/0x40 [ 5840.166208] ? hrtimer_interrupt+0x128/0x2c0 [ 5840.168217] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5840.170635] ? apic_timer_interrupt+0xf/0x20 [ 5840.172742] [ 5840.173941] ? native_safe_halt+0x2e/0x50 [ 5840.176210] kvm_wait+0xcd/0xe0 [ 5840.177629] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5840.180404] do_raw_spin_lock+0x177/0x190 [ 5840.182522] _raw_spin_lock+0x21/0x30 [ 5840.184498] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5840.187347] ? do_raw_spin_unlock+0x75/0x190 [ 5840.189531] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5840.192190] ? do_wait_intr+0xf0/0xf0 [ 5840.193968] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5840.197460] mdc_close+0x53b/0xd70 [mdc] [ 5840.202421] lmv_close+0x18d/0x490 [lmv] [ 5840.204635] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5840.211657] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5840.214866] ll_md_close+0x274/0x800 [lustre] [ 5840.220845] ll_file_release+0x128/0xc50 [lustre] [ 5840.231875] __fput+0x10d/0x440 [ 5840.233739] ____fput+0x12/0x20 [ 5840.236917] task_work_run+0xc6/0x110 [ 5840.238714] exit_to_usermode_loop+0x1dd/0x1f0 [ 5840.241055] do_syscall_64+0x3d6/0x440 [ 5840.242871] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5840.246368] RIP: 0033:0x7fa749552cd8 [ 5840.248578] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5840.259763] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5840.263593] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5840.267540] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5840.270866] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5840.274562] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5840.277574] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5852.054055] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5852.062187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5852.109697] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5852.117128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5852.129056] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5852.132682] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5852.141538] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5852.144853] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5852.147621] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5852.150382] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5852.152932] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5852.155545] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5852.158350] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5852.161360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5852.163806] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5852.169790] Call Trace: [ 5852.170736] [ 5852.171804] ? show_regs.cold.9+0x22/0x2f [ 5852.174248] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5852.175924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5852.187901] ? watchdog+0x40/0x40 [ 5852.194763] ? hrtimer_interrupt+0x128/0x2c0 [ 5852.199699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5852.201338] ? apic_timer_interrupt+0xf/0x20 [ 5852.215479] [ 5852.216203] ? native_safe_halt+0x2e/0x50 [ 5852.217327] kvm_wait+0xcd/0xe0 [ 5852.226144] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5852.227835] do_raw_spin_lock+0x177/0x190 [ 5852.232632] _raw_spin_lock+0x21/0x30 [ 5852.234636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5852.236875] ? do_raw_spin_unlock+0x75/0x190 [ 5852.238583] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5852.242979] ? do_wait_intr+0xf0/0xf0 [ 5852.244461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5852.248820] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5852.254696] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5852.259751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.261910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5852.264695] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5852.266867] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5852.268994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.271731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5852.275904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5852.281723] lmv_intent_open+0x338/0x1150 [lmv] [ 5852.287951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.294591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.300929] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5852.308545] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5852.319076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.320653] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5852.322211] ? do_raw_spin_unlock+0x75/0x190 [ 5852.329720] ? _raw_spin_unlock+0x12/0x30 [ 5852.331165] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5852.336644] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5852.339252] ll_atomic_open+0x36a/0xd70 [lustre] [ 5852.341128] lookup_open+0x356/0xb40 [ 5852.343570] do_last+0x313/0xfc0 [ 5852.348094] ? nd_jump_root+0xe5/0x160 [ 5852.349841] ? path_init+0x437/0x520 [ 5852.351730] path_openat+0xf7/0x500 [ 5852.353457] ? do_raw_spin_unlock+0x75/0x190 [ 5852.355740] do_filp_open+0x99/0x140 [ 5852.359559] ? getname_flags+0x6e/0x330 [ 5852.361260] ? __check_object_size+0xff/0x256 [ 5852.363045] ? do_raw_spin_unlock+0x75/0x190 [ 5852.369907] ? _raw_spin_unlock+0x12/0x30 [ 5852.371056] do_sys_openat2+0x2b4/0x410 [ 5852.373581] do_sys_open+0x73/0xa0 [ 5852.374675] __x64_sys_openat+0x24/0x30 [ 5852.376066] do_syscall_64+0xc1/0x440 [ 5852.384291] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5852.385824] RIP: 0033:0x7fa749552332 [ 5852.387653] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5852.405909] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5852.409700] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5852.414283] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5852.419882] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5852.425808] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5852.431530] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5854.181882] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3937s! [ 5854.187888] Showing busy workqueues and worker pools: [ 5854.192344] workqueue events: flags=0x0 [ 5854.194191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5854.196599] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5854.204866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5854.212473] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5854.220903] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.230831] pending: kfree_rcu_monitor [ 5854.236450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5854.246854] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5854.256057] workqueue events_long: flags=0x0 [ 5854.258849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.262839] pending: lu_objects_flush [obdclass] [ 5854.269790] workqueue events_unbound: flags=0x2 [ 5854.277595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5854.288361] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5854.298883] workqueue events_power_efficient: flags=0x80 [ 5854.302030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.313348] pending: check_lifetime [ 5854.316983] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.320786] pending: neigh_periodic_work [ 5854.324518] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.327512] pending: neigh_periodic_work [ 5854.331677] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.337341] pending: do_cache_clean [ 5854.341314] workqueue mm_percpu_wq: flags=0x8 [ 5854.345209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.349850] pending: vmstat_update [ 5854.355964] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.358679] pending: vmstat_update [ 5854.360369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5854.363323] pending: vmstat_update [ 5854.364602] workqueue writeback: flags=0x4a [ 5854.366550] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5854.369939] pending: wb_workfn, wb_workfn [ 5854.372304] workqueue xprtiod: flags=0xa [ 5854.374746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5854.377269] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5854.381024] workqueue ipv6_addrconf: flags=0x40008 [ 5854.383252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5854.385982] pending: addrconf_verify_work [ 5854.387835] workqueue knbd0-recv: flags=0x1a [ 5854.389370] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5854.391704] in-flight: 651:recv_work [ 5854.393407] workqueue ptlrpc_pinger: flags=0x2 [ 5854.400474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5854.403758] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5854.409238] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3937s workers=4 idle: 37 31846 28897 [ 5854.417390] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5844s workers=2 idle: 653 [ 5856.057304] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5856.066475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5856.099562] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5856.105657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5856.110382] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5856.114919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5856.119327] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5856.128435] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5856.132012] RAX: 0000000000007b26 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5856.134719] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5856.137275] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5856.139701] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5856.142687] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5856.146322] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5856.150219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5856.152482] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5856.156100] Call Trace: [ 5856.157987] [ 5856.160926] ? show_regs.cold.9+0x22/0x2f [ 5856.164787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5856.169553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5856.177243] ? watchdog+0x40/0x40 [ 5856.181334] ? hrtimer_interrupt+0x128/0x2c0 [ 5856.183692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5856.186954] ? apic_timer_interrupt+0xf/0x20 [ 5856.189546] [ 5856.192436] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5856.198438] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5856.207053] do_raw_spin_lock+0x177/0x190 [ 5856.210845] _raw_spin_lock+0x21/0x30 [ 5856.214092] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5856.222833] process_one_work+0x2c8/0x700 [ 5856.224176] worker_thread+0x296/0x6e0 [ 5856.229457] ? rescuer_thread+0x570/0x570 [ 5856.232937] kthread+0x1d1/0x200 [ 5856.238970] ? set_kthread_struct+0x70/0x70 [ 5856.241675] ret_from_fork+0x1f/0x30 [ 5864.058164] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 5864.067923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.119393] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.129789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.145365] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5864.149321] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5864.169739] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5864.173892] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5864.177580] RDX: 00000000000014ec RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5864.181277] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5864.184644] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5864.187917] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5864.191127] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5864.194731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.198075] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5864.201555] Call Trace: [ 5864.202774] [ 5864.203763] ? show_regs.cold.9+0x22/0x2f [ 5864.205446] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.207644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.209813] ? watchdog+0x40/0x40 [ 5864.211192] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.213171] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.215852] ? apic_timer_interrupt+0xf/0x20 [ 5864.217896] [ 5864.218935] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5864.221155] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5864.223478] do_raw_spin_lock+0x177/0x190 [ 5864.225480] _raw_spin_lock+0x21/0x30 [ 5864.227442] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5864.230614] obd_mod_free+0x46/0x140 [ptlrpc] [ 5864.233040] mdc_commit_open+0x91/0xb0 [mdc] [ 5864.235134] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5864.238831] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5864.241611] after_reply+0xe2c/0x1780 [ptlrpc] [ 5864.244182] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5864.246904] ? __next_timer_interrupt+0x160/0x160 [ 5864.248936] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5864.251578] ? do_wait_intr+0xf0/0xf0 [ 5864.253956] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5864.256569] kthread+0x1d1/0x200 [ 5864.259779] ? set_kthread_struct+0x70/0x70 [ 5864.261730] ret_from_fork+0x1f/0x30 [ 5868.059532] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5868.064895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5868.124601] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5868.142415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5868.158563] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5868.165442] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5868.190474] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5868.195411] RAX: 0000000000006330 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5868.201209] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5868.216329] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5868.228479] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5868.235868] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5868.244531] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5868.256356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5868.266138] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5868.278802] Call Trace: [ 5868.281144] [ 5868.282375] ? show_regs.cold.9+0x22/0x2f [ 5868.285075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5868.293236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5868.306452] ? watchdog+0x40/0x40 [ 5868.308419] ? hrtimer_interrupt+0x128/0x2c0 [ 5868.315093] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5868.320193] ? apic_timer_interrupt+0xf/0x20 [ 5868.325384] [ 5868.326956] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5868.339476] do_raw_spin_lock+0x177/0x190 [ 5868.344487] _raw_spin_lock+0x21/0x30 [ 5868.347553] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5868.352100] ? do_raw_spin_unlock+0x75/0x190 [ 5868.354969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5868.361077] ? do_wait_intr+0xf0/0xf0 [ 5868.371803] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5868.376235] mdc_close+0x53b/0xd70 [mdc] [ 5868.379898] lmv_close+0x18d/0x490 [lmv] [ 5868.381331] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5868.389156] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5868.390705] ll_md_close+0x274/0x800 [lustre] [ 5868.396099] ll_file_release+0x128/0xc50 [lustre] [ 5868.397990] __fput+0x10d/0x440 [ 5868.400048] ____fput+0x12/0x20 [ 5868.402000] task_work_run+0xc6/0x110 [ 5868.404048] exit_to_usermode_loop+0x1dd/0x1f0 [ 5868.406174] do_syscall_64+0x3d6/0x440 [ 5868.407952] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5868.410249] RIP: 0033:0x7fa749552cd8 [ 5868.411862] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5868.420435] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5868.424716] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5868.429776] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5868.432900] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5868.439748] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5868.443139] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5880.061114] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5880.071920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5880.145572] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5880.167615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5880.182372] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5880.189517] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5880.214241] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5880.225668] RAX: 0000000000002d82 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5880.229801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5880.233940] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5880.237401] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5880.241074] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5880.244659] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5880.252237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5880.255216] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5880.265280] Call Trace: [ 5880.273820] [ 5880.275367] ? show_regs.cold.9+0x22/0x2f [ 5880.281631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5880.283796] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5880.287798] ? watchdog+0x40/0x40 [ 5880.290969] ? hrtimer_interrupt+0x128/0x2c0 [ 5880.296014] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5880.297754] ? apic_timer_interrupt+0xf/0x20 [ 5880.300886] [ 5880.302198] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5880.312451] do_raw_spin_lock+0x177/0x190 [ 5880.318270] _raw_spin_lock+0x21/0x30 [ 5880.326314] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5880.332463] ? do_raw_spin_unlock+0x75/0x190 [ 5880.334377] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5880.336505] ? do_wait_intr+0xf0/0xf0 [ 5880.337876] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5880.339921] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5880.341617] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5880.345731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.347680] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5880.349607] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5880.351625] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5880.353008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.355157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5880.357128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5880.361097] lmv_intent_open+0x338/0x1150 [lmv] [ 5880.362619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.366438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.371575] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5880.374673] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5880.376297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.378813] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5880.381061] ? do_raw_spin_unlock+0x75/0x190 [ 5880.382800] ? _raw_spin_unlock+0x12/0x30 [ 5880.385345] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5880.387787] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5880.389775] ll_atomic_open+0x36a/0xd70 [lustre] [ 5880.391411] lookup_open+0x356/0xb40 [ 5880.392753] do_last+0x313/0xfc0 [ 5880.393928] ? nd_jump_root+0xe5/0x160 [ 5880.395294] ? path_init+0x437/0x520 [ 5880.396295] path_openat+0xf7/0x500 [ 5880.397255] ? do_raw_spin_unlock+0x75/0x190 [ 5880.398561] do_filp_open+0x99/0x140 [ 5880.399579] ? getname_flags+0x6e/0x330 [ 5880.400771] ? __check_object_size+0xff/0x256 [ 5880.402174] ? do_raw_spin_unlock+0x75/0x190 [ 5880.403383] ? _raw_spin_unlock+0x12/0x30 [ 5880.404487] do_sys_openat2+0x2b4/0x410 [ 5880.405591] do_sys_open+0x73/0xa0 [ 5880.406583] __x64_sys_openat+0x24/0x30 [ 5880.407614] do_syscall_64+0xc1/0x440 [ 5880.408651] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5880.410241] RIP: 0033:0x7fa749552332 [ 5880.411299] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5880.416658] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5880.418960] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5880.421040] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5880.423192] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5880.425325] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5880.427532] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5884.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5884.060279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5884.084245] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5884.090094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5884.093246] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5884.096463] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5884.098943] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5884.108178] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5884.112009] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5884.115260] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5884.118482] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5884.122228] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5884.126178] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5884.129490] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5884.134226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5884.137066] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5884.140100] Call Trace: [ 5884.141190] [ 5884.142260] ? show_regs.cold.9+0x22/0x2f [ 5884.144409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5884.146387] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5884.148615] ? watchdog+0x40/0x40 [ 5884.150325] ? hrtimer_interrupt+0x128/0x2c0 [ 5884.152697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5884.155074] ? apic_timer_interrupt+0xf/0x20 [ 5884.157283] [ 5884.158396] ? native_safe_halt+0x2e/0x50 [ 5884.160235] kvm_wait+0xcd/0xe0 [ 5884.161766] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5884.164289] do_raw_spin_lock+0x177/0x190 [ 5884.166079] _raw_spin_lock+0x21/0x30 [ 5884.168219] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5884.170995] process_one_work+0x2c8/0x700 [ 5884.172607] worker_thread+0x296/0x6e0 [ 5884.174867] ? rescuer_thread+0x570/0x570 [ 5884.176862] kthread+0x1d1/0x200 [ 5884.178328] ? set_kthread_struct+0x70/0x70 [ 5884.180326] ret_from_fork+0x1f/0x30 [ 5884.900097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3967s! [ 5884.903083] Showing busy workqueues and worker pools: [ 5884.904308] workqueue events: flags=0x0 [ 5884.906753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5884.909424] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5884.915351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5884.918414] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5884.922489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5884.928033] pending: kfree_rcu_monitor [ 5884.930156] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5884.932969] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5884.941739] workqueue events_long: flags=0x0 [ 5884.944035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5884.946642] pending: lu_objects_flush [obdclass] [ 5884.951029] workqueue events_unbound: flags=0x2 [ 5884.952814] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5884.958981] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5884.964815] workqueue events_power_efficient: flags=0x80 [ 5884.966854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5884.969975] pending: check_lifetime [ 5884.974077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5884.975690] pending: neigh_periodic_work [ 5884.980700] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5884.985935] pending: neigh_periodic_work [ 5884.989663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5884.994079] pending: do_cache_clean [ 5885.000667] workqueue mm_percpu_wq: flags=0x8 [ 5885.006375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5885.014455] pending: vmstat_update [ 5885.016173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5885.018908] pending: vmstat_update [ 5885.020304] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5885.022738] pending: vmstat_update [ 5885.024388] workqueue writeback: flags=0x4a [ 5885.025545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5885.027488] pending: wb_workfn, wb_workfn [ 5885.028940] workqueue xprtiod: flags=0xa [ 5885.030183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5885.033080] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5885.035631] workqueue ipv6_addrconf: flags=0x40008 [ 5885.037142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5885.038929] pending: addrconf_verify_work [ 5885.040309] workqueue knbd0-recv: flags=0x1a [ 5885.041819] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5885.049767] in-flight: 651:recv_work [ 5885.054963] workqueue ptlrpc_pinger: flags=0x2 [ 5885.057460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5885.059748] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5885.067020] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3967s workers=4 idle: 37 31846 28897 [ 5885.079227] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5874s workers=2 idle: 653 [ 5892.062309] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5892.085407] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.159094] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.175153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5892.193296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5892.197621] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5892.252417] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5892.255105] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5892.282295] RDX: 0000000000001d4e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5892.289079] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5892.303412] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5892.315843] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5892.325577] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5892.342294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.351927] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5892.359879] Call Trace: [ 5892.364300] [ 5892.368973] ? show_regs.cold.9+0x22/0x2f [ 5892.380033] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.381709] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.396861] ? watchdog+0x40/0x40 [ 5892.407365] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.417856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.429287] ? apic_timer_interrupt+0xf/0x20 [ 5892.439321] [ 5892.446964] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5892.459203] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5892.467773] do_raw_spin_lock+0x177/0x190 [ 5892.484461] _raw_spin_lock+0x21/0x30 [ 5892.493921] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5892.505820] obd_mod_free+0x46/0x140 [ptlrpc] [ 5892.512229] mdc_commit_open+0x91/0xb0 [mdc] [ 5892.518457] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5892.527266] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5892.533860] after_reply+0xe2c/0x1780 [ptlrpc] [ 5892.543409] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5892.549339] ? __next_timer_interrupt+0x160/0x160 [ 5892.558653] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5892.565525] ? do_wait_intr+0xf0/0xf0 [ 5892.567880] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5892.574896] kthread+0x1d1/0x200 [ 5892.581293] ? set_kthread_struct+0x70/0x70 [ 5892.584163] ret_from_fork+0x1f/0x30 [ 5896.059013] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5896.062094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5896.085339] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5896.112588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5896.127087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5896.130139] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5896.137960] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5896.142168] RAX: 0000000000004485 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5896.145256] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5896.148206] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5896.151970] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5896.155970] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5896.158292] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5896.162644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5896.166171] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5896.169938] Call Trace: [ 5896.171172] [ 5896.172344] ? show_regs.cold.9+0x22/0x2f [ 5896.174301] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5896.177016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5896.179317] ? watchdog+0x40/0x40 [ 5896.180846] ? hrtimer_interrupt+0x128/0x2c0 [ 5896.183166] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5896.185374] ? apic_timer_interrupt+0xf/0x20 [ 5896.189532] [ 5896.193067] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5896.194977] do_raw_spin_lock+0x177/0x190 [ 5896.196110] _raw_spin_lock+0x21/0x30 [ 5896.199590] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5896.205642] ? do_raw_spin_unlock+0x75/0x190 [ 5896.206817] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5896.208530] ? do_wait_intr+0xf0/0xf0 [ 5896.212056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5896.220291] mdc_close+0x53b/0xd70 [mdc] [ 5896.226928] lmv_close+0x18d/0x490 [lmv] [ 5896.231203] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5896.235088] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5896.236834] ll_md_close+0x274/0x800 [lustre] [ 5896.243794] ll_file_release+0x128/0xc50 [lustre] [ 5896.250878] __fput+0x10d/0x440 [ 5896.256304] ____fput+0x12/0x20 [ 5896.260384] task_work_run+0xc6/0x110 [ 5896.263865] exit_to_usermode_loop+0x1dd/0x1f0 [ 5896.265991] do_syscall_64+0x3d6/0x440 [ 5896.267853] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5896.270458] RIP: 0033:0x7fa749552cd8 [ 5896.272037] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5896.280640] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5896.284087] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5896.286836] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5896.289262] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5896.292094] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5896.294320] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5912.056808] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5912.057138] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 5912.060563] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5912.060705] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5912.060713] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5912.060716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5912.060730] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5912.060734] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5912.060738] RAX: 0000000000005a82 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5912.060741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5912.060744] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5912.060746] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5912.060749] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5912.060752] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5912.060754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5912.060756] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5912.060765] Call Trace: [ 5912.060771] [ 5912.060776] ? show_regs.cold.9+0x22/0x2f [ 5912.060786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5912.060793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5912.060797] ? watchdog+0x40/0x40 [ 5912.060802] ? hrtimer_interrupt+0x128/0x2c0 [ 5912.060807] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5912.060814] ? apic_timer_interrupt+0xf/0x20 [ 5912.060819] [ 5912.060821] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5912.060828] do_raw_spin_lock+0x177/0x190 [ 5912.060833] _raw_spin_lock+0x21/0x30 [ 5912.060838] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5912.061112] ? do_raw_spin_unlock+0x75/0x190 [ 5912.061119] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5912.064526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 5912.101911] ? do_wait_intr+0xf0/0xf0 [ 5912.101938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5912.102286] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5912.102702] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5912.102933] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5912.103052] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5912.103093] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5912.103126] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5912.112000] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5912.112111] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5912.112119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5912.112192] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5912.112691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5912.112923] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5912.112928] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5912.112933] RAX: 000000000000529f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5912.112936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5912.112939] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5912.112941] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5912.112944] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5912.112947] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5912.112950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5912.112953] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5912.112961] Call Trace: [ 5912.112966] [ 5912.112971] ? show_regs.cold.9+0x22/0x2f [ 5912.112980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5912.112987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5912.112991] ? watchdog+0x40/0x40 [ 5912.112996] ? hrtimer_interrupt+0x128/0x2c0 [ 5912.113000] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5912.113009] ? apic_timer_interrupt+0xf/0x20 [ 5912.113014] [ 5912.113017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5912.113022] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5912.113028] do_raw_spin_lock+0x177/0x190 [ 5912.113033] _raw_spin_lock+0x21/0x30 [ 5912.113038] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5912.113368] process_one_work+0x2c8/0x700 [ 5912.113375] worker_thread+0x296/0x6e0 [ 5912.113380] ? rescuer_thread+0x570/0x570 [ 5912.113384] kthread+0x1d1/0x200 [ 5912.113389] ? set_kthread_struct+0x70/0x70 [ 5912.113393] ret_from_fork+0x1f/0x30 [ 5912.381500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5912.386522] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5912.390127] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5912.392122] lmv_intent_open+0x338/0x1150 [lmv] [ 5912.396857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5912.403698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5912.412331] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5912.418486] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5912.423277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5912.427008] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5912.430488] ? do_raw_spin_unlock+0x75/0x190 [ 5912.432846] ? _raw_spin_unlock+0x12/0x30 [ 5912.435063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5912.438030] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5912.441981] ll_atomic_open+0x36a/0xd70 [lustre] [ 5912.445541] lookup_open+0x356/0xb40 [ 5912.447540] do_last+0x313/0xfc0 [ 5912.449370] ? nd_jump_root+0xe5/0x160 [ 5912.452254] ? path_init+0x437/0x520 [ 5912.454837] path_openat+0xf7/0x500 [ 5912.456781] ? do_raw_spin_unlock+0x75/0x190 [ 5912.459110] do_filp_open+0x99/0x140 [ 5912.460837] ? getname_flags+0x6e/0x330 [ 5912.462425] ? __check_object_size+0xff/0x256 [ 5912.464568] ? do_raw_spin_unlock+0x75/0x190 [ 5912.466113] ? _raw_spin_unlock+0x12/0x30 [ 5912.468019] do_sys_openat2+0x2b4/0x410 [ 5912.473024] do_sys_open+0x73/0xa0 [ 5912.475726] __x64_sys_openat+0x24/0x30 [ 5912.477918] do_syscall_64+0xc1/0x440 [ 5912.480057] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5912.482802] RIP: 0033:0x7fa749552332 [ 5912.485016] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5912.500889] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5912.508672] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5912.514896] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5912.521386] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5912.527945] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5912.531781] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5915.104743] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3997s! [ 5915.111563] Showing busy workqueues and worker pools: [ 5915.118153] workqueue events: flags=0x0 [ 5915.119484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5915.122298] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5915.132234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5915.136926] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5915.146293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.150235] pending: kfree_rcu_monitor [ 5915.155782] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5915.162901] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5915.171497] workqueue events_long: flags=0x0 [ 5915.174683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.190679] pending: lu_objects_flush [obdclass] [ 5915.192387] workqueue events_unbound: flags=0x2 [ 5915.201557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5915.204636] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5915.209469] workqueue events_power_efficient: flags=0x80 [ 5915.211621] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.214550] pending: check_lifetime [ 5915.215799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.220653] pending: neigh_periodic_work [ 5915.222289] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.226574] pending: neigh_periodic_work [ 5915.230435] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.235474] pending: do_cache_clean [ 5915.237594] workqueue mm_percpu_wq: flags=0x8 [ 5915.239503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.249495] pending: vmstat_update [ 5915.252236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.255604] pending: vmstat_update [ 5915.258675] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5915.266810] pending: vmstat_update [ 5915.268014] workqueue writeback: flags=0x4a [ 5915.271216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5915.275921] pending: wb_workfn, wb_workfn [ 5915.278741] workqueue xprtiod: flags=0xa [ 5915.280906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5915.283867] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5915.288597] workqueue ipv6_addrconf: flags=0x40008 [ 5915.290936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5915.299548] pending: addrconf_verify_work [ 5915.301602] workqueue knbd0-recv: flags=0x1a [ 5915.306322] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5915.312334] in-flight: 651:recv_work [ 5915.319253] workqueue ptlrpc_pinger: flags=0x2 [ 5915.320786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5915.324565] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5915.328772] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3998s workers=4 idle: 37 31846 28897 [ 5915.335791] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5905s workers=2 idle: 653 [ 5920.058590] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5920.063745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.098662] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.107805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5920.114766] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5920.117777] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5920.129175] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5920.132635] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5920.136646] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5920.141646] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5920.151568] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5920.153911] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5920.162833] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5920.171871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.181108] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5920.188691] Call Trace: [ 5920.190071] [ 5920.191328] ? show_regs.cold.9+0x22/0x2f [ 5920.193176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.195406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.197223] ? watchdog+0x40/0x40 [ 5920.202618] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.204817] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.208447] ? apic_timer_interrupt+0xf/0x20 [ 5920.211773] [ 5920.212825] ? native_safe_halt+0x2e/0x50 [ 5920.215060] kvm_wait+0xcd/0xe0 [ 5920.216730] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5920.218873] do_raw_spin_lock+0x177/0x190 [ 5920.220102] _raw_spin_lock+0x21/0x30 [ 5920.221852] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5920.224384] obd_mod_free+0x46/0x140 [ptlrpc] [ 5920.226664] mdc_commit_open+0x91/0xb0 [mdc] [ 5920.228471] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5920.230934] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5920.233693] after_reply+0xe2c/0x1780 [ptlrpc] [ 5920.236579] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5920.241198] ? __next_timer_interrupt+0x160/0x160 [ 5920.246171] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5920.250380] ? do_wait_intr+0xf0/0xf0 [ 5920.252185] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5920.258883] kthread+0x1d1/0x200 [ 5920.260891] ? set_kthread_struct+0x70/0x70 [ 5920.262909] ret_from_fork+0x1f/0x30 [ 5924.059177] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5924.067918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5924.099816] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5924.104304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5924.112006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5924.115371] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5924.132670] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5924.137329] RAX: 0000000000005984 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5924.146639] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5924.152303] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5924.157186] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5924.162184] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5924.166576] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5924.171768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5924.175082] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5924.178980] Call Trace: [ 5924.180383] [ 5924.181699] ? show_regs.cold.9+0x22/0x2f [ 5924.185770] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5924.189878] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5924.195647] ? watchdog+0x40/0x40 [ 5924.197744] ? hrtimer_interrupt+0x128/0x2c0 [ 5924.201553] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5924.206358] ? apic_timer_interrupt+0xf/0x20 [ 5924.209226] [ 5924.211177] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5924.215997] do_raw_spin_lock+0x177/0x190 [ 5924.222305] _raw_spin_lock+0x21/0x30 [ 5924.226701] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5924.233153] ? do_raw_spin_unlock+0x75/0x190 [ 5924.238715] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5924.245916] ? do_wait_intr+0xf0/0xf0 [ 5924.250037] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5924.255702] mdc_close+0x53b/0xd70 [mdc] [ 5924.260798] lmv_close+0x18d/0x490 [lmv] [ 5924.262709] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5924.269373] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5924.272656] ll_md_close+0x274/0x800 [lustre] [ 5924.280292] ll_file_release+0x128/0xc50 [lustre] [ 5924.286228] __fput+0x10d/0x440 [ 5924.291619] ____fput+0x12/0x20 [ 5924.296639] task_work_run+0xc6/0x110 [ 5924.299376] exit_to_usermode_loop+0x1dd/0x1f0 [ 5924.303347] do_syscall_64+0x3d6/0x440 [ 5924.305720] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5924.309069] RIP: 0033:0x7fa749552cd8 [ 5924.310730] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5924.326042] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5924.330970] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5924.335816] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5924.342272] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5924.346000] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5924.350629] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5933.772244] rcu: INFO: rcu_sched self-detected stall on CPU [ 5933.774722] rcu: 2-...!: (2220933 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 5933.785792] (t=4020083 jiffies g=258701 q=2651) [ 5933.788425] rcu: rcu_sched kthread starved for 4016878 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5933.793687] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5933.798448] rcu: RCU grace-period kthread stack dump: [ 5933.800725] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5933.804765] Call Trace: [ 5933.805885] __schedule+0x351/0xcb0 [ 5933.807366] schedule+0xc0/0x180 [ 5933.808514] schedule_timeout+0xb4/0x190 [ 5933.810374] ? do_raw_spin_unlock+0x75/0x190 [ 5933.812226] ? __next_timer_interrupt+0x160/0x160 [ 5933.820008] ? prepare_to_swait_event+0xf3/0x140 [ 5933.822557] rcu_gp_kthread+0x8dd/0x15b0 [ 5933.826025] ? force_qs_rnp+0x2c0/0x2c0 [ 5933.827407] kthread+0x1d1/0x200 [ 5933.830990] ? set_kthread_struct+0x70/0x70 [ 5933.835068] ret_from_fork+0x1f/0x30 [ 5933.837340] rcu: Stack dump where RCU GP kthread last ran: [ 5933.841553] Sending NMI from CPU 2 to CPUs 1: [ 5933.849099] NMI backtrace for cpu 1 [ 5933.849106] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5933.849109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5933.849111] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5933.849113] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5933.849115] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5933.849117] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 [ 5933.849119] RAX: 0000000000000821 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5933.849121] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5933.849122] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5933.849124] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5933.849125] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5933.849127] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5933.849128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5933.849130] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5933.849131] Call Trace: [ 5933.849132] [ 5933.849133] ? show_regs.cold.9+0x22/0x2f [ 5933.849134] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5933.849135] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5933.849137] ? nmi_handle+0xbd/0x210 [ 5933.849138] ? default_do_nmi+0x4b/0x1b0 [ 5933.849139] ? do_nmi+0x280/0x380 [ 5933.849140] ? end_repeat_nmi+0x16/0x1f [ 5933.849141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5933.849143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5933.849144] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5933.849145] [ 5933.849146] do_raw_spin_lock+0x177/0x190 [ 5933.849147] _raw_spin_lock+0x21/0x30 [ 5933.849148] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5933.849149] process_one_work+0x2c8/0x700 [ 5933.849150] worker_thread+0x296/0x6e0 [ 5933.849151] ? rescuer_thread+0x570/0x570 [ 5933.849153] kthread+0x1d1/0x200 [ 5933.849154] ? set_kthread_struct+0x70/0x70 [ 5933.849155] ret_from_fork+0x1f/0x30 [ 5933.849866] NMI backtrace for cpu 2 [ 5933.989184] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5933.994603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5933.998920] Call Trace: [ 5934.000909] [ 5934.004485] ? dump_stack+0xbb/0x10e [ 5934.009352] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5934.012129] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5934.017415] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5934.022339] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5934.027591] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5934.030550] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5934.032107] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5934.040955] ? update_process_times+0x6d/0xc0 [ 5934.044921] ? tick_sched_handle.isra.18+0x27/0x90 [ 5934.049548] ? tick_sched_timer+0x53/0xb0 [ 5934.054365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5934.055995] ? tick_sched_handle.isra.18+0x90/0x90 [ 5934.058397] ? hrtimer_interrupt+0x128/0x2c0 [ 5934.060476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5934.065850] ? apic_timer_interrupt+0xf/0x20 [ 5934.073941] [ 5934.074827] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5934.076574] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5934.082019] ? do_raw_spin_lock+0x177/0x190 [ 5934.084482] ? _raw_spin_lock+0x21/0x30 [ 5934.086002] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5934.092126] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5934.097733] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5934.100680] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5934.104918] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5934.111990] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5934.117883] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5934.122583] ? __next_timer_interrupt+0x160/0x160 [ 5934.125005] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5934.127442] ? do_wait_intr+0xf0/0xf0 [ 5934.129383] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5934.132073] ? kthread+0x1d1/0x200 [ 5934.133527] ? set_kthread_struct+0x70/0x70 [ 5934.135459] ? ret_from_fork+0x1f/0x30 [ 5940.055594] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 5940.065540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5940.105464] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5940.112994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5940.119619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.128912] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5940.139199] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5940.141874] RAX: 000000000000682f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5940.145008] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5940.148291] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5940.151241] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5940.154410] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5940.157592] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5940.161329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5940.163863] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5940.169026] Call Trace: [ 5940.171272] [ 5940.175266] ? show_regs.cold.9+0x22/0x2f [ 5940.177074] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5940.179440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5940.181997] ? watchdog+0x40/0x40 [ 5940.184285] ? hrtimer_interrupt+0x128/0x2c0 [ 5940.186901] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5940.189818] ? apic_timer_interrupt+0xf/0x20 [ 5940.192072] [ 5940.193365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.196316] do_raw_spin_lock+0x177/0x190 [ 5940.198787] _raw_spin_lock+0x21/0x30 [ 5940.200888] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5940.204633] ? do_raw_spin_unlock+0x75/0x190 [ 5940.207339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5940.210804] ? do_wait_intr+0xf0/0xf0 [ 5940.213818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5940.216889] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5940.220412] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5940.224393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5940.226942] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5940.229066] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5940.231087] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5940.233214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5940.236303] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5940.239861] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5940.247430] lmv_intent_open+0x338/0x1150 [lmv] [ 5940.251380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5940.255064] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5940.259459] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5940.262724] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5940.266495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5940.274089] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5940.283049] ? do_raw_spin_unlock+0x75/0x190 [ 5940.286226] ? _raw_spin_unlock+0x12/0x30 [ 5940.289819] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5940.293369] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5940.296872] ll_atomic_open+0x36a/0xd70 [lustre] [ 5940.300473] lookup_open+0x356/0xb40 [ 5940.303069] do_last+0x313/0xfc0 [ 5940.305386] ? nd_jump_root+0xe5/0x160 [ 5940.308713] ? path_init+0x437/0x520 [ 5940.311571] path_openat+0xf7/0x500 [ 5940.315067] ? do_raw_spin_unlock+0x75/0x190 [ 5940.317505] do_filp_open+0x99/0x140 [ 5940.322350] ? getname_flags+0x6e/0x330 [ 5940.325286] ? __check_object_size+0xff/0x256 [ 5940.328307] ? do_raw_spin_unlock+0x75/0x190 [ 5940.330325] ? _raw_spin_unlock+0x12/0x30 [ 5940.331844] do_sys_openat2+0x2b4/0x410 [ 5940.334823] do_sys_open+0x73/0xa0 [ 5940.337901] __x64_sys_openat+0x24/0x30 [ 5940.340278] do_syscall_64+0xc1/0x440 [ 5940.344327] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5940.347221] RIP: 0033:0x7fa749552332 [ 5940.348394] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5940.368084] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5940.373051] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5940.377445] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5940.381940] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5940.386483] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5940.394272] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5945.830325] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4028s! [ 5945.849974] Showing busy workqueues and worker pools: [ 5945.852763] workqueue events: flags=0x0 [ 5945.859968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5945.865926] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5945.882328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5945.890791] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5945.906746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5945.910511] pending: kfree_rcu_monitor [ 5945.919753] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5945.926145] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5945.937021] workqueue events_long: flags=0x0 [ 5945.939891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5945.953136] pending: lu_objects_flush [obdclass] [ 5945.956960] workqueue events_unbound: flags=0x2 [ 5945.964806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5945.974818] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5945.981963] workqueue events_power_efficient: flags=0x80 [ 5945.993734] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5946.000718] pending: check_lifetime [ 5946.002819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5946.017171] pending: neigh_periodic_work [ 5946.020384] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5946.030581] pending: neigh_periodic_work [ 5946.035246] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5946.043242] pending: do_cache_clean [ 5946.047224] workqueue mm_percpu_wq: flags=0x8 [ 5946.051973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5946.058281] pending: vmstat_update [ 5946.059657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5946.079974] pending: vmstat_update [ 5946.087765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5946.097961] pending: vmstat_update [ 5946.104341] workqueue writeback: flags=0x4a [ 5946.107841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5946.113120] pending: wb_workfn, wb_workfn [ 5946.120358] workqueue xprtiod: flags=0xa [ 5946.129976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5946.142149] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5946.155085] workqueue ipv6_addrconf: flags=0x40008 [ 5946.164518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5946.179874] pending: addrconf_verify_work [ 5946.187014] workqueue knbd0-recv: flags=0x1a [ 5946.188506] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5946.200503] in-flight: 651:recv_work [ 5946.206941] workqueue ptlrpc_pinger: flags=0x2 [ 5946.210276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5946.221837] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5946.233775] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4029s workers=4 idle: 37 31846 28897 [ 5946.240512] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5936s workers=2 idle: 653 [ 5952.059476] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 5952.064061] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5952.108691] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5952.112233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5952.117770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5952.119677] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5952.124957] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5952.130299] RAX: 0000000000004c74 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5952.132786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5952.136999] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5952.138569] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5952.142700] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5952.144321] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5952.147087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5952.150277] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5952.152088] Call Trace: [ 5952.155415] [ 5952.155908] ? show_regs.cold.9+0x22/0x2f [ 5952.157085] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5952.162889] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5952.168006] ? watchdog+0x40/0x40 [ 5952.171232] ? hrtimer_interrupt+0x128/0x2c0 [ 5952.172265] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5952.173333] ? apic_timer_interrupt+0xf/0x20 [ 5952.175440] [ 5952.176409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5952.182924] do_raw_spin_lock+0x177/0x190 [ 5952.188640] _raw_spin_lock+0x21/0x30 [ 5952.193059] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5952.194424] ? do_raw_spin_unlock+0x75/0x190 [ 5952.195472] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5952.197841] ? do_wait_intr+0xf0/0xf0 [ 5952.199194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5952.200552] mdc_close+0x53b/0xd70 [mdc] [ 5952.201452] lmv_close+0x18d/0x490 [lmv] [ 5952.209092] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5952.215319] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5952.218038] ll_md_close+0x274/0x800 [lustre] [ 5952.221418] ll_file_release+0x128/0xc50 [lustre] [ 5952.223174] __fput+0x10d/0x440 [ 5952.224483] ____fput+0x12/0x20 [ 5952.225323] task_work_run+0xc6/0x110 [ 5952.228572] exit_to_usermode_loop+0x1dd/0x1f0 [ 5952.229867] do_syscall_64+0x3d6/0x440 [ 5952.230894] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5952.232678] RIP: 0033:0x7fa749552cd8 [ 5952.233647] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5952.239176] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5952.243134] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5952.244702] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5952.249579] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5952.252857] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5952.257967] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5960.057287] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5960.058020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5960.061106] Modules linked in: [ 5960.065507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 5960.068549] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5960.069414] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5960.069421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5960.069427] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5960.070324] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5960.070336] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5960.070341] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5960.070345] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5960.070348] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5960.070350] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5960.070352] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5960.070354] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5960.070357] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5960.070360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5960.070362] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5960.070371] Call Trace: [ 5960.070376] [ 5960.070382] ? show_regs.cold.9+0x22/0x2f [ 5960.070392] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5960.070399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5960.070403] ? watchdog+0x40/0x40 [ 5960.070408] ? hrtimer_interrupt+0x128/0x2c0 [ 5960.070412] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5960.070418] ? apic_timer_interrupt+0xf/0x20 [ 5960.070423] [ 5960.070426] ? native_safe_halt+0x2e/0x50 [ 5960.070430] kvm_wait+0xcd/0xe0 [ 5960.070438] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5960.070447] do_raw_spin_lock+0x177/0x190 [ 5960.070452] _raw_spin_lock+0x21/0x30 [ 5960.070456] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5960.071290] process_one_work+0x2c8/0x700 [ 5960.071299] worker_thread+0x296/0x6e0 [ 5960.071303] ? rescuer_thread+0x570/0x570 [ 5960.071312] kthread+0x1d1/0x200 [ 5960.071317] ? set_kthread_struct+0x70/0x70 [ 5960.071320] ret_from_fork+0x1f/0x30 [ 5960.325768] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5960.359104] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5960.367036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5960.376350] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5960.379355] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5960.392066] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5960.399227] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5960.410481] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5960.420207] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5960.425366] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5960.430753] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5960.437136] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5960.440277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5960.447451] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5960.452403] Call Trace: [ 5960.455279] [ 5960.456731] ? show_regs.cold.9+0x22/0x2f [ 5960.459204] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5960.471270] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5960.478485] ? watchdog+0x40/0x40 [ 5960.486161] ? hrtimer_interrupt+0x128/0x2c0 [ 5960.490239] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5960.499430] ? apic_timer_interrupt+0xf/0x20 [ 5960.502663] [ 5960.504204] ? native_safe_halt+0x2e/0x50 [ 5960.511962] kvm_wait+0xcd/0xe0 [ 5960.516406] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5960.523634] do_raw_spin_lock+0x177/0x190 [ 5960.537269] _raw_spin_lock+0x21/0x30 [ 5960.540190] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5960.543548] obd_mod_free+0x46/0x140 [ptlrpc] [ 5960.548054] mdc_commit_open+0x91/0xb0 [mdc] [ 5960.549888] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5960.552326] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5960.555405] after_reply+0xe2c/0x1780 [ptlrpc] [ 5960.558049] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5960.563256] ? __next_timer_interrupt+0x160/0x160 [ 5960.567859] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5960.571031] ? do_wait_intr+0xf0/0xf0 [ 5960.572943] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5960.579396] kthread+0x1d1/0x200 [ 5960.583448] ? set_kthread_struct+0x70/0x70 [ 5960.586382] ret_from_fork+0x1f/0x30 [ 5972.054109] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 5972.060631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5972.083213] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5972.089620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5972.094661] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5972.097969] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5972.108755] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5972.116192] RAX: 0000000000005385 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5972.124310] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 5972.128131] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 5972.131580] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 5972.134751] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 5972.142695] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 5972.146603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5972.149704] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 5972.154502] Call Trace: [ 5972.155572] [ 5972.156290] ? show_regs.cold.9+0x22/0x2f [ 5972.158684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5972.160810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5972.162787] ? watchdog+0x40/0x40 [ 5972.164741] ? hrtimer_interrupt+0x128/0x2c0 [ 5972.166563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5972.168578] ? apic_timer_interrupt+0xf/0x20 [ 5972.170556] [ 5972.171636] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5972.174354] do_raw_spin_lock+0x177/0x190 [ 5972.176333] _raw_spin_lock+0x21/0x30 [ 5972.178131] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5972.180884] ? do_raw_spin_unlock+0x75/0x190 [ 5972.182877] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5972.185411] ? do_wait_intr+0xf0/0xf0 [ 5972.186877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5972.189608] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5972.191941] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5972.194910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5972.198517] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5972.200359] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5972.202066] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5972.204090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5972.206754] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5972.209741] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5972.215731] lmv_intent_open+0x338/0x1150 [lmv] [ 5972.217687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5972.220354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5972.222950] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5972.225329] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5972.227756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5972.230071] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5972.232382] ? do_raw_spin_unlock+0x75/0x190 [ 5972.234630] ? _raw_spin_unlock+0x12/0x30 [ 5972.236615] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5972.238871] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5972.241608] ll_atomic_open+0x36a/0xd70 [lustre] [ 5972.243714] lookup_open+0x356/0xb40 [ 5972.245433] do_last+0x313/0xfc0 [ 5972.246692] ? nd_jump_root+0xe5/0x160 [ 5972.248547] ? path_init+0x437/0x520 [ 5972.250067] path_openat+0xf7/0x500 [ 5972.251885] ? do_raw_spin_unlock+0x75/0x190 [ 5972.254353] do_filp_open+0x99/0x140 [ 5972.256055] ? getname_flags+0x6e/0x330 [ 5972.257770] ? __check_object_size+0xff/0x256 [ 5972.259700] ? do_raw_spin_unlock+0x75/0x190 [ 5972.261902] ? _raw_spin_unlock+0x12/0x30 [ 5972.263743] do_sys_openat2+0x2b4/0x410 [ 5972.266195] do_sys_open+0x73/0xa0 [ 5972.267646] __x64_sys_openat+0x24/0x30 [ 5972.269746] do_syscall_64+0xc1/0x440 [ 5972.271624] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5972.274495] RIP: 0033:0x7fa749552332 [ 5972.275909] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5972.290193] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5972.294450] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 5972.297957] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 5972.302877] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 5972.306917] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 5972.310591] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 5976.552427] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4059s! [ 5976.566200] Showing busy workqueues and worker pools: [ 5976.569235] workqueue events: flags=0x0 [ 5976.571294] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5976.575206] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 5976.583311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5976.591387] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5976.599400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.613278] pending: kfree_rcu_monitor [ 5976.618929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5976.621256] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 5976.634802] workqueue events_long: flags=0x0 [ 5976.638641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.645788] pending: lu_objects_flush [obdclass] [ 5976.652335] workqueue events_unbound: flags=0x2 [ 5976.660322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 5976.673080] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5976.688730] workqueue events_power_efficient: flags=0x80 [ 5976.690124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.712115] pending: check_lifetime [ 5976.718628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.721390] pending: neigh_periodic_work [ 5976.723089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.732376] pending: neigh_periodic_work [ 5976.739825] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.744392] pending: do_cache_clean [ 5976.750089] workqueue mm_percpu_wq: flags=0x8 [ 5976.751528] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.757984] pending: vmstat_update [ 5976.762327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.765666] pending: vmstat_update [ 5976.771416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5976.776616] pending: vmstat_update [ 5976.779745] workqueue writeback: flags=0x4a [ 5976.781629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5976.786522] pending: wb_workfn, wb_workfn [ 5976.791683] workqueue xprtiod: flags=0xa [ 5976.794647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5976.800228] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5976.809635] workqueue ipv6_addrconf: flags=0x40008 [ 5976.814198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5976.820210] pending: addrconf_verify_work [ 5976.824179] workqueue knbd0-recv: flags=0x1a [ 5976.827218] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5976.833164] in-flight: 651:recv_work [ 5976.836620] workqueue ptlrpc_pinger: flags=0x2 [ 5976.838682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5976.846948] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5976.858822] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4059s workers=4 idle: 37 31846 28897 [ 5976.867611] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5966s workers=2 idle: 653 [ 5980.059103] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 5980.062533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5980.084748] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5980.090353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5980.094480] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5980.096724] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5980.106605] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5980.110135] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5980.113806] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 5980.116907] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5980.120528] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 5980.123893] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 5980.127513] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 5980.131668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5980.133929] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 5980.137562] Call Trace: [ 5980.138776] [ 5980.139753] ? show_regs.cold.9+0x22/0x2f [ 5980.141582] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5980.143665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5980.145431] ? watchdog+0x40/0x40 [ 5980.147174] ? hrtimer_interrupt+0x128/0x2c0 [ 5980.149131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5980.151535] ? apic_timer_interrupt+0xf/0x20 [ 5980.153595] [ 5980.154549] ? native_safe_halt+0x2e/0x50 [ 5980.156143] kvm_wait+0xcd/0xe0 [ 5980.157719] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5980.160380] do_raw_spin_lock+0x177/0x190 [ 5980.162327] _raw_spin_lock+0x21/0x30 [ 5980.163887] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5980.166510] ? do_raw_spin_unlock+0x75/0x190 [ 5980.168519] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5980.171535] ? do_wait_intr+0xf0/0xf0 [ 5980.173002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5980.175677] mdc_close+0x53b/0xd70 [mdc] [ 5980.177622] lmv_close+0x18d/0x490 [lmv] [ 5980.179596] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5980.182520] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5980.185061] ll_md_close+0x274/0x800 [lustre] [ 5980.187403] ll_file_release+0x128/0xc50 [lustre] [ 5980.189961] __fput+0x10d/0x440 [ 5980.191667] ____fput+0x12/0x20 [ 5980.192931] task_work_run+0xc6/0x110 [ 5980.194823] exit_to_usermode_loop+0x1dd/0x1f0 [ 5980.197068] do_syscall_64+0x3d6/0x440 [ 5980.198831] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5980.201521] RIP: 0033:0x7fa749552cd8 [ 5980.203216] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5980.212419] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5980.216064] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 5980.219755] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 5980.223003] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 5980.226597] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 5980.229607] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5988.058206] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 5988.064147] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 5988.066928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5988.067008] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5988.067015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5988.067018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5988.067032] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5988.067036] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5988.067041] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 5988.067044] RDX: 0000000000007aac RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 5988.067046] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 5988.067049] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 5988.067052] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5988.067055] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 5988.067058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5988.067061] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 5988.067070] Call Trace: [ 5988.067075] [ 5988.067082] ? show_regs.cold.9+0x22/0x2f [ 5988.067103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5988.067109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5988.067114] ? watchdog+0x40/0x40 [ 5988.067118] ? hrtimer_interrupt+0x128/0x2c0 [ 5988.067122] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5988.067131] ? apic_timer_interrupt+0xf/0x20 [ 5988.067135] [ 5988.067138] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5988.067144] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5988.067149] do_raw_spin_lock+0x177/0x190 [ 5988.067153] _raw_spin_lock+0x21/0x30 [ 5988.067160] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5988.068158] obd_mod_free+0x46/0x140 [ptlrpc] [ 5988.068469] mdc_commit_open+0x91/0xb0 [mdc] [ 5988.068787] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5988.069295] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5988.069491] after_reply+0xe2c/0x1780 [ptlrpc] [ 5988.069697] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5988.070492] ? __next_timer_interrupt+0x160/0x160 [ 5988.070561] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5988.070800] ? do_wait_intr+0xf0/0xf0 [ 5988.070810] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5988.083249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5988.083326] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5988.083334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5988.083341] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5988.085332] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5988.085346] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5988.085350] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5988.085354] RAX: 0000000000007905 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 5988.085357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 5988.086941] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5988.086944] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 5988.086947] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 5988.086951] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 5988.086954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5988.086957] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 5988.086968] Call Trace: [ 5988.086973] [ 5988.086978] ? show_regs.cold.9+0x22/0x2f [ 5988.086989] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5988.086996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5988.087000] ? watchdog+0x40/0x40 [ 5988.087005] ? hrtimer_interrupt+0x128/0x2c0 [ 5988.087009] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5988.087017] ? apic_timer_interrupt+0xf/0x20 [ 5988.087022] [ 5988.087024] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5988.087030] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5988.087120] do_raw_spin_lock+0x177/0x190 [ 5988.087127] _raw_spin_lock+0x21/0x30 [ 5988.087132] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5988.087687] process_one_work+0x2c8/0x700 [ 5988.087695] worker_thread+0x296/0x6e0 [ 5988.087700] ? rescuer_thread+0x570/0x570 [ 5988.087704] kthread+0x1d1/0x200 [ 5988.087709] ? set_kthread_struct+0x70/0x70 [ 5988.087712] ret_from_fork+0x1f/0x30 [ 5988.608778] kthread+0x1d1/0x200 [ 5988.610768] ? set_kthread_struct+0x70/0x70 [ 5988.613803] ret_from_fork+0x1f/0x30 [ 6004.054241] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6004.060418] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6004.114569] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6004.131171] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6004.150091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6004.158609] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6004.181494] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6004.190133] RAX: 0000000000001d84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6004.197771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6004.204427] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6004.209157] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6004.223283] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6004.231426] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6004.238306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6004.247369] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6004.257239] Call Trace: [ 6004.258907] [ 6004.263690] ? show_regs.cold.9+0x22/0x2f [ 6004.265759] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6004.273541] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6004.294293] ? watchdog+0x40/0x40 [ 6004.296074] ? hrtimer_interrupt+0x128/0x2c0 [ 6004.304314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6004.309866] ? apic_timer_interrupt+0xf/0x20 [ 6004.316757] [ 6004.318680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6004.326655] do_raw_spin_lock+0x177/0x190 [ 6004.333559] _raw_spin_lock+0x21/0x30 [ 6004.341428] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6004.356117] ? do_raw_spin_unlock+0x75/0x190 [ 6004.369402] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6004.375676] ? do_wait_intr+0xf0/0xf0 [ 6004.383217] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6004.397813] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6004.409219] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6004.422102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6004.429414] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6004.431525] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6004.449053] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6004.464430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6004.469491] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6004.478459] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6004.484880] lmv_intent_open+0x338/0x1150 [lmv] [ 6004.496874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6004.501268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6004.508449] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6004.515140] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6004.517762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6004.522178] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6004.528320] ? do_raw_spin_unlock+0x75/0x190 [ 6004.530310] ? _raw_spin_unlock+0x12/0x30 [ 6004.536111] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6004.538082] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6004.546019] ll_atomic_open+0x36a/0xd70 [lustre] [ 6004.553884] lookup_open+0x356/0xb40 [ 6004.560296] do_last+0x313/0xfc0 [ 6004.564533] ? nd_jump_root+0xe5/0x160 [ 6004.565983] ? path_init+0x437/0x520 [ 6004.571883] path_openat+0xf7/0x500 [ 6004.577418] ? do_raw_spin_unlock+0x75/0x190 [ 6004.583381] do_filp_open+0x99/0x140 [ 6004.585084] ? getname_flags+0x6e/0x330 [ 6004.593719] ? __check_object_size+0xff/0x256 [ 6004.595579] ? do_raw_spin_unlock+0x75/0x190 [ 6004.605712] ? _raw_spin_unlock+0x12/0x30 [ 6004.607745] do_sys_openat2+0x2b4/0x410 [ 6004.619912] do_sys_open+0x73/0xa0 [ 6004.622409] __x64_sys_openat+0x24/0x30 [ 6004.628235] do_syscall_64+0xc1/0x440 [ 6004.629618] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6004.645091] RIP: 0033:0x7fa749552332 [ 6004.646399] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6004.681775] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6004.693415] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6004.703346] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6004.713149] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6004.728103] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6004.738251] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6007.277864] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4090s! [ 6007.290304] Showing busy workqueues and worker pools: [ 6007.296128] workqueue events: flags=0x0 [ 6007.298428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6007.320092] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6007.337346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6007.342007] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6007.357430] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.374096] pending: kfree_rcu_monitor [ 6007.375426] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6007.389287] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6007.400433] workqueue events_long: flags=0x0 [ 6007.415670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.432783] pending: lu_objects_flush [obdclass] [ 6007.446359] workqueue events_unbound: flags=0x2 [ 6007.451898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6007.465398] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6007.491380] workqueue events_power_efficient: flags=0x80 [ 6007.497287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.517313] pending: check_lifetime [ 6007.518429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.529976] pending: neigh_periodic_work [ 6007.542382] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.552614] pending: neigh_periodic_work [ 6007.562033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.568021] pending: do_cache_clean [ 6007.571905] workqueue mm_percpu_wq: flags=0x8 [ 6007.574349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.580641] pending: vmstat_update [ 6007.586454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.593155] pending: vmstat_update [ 6007.594530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6007.602226] pending: vmstat_update [ 6007.606163] workqueue writeback: flags=0x4a [ 6007.613343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6007.621196] pending: wb_workfn, wb_workfn [ 6007.626233] workqueue xprtiod: flags=0xa [ 6007.629193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6007.636022] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6007.639684] workqueue ipv6_addrconf: flags=0x40008 [ 6007.644876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6007.650225] pending: addrconf_verify_work [ 6007.652686] workqueue knbd0-recv: flags=0x1a [ 6007.655179] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6007.658939] in-flight: 651:recv_work [ 6007.662208] workqueue ptlrpc_pinger: flags=0x2 [ 6007.668309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6007.671157] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6007.675327] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4090s workers=4 idle: 37 31846 28897 [ 6007.678217] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5997s workers=2 idle: 653 [ 6008.061378] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6008.071848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6008.127167] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6008.138993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6008.149226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6008.155076] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6008.169645] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6008.173670] RAX: 000000000000722d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6008.176690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6008.182314] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6008.187000] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6008.190148] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6008.197394] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6008.202649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6008.205050] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6008.208240] Call Trace: [ 6008.209739] [ 6008.210636] ? show_regs.cold.9+0x22/0x2f [ 6008.212796] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6008.215109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6008.221965] ? watchdog+0x40/0x40 [ 6008.226803] ? hrtimer_interrupt+0x128/0x2c0 [ 6008.230103] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6008.236532] ? apic_timer_interrupt+0xf/0x20 [ 6008.240520] [ 6008.244035] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6008.251707] do_raw_spin_lock+0x177/0x190 [ 6008.257577] _raw_spin_lock+0x21/0x30 [ 6008.263203] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6008.269563] ? do_raw_spin_unlock+0x75/0x190 [ 6008.273751] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6008.285343] ? do_wait_intr+0xf0/0xf0 [ 6008.297213] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6008.301068] mdc_close+0x53b/0xd70 [mdc] [ 6008.303153] lmv_close+0x18d/0x490 [lmv] [ 6008.310070] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6008.314005] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6008.323611] ll_md_close+0x274/0x800 [lustre] [ 6008.331465] ll_file_release+0x128/0xc50 [lustre] [ 6008.335685] __fput+0x10d/0x440 [ 6008.339942] ____fput+0x12/0x20 [ 6008.342410] task_work_run+0xc6/0x110 [ 6008.347448] exit_to_usermode_loop+0x1dd/0x1f0 [ 6008.350904] do_syscall_64+0x3d6/0x440 [ 6008.355167] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6008.359459] RIP: 0033:0x7fa749552cd8 [ 6008.363553] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6008.384029] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6008.390977] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6008.395537] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6008.406434] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6008.410696] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6008.421145] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6016.059214] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6016.064362] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6016.064902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6016.064975] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6016.064980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6016.064986] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6016.065276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6016.065288] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6016.065292] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6016.065296] RAX: 0000000000004076 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6016.065299] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6016.065301] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6016.065304] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6016.065306] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6016.065309] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6016.065311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6016.065314] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6016.065322] Call Trace: [ 6016.065328] [ 6016.065333] ? show_regs.cold.9+0x22/0x2f [ 6016.065342] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6016.065349] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6016.065353] ? watchdog+0x40/0x40 [ 6016.065358] ? hrtimer_interrupt+0x128/0x2c0 [ 6016.065362] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6016.065370] ? apic_timer_interrupt+0xf/0x20 [ 6016.065375] [ 6016.065378] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6016.065382] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6016.065388] do_raw_spin_lock+0x177/0x190 [ 6016.065392] _raw_spin_lock+0x21/0x30 [ 6016.065397] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6016.075675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6016.075746] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6016.075753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6016.075756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6016.075770] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6016.075774] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6016.075778] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6016.075780] RDX: 0000000000006073 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6016.075782] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6016.075784] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6016.075786] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6016.075790] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6016.075793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6016.075795] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6016.075804] Call Trace: [ 6016.075811] [ 6016.075817] ? show_regs.cold.9+0x22/0x2f [ 6016.075828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6016.075835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6016.075839] ? watchdog+0x40/0x40 [ 6016.075843] ? hrtimer_interrupt+0x128/0x2c0 [ 6016.075847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6016.075855] ? apic_timer_interrupt+0xf/0x20 [ 6016.075859] [ 6016.075862] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6016.075867] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6016.075872] do_raw_spin_lock+0x177/0x190 [ 6016.075992] _raw_spin_lock+0x21/0x30 [ 6016.076000] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6016.076288] obd_mod_free+0x46/0x140 [ptlrpc] [ 6016.076478] mdc_commit_open+0x91/0xb0 [mdc] [ 6016.076515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6016.122356] process_one_work+0x2c8/0x700 [ 6016.122394] worker_thread+0x296/0x6e0 [ 6016.122399] ? rescuer_thread+0x570/0x570 [ 6016.122403] kthread+0x1d1/0x200 [ 6016.122408] ? set_kthread_struct+0x70/0x70 [ 6016.122412] ret_from_fork+0x1f/0x30 [ 6016.619740] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6016.624338] after_reply+0xe2c/0x1780 [ptlrpc] [ 6016.628421] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6016.636192] ? __next_timer_interrupt+0x160/0x160 [ 6016.640350] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6016.646067] ? do_wait_intr+0xf0/0xf0 [ 6016.649852] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6016.653124] kthread+0x1d1/0x200 [ 6016.659776] ? set_kthread_struct+0x70/0x70 [ 6016.662191] ret_from_fork+0x1f/0x30 [ 6032.054389] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6032.059473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6032.123191] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6032.137962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6032.154078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6032.156308] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6032.188017] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6032.196549] RAX: 0000000000000c7b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6032.208019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6032.215731] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6032.222890] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6032.240898] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6032.251281] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6032.269812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6032.283578] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6032.297684] Call Trace: [ 6032.298598] [ 6032.306680] ? show_regs.cold.9+0x22/0x2f [ 6032.318280] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6032.327410] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6032.336081] ? watchdog+0x40/0x40 [ 6032.344378] ? hrtimer_interrupt+0x128/0x2c0 [ 6032.358181] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6032.359950] ? apic_timer_interrupt+0xf/0x20 [ 6032.371161] [ 6032.374546] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6032.386960] do_raw_spin_lock+0x177/0x190 [ 6032.392256] _raw_spin_lock+0x21/0x30 [ 6032.397946] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6032.404954] ? do_raw_spin_unlock+0x75/0x190 [ 6032.408960] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6032.419936] ? do_wait_intr+0xf0/0xf0 [ 6032.426956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6032.432096] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6032.438911] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6032.443758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6032.453545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6032.463153] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6032.465872] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6032.477738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6032.483328] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6032.493796] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6032.495995] lmv_intent_open+0x338/0x1150 [lmv] [ 6032.504656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6032.513939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6032.517983] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6032.521236] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6032.527895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6032.532134] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6032.534879] ? do_raw_spin_unlock+0x75/0x190 [ 6032.541875] ? _raw_spin_unlock+0x12/0x30 [ 6032.543362] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6032.549948] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6032.557445] ll_atomic_open+0x36a/0xd70 [lustre] [ 6032.560985] lookup_open+0x356/0xb40 [ 6032.564502] do_last+0x313/0xfc0 [ 6032.568331] ? nd_jump_root+0xe5/0x160 [ 6032.577596] ? path_init+0x437/0x520 [ 6032.588189] path_openat+0xf7/0x500 [ 6032.591749] ? do_raw_spin_unlock+0x75/0x190 [ 6032.598777] do_filp_open+0x99/0x140 [ 6032.603822] ? getname_flags+0x6e/0x330 [ 6032.609900] ? __check_object_size+0xff/0x256 [ 6032.614816] ? do_raw_spin_unlock+0x75/0x190 [ 6032.621498] ? _raw_spin_unlock+0x12/0x30 [ 6032.628602] do_sys_openat2+0x2b4/0x410 [ 6032.630022] do_sys_open+0x73/0xa0 [ 6032.637386] __x64_sys_openat+0x24/0x30 [ 6032.641606] do_syscall_64+0xc1/0x440 [ 6032.648267] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6032.654112] RIP: 0033:0x7fa749552332 [ 6032.655243] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6032.679385] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6032.685394] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6032.698803] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6032.707820] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6032.714084] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6032.722233] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6036.059129] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6036.061841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6036.102255] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6036.109917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6036.121344] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6036.124341] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6036.136842] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6036.150201] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6036.160484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6036.170848] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6036.176693] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6036.189618] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6036.193346] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6036.203874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6036.213401] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6036.215009] Call Trace: [ 6036.215618] [ 6036.216098] ? show_regs.cold.9+0x22/0x2f [ 6036.228452] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6036.234593] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6036.241264] ? watchdog+0x40/0x40 [ 6036.242433] ? hrtimer_interrupt+0x128/0x2c0 [ 6036.247698] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6036.252470] ? apic_timer_interrupt+0xf/0x20 [ 6036.253383] [ 6036.253861] ? native_safe_halt+0x2e/0x50 [ 6036.262855] kvm_wait+0xcd/0xe0 [ 6036.264938] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6036.266133] do_raw_spin_lock+0x177/0x190 [ 6036.275431] _raw_spin_lock+0x21/0x30 [ 6036.278404] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6036.285821] ? do_raw_spin_unlock+0x75/0x190 [ 6036.289185] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6036.294198] ? do_wait_intr+0xf0/0xf0 [ 6036.295074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6036.300412] mdc_close+0x53b/0xd70 [mdc] [ 6036.303107] lmv_close+0x18d/0x490 [lmv] [ 6036.304597] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6036.310502] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6036.312474] ll_md_close+0x274/0x800 [lustre] [ 6036.317431] ll_file_release+0x128/0xc50 [lustre] [ 6036.319726] __fput+0x10d/0x440 [ 6036.321664] ____fput+0x12/0x20 [ 6036.323287] task_work_run+0xc6/0x110 [ 6036.324725] exit_to_usermode_loop+0x1dd/0x1f0 [ 6036.327115] do_syscall_64+0x3d6/0x440 [ 6036.328511] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6036.335767] RIP: 0033:0x7fa749552cd8 [ 6036.336659] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6036.350440] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6036.358572] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6036.365981] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6036.367537] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6036.374834] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6036.377716] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6037.986394] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4120s! [ 6038.000831] Showing busy workqueues and worker pools: [ 6038.005591] workqueue events: flags=0x0 [ 6038.013511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6038.022345] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6038.036437] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6038.049908] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6038.062682] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.075178] pending: kfree_rcu_monitor [ 6038.082438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6038.089348] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6038.094065] workqueue events_long: flags=0x0 [ 6038.096071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.103504] pending: lu_objects_flush [obdclass] [ 6038.110906] workqueue events_unbound: flags=0x2 [ 6038.112555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6038.122620] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6038.129923] workqueue events_power_efficient: flags=0x80 [ 6038.131819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.138791] pending: check_lifetime [ 6038.144867] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.150558] pending: neigh_periodic_work [ 6038.152464] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.161883] pending: neigh_periodic_work [ 6038.163403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.176308] pending: do_cache_clean [ 6038.178756] workqueue mm_percpu_wq: flags=0x8 [ 6038.182014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.186405] pending: vmstat_update [ 6038.189550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.193734] pending: vmstat_update [ 6038.196486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6038.208110] pending: vmstat_update [ 6038.217409] workqueue writeback: flags=0x4a [ 6038.219394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6038.227395] pending: wb_workfn, wb_workfn [ 6038.231932] workqueue xprtiod: flags=0xa [ 6038.233670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6038.238377] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6038.241659] workqueue ipv6_addrconf: flags=0x40008 [ 6038.245691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6038.248046] pending: addrconf_verify_work [ 6038.257297] workqueue knbd0-recv: flags=0x1a [ 6038.261356] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6038.267558] in-flight: 651:recv_work [ 6038.270960] workqueue ptlrpc_pinger: flags=0x2 [ 6038.272215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6038.279095] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6038.283418] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4121s workers=4 idle: 37 31846 28897 [ 6038.295190] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6028s workers=2 idle: 653 [ 6044.060533] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6044.064340] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6044.087087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6044.087177] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6044.087184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6044.087190] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6044.087487] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6044.087499] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6044.087503] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6044.087508] RAX: 0000000000007d08 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6044.087510] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6044.087512] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6044.087514] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6044.087517] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6044.087521] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6044.087524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6044.087527] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6044.087537] Call Trace: [ 6044.087544] [ 6044.087550] ? show_regs.cold.9+0x22/0x2f [ 6044.087561] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6044.087568] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6044.087572] ? watchdog+0x40/0x40 [ 6044.087576] ? hrtimer_interrupt+0x128/0x2c0 [ 6044.087580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6044.087588] ? apic_timer_interrupt+0xf/0x20 [ 6044.087593] [ 6044.087606] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6044.087611] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6044.087617] do_raw_spin_lock+0x177/0x190 [ 6044.087622] _raw_spin_lock+0x21/0x30 [ 6044.087627] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6044.087830] process_one_work+0x2c8/0x700 [ 6044.087838] worker_thread+0x296/0x6e0 [ 6044.087842] ? rescuer_thread+0x570/0x570 [ 6044.087846] kthread+0x1d1/0x200 [ 6044.087851] ? set_kthread_struct+0x70/0x70 [ 6044.087854] ret_from_fork+0x1f/0x30 [ 6044.412141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6044.454300] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6044.482645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6044.497701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6044.507076] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6044.533878] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6044.549448] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6044.557448] RDX: 0000000000003d65 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6044.568320] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6044.580205] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6044.589940] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6044.598411] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6044.606417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6044.610954] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6044.618287] Call Trace: [ 6044.619416] [ 6044.621074] ? show_regs.cold.9+0x22/0x2f [ 6044.628600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6044.633578] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6044.636387] ? watchdog+0x40/0x40 [ 6044.638879] ? hrtimer_interrupt+0x128/0x2c0 [ 6044.646486] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6044.655524] ? apic_timer_interrupt+0xf/0x20 [ 6044.662046] [ 6044.663393] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6044.675571] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6044.685401] do_raw_spin_lock+0x177/0x190 [ 6044.689155] _raw_spin_lock+0x21/0x30 [ 6044.692529] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6044.705129] obd_mod_free+0x46/0x140 [ptlrpc] [ 6044.706982] mdc_commit_open+0x91/0xb0 [mdc] [ 6044.716189] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6044.728484] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6044.733920] after_reply+0xe2c/0x1780 [ptlrpc] [ 6044.746137] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6044.760716] ? __next_timer_interrupt+0x160/0x160 [ 6044.767396] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6044.770411] ? do_wait_intr+0xf0/0xf0 [ 6044.779474] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6044.788609] kthread+0x1d1/0x200 [ 6044.789774] ? set_kthread_struct+0x70/0x70 [ 6044.799814] ret_from_fork+0x1f/0x30 [ 6064.059139] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6064.059390] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6064.070414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6064.070494] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6064.070501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6064.070505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6064.070519] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6064.070523] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6064.070528] RAX: 000000000000458a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6064.070530] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6064.070532] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6064.070535] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6064.070538] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6064.070541] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6064.070544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6064.070546] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6064.070556] Call Trace: [ 6064.070561] [ 6064.070567] ? show_regs.cold.9+0x22/0x2f [ 6064.070576] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6064.070583] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6064.070587] ? watchdog+0x40/0x40 [ 6064.070591] ? hrtimer_interrupt+0x128/0x2c0 [ 6064.070595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6064.070603] ? apic_timer_interrupt+0xf/0x20 [ 6064.070607] [ 6064.070611] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6064.070619] do_raw_spin_lock+0x177/0x190 [ 6064.070623] _raw_spin_lock+0x21/0x30 [ 6064.070629] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6064.070924] ? do_raw_spin_unlock+0x75/0x190 [ 6064.070930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6064.071130] ? do_wait_intr+0xf0/0xf0 [ 6064.071139] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6064.088579] Modules linked in: [ 6064.121544] mdc_close+0x53b/0xd70 [mdc] [ 6064.121596] lmv_close+0x18d/0x490 [lmv] [ 6064.121626] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6064.121740] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6064.121830] ll_md_close+0x274/0x800 [lustre] [ 6064.121915] ll_file_release+0x128/0xc50 [lustre] [ 6064.121998] __fput+0x10d/0x440 [ 6064.122009] ____fput+0x12/0x20 [ 6064.122014] task_work_run+0xc6/0x110 [ 6064.122022] exit_to_usermode_loop+0x1dd/0x1f0 [ 6064.122030] do_syscall_64+0x3d6/0x440 [ 6064.122034] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6064.122046] RIP: 0033:0x7fa749552cd8 [ 6064.122053] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6064.122057] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6064.122062] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6064.122064] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6064.122066] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6064.122069] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6064.122071] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6064.554620] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6064.582058] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6064.595576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6064.606388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6064.612703] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6064.638534] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6064.645940] RAX: 0000000000006719 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6064.653151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6064.658090] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6064.668612] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6064.673139] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6064.679854] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6064.684756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6064.700085] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6064.716804] Call Trace: [ 6064.724706] [ 6064.728105] ? show_regs.cold.9+0x22/0x2f [ 6064.736822] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6064.742504] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6064.760952] ? watchdog+0x40/0x40 [ 6064.769206] ? hrtimer_interrupt+0x128/0x2c0 [ 6064.784524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6064.792251] ? apic_timer_interrupt+0xf/0x20 [ 6064.795868] [ 6064.800047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6064.809084] do_raw_spin_lock+0x177/0x190 [ 6064.817870] _raw_spin_lock+0x21/0x30 [ 6064.819991] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6064.830001] ? do_raw_spin_unlock+0x75/0x190 [ 6064.837294] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6064.844834] ? do_wait_intr+0xf0/0xf0 [ 6064.846794] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6064.854829] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6064.862526] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6064.875431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6064.884085] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6064.888105] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6064.893413] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6064.900595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6064.914494] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6064.920314] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6064.933248] lmv_intent_open+0x338/0x1150 [lmv] [ 6064.939606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6064.947773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6064.950634] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6064.975113] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6064.983094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6064.986822] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6064.989362] ? do_raw_spin_unlock+0x75/0x190 [ 6064.994526] ? _raw_spin_unlock+0x12/0x30 [ 6064.997563] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6065.011496] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6065.019085] ll_atomic_open+0x36a/0xd70 [lustre] [ 6065.024634] lookup_open+0x356/0xb40 [ 6065.027886] do_last+0x313/0xfc0 [ 6065.031647] ? nd_jump_root+0xe5/0x160 [ 6065.036191] ? path_init+0x437/0x520 [ 6065.040327] path_openat+0xf7/0x500 [ 6065.041595] ? do_raw_spin_unlock+0x75/0x190 [ 6065.049277] do_filp_open+0x99/0x140 [ 6065.053895] ? getname_flags+0x6e/0x330 [ 6065.057652] ? __check_object_size+0xff/0x256 [ 6065.062107] ? do_raw_spin_unlock+0x75/0x190 [ 6065.064274] ? _raw_spin_unlock+0x12/0x30 [ 6065.065678] do_sys_openat2+0x2b4/0x410 [ 6065.074818] do_sys_open+0x73/0xa0 [ 6065.079714] __x64_sys_openat+0x24/0x30 [ 6065.083762] do_syscall_64+0xc1/0x440 [ 6065.089257] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6065.097232] RIP: 0033:0x7fa749552332 [ 6065.099184] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6065.116265] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6065.123853] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6065.130892] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6065.138476] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6065.149184] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6065.156441] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6068.704087] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4151s! [ 6068.707542] Showing busy workqueues and worker pools: [ 6068.709203] workqueue events: flags=0x0 [ 6068.710418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6068.715020] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6068.721767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6068.724802] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6068.731233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.736139] pending: kfree_rcu_monitor [ 6068.740379] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6068.746145] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6068.756166] workqueue events_long: flags=0x0 [ 6068.758969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.763274] pending: lu_objects_flush [obdclass] [ 6068.766331] workqueue events_unbound: flags=0x2 [ 6068.769191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6068.773230] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6068.778269] workqueue events_power_efficient: flags=0x80 [ 6068.781042] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.784351] pending: check_lifetime [ 6068.786353] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.789915] pending: neigh_periodic_work [ 6068.791992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.795165] pending: neigh_periodic_work [ 6068.796892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.798889] pending: do_cache_clean [ 6068.800013] workqueue mm_percpu_wq: flags=0x8 [ 6068.801792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.804732] pending: vmstat_update [ 6068.806300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.809096] pending: vmstat_update [ 6068.810890] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6068.813715] pending: vmstat_update [ 6068.815310] workqueue writeback: flags=0x4a [ 6068.817048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6068.820143] pending: wb_workfn, wb_workfn [ 6068.822110] workqueue xprtiod: flags=0xa [ 6068.823955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6068.826455] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6068.829870] workqueue ipv6_addrconf: flags=0x40008 [ 6068.831788] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6068.834349] pending: addrconf_verify_work [ 6068.836220] workqueue knbd0-recv: flags=0x1a [ 6068.837995] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6068.840748] in-flight: 651:recv_work [ 6068.842220] workqueue ptlrpc_pinger: flags=0x2 [ 6068.844055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6068.846647] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6068.849146] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4151s workers=4 idle: 37 31846 28897 [ 6068.852075] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6058s workers=2 idle: 653 [ 6072.061020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6072.073539] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6072.079900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6072.080042] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6072.080064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6072.080067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6072.080082] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6072.080086] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6072.080091] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6072.080093] RDX: 000000000000669a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6072.080096] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6072.080098] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6072.080101] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6072.080104] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6072.080107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6072.080110] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6072.080118] Call Trace: [ 6072.080123] [ 6072.080129] ? show_regs.cold.9+0x22/0x2f [ 6072.080139] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6072.080145] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6072.080150] ? watchdog+0x40/0x40 [ 6072.080154] ? hrtimer_interrupt+0x128/0x2c0 [ 6072.080158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6072.080166] ? apic_timer_interrupt+0xf/0x20 [ 6072.080170] [ 6072.080173] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6072.080177] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6072.080181] do_raw_spin_lock+0x177/0x190 [ 6072.080186] _raw_spin_lock+0x21/0x30 [ 6072.080192] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6072.080506] obd_mod_free+0x46/0x140 [ptlrpc] [ 6072.080965] mdc_commit_open+0x91/0xb0 [mdc] [ 6072.081003] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6072.081193] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6072.081598] after_reply+0xe2c/0x1780 [ptlrpc] [ 6072.082656] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6072.083013] ? __next_timer_interrupt+0x160/0x160 [ 6072.083023] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6072.084573] ? do_wait_intr+0xf0/0xf0 [ 6072.084584] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6072.085120] kthread+0x1d1/0x200 [ 6072.085128] ? set_kthread_struct+0x70/0x70 [ 6072.085132] ret_from_fork+0x1f/0x30 [ 6072.619302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6072.672614] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6072.683901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6072.691855] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6072.712149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6072.722326] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6072.739027] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6072.748078] RAX: 0000000000007e16 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6072.762647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6072.772387] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6072.780609] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6072.787496] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6072.798502] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6072.804251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6072.811004] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6072.815267] Call Trace: [ 6072.816474] [ 6072.817265] ? show_regs.cold.9+0x22/0x2f [ 6072.819624] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6072.823191] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6072.827137] ? watchdog+0x40/0x40 [ 6072.828721] ? hrtimer_interrupt+0x128/0x2c0 [ 6072.834422] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6072.837034] ? apic_timer_interrupt+0xf/0x20 [ 6072.843797] [ 6072.844863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6072.851348] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6072.854576] do_raw_spin_lock+0x177/0x190 [ 6072.864183] _raw_spin_lock+0x21/0x30 [ 6072.874613] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6072.878399] process_one_work+0x2c8/0x700 [ 6072.887814] worker_thread+0x296/0x6e0 [ 6072.893832] ? rescuer_thread+0x570/0x570 [ 6072.895825] kthread+0x1d1/0x200 [ 6072.899729] ? set_kthread_struct+0x70/0x70 [ 6072.904826] ret_from_fork+0x1f/0x30 [ 6092.060784] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6092.065527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6092.111370] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6092.125689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6092.132555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6092.142831] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6092.159326] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6092.164927] RAX: 0000000000005585 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6092.169600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6092.176884] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6092.181356] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6092.187276] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6092.192289] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6092.197405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6092.202288] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6092.210364] Call Trace: [ 6092.211737] [ 6092.214191] ? show_regs.cold.9+0x22/0x2f [ 6092.219206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6092.227185] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6092.228759] ? watchdog+0x40/0x40 [ 6092.244371] ? hrtimer_interrupt+0x128/0x2c0 [ 6092.248042] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6092.255737] ? apic_timer_interrupt+0xf/0x20 [ 6092.257964] [ 6092.261343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6092.266740] do_raw_spin_lock+0x177/0x190 [ 6092.268633] _raw_spin_lock+0x21/0x30 [ 6092.278085] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6092.281865] ? do_raw_spin_unlock+0x75/0x190 [ 6092.285942] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6092.296429] ? do_wait_intr+0xf0/0xf0 [ 6092.305214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6092.317716] mdc_close+0x53b/0xd70 [mdc] [ 6092.325191] lmv_close+0x18d/0x490 [lmv] [ 6092.328844] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6092.344243] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6092.355771] ll_md_close+0x274/0x800 [lustre] [ 6092.359599] ll_file_release+0x128/0xc50 [lustre] [ 6092.362779] __fput+0x10d/0x440 [ 6092.366797] ____fput+0x12/0x20 [ 6092.371113] task_work_run+0xc6/0x110 [ 6092.372846] exit_to_usermode_loop+0x1dd/0x1f0 [ 6092.375211] do_syscall_64+0x3d6/0x440 [ 6092.377226] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6092.380454] RIP: 0033:0x7fa749552cd8 [ 6092.383834] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6092.400253] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6092.406886] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6092.412386] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6092.417293] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6092.429510] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6092.433468] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6096.058302] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6096.074506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6096.133470] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6096.151563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6096.166103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6096.169465] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6096.196774] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6096.203346] RAX: 0000000000006eed RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6096.214058] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6096.224344] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6096.231437] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6096.237411] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6096.241740] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6096.251822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6096.259091] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6096.272558] Call Trace: [ 6096.274030] [ 6096.275395] ? show_regs.cold.9+0x22/0x2f [ 6096.284152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6096.287264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6096.297476] ? watchdog+0x40/0x40 [ 6096.300820] ? hrtimer_interrupt+0x128/0x2c0 [ 6096.307316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6096.312264] ? apic_timer_interrupt+0xf/0x20 [ 6096.320327] [ 6096.321790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6096.328349] do_raw_spin_lock+0x177/0x190 [ 6096.331255] _raw_spin_lock+0x21/0x30 [ 6096.336469] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6096.341409] ? do_raw_spin_unlock+0x75/0x190 [ 6096.343278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6096.346171] ? do_wait_intr+0xf0/0xf0 [ 6096.347277] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6096.352682] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6096.356375] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6096.359170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6096.363353] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6096.366717] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6096.369072] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6096.371132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6096.374085] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6096.378374] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6096.384166] lmv_intent_open+0x338/0x1150 [lmv] [ 6096.390430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6096.396451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6096.402144] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6096.418830] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6096.420546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6096.427094] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6096.431460] ? do_raw_spin_unlock+0x75/0x190 [ 6096.433574] ? _raw_spin_unlock+0x12/0x30 [ 6096.436148] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6096.438614] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6096.441576] ll_atomic_open+0x36a/0xd70 [lustre] [ 6096.444366] lookup_open+0x356/0xb40 [ 6096.448436] do_last+0x313/0xfc0 [ 6096.449675] ? nd_jump_root+0xe5/0x160 [ 6096.453283] ? path_init+0x437/0x520 [ 6096.455430] path_openat+0xf7/0x500 [ 6096.459582] ? do_raw_spin_unlock+0x75/0x190 [ 6096.462020] do_filp_open+0x99/0x140 [ 6096.462911] ? getname_flags+0x6e/0x330 [ 6096.464540] ? __check_object_size+0xff/0x256 [ 6096.470648] ? do_raw_spin_unlock+0x75/0x190 [ 6096.473011] ? _raw_spin_unlock+0x12/0x30 [ 6096.475563] do_sys_openat2+0x2b4/0x410 [ 6096.480445] do_sys_open+0x73/0xa0 [ 6096.485506] __x64_sys_openat+0x24/0x30 [ 6096.492220] do_syscall_64+0xc1/0x440 [ 6096.499643] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6096.501983] RIP: 0033:0x7fa749552332 [ 6096.503840] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6096.516582] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6096.521752] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6096.526614] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6096.537138] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6096.542315] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6096.544037] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6098.913024] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4181s! [ 6098.915805] Showing busy workqueues and worker pools: [ 6098.926831] workqueue events: flags=0x0 [ 6098.929195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6098.946576] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6098.961082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6098.980510] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6098.986905] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6098.992911] pending: kfree_rcu_monitor [ 6098.996761] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6099.000435] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6099.009891] workqueue events_long: flags=0x0 [ 6099.010936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6099.017755] pending: lu_objects_flush [obdclass] [ 6099.021626] workqueue events_unbound: flags=0x2 [ 6099.023802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6099.029559] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6099.034453] workqueue events_power_efficient: flags=0x80 [ 6099.046394] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6099.052098] pending: check_lifetime [ 6099.056156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6099.061285] pending: neigh_periodic_work [ 6099.068751] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6099.074129] pending: neigh_periodic_work [ 6099.077097] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6099.080740] pending: do_cache_clean [ 6099.082673] workqueue mm_percpu_wq: flags=0x8 [ 6099.084466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6099.091176] pending: vmstat_update [ 6099.093741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6099.098584] pending: vmstat_update [ 6099.101108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6099.104669] pending: vmstat_update [ 6099.107472] workqueue writeback: flags=0x4a [ 6099.110672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6099.119586] pending: wb_workfn, wb_workfn [ 6099.122779] workqueue xprtiod: flags=0xa [ 6099.128780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6099.131688] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6099.134653] workqueue ipv6_addrconf: flags=0x40008 [ 6099.145789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6099.149847] pending: addrconf_verify_work [ 6099.155289] workqueue knbd0-recv: flags=0x1a [ 6099.157570] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6099.162278] in-flight: 651:recv_work [ 6099.165325] workqueue ptlrpc_pinger: flags=0x2 [ 6099.167765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6099.174012] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6099.180376] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4182s workers=4 idle: 37 31846 28897 [ 6099.184601] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6089s workers=2 idle: 653 [ 6100.057209] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6100.058494] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6100.060208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6100.060284] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6100.060291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6100.060297] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6100.060576] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6100.060586] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6100.060590] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6100.060595] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6100.060597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6100.060599] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6100.060602] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6100.060603] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6100.060606] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6100.060609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6100.060611] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6100.060906] Call Trace: [ 6100.060915] [ 6100.060921] ? show_regs.cold.9+0x22/0x2f [ 6100.060931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6100.060936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6100.060941] ? watchdog+0x40/0x40 [ 6100.060945] ? hrtimer_interrupt+0x128/0x2c0 [ 6100.060948] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6100.060954] ? apic_timer_interrupt+0xf/0x20 [ 6100.060958] [ 6100.060961] ? native_safe_halt+0x2e/0x50 [ 6100.060965] kvm_wait+0xcd/0xe0 [ 6100.061031] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6100.061041] do_raw_spin_lock+0x177/0x190 [ 6100.061046] _raw_spin_lock+0x21/0x30 [ 6100.061050] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6100.061280] process_one_work+0x2c8/0x700 [ 6100.061288] worker_thread+0x296/0x6e0 [ 6100.061292] ? rescuer_thread+0x570/0x570 [ 6100.061296] kthread+0x1d1/0x200 [ 6100.061301] ? set_kthread_struct+0x70/0x70 [ 6100.061304] ret_from_fork+0x1f/0x30 [ 6100.313293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6100.360111] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6100.370033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6100.377723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6100.382369] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6100.402050] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6100.410241] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6100.418925] RDX: 00000000000054e0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6100.423594] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6100.431228] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6100.439337] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6100.450527] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6100.457313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6100.461356] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6100.474893] Call Trace: [ 6100.476565] [ 6100.477663] ? show_regs.cold.9+0x22/0x2f [ 6100.484267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6100.490488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6100.492876] ? watchdog+0x40/0x40 [ 6100.496244] ? hrtimer_interrupt+0x128/0x2c0 [ 6100.498905] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6100.502551] ? apic_timer_interrupt+0xf/0x20 [ 6100.505542] [ 6100.510487] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6100.512955] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6100.520570] do_raw_spin_lock+0x177/0x190 [ 6100.528411] _raw_spin_lock+0x21/0x30 [ 6100.532803] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6100.540974] obd_mod_free+0x46/0x140 [ptlrpc] [ 6100.548427] mdc_commit_open+0x91/0xb0 [mdc] [ 6100.553584] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6100.562101] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6100.571308] after_reply+0xe2c/0x1780 [ptlrpc] [ 6100.578169] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6100.587956] ? __next_timer_interrupt+0x160/0x160 [ 6100.597168] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6100.610349] ? do_wait_intr+0xf0/0xf0 [ 6100.616268] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6100.623293] kthread+0x1d1/0x200 [ 6100.625687] ? set_kthread_struct+0x70/0x70 [ 6100.632449] ret_from_fork+0x1f/0x30 [ 6113.785834] rcu: INFO: rcu_sched self-detected stall on CPU [ 6113.795412] rcu: 2-...!: (2309090 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 6113.802588] (t=4200096 jiffies g=258701 q=2752) [ 6113.809177] rcu: rcu_sched kthread starved for 4196891 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 6113.818895] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6113.822986] rcu: RCU grace-period kthread stack dump: [ 6113.826501] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6113.841581] Call Trace: [ 6113.845436] __schedule+0x351/0xcb0 [ 6113.855051] schedule+0xc0/0x180 [ 6113.860963] schedule_timeout+0xb4/0x190 [ 6113.868294] ? do_raw_spin_unlock+0x75/0x190 [ 6113.874833] ? __next_timer_interrupt+0x160/0x160 [ 6113.879192] ? prepare_to_swait_event+0xf3/0x140 [ 6113.880930] rcu_gp_kthread+0x8dd/0x15b0 [ 6113.882424] ? force_qs_rnp+0x2c0/0x2c0 [ 6113.884471] kthread+0x1d1/0x200 [ 6113.886462] ? set_kthread_struct+0x70/0x70 [ 6113.888943] ret_from_fork+0x1f/0x30 [ 6113.890402] rcu: Stack dump where RCU GP kthread last ran: [ 6113.892551] Sending NMI from CPU 2 to CPUs 0: [ 6113.904737] NMI backtrace for cpu 0 [ 6113.904744] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6113.904747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6113.904748] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6113.904750] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6113.904752] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [ 6113.904755] RAX: 0000000000004551 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6113.904756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6113.904758] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6113.904759] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6113.904760] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6113.904762] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6113.904763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6113.904765] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6113.904766] Call Trace: [ 6113.904767] [ 6113.904768] ? show_regs.cold.9+0x22/0x2f [ 6113.904769] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6113.904770] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6113.904771] ? nmi_handle+0xbd/0x210 [ 6113.904772] ? default_do_nmi+0x4b/0x1b0 [ 6113.904773] ? do_nmi+0x280/0x380 [ 6113.904774] ? end_repeat_nmi+0x16/0x1f [ 6113.904775] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6113.904776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6113.904777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6113.904778] [ 6113.904779] do_raw_spin_lock+0x177/0x190 [ 6113.904780] _raw_spin_lock+0x21/0x30 [ 6113.904782] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6113.904783] ? do_raw_spin_unlock+0x75/0x190 [ 6113.904784] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6113.904785] ? do_wait_intr+0xf0/0xf0 [ 6113.904786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6113.904787] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6113.904788] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6113.904789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6113.904790] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6113.904792] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6113.904793] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6113.904794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6113.904796] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6113.904797] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6113.904798] lmv_intent_open+0x338/0x1150 [lmv] [ 6113.904799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6113.904800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6113.904801] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6113.904803] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6113.904804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6113.904805] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6113.904806] ? do_raw_spin_unlock+0x75/0x190 [ 6113.904807] ? _raw_spin_unlock+0x12/0x30 [ 6113.904808] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6113.904809] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6113.904810] ll_atomic_open+0x36a/0xd70 [lustre] [ 6113.904812] lookup_open+0x356/0xb40 [ 6113.904813] do_last+0x313/0xfc0 [ 6113.904814] ? nd_jump_root+0xe5/0x160 [ 6113.904815] ? path_init+0x437/0x520 [ 6113.904816] path_openat+0xf7/0x500 [ 6113.904817] ? do_raw_spin_unlock+0x75/0x190 [ 6113.904818] do_filp_open+0x99/0x140 [ 6113.904819] ? getname_flags+0x6e/0x330 [ 6113.904820] ? __check_object_size+0xff/0x256 [ 6113.904822] ? do_raw_spin_unlock+0x75/0x190 [ 6113.904822] ? _raw_spin_unlock+0x12/0x30 [ 6113.904824] do_sys_openat2+0x2b4/0x410 [ 6113.904825] do_sys_open+0x73/0xa0 [ 6113.904826] __x64_sys_openat+0x24/0x30 [ 6113.904827] do_syscall_64+0xc1/0x440 [ 6113.904828] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6113.904829] RIP: 0033:0x7fa749552332 [ 6113.904830] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6113.904833] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6113.904836] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6113.904837] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6113.904839] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6113.904841] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6113.904842] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6113.906250] NMI backtrace for cpu 2 [ 6114.376516] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6114.396635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6114.403093] Call Trace: [ 6114.403986] [ 6114.404611] ? dump_stack+0xbb/0x10e [ 6114.413483] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6114.419940] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6114.424839] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6114.431812] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6114.433776] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6114.441156] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6114.448174] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6114.450302] ? update_process_times+0x6d/0xc0 [ 6114.455372] ? tick_sched_handle.isra.18+0x27/0x90 [ 6114.463164] ? tick_sched_timer+0x53/0xb0 [ 6114.469200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6114.473609] ? tick_sched_handle.isra.18+0x90/0x90 [ 6114.480681] ? hrtimer_interrupt+0x128/0x2c0 [ 6114.487760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6114.496636] ? apic_timer_interrupt+0xf/0x20 [ 6114.503400] [ 6114.505431] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6114.516963] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6114.525474] ? do_raw_spin_lock+0x177/0x190 [ 6114.536269] ? _raw_spin_lock+0x21/0x30 [ 6114.548387] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6114.551468] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6114.566919] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6114.569845] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6114.580697] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6114.592847] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6114.602943] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6114.617739] ? __next_timer_interrupt+0x160/0x160 [ 6114.623337] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6114.628693] ? do_wait_intr+0xf0/0xf0 [ 6114.632618] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6114.638931] ? kthread+0x1d1/0x200 [ 6114.642852] ? set_kthread_struct+0x70/0x70 [ 6114.645032] ? ret_from_fork+0x1f/0x30 [ 6120.062303] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6120.070011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6120.109542] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6120.119499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6120.131186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6120.140249] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6120.189282] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6120.211120] RAX: 0000000000000be7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6120.215777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6120.223273] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6120.236606] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6120.245275] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6120.252166] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6120.264383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6120.268671] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6120.273430] Call Trace: [ 6120.274314] [ 6120.275744] ? show_regs.cold.9+0x22/0x2f [ 6120.278047] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6120.280213] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6120.282515] ? watchdog+0x40/0x40 [ 6120.284089] ? hrtimer_interrupt+0x128/0x2c0 [ 6120.288722] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6120.291131] ? apic_timer_interrupt+0xf/0x20 [ 6120.293104] [ 6120.294034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6120.303308] do_raw_spin_lock+0x177/0x190 [ 6120.306901] _raw_spin_lock+0x21/0x30 [ 6120.309979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6120.318240] ? do_raw_spin_unlock+0x75/0x190 [ 6120.322618] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6120.330852] ? do_wait_intr+0xf0/0xf0 [ 6120.333804] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6120.347235] mdc_close+0x53b/0xd70 [mdc] [ 6120.360868] lmv_close+0x18d/0x490 [lmv] [ 6120.366804] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6120.378516] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6120.395410] ll_md_close+0x274/0x800 [lustre] [ 6120.399329] ll_file_release+0x128/0xc50 [lustre] [ 6120.414751] __fput+0x10d/0x440 [ 6120.422070] ____fput+0x12/0x20 [ 6120.425963] task_work_run+0xc6/0x110 [ 6120.428930] exit_to_usermode_loop+0x1dd/0x1f0 [ 6120.433685] do_syscall_64+0x3d6/0x440 [ 6120.436357] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6120.451583] RIP: 0033:0x7fa749552cd8 [ 6120.455709] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6120.480798] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6120.498280] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6120.504690] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6120.514413] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6120.520689] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6120.530576] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6128.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6128.063699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6128.100968] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6128.105844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6128.109817] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6128.113056] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6128.115047] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6128.122898] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6128.125671] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6128.129034] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6128.132408] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6128.135152] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6128.138290] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6128.141460] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6128.144798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6128.147559] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6128.154654] Call Trace: [ 6128.155693] [ 6128.156578] ? show_regs.cold.9+0x22/0x2f [ 6128.160689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6128.167592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6128.170992] ? watchdog+0x40/0x40 [ 6128.175289] ? hrtimer_interrupt+0x128/0x2c0 [ 6128.177699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6128.183028] ? apic_timer_interrupt+0xf/0x20 [ 6128.185304] [ 6128.186397] ? native_safe_halt+0x2e/0x50 [ 6128.188490] kvm_wait+0xcd/0xe0 [ 6128.192216] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6128.197478] do_raw_spin_lock+0x177/0x190 [ 6128.202199] _raw_spin_lock+0x21/0x30 [ 6128.205423] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6128.210292] process_one_work+0x2c8/0x700 [ 6128.211988] worker_thread+0x296/0x6e0 [ 6128.217807] ? rescuer_thread+0x570/0x570 [ 6128.219986] kthread+0x1d1/0x200 [ 6128.224305] ? set_kthread_struct+0x70/0x70 [ 6128.225895] ret_from_fork+0x1f/0x30 [ 6129.633051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4212s! [ 6129.636069] Showing busy workqueues and worker pools: [ 6129.639162] workqueue events: flags=0x0 [ 6129.641679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6129.645739] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6129.651307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6129.654161] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6129.658701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.663581] pending: kfree_rcu_monitor [ 6129.667940] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6129.677011] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6129.682867] workqueue events_long: flags=0x0 [ 6129.686746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.693184] pending: lu_objects_flush [obdclass] [ 6129.695925] workqueue events_unbound: flags=0x2 [ 6129.698625] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6129.702302] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6129.707905] workqueue events_power_efficient: flags=0x80 [ 6129.710930] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.714925] pending: check_lifetime [ 6129.717280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.721581] pending: neigh_periodic_work [ 6129.725887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.731410] pending: neigh_periodic_work [ 6129.734957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.740476] pending: do_cache_clean [ 6129.741932] workqueue mm_percpu_wq: flags=0x8 [ 6129.743432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.746855] pending: vmstat_update [ 6129.748809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.752000] pending: vmstat_update [ 6129.754723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6129.765525] pending: vmstat_update [ 6129.770120] workqueue writeback: flags=0x4a [ 6129.775577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6129.784285] pending: wb_workfn, wb_workfn [ 6129.787467] workqueue xprtiod: flags=0xa [ 6129.791693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6129.797945] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6129.805160] workqueue ipv6_addrconf: flags=0x40008 [ 6129.808496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6129.813899] pending: addrconf_verify_work [ 6129.819961] workqueue knbd0-recv: flags=0x1a [ 6129.824729] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6129.833503] in-flight: 651:recv_work [ 6129.837582] workqueue ptlrpc_pinger: flags=0x2 [ 6129.841270] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6129.846139] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6129.850586] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4212s workers=4 idle: 37 31846 28897 [ 6129.857493] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6119s workers=2 idle: 653 [ 6140.058028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6140.061180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6140.086501] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6140.091907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6140.095879] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6140.097872] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6140.105547] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6140.108937] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6140.112078] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6140.115238] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6140.118379] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6140.121666] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6140.124861] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6140.128432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6140.131330] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6140.134886] Call Trace: [ 6140.135903] [ 6140.136953] ? show_regs.cold.9+0x22/0x2f [ 6140.138844] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6140.141235] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6140.143275] ? watchdog+0x40/0x40 [ 6140.144588] ? hrtimer_interrupt+0x128/0x2c0 [ 6140.146415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6140.148592] ? apic_timer_interrupt+0xf/0x20 [ 6140.150514] [ 6140.151448] ? native_safe_halt+0x2e/0x50 [ 6140.153226] kvm_wait+0xcd/0xe0 [ 6140.154558] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6140.157273] do_raw_spin_lock+0x177/0x190 [ 6140.159404] _raw_spin_lock+0x21/0x30 [ 6140.161438] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6140.166513] obd_mod_free+0x46/0x140 [ptlrpc] [ 6140.170646] mdc_commit_open+0x91/0xb0 [mdc] [ 6140.175691] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6140.178383] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6140.182252] after_reply+0xe2c/0x1780 [ptlrpc] [ 6140.185294] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6140.188086] ? __next_timer_interrupt+0x160/0x160 [ 6140.190233] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6140.192242] ? do_wait_intr+0xf0/0xf0 [ 6140.194096] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6140.196858] kthread+0x1d1/0x200 [ 6140.197907] ? set_kthread_struct+0x70/0x70 [ 6140.199621] ret_from_fork+0x1f/0x30 [ 6148.061108] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6148.072873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6148.120434] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6148.132584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6148.139882] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6148.143713] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6148.166600] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6148.172248] RAX: 0000000000006907 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6148.179535] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6148.181964] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6148.194553] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6148.201221] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6148.207944] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6148.212965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6148.219317] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6148.226353] Call Trace: [ 6148.229347] [ 6148.230076] ? show_regs.cold.9+0x22/0x2f [ 6148.232360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6148.235918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6148.238343] ? watchdog+0x40/0x40 [ 6148.240307] ? hrtimer_interrupt+0x128/0x2c0 [ 6148.242511] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6148.245370] ? apic_timer_interrupt+0xf/0x20 [ 6148.248336] [ 6148.249313] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6148.252250] do_raw_spin_lock+0x177/0x190 [ 6148.253905] _raw_spin_lock+0x21/0x30 [ 6148.255902] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6148.258681] ? do_raw_spin_unlock+0x75/0x190 [ 6148.260435] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6148.263722] ? do_wait_intr+0xf0/0xf0 [ 6148.265517] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6148.267833] mdc_close+0x53b/0xd70 [mdc] [ 6148.269832] lmv_close+0x18d/0x490 [lmv] [ 6148.271125] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6148.277871] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6148.281393] ll_md_close+0x274/0x800 [lustre] [ 6148.283565] ll_file_release+0x128/0xc50 [lustre] [ 6148.285903] __fput+0x10d/0x440 [ 6148.287687] ____fput+0x12/0x20 [ 6148.289047] task_work_run+0xc6/0x110 [ 6148.290975] exit_to_usermode_loop+0x1dd/0x1f0 [ 6148.293219] do_syscall_64+0x3d6/0x440 [ 6148.294938] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6148.297879] RIP: 0033:0x7fa749552cd8 [ 6148.299682] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6148.311504] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6148.315179] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6148.318367] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6148.321349] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6148.324320] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6148.328145] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6156.055296] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6156.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6156.059064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 6156.063136] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6156.063223] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6156.063231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6156.063237] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6156.063644] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6156.063654] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6156.063658] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6156.063663] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6156.063665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6156.063668] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6156.063669] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6156.063672] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6156.063674] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6156.063677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6156.063679] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6156.063689] Call Trace: [ 6156.063695] [ 6156.063701] ? show_regs.cold.9+0x22/0x2f [ 6156.063713] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6156.063719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6156.063724] ? watchdog+0x40/0x40 [ 6156.063728] ? hrtimer_interrupt+0x128/0x2c0 [ 6156.063732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6156.063740] ? apic_timer_interrupt+0xf/0x20 [ 6156.063744] [ 6156.063747] ? native_safe_halt+0x2e/0x50 [ 6156.063752] kvm_wait+0xcd/0xe0 [ 6156.063905] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6156.063913] do_raw_spin_lock+0x177/0x190 [ 6156.063918] _raw_spin_lock+0x21/0x30 [ 6156.063922] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6156.064271] process_one_work+0x2c8/0x700 [ 6156.064278] worker_thread+0x296/0x6e0 [ 6156.064283] ? rescuer_thread+0x570/0x570 [ 6156.064287] kthread+0x1d1/0x200 [ 6156.064292] ? set_kthread_struct+0x70/0x70 [ 6156.064295] ret_from_fork+0x1f/0x30 [ 6156.333977] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6156.364943] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6156.376004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6156.382259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6156.386169] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6156.397840] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6156.401473] RAX: 0000000000007f26 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6156.409903] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6156.414950] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6156.421644] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6156.431402] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6156.439614] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6156.443676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6156.446671] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6156.453053] Call Trace: [ 6156.453898] [ 6156.454643] ? show_regs.cold.9+0x22/0x2f [ 6156.457607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6156.459097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6156.461507] ? watchdog+0x40/0x40 [ 6156.463174] ? hrtimer_interrupt+0x128/0x2c0 [ 6156.465183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6156.467659] ? apic_timer_interrupt+0xf/0x20 [ 6156.469961] [ 6156.471189] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6156.473382] do_raw_spin_lock+0x177/0x190 [ 6156.476555] _raw_spin_lock+0x21/0x30 [ 6156.479807] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6156.482992] ? do_raw_spin_unlock+0x75/0x190 [ 6156.484718] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6156.486773] ? do_wait_intr+0xf0/0xf0 [ 6156.488006] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6156.490924] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6156.493454] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6156.496474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.500547] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6156.503490] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6156.506274] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6156.508701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.511638] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6156.515173] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6156.517790] lmv_intent_open+0x338/0x1150 [lmv] [ 6156.521802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.524950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.527497] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6156.531730] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6156.536229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.543056] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6156.548729] ? do_raw_spin_unlock+0x75/0x190 [ 6156.552503] ? _raw_spin_unlock+0x12/0x30 [ 6156.558644] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6156.564617] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6156.569746] ll_atomic_open+0x36a/0xd70 [lustre] [ 6156.575243] lookup_open+0x356/0xb40 [ 6156.580736] do_last+0x313/0xfc0 [ 6156.583681] ? nd_jump_root+0xe5/0x160 [ 6156.588236] ? path_init+0x437/0x520 [ 6156.592334] path_openat+0xf7/0x500 [ 6156.595934] ? do_raw_spin_unlock+0x75/0x190 [ 6156.600723] do_filp_open+0x99/0x140 [ 6156.606421] ? getname_flags+0x6e/0x330 [ 6156.609547] ? __check_object_size+0xff/0x256 [ 6156.616425] ? do_raw_spin_unlock+0x75/0x190 [ 6156.618629] ? _raw_spin_unlock+0x12/0x30 [ 6156.625604] do_sys_openat2+0x2b4/0x410 [ 6156.628831] do_sys_open+0x73/0xa0 [ 6156.632681] __x64_sys_openat+0x24/0x30 [ 6156.635469] do_syscall_64+0xc1/0x440 [ 6156.637172] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6156.639461] RIP: 0033:0x7fa749552332 [ 6156.643497] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6156.652692] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6156.659408] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6156.661913] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6156.667816] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6156.674544] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6156.681059] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6160.354073] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4243s! [ 6160.358685] Showing busy workqueues and worker pools: [ 6160.361686] workqueue events: flags=0x0 [ 6160.364291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6160.368084] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6160.374135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6160.378030] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6160.384630] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.389244] pending: kfree_rcu_monitor [ 6160.392378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6160.396344] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6160.402365] workqueue events_long: flags=0x0 [ 6160.405144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.409228] pending: lu_objects_flush [obdclass] [ 6160.412362] workqueue events_unbound: flags=0x2 [ 6160.415124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6160.419013] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6160.424641] workqueue events_power_efficient: flags=0x80 [ 6160.428189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.432486] pending: check_lifetime [ 6160.436149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.440629] pending: neigh_periodic_work [ 6160.443598] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.447267] pending: neigh_periodic_work [ 6160.449866] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.454274] pending: do_cache_clean [ 6160.456572] workqueue mm_percpu_wq: flags=0x8 [ 6160.458638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.462465] pending: vmstat_update [ 6160.464550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.468233] pending: vmstat_update [ 6160.469772] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6160.473025] pending: vmstat_update [ 6160.474520] workqueue writeback: flags=0x4a [ 6160.476459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6160.479347] pending: wb_workfn, wb_workfn [ 6160.481403] workqueue xprtiod: flags=0xa [ 6160.483497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6160.486241] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6160.489765] workqueue ipv6_addrconf: flags=0x40008 [ 6160.492463] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6160.495425] pending: addrconf_verify_work [ 6160.497533] workqueue knbd0-recv: flags=0x1a [ 6160.499323] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6160.502188] in-flight: 651:recv_work [ 6160.504359] workqueue ptlrpc_pinger: flags=0x2 [ 6160.506316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6160.509246] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6160.512028] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4243s workers=4 idle: 37 31846 28897 [ 6160.515334] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6150s workers=2 idle: 653 [ 6168.060732] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6168.066473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6168.114411] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6168.128831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6168.134267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6168.137533] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6168.157448] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6168.159888] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6168.170498] RDX: 00000000000073f5 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6168.181133] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6168.190057] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6168.192707] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6168.196755] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6168.216528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6168.223912] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6168.228623] Call Trace: [ 6168.233208] [ 6168.234622] ? show_regs.cold.9+0x22/0x2f [ 6168.237331] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6168.240562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6168.246717] ? watchdog+0x40/0x40 [ 6168.249039] ? hrtimer_interrupt+0x128/0x2c0 [ 6168.250541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6168.254153] ? apic_timer_interrupt+0xf/0x20 [ 6168.256300] [ 6168.257066] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6168.258820] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6168.260892] do_raw_spin_lock+0x177/0x190 [ 6168.262769] _raw_spin_lock+0x21/0x30 [ 6168.264242] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6168.266907] obd_mod_free+0x46/0x140 [ptlrpc] [ 6168.268097] mdc_commit_open+0x91/0xb0 [mdc] [ 6168.271273] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6168.273823] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6168.277215] after_reply+0xe2c/0x1780 [ptlrpc] [ 6168.279261] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6168.281596] ? __next_timer_interrupt+0x160/0x160 [ 6168.283379] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6168.285529] ? do_wait_intr+0xf0/0xf0 [ 6168.287035] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6168.289499] kthread+0x1d1/0x200 [ 6168.290765] ? set_kthread_struct+0x70/0x70 [ 6168.292395] ret_from_fork+0x1f/0x30 [ 6176.059105] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6176.062699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6176.085225] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6176.092232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6176.097701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6176.101399] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6176.110438] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6176.113713] RAX: 00000000000020a9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6176.116834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6176.120326] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6176.123689] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6176.126947] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6176.130361] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6176.135285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6176.137763] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6176.141387] Call Trace: [ 6176.142454] [ 6176.143638] ? show_regs.cold.9+0x22/0x2f [ 6176.145487] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6176.147772] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6176.150005] ? watchdog+0x40/0x40 [ 6176.151644] ? hrtimer_interrupt+0x128/0x2c0 [ 6176.153661] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6176.155814] ? apic_timer_interrupt+0xf/0x20 [ 6176.158033] [ 6176.158991] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6176.161749] do_raw_spin_lock+0x177/0x190 [ 6176.163765] _raw_spin_lock+0x21/0x30 [ 6176.165590] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6176.168379] ? do_raw_spin_unlock+0x75/0x190 [ 6176.170438] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6176.173272] ? do_wait_intr+0xf0/0xf0 [ 6176.174974] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6176.177741] mdc_close+0x53b/0xd70 [mdc] [ 6176.179855] lmv_close+0x18d/0x490 [lmv] [ 6176.181770] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6176.184925] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6176.187448] ll_md_close+0x274/0x800 [lustre] [ 6176.189718] ll_file_release+0x128/0xc50 [lustre] [ 6176.192410] __fput+0x10d/0x440 [ 6176.193927] ____fput+0x12/0x20 [ 6176.195549] task_work_run+0xc6/0x110 [ 6176.197331] exit_to_usermode_loop+0x1dd/0x1f0 [ 6176.199552] do_syscall_64+0x3d6/0x440 [ 6176.201661] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6176.203986] RIP: 0033:0x7fa749552cd8 [ 6176.205735] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6176.215289] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6176.219316] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6176.222818] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6176.226534] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6176.229894] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6176.233065] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6184.057039] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6184.059733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6184.084529] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6184.091287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6184.099181] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6184.108316] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6184.110192] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6184.128415] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6184.138219] RAX: 0000000000002877 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6184.148198] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6184.153056] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6184.157802] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6184.160184] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6184.168287] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6184.172411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6184.174296] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6184.181966] Call Trace: [ 6184.183305] [ 6184.184921] ? show_regs.cold.9+0x22/0x2f [ 6184.187557] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6184.190201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6184.192684] ? watchdog+0x40/0x40 [ 6184.194944] ? hrtimer_interrupt+0x128/0x2c0 [ 6184.199237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6184.201503] ? apic_timer_interrupt+0xf/0x20 [ 6184.202589] [ 6184.204859] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6184.206434] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6184.209191] do_raw_spin_lock+0x177/0x190 [ 6184.211014] _raw_spin_lock+0x21/0x30 [ 6184.218502] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6184.222863] process_one_work+0x2c8/0x700 [ 6184.224130] worker_thread+0x296/0x6e0 [ 6184.227018] ? rescuer_thread+0x570/0x570 [ 6184.228605] kthread+0x1d1/0x200 [ 6184.229729] ? set_kthread_struct+0x70/0x70 [ 6184.231173] ret_from_fork+0x1f/0x30 [ 6188.054571] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6188.065147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6188.106741] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6188.119911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6188.128982] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6188.136673] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6188.148952] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6188.152653] RAX: 0000000000006f91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6188.156226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6188.159341] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6188.164523] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6188.168986] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6188.173426] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6188.178517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6188.180826] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6188.184406] Call Trace: [ 6188.187455] [ 6188.188467] ? show_regs.cold.9+0x22/0x2f [ 6188.190143] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6188.192339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6188.196050] ? watchdog+0x40/0x40 [ 6188.198375] ? hrtimer_interrupt+0x128/0x2c0 [ 6188.202180] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6188.205746] ? apic_timer_interrupt+0xf/0x20 [ 6188.207875] [ 6188.208874] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6188.211939] do_raw_spin_lock+0x177/0x190 [ 6188.218393] _raw_spin_lock+0x21/0x30 [ 6188.220281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6188.224721] ? do_raw_spin_unlock+0x75/0x190 [ 6188.227277] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6188.234614] ? do_wait_intr+0xf0/0xf0 [ 6188.241872] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6188.245998] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6188.255440] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6188.260757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.263944] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6188.268456] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6188.270984] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6188.275486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.281008] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6188.288434] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6188.293056] lmv_intent_open+0x338/0x1150 [lmv] [ 6188.298312] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.300287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.305427] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6188.309763] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6188.311479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.315789] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6188.319002] ? do_raw_spin_unlock+0x75/0x190 [ 6188.320467] ? _raw_spin_unlock+0x12/0x30 [ 6188.322572] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6188.324811] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6188.326664] ll_atomic_open+0x36a/0xd70 [lustre] [ 6188.329864] lookup_open+0x356/0xb40 [ 6188.335894] do_last+0x313/0xfc0 [ 6188.337896] ? nd_jump_root+0xe5/0x160 [ 6188.341708] ? path_init+0x437/0x520 [ 6188.343799] path_openat+0xf7/0x500 [ 6188.348999] ? do_raw_spin_unlock+0x75/0x190 [ 6188.352966] do_filp_open+0x99/0x140 [ 6188.354389] ? getname_flags+0x6e/0x330 [ 6188.357706] ? __check_object_size+0xff/0x256 [ 6188.359424] ? do_raw_spin_unlock+0x75/0x190 [ 6188.363499] ? _raw_spin_unlock+0x12/0x30 [ 6188.365263] do_sys_openat2+0x2b4/0x410 [ 6188.371046] do_sys_open+0x73/0xa0 [ 6188.372470] __x64_sys_openat+0x24/0x30 [ 6188.375468] do_syscall_64+0xc1/0x440 [ 6188.376549] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6188.379667] RIP: 0033:0x7fa749552332 [ 6188.380986] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6188.395154] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6188.400583] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6188.406043] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6188.411953] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6188.416782] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6188.421725] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6190.569383] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4273s! [ 6190.577151] Showing busy workqueues and worker pools: [ 6190.580198] workqueue events: flags=0x0 [ 6190.583568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6190.588939] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6190.607646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6190.610738] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6190.614929] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.618786] pending: kfree_rcu_monitor [ 6190.621036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6190.623080] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6190.634217] workqueue events_long: flags=0x0 [ 6190.636994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.640590] pending: lu_objects_flush [obdclass] [ 6190.644377] workqueue events_unbound: flags=0x2 [ 6190.649558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6190.657480] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6190.669979] workqueue events_power_efficient: flags=0x80 [ 6190.672740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.678978] pending: check_lifetime [ 6190.681203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.691060] pending: neigh_periodic_work [ 6190.699496] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.706259] pending: neigh_periodic_work [ 6190.709217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.714648] pending: do_cache_clean [ 6190.723023] workqueue mm_percpu_wq: flags=0x8 [ 6190.730052] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.740342] pending: vmstat_update [ 6190.742923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.753125] pending: vmstat_update [ 6190.764906] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6190.769904] pending: vmstat_update [ 6190.773164] workqueue writeback: flags=0x4a [ 6190.776947] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6190.786123] pending: wb_workfn, wb_workfn [ 6190.789311] workqueue xprtiod: flags=0xa [ 6190.801817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6190.812664] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6190.823868] workqueue ipv6_addrconf: flags=0x40008 [ 6190.827748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6190.832799] pending: addrconf_verify_work [ 6190.838051] workqueue knbd0-recv: flags=0x1a [ 6190.841203] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6190.847544] in-flight: 651:recv_work [ 6190.851577] workqueue ptlrpc_pinger: flags=0x2 [ 6190.855271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6190.871658] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6190.885402] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4273s workers=4 idle: 37 31846 28897 [ 6190.904129] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6180s workers=2 idle: 653 [ 6196.059185] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6196.067233] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6196.124666] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6196.133140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6196.140363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6196.146472] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6196.169922] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6196.182767] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6196.193432] RDX: 0000000000004c86 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6196.201312] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6196.209783] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6196.218538] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6196.229786] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6196.250548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6196.261828] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6196.272543] Call Trace: [ 6196.278869] [ 6196.280999] ? show_regs.cold.9+0x22/0x2f [ 6196.283666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6196.294858] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6196.306558] ? watchdog+0x40/0x40 [ 6196.312782] ? hrtimer_interrupt+0x128/0x2c0 [ 6196.317271] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6196.321681] ? apic_timer_interrupt+0xf/0x20 [ 6196.323659] [ 6196.324330] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6196.326465] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6196.332671] do_raw_spin_lock+0x177/0x190 [ 6196.335719] _raw_spin_lock+0x21/0x30 [ 6196.339807] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6196.343956] obd_mod_free+0x46/0x140 [ptlrpc] [ 6196.357624] mdc_commit_open+0x91/0xb0 [mdc] [ 6196.366188] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6196.374846] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6196.386606] after_reply+0xe2c/0x1780 [ptlrpc] [ 6196.393618] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6196.403428] ? __next_timer_interrupt+0x160/0x160 [ 6196.410746] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6196.423553] ? do_wait_intr+0xf0/0xf0 [ 6196.429225] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6196.443214] kthread+0x1d1/0x200 [ 6196.446691] ? set_kthread_struct+0x70/0x70 [ 6196.455168] ret_from_fork+0x1f/0x30 [ 6204.061922] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6204.064032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6204.093442] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6204.107199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6204.115078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6204.118926] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6204.137775] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6204.143320] RAX: 0000000000007d8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6204.148354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6204.157018] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6204.161174] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6204.165221] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6204.168886] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6204.173995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6204.176647] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6204.181839] Call Trace: [ 6204.183154] [ 6204.187231] ? show_regs.cold.9+0x22/0x2f [ 6204.189138] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6204.196108] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6204.197388] ? watchdog+0x40/0x40 [ 6204.201108] ? hrtimer_interrupt+0x128/0x2c0 [ 6204.206190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6204.210191] ? apic_timer_interrupt+0xf/0x20 [ 6204.218294] [ 6204.219026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6204.224617] do_raw_spin_lock+0x177/0x190 [ 6204.225865] _raw_spin_lock+0x21/0x30 [ 6204.230659] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6204.235037] ? do_raw_spin_unlock+0x75/0x190 [ 6204.236527] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6204.239137] ? do_wait_intr+0xf0/0xf0 [ 6204.240795] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6204.243631] mdc_close+0x53b/0xd70 [mdc] [ 6204.245873] lmv_close+0x18d/0x490 [lmv] [ 6204.249286] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6204.252494] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6204.259182] ll_md_close+0x274/0x800 [lustre] [ 6204.263121] ll_file_release+0x128/0xc50 [lustre] [ 6204.265820] __fput+0x10d/0x440 [ 6204.269078] ____fput+0x12/0x20 [ 6204.270201] task_work_run+0xc6/0x110 [ 6204.272581] exit_to_usermode_loop+0x1dd/0x1f0 [ 6204.274824] do_syscall_64+0x3d6/0x440 [ 6204.276771] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6204.286833] RIP: 0033:0x7fa749552cd8 [ 6204.289800] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6204.304546] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6204.312568] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6204.316944] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6204.321924] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6204.325465] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6204.327204] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6212.057036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6212.060539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6212.078761] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6212.083014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6212.086228] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6212.089311] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6212.091139] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6212.098796] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6212.102107] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6212.105172] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6212.108223] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6212.111252] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6212.114365] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6212.117693] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6212.121709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6212.124598] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6212.127693] Call Trace: [ 6212.128787] [ 6212.130119] ? show_regs.cold.9+0x22/0x2f [ 6212.132775] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6212.136277] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6212.139368] ? watchdog+0x40/0x40 [ 6212.140739] ? hrtimer_interrupt+0x128/0x2c0 [ 6212.142468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6212.144446] ? apic_timer_interrupt+0xf/0x20 [ 6212.146134] [ 6212.147087] ? native_safe_halt+0x2e/0x50 [ 6212.148740] kvm_wait+0xcd/0xe0 [ 6212.149966] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6212.152122] do_raw_spin_lock+0x177/0x190 [ 6212.153841] _raw_spin_lock+0x21/0x30 [ 6212.155308] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6212.157739] process_one_work+0x2c8/0x700 [ 6212.159596] worker_thread+0x296/0x6e0 [ 6212.161209] ? rescuer_thread+0x570/0x570 [ 6212.162589] kthread+0x1d1/0x200 [ 6212.163748] ? set_kthread_struct+0x70/0x70 [ 6212.165130] ret_from_fork+0x1f/0x30 [ 6216.055791] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6216.064976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6216.121621] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6216.145009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6216.153154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6216.156324] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6216.170159] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6216.173663] RAX: 000000000000270a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6216.177361] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6216.180961] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6216.184069] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6216.187167] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6216.190576] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6216.193780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6216.196267] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6216.199321] Call Trace: [ 6216.200328] [ 6216.201169] ? show_regs.cold.9+0x22/0x2f [ 6216.202779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6216.204882] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6216.206727] ? watchdog+0x40/0x40 [ 6216.208053] ? hrtimer_interrupt+0x128/0x2c0 [ 6216.209964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6216.212069] ? apic_timer_interrupt+0xf/0x20 [ 6216.214056] [ 6216.214861] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6216.217432] do_raw_spin_lock+0x177/0x190 [ 6216.219317] _raw_spin_lock+0x21/0x30 [ 6216.221682] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6216.225850] ? do_raw_spin_unlock+0x75/0x190 [ 6216.229351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6216.231352] ? do_wait_intr+0xf0/0xf0 [ 6216.234116] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6216.236997] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6216.239601] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6216.242532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.244958] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6216.247447] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6216.249938] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6216.251882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.254205] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6216.256918] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6216.259318] lmv_intent_open+0x338/0x1150 [lmv] [ 6216.261574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.264181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.266784] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6216.268884] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6216.271108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.273647] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6216.276000] ? do_raw_spin_unlock+0x75/0x190 [ 6216.277956] ? _raw_spin_unlock+0x12/0x30 [ 6216.279587] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6216.281912] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6216.284655] ll_atomic_open+0x36a/0xd70 [lustre] [ 6216.286997] lookup_open+0x356/0xb40 [ 6216.288421] do_last+0x313/0xfc0 [ 6216.289953] ? nd_jump_root+0xe5/0x160 [ 6216.291616] ? path_init+0x437/0x520 [ 6216.293319] path_openat+0xf7/0x500 [ 6216.295012] ? do_raw_spin_unlock+0x75/0x190 [ 6216.296919] do_filp_open+0x99/0x140 [ 6216.298374] ? getname_flags+0x6e/0x330 [ 6216.300221] ? __check_object_size+0xff/0x256 [ 6216.302351] ? do_raw_spin_unlock+0x75/0x190 [ 6216.304898] ? _raw_spin_unlock+0x12/0x30 [ 6216.306441] do_sys_openat2+0x2b4/0x410 [ 6216.308297] do_sys_open+0x73/0xa0 [ 6216.310616] __x64_sys_openat+0x24/0x30 [ 6216.312972] do_syscall_64+0xc1/0x440 [ 6216.314789] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6216.317137] RIP: 0033:0x7fa749552332 [ 6216.318995] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6216.327445] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6216.331043] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6216.334199] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6216.337623] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6216.341325] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6216.344935] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6221.280140] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4304s! [ 6221.286453] Showing busy workqueues and worker pools: [ 6221.291373] workqueue events: flags=0x0 [ 6221.295193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6221.298629] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6221.304144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6221.309913] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6221.316044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.321873] pending: kfree_rcu_monitor [ 6221.326286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6221.331162] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6221.338663] workqueue events_long: flags=0x0 [ 6221.342345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.347922] pending: lu_objects_flush [obdclass] [ 6221.350643] workqueue events_unbound: flags=0x2 [ 6221.355102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6221.363811] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6221.372650] workqueue events_power_efficient: flags=0x80 [ 6221.376971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.380077] pending: check_lifetime [ 6221.384128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.389505] pending: neigh_periodic_work [ 6221.397573] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.404065] pending: neigh_periodic_work [ 6221.407538] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.412139] pending: do_cache_clean [ 6221.417190] workqueue mm_percpu_wq: flags=0x8 [ 6221.426519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.431113] pending: vmstat_update [ 6221.433864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.444124] pending: vmstat_update [ 6221.445805] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6221.448973] pending: vmstat_update [ 6221.451244] workqueue writeback: flags=0x4a [ 6221.453143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6221.456525] pending: wb_workfn, wb_workfn [ 6221.465763] workqueue xprtiod: flags=0xa [ 6221.469416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6221.476919] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6221.485905] workqueue ipv6_addrconf: flags=0x40008 [ 6221.489843] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6221.502543] pending: addrconf_verify_work [ 6221.507336] workqueue knbd0-recv: flags=0x1a [ 6221.511916] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6221.517304] in-flight: 651:recv_work [ 6221.524043] workqueue ptlrpc_pinger: flags=0x2 [ 6221.530147] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6221.538939] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6221.543280] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4304s workers=4 idle: 37 31846 28897 [ 6221.546349] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6211s workers=2 idle: 653 [ 6224.058120] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6224.061877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6224.081370] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6224.087446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6224.091499] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6224.094752] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6224.108616] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6224.112261] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6224.116411] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6224.119470] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6224.122456] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6224.126302] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6224.129507] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6224.133246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6224.136316] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6224.142872] Call Trace: [ 6224.144347] [ 6224.145806] ? show_regs.cold.9+0x22/0x2f [ 6224.149476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6224.160681] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6224.162022] ? watchdog+0x40/0x40 [ 6224.165654] ? hrtimer_interrupt+0x128/0x2c0 [ 6224.168092] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6224.170653] ? apic_timer_interrupt+0xf/0x20 [ 6224.173408] [ 6224.174484] ? native_safe_halt+0x2e/0x50 [ 6224.176763] kvm_wait+0xcd/0xe0 [ 6224.178412] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6224.181089] do_raw_spin_lock+0x177/0x190 [ 6224.183056] _raw_spin_lock+0x21/0x30 [ 6224.184711] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6224.187142] obd_mod_free+0x46/0x140 [ptlrpc] [ 6224.189366] mdc_commit_open+0x91/0xb0 [mdc] [ 6224.191245] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6224.193663] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6224.196683] after_reply+0xe2c/0x1780 [ptlrpc] [ 6224.198960] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6224.201504] ? __next_timer_interrupt+0x160/0x160 [ 6224.203382] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6224.205187] ? do_wait_intr+0xf0/0xf0 [ 6224.206432] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6224.209217] kthread+0x1d1/0x200 [ 6224.210466] ? set_kthread_struct+0x70/0x70 [ 6224.213904] ret_from_fork+0x1f/0x30 [ 6232.059267] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6232.068161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6232.100419] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6232.107966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6232.115208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6232.119377] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6232.130781] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6232.135789] RAX: 00000000000079de RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6232.139185] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6232.143441] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6232.149255] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6232.152187] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6232.161179] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6232.164767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6232.169608] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6232.175163] Call Trace: [ 6232.176008] [ 6232.176968] ? show_regs.cold.9+0x22/0x2f [ 6232.181024] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6232.184655] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6232.186656] ? watchdog+0x40/0x40 [ 6232.193476] ? hrtimer_interrupt+0x128/0x2c0 [ 6232.196618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6232.201028] ? apic_timer_interrupt+0xf/0x20 [ 6232.205126] [ 6232.210327] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6232.214069] do_raw_spin_lock+0x177/0x190 [ 6232.222502] _raw_spin_lock+0x21/0x30 [ 6232.230508] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6232.233976] ? do_raw_spin_unlock+0x75/0x190 [ 6232.240940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6232.243495] ? do_wait_intr+0xf0/0xf0 [ 6232.246709] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6232.250140] mdc_close+0x53b/0xd70 [mdc] [ 6232.251293] lmv_close+0x18d/0x490 [lmv] [ 6232.254285] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6232.258498] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6232.267463] ll_md_close+0x274/0x800 [lustre] [ 6232.272562] ll_file_release+0x128/0xc50 [lustre] [ 6232.279421] __fput+0x10d/0x440 [ 6232.280503] ____fput+0x12/0x20 [ 6232.290077] task_work_run+0xc6/0x110 [ 6232.291308] exit_to_usermode_loop+0x1dd/0x1f0 [ 6232.292939] do_syscall_64+0x3d6/0x440 [ 6232.294305] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6232.297916] RIP: 0033:0x7fa749552cd8 [ 6232.299174] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6232.313101] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6232.318037] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6232.321091] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6232.323668] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6232.326375] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6232.329718] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6240.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6240.061093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6240.083582] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6240.089393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6240.093611] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6240.097751] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6240.099972] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6240.109874] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6240.113925] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6240.117346] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6240.120243] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6240.123338] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6240.125839] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6240.128891] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6240.132972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6240.135376] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6240.138546] Call Trace: [ 6240.139642] [ 6240.140551] ? show_regs.cold.9+0x22/0x2f [ 6240.142202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6240.144313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6240.146737] ? watchdog+0x40/0x40 [ 6240.148341] ? hrtimer_interrupt+0x128/0x2c0 [ 6240.152206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6240.153873] ? apic_timer_interrupt+0xf/0x20 [ 6240.156315] [ 6240.157353] ? native_safe_halt+0x2e/0x50 [ 6240.158929] kvm_wait+0xcd/0xe0 [ 6240.160240] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6240.164437] do_raw_spin_lock+0x177/0x190 [ 6240.166192] _raw_spin_lock+0x21/0x30 [ 6240.167861] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6240.169912] process_one_work+0x2c8/0x700 [ 6240.171869] worker_thread+0x296/0x6e0 [ 6240.173558] ? rescuer_thread+0x570/0x570 [ 6240.175191] kthread+0x1d1/0x200 [ 6240.176730] ? set_kthread_struct+0x70/0x70 [ 6240.178718] ret_from_fork+0x1f/0x30 [ 6248.054024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6248.056494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6248.089098] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6248.096483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6248.103518] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6248.106102] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6248.125141] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6248.134174] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6248.142170] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6248.148926] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6248.157688] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6248.162610] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6248.168835] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6248.175159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6248.181523] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6248.187122] Call Trace: [ 6248.189173] [ 6248.190985] ? show_regs.cold.9+0x22/0x2f [ 6248.193321] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6248.197821] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6248.201623] ? watchdog+0x40/0x40 [ 6248.202740] ? hrtimer_interrupt+0x128/0x2c0 [ 6248.209203] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6248.213690] ? apic_timer_interrupt+0xf/0x20 [ 6248.220099] [ 6248.222174] ? native_safe_halt+0x2e/0x50 [ 6248.226823] kvm_wait+0xcd/0xe0 [ 6248.231807] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6248.241201] do_raw_spin_lock+0x177/0x190 [ 6248.246801] _raw_spin_lock+0x21/0x30 [ 6248.252831] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6248.264787] ? do_raw_spin_unlock+0x75/0x190 [ 6248.274063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6248.281140] ? do_wait_intr+0xf0/0xf0 [ 6248.283841] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6248.286518] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6248.296518] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6248.311171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6248.322625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6248.328698] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6248.335138] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6248.344501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6248.360225] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6248.368073] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6248.373720] lmv_intent_open+0x338/0x1150 [lmv] [ 6248.381735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6248.387972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6248.392625] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6248.402132] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6248.408540] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6248.419725] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6248.429745] ? do_raw_spin_unlock+0x75/0x190 [ 6248.437401] ? _raw_spin_unlock+0x12/0x30 [ 6248.444475] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6248.453513] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6248.461191] ll_atomic_open+0x36a/0xd70 [lustre] [ 6248.466699] lookup_open+0x356/0xb40 [ 6248.472185] do_last+0x313/0xfc0 [ 6248.474355] ? nd_jump_root+0xe5/0x160 [ 6248.479922] ? path_init+0x437/0x520 [ 6248.485313] path_openat+0xf7/0x500 [ 6248.494847] ? do_raw_spin_unlock+0x75/0x190 [ 6248.497539] do_filp_open+0x99/0x140 [ 6248.501289] ? getname_flags+0x6e/0x330 [ 6248.503246] ? __check_object_size+0xff/0x256 [ 6248.506350] ? do_raw_spin_unlock+0x75/0x190 [ 6248.511375] ? _raw_spin_unlock+0x12/0x30 [ 6248.516449] do_sys_openat2+0x2b4/0x410 [ 6248.517859] do_sys_open+0x73/0xa0 [ 6248.522203] __x64_sys_openat+0x24/0x30 [ 6248.525672] do_syscall_64+0xc1/0x440 [ 6248.531446] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6248.534610] RIP: 0033:0x7fa749552332 [ 6248.539201] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6248.553672] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6248.560091] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6248.565485] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6248.571145] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6248.576147] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6248.580602] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6252.003159] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4334s! [ 6252.007035] Showing busy workqueues and worker pools: [ 6252.009451] workqueue events: flags=0x0 [ 6252.011028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6252.014075] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6252.020048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6252.030223] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6252.040974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.045054] pending: kfree_rcu_monitor [ 6252.047378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6252.050674] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6252.054347] workqueue events_long: flags=0x0 [ 6252.055920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.058177] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6252.059211] pending: [ 6252.062432] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6252.062525] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6252.062532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6252.062535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6252.062551] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6252.062556] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6252.062560] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6252.062562] RDX: 0000000000006525 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6252.062564] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6252.062567] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6252.062569] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6252.062572] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6252.062574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6252.062577] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6252.062585] Call Trace: [ 6252.062591] [ 6252.062598] ? show_regs.cold.9+0x22/0x2f [ 6252.062608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6252.062615] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6252.062620] ? watchdog+0x40/0x40 [ 6252.062624] ? hrtimer_interrupt+0x128/0x2c0 [ 6252.062628] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6252.062636] ? apic_timer_interrupt+0xf/0x20 [ 6252.062641] [ 6252.062644] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6252.062649] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6252.062654] do_raw_spin_lock+0x177/0x190 [ 6252.062907] _raw_spin_lock+0x21/0x30 [ 6252.064553] lu_objects_flush [obdclass] [ 6252.064773] workqueue events_unbound: flags=0x2 [ 6252.064777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6252.064787] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6252.064808] workqueue events_power_efficient: flags=0x80 [ 6252.064811] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.064819] pending: check_lifetime [ 6252.064829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.064836] pending: neigh_periodic_work [ 6252.064844] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.064850] pending: neigh_periodic_work [ 6252.064855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.064862] pending: do_cache_clean [ 6252.064871] workqueue mm_percpu_wq: flags=0x8 [ 6252.064874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.064879] pending: vmstat_update [ 6252.064884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.064890] pending: vmstat_update [ 6252.064894] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6252.064899] pending: vmstat_update [ 6252.064905] workqueue writeback: flags=0x4a [ 6252.064907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6252.064913] pending: wb_workfn, wb_workfn [ 6252.064948] workqueue xprtiod: flags=0xa [ 6252.064951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6252.064957] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6252.064975] workqueue ipv6_addrconf: flags=0x40008 [ 6252.064978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6252.064985] pending: addrconf_verify_work [ 6252.065022] workqueue knbd0-recv: flags=0x1a [ 6252.065026] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6252.065033] in-flight: 651:recv_work [ 6252.065043] workqueue ptlrpc_pinger: flags=0x2 [ 6252.065045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6252.065050] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6252.065576] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4334s workers=4 idle: 37 31846 28897 [ 6252.065590] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6241s workers=2 idle: 653 [ 6252.517032] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6252.528764] obd_mod_free+0x46/0x140 [ptlrpc] [ 6252.537204] mdc_commit_open+0x91/0xb0 [mdc] [ 6252.543553] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6252.559421] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6252.571287] after_reply+0xe2c/0x1780 [ptlrpc] [ 6252.573237] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6252.587236] ? __next_timer_interrupt+0x160/0x160 [ 6252.591343] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6252.599395] ? do_wait_intr+0xf0/0xf0 [ 6252.607972] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6252.614059] kthread+0x1d1/0x200 [ 6252.619286] ? set_kthread_struct+0x70/0x70 [ 6252.627969] ret_from_fork+0x1f/0x30 [ 6260.060548] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6260.075357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6260.119248] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6260.128387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6260.141702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6260.145355] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6260.164924] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6260.169327] RAX: 0000000000007e14 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6260.171654] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6260.174023] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6260.176632] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6260.179577] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6260.182081] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6260.184779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6260.187438] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6260.190557] Call Trace: [ 6260.191521] [ 6260.192267] ? show_regs.cold.9+0x22/0x2f [ 6260.193937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6260.195870] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6260.197532] ? watchdog+0x40/0x40 [ 6260.198650] ? hrtimer_interrupt+0x128/0x2c0 [ 6260.199944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6260.201418] ? apic_timer_interrupt+0xf/0x20 [ 6260.202965] [ 6260.203791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6260.205914] do_raw_spin_lock+0x177/0x190 [ 6260.207452] _raw_spin_lock+0x21/0x30 [ 6260.208868] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6260.211167] ? do_raw_spin_unlock+0x75/0x190 [ 6260.212850] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6260.214988] ? do_wait_intr+0xf0/0xf0 [ 6260.216813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6260.219529] mdc_close+0x53b/0xd70 [mdc] [ 6260.221342] lmv_close+0x18d/0x490 [lmv] [ 6260.222983] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6260.225435] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6260.228193] ll_md_close+0x274/0x800 [lustre] [ 6260.230506] ll_file_release+0x128/0xc50 [lustre] [ 6260.237082] __fput+0x10d/0x440 [ 6260.239587] ____fput+0x12/0x20 [ 6260.241736] task_work_run+0xc6/0x110 [ 6260.244231] exit_to_usermode_loop+0x1dd/0x1f0 [ 6260.249542] do_syscall_64+0x3d6/0x440 [ 6260.252610] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6260.256770] RIP: 0033:0x7fa749552cd8 [ 6260.260209] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6260.268705] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6260.272866] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6260.275730] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6260.278245] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6260.283919] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6260.287694] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6268.057743] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6268.072123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6268.126569] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6268.139635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6268.148255] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6268.153910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6268.163651] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6268.176506] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6268.185767] RAX: 0000000000007f8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6268.193537] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6268.202770] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6268.207313] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6268.211232] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6268.215699] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6268.218714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6268.221064] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6268.223668] Call Trace: [ 6268.225689] [ 6268.226336] ? show_regs.cold.9+0x22/0x2f [ 6268.228156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6268.230581] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6268.232619] ? watchdog+0x40/0x40 [ 6268.233827] ? hrtimer_interrupt+0x128/0x2c0 [ 6268.235470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6268.237302] ? apic_timer_interrupt+0xf/0x20 [ 6268.239211] [ 6268.240160] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6268.242603] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6268.244755] do_raw_spin_lock+0x177/0x190 [ 6268.246546] _raw_spin_lock+0x21/0x30 [ 6268.248027] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6268.250612] process_one_work+0x2c8/0x700 [ 6268.252021] worker_thread+0x296/0x6e0 [ 6268.253872] ? rescuer_thread+0x570/0x570 [ 6268.255709] kthread+0x1d1/0x200 [ 6268.256685] ? set_kthread_struct+0x70/0x70 [ 6268.257827] ret_from_fork+0x1f/0x30 [ 6280.057261] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6280.063863] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6280.064509] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6280.064577] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6280.064583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6280.064586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6280.064599] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6280.064603] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6280.064607] RAX: 0000000000006b10 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6280.064609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6280.064611] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6280.064613] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6280.064793] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6280.064805] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6280.064807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6280.064810] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6280.064817] Call Trace: [ 6280.064822] [ 6280.064827] ? show_regs.cold.9+0x22/0x2f [ 6280.064836] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6280.064842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6280.064845] ? watchdog+0x40/0x40 [ 6280.064849] ? hrtimer_interrupt+0x128/0x2c0 [ 6280.064853] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6280.064860] ? apic_timer_interrupt+0xf/0x20 [ 6280.064865] [ 6280.064867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6280.064873] do_raw_spin_lock+0x177/0x190 [ 6280.064877] _raw_spin_lock+0x21/0x30 [ 6280.064882] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6280.065278] ? do_raw_spin_unlock+0x75/0x190 [ 6280.065285] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6280.065484] ? do_wait_intr+0xf0/0xf0 [ 6280.065493] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6280.065685] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6280.066042] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6280.066317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.066432] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6280.066471] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6280.066501] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6280.066531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.066616] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6280.067039] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6280.067067] lmv_intent_open+0x338/0x1150 [lmv] [ 6280.067092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.067434] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.067516] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6280.067540] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6280.067623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.067903] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6280.067997] ? do_raw_spin_unlock+0x75/0x190 [ 6280.068005] ? _raw_spin_unlock+0x12/0x30 [ 6280.068011] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6280.068038] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6280.068057] ll_atomic_open+0x36a/0xd70 [lustre] [ 6280.068138] lookup_open+0x356/0xb40 [ 6280.068863] do_last+0x313/0xfc0 [ 6280.068871] ? nd_jump_root+0xe5/0x160 [ 6280.068876] ? path_init+0x437/0x520 [ 6280.068881] path_openat+0xf7/0x500 [ 6280.068886] ? do_raw_spin_unlock+0x75/0x190 [ 6280.068891] do_filp_open+0x99/0x140 [ 6280.068897] ? getname_flags+0x6e/0x330 [ 6280.068902] ? __check_object_size+0xff/0x256 [ 6280.068909] ? do_raw_spin_unlock+0x75/0x190 [ 6280.068912] ? _raw_spin_unlock+0x12/0x30 [ 6280.068916] do_sys_openat2+0x2b4/0x410 [ 6280.068921] do_sys_open+0x73/0xa0 [ 6280.068925] __x64_sys_openat+0x24/0x30 [ 6280.068929] do_syscall_64+0xc1/0x440 [ 6280.068935] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6280.068940] RIP: 0033:0x7fa749552332 [ 6280.068948] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6280.068952] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6280.068958] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6280.068960] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6280.068963] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6280.068965] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6280.068983] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6280.483352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6280.528740] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6280.537130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6280.548520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6280.555625] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6280.568237] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6280.572789] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6280.579332] RDX: 000000000000718a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6280.582658] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6280.593697] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6280.601011] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6280.605512] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6280.612286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6280.620920] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6280.630375] Call Trace: [ 6280.631774] [ 6280.633120] ? show_regs.cold.9+0x22/0x2f [ 6280.651726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6280.653509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6280.661022] ? watchdog+0x40/0x40 [ 6280.663200] ? hrtimer_interrupt+0x128/0x2c0 [ 6280.666639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6280.672182] ? apic_timer_interrupt+0xf/0x20 [ 6280.678233] [ 6280.679268] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6280.686659] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6280.690167] do_raw_spin_lock+0x177/0x190 [ 6280.695862] _raw_spin_lock+0x21/0x30 [ 6280.699140] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6280.703556] obd_mod_free+0x46/0x140 [ptlrpc] [ 6280.706002] mdc_commit_open+0x91/0xb0 [mdc] [ 6280.709732] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6280.718066] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6280.722150] after_reply+0xe2c/0x1780 [ptlrpc] [ 6280.724772] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6280.728517] ? __next_timer_interrupt+0x160/0x160 [ 6280.731204] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6280.739178] ? do_wait_intr+0xf0/0xf0 [ 6280.741345] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6280.746531] kthread+0x1d1/0x200 [ 6280.747648] ? set_kthread_struct+0x70/0x70 [ 6280.756876] ret_from_fork+0x1f/0x30 [ 6282.211047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4365s! [ 6282.214359] Showing busy workqueues and worker pools: [ 6282.216371] workqueue events: flags=0x0 [ 6282.229631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6282.232778] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6282.239206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6282.242574] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6282.246992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.249904] pending: kfree_rcu_monitor [ 6282.251562] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6282.254632] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6282.262381] workqueue events_long: flags=0x0 [ 6282.264731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.267213] pending: lu_objects_flush [obdclass] [ 6282.269405] workqueue events_unbound: flags=0x2 [ 6282.271083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6282.273886] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6282.277771] workqueue events_power_efficient: flags=0x80 [ 6282.279879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.296801] pending: check_lifetime [ 6282.298472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.301203] pending: neigh_periodic_work [ 6282.302963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.305300] pending: neigh_periodic_work [ 6282.310567] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.316606] pending: do_cache_clean [ 6282.318582] workqueue mm_percpu_wq: flags=0x8 [ 6282.320754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.323450] pending: vmstat_update [ 6282.325362] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.328481] pending: vmstat_update [ 6282.330542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6282.333458] pending: vmstat_update [ 6282.335177] workqueue writeback: flags=0x4a [ 6282.336805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6282.339772] pending: wb_workfn, wb_workfn [ 6282.342106] workqueue xprtiod: flags=0xa [ 6282.343816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6282.346506] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6282.349981] workqueue ipv6_addrconf: flags=0x40008 [ 6282.351855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6282.354323] pending: addrconf_verify_work [ 6282.356405] workqueue knbd0-recv: flags=0x1a [ 6282.358257] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6282.364108] in-flight: 651:recv_work [ 6282.366076] workqueue ptlrpc_pinger: flags=0x2 [ 6282.368092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6282.371159] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6282.381557] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4365s workers=4 idle: 37 31846 28897 [ 6282.389615] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6272s workers=2 idle: 653 [ 6288.059053] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6288.067944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6288.112636] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6288.126673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6288.133147] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6288.136212] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6288.149136] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6288.156799] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6288.162194] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6288.166562] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6288.178909] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6288.184621] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6288.193503] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6288.201254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6288.204775] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6288.213248] Call Trace: [ 6288.216103] [ 6288.216933] ? show_regs.cold.9+0x22/0x2f [ 6288.220937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6288.225517] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6288.231312] ? watchdog+0x40/0x40 [ 6288.234741] ? hrtimer_interrupt+0x128/0x2c0 [ 6288.238210] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6288.241856] ? apic_timer_interrupt+0xf/0x20 [ 6288.246051] [ 6288.247738] ? native_safe_halt+0x2e/0x50 [ 6288.253268] kvm_wait+0xcd/0xe0 [ 6288.258032] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6288.263071] do_raw_spin_lock+0x177/0x190 [ 6288.266990] _raw_spin_lock+0x21/0x30 [ 6288.270744] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6288.277889] ? do_raw_spin_unlock+0x75/0x190 [ 6288.280243] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6288.288401] ? do_wait_intr+0xf0/0xf0 [ 6288.292231] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6288.301188] mdc_close+0x53b/0xd70 [mdc] [ 6288.305484] lmv_close+0x18d/0x490 [lmv] [ 6288.310411] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6288.318467] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6288.325303] ll_md_close+0x274/0x800 [lustre] [ 6288.329721] ll_file_release+0x128/0xc50 [lustre] [ 6288.334708] __fput+0x10d/0x440 [ 6288.338443] ____fput+0x12/0x20 [ 6288.342242] task_work_run+0xc6/0x110 [ 6288.348275] exit_to_usermode_loop+0x1dd/0x1f0 [ 6288.354417] do_syscall_64+0x3d6/0x440 [ 6288.357095] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6288.360853] RIP: 0033:0x7fa749552cd8 [ 6288.363815] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6288.384523] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6288.394497] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6288.401312] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6288.408689] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6288.414761] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6288.421690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6293.788643] rcu: INFO: rcu_sched self-detected stall on CPU [ 6293.794396] rcu: 2-...!: (2402133 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 6293.810027] (t=4380099 jiffies g=258701 q=2785) [ 6293.822247] rcu: rcu_sched kthread starved for 4376894 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6293.835987] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6293.848458] rcu: RCU grace-period kthread stack dump: [ 6293.850737] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6293.854792] Call Trace: [ 6293.856885] __schedule+0x351/0xcb0 [ 6293.858663] schedule+0xc0/0x180 [ 6293.860024] schedule_timeout+0xb4/0x190 [ 6293.861957] ? do_raw_spin_unlock+0x75/0x190 [ 6293.864227] ? __next_timer_interrupt+0x160/0x160 [ 6293.866899] ? prepare_to_swait_event+0xf3/0x140 [ 6293.871138] rcu_gp_kthread+0x8dd/0x15b0 [ 6293.873019] ? force_qs_rnp+0x2c0/0x2c0 [ 6293.876379] kthread+0x1d1/0x200 [ 6293.877948] ? set_kthread_struct+0x70/0x70 [ 6293.879948] ret_from_fork+0x1f/0x30 [ 6293.881857] rcu: Stack dump where RCU GP kthread last ran: [ 6293.884987] Sending NMI from CPU 2 to CPUs 1: [ 6293.887255] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 6293.888259] NMI backtrace for cpu 2 [ 6293.892032] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6293.898169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6293.902888] Call Trace: [ 6293.903962] [ 6293.904952] ? dump_stack+0xbb/0x10e [ 6293.906856] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6293.909347] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6293.911548] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6293.919666] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6293.926557] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6293.928578] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6293.929865] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6293.931272] ? update_process_times+0x6d/0xc0 [ 6293.932913] ? tick_sched_handle.isra.18+0x27/0x90 [ 6293.935804] ? tick_sched_timer+0x53/0xb0 [ 6293.937849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6293.940046] ? tick_sched_handle.isra.18+0x90/0x90 [ 6293.946400] ? hrtimer_interrupt+0x128/0x2c0 [ 6293.951899] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6293.953538] ? apic_timer_interrupt+0xf/0x20 [ 6293.963049] [ 6293.963822] ? native_safe_halt+0x2e/0x50 [ 6293.965121] ? kvm_wait+0xcd/0xe0 [ 6293.968612] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6293.970518] ? do_raw_spin_lock+0x177/0x190 [ 6293.974126] ? _raw_spin_lock+0x21/0x30 [ 6293.976293] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6293.980992] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6293.983603] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6293.986287] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6293.991688] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6293.995538] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6294.002652] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6294.007305] ? __next_timer_interrupt+0x160/0x160 [ 6294.010916] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6294.013685] ? do_wait_intr+0xf0/0xf0 [ 6294.015794] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6294.020676] ? kthread+0x1d1/0x200 [ 6294.022241] ? set_kthread_struct+0x70/0x70 [ 6294.027367] ? ret_from_fork+0x1f/0x30 [ 6308.054024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6308.056816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6308.072377] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6308.087217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6308.098238] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6308.099877] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6308.121285] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6308.128845] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6308.135594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6308.147697] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6308.152864] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6308.160146] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6308.166418] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6308.172599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6308.178657] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6308.184812] Call Trace: [ 6308.188170] [ 6308.194087] ? show_regs.cold.9+0x22/0x2f [ 6308.201544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6308.206685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6308.213544] ? watchdog+0x40/0x40 [ 6308.221557] ? hrtimer_interrupt+0x128/0x2c0 [ 6308.228853] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6308.239126] ? apic_timer_interrupt+0xf/0x20 [ 6308.248824] [ 6308.256679] ? native_safe_halt+0x2e/0x50 [ 6308.262106] kvm_wait+0xcd/0xe0 [ 6308.264811] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6308.277406] do_raw_spin_lock+0x177/0x190 [ 6308.278831] _raw_spin_lock+0x21/0x30 [ 6308.283656] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6308.290177] ? do_raw_spin_unlock+0x75/0x190 [ 6308.295988] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6308.305943] ? do_wait_intr+0xf0/0xf0 [ 6308.309989] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6308.313743] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6308.322361] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6308.328817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6308.334195] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6308.337312] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6308.341837] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6308.347203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6308.351825] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6308.360479] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6308.363404] lmv_intent_open+0x338/0x1150 [lmv] [ 6308.365517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6308.368035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6308.370620] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6308.372532] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6308.374628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6308.377431] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6308.378938] ? do_raw_spin_unlock+0x75/0x190 [ 6308.380707] ? _raw_spin_unlock+0x12/0x30 [ 6308.382321] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6308.384558] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6308.386653] ll_atomic_open+0x36a/0xd70 [lustre] [ 6308.389057] lookup_open+0x356/0xb40 [ 6308.390320] do_last+0x313/0xfc0 [ 6308.391932] ? nd_jump_root+0xe5/0x160 [ 6308.393973] ? path_init+0x437/0x520 [ 6308.396732] path_openat+0xf7/0x500 [ 6308.398777] ? do_raw_spin_unlock+0x75/0x190 [ 6308.401682] do_filp_open+0x99/0x140 [ 6308.404970] ? getname_flags+0x6e/0x330 [ 6308.408380] ? __check_object_size+0xff/0x256 [ 6308.409737] ? do_raw_spin_unlock+0x75/0x190 [ 6308.412679] ? _raw_spin_unlock+0x12/0x30 [ 6308.416829] do_sys_openat2+0x2b4/0x410 [ 6308.419029] do_sys_open+0x73/0xa0 [ 6308.420162] __x64_sys_openat+0x24/0x30 [ 6308.421434] do_syscall_64+0xc1/0x440 [ 6308.423680] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6308.430282] RIP: 0033:0x7fa749552332 [ 6308.433805] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6308.446313] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6308.451078] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6308.457590] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6308.461620] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6308.466176] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6308.471804] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6312.418854] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4395s! [ 6312.423780] Showing busy workqueues and worker pools: [ 6312.426695] workqueue events: flags=0x0 [ 6312.428854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6312.433815] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6312.446855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6312.459859] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6312.469708] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.472022] pending: kfree_rcu_monitor [ 6312.473127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6312.493947] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6312.498031] workqueue events_long: flags=0x0 [ 6312.505408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.510626] pending: lu_objects_flush [obdclass] [ 6312.517289] workqueue events_unbound: flags=0x2 [ 6312.523953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6312.537075] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6312.551630] workqueue events_power_efficient: flags=0x80 [ 6312.560861] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.565462] pending: check_lifetime [ 6312.578426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.588445] pending: neigh_periodic_work [ 6312.593918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.599389] pending: neigh_periodic_work [ 6312.603667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.608031] pending: do_cache_clean [ 6312.610134] workqueue mm_percpu_wq: flags=0x8 [ 6312.620082] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.626776] pending: vmstat_update [ 6312.628862] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.635745] pending: vmstat_update [ 6312.642454] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6312.650835] pending: vmstat_update [ 6312.656975] workqueue writeback: flags=0x4a [ 6312.660081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6312.666082] pending: wb_workfn, wb_workfn [ 6312.672011] workqueue xprtiod: flags=0xa [ 6312.675990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6312.681344] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6312.685764] workqueue ipv6_addrconf: flags=0x40008 [ 6312.696841] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6312.705458] pending: addrconf_verify_work [ 6312.713777] workqueue knbd0-recv: flags=0x1a [ 6312.720364] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6312.723610] in-flight: 651:recv_work [ 6312.725133] workqueue ptlrpc_pinger: flags=0x2 [ 6312.727078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6312.729690] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6312.731975] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4395s workers=4 idle: 37 31846 28897 [ 6312.735360] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6302s workers=2 idle: 653 [ 6316.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6316.065024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6316.121472] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6316.126243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6316.130674] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6316.134038] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6316.148883] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6316.153166] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6316.158426] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6316.162061] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6316.176958] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6316.185711] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6316.192987] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6316.196119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6316.198531] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6316.201696] Call Trace: [ 6316.203714] [ 6316.209924] ? show_regs.cold.9+0x22/0x2f [ 6316.212828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6316.216349] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6316.218167] ? watchdog+0x40/0x40 [ 6316.219757] ? hrtimer_interrupt+0x128/0x2c0 [ 6316.222409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6316.230699] ? apic_timer_interrupt+0xf/0x20 [ 6316.237991] [ 6316.242432] ? native_safe_halt+0x2e/0x50 [ 6316.244067] kvm_wait+0xcd/0xe0 [ 6316.246514] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6316.249654] do_raw_spin_lock+0x177/0x190 [ 6316.257601] _raw_spin_lock+0x21/0x30 [ 6316.261425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6316.264600] ? do_raw_spin_unlock+0x75/0x190 [ 6316.267041] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6316.269680] ? do_wait_intr+0xf0/0xf0 [ 6316.271295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6316.274288] mdc_close+0x53b/0xd70 [mdc] [ 6316.276318] lmv_close+0x18d/0x490 [lmv] [ 6316.278479] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6316.281250] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6316.286682] ll_md_close+0x274/0x800 [lustre] [ 6316.293298] ll_file_release+0x128/0xc50 [lustre] [ 6316.298441] __fput+0x10d/0x440 [ 6316.302667] ____fput+0x12/0x20 [ 6316.304881] task_work_run+0xc6/0x110 [ 6316.310009] exit_to_usermode_loop+0x1dd/0x1f0 [ 6316.325504] do_syscall_64+0x3d6/0x440 [ 6316.333270] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6316.345445] RIP: 0033:0x7fa749552cd8 [ 6316.351253] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6316.370663] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6316.386143] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6316.393206] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6316.398990] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6316.404272] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6316.407107] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6320.058018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6320.062735] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6320.066974] Modules linked in: [ 6320.073891] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.073962] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.073969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.073972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6320.073988] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6320.073992] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6320.073996] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6320.073998] RDX: 000000000000602e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6320.074000] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6320.074002] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6320.074004] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6320.074008] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6320.074012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.074014] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6320.074024] Call Trace: [ 6320.074030] [ 6320.074034] ? show_regs.cold.9+0x22/0x2f [ 6320.074050] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.074059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.074272] ? watchdog+0x40/0x40 [ 6320.074279] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.074284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.074292] ? apic_timer_interrupt+0xf/0x20 [ 6320.074297] [ 6320.074301] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6320.074306] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6320.074310] do_raw_spin_lock+0x177/0x190 [ 6320.074315] _raw_spin_lock+0x21/0x30 [ 6320.074321] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6320.075716] obd_mod_free+0x46/0x140 [ptlrpc] [ 6320.076624] mdc_commit_open+0x91/0xb0 [mdc] [ 6320.076886] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6320.077225] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6320.077642] after_reply+0xe2c/0x1780 [ptlrpc] [ 6320.085540] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.085620] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.085627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.085633] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6320.087431] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.087444] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6320.087449] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6320.087457] RAX: 00000000000038d4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6320.087459] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6320.087461] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6320.087463] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6320.087465] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6320.087469] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6320.087471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.087473] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6320.087481] Call Trace: [ 6320.087487] [ 6320.087491] ? show_regs.cold.9+0x22/0x2f [ 6320.087500] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.087506] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.087718] ? watchdog+0x40/0x40 [ 6320.087722] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.087726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.087733] ? apic_timer_interrupt+0xf/0x20 [ 6320.087738] [ 6320.087741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.087746] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6320.087751] do_raw_spin_lock+0x177/0x190 [ 6320.087756] _raw_spin_lock+0x21/0x30 [ 6320.087761] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6320.089587] process_one_work+0x2c8/0x700 [ 6320.089598] worker_thread+0x296/0x6e0 [ 6320.089603] ? rescuer_thread+0x570/0x570 [ 6320.089607] kthread+0x1d1/0x200 [ 6320.089611] ? set_kthread_struct+0x70/0x70 [ 6320.089614] ret_from_fork+0x1f/0x30 [ 6320.509710] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6320.513054] ? __next_timer_interrupt+0x160/0x160 [ 6320.515683] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6320.518711] ? do_wait_intr+0xf0/0xf0 [ 6320.520832] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6320.524229] kthread+0x1d1/0x200 [ 6320.529702] ? set_kthread_struct+0x70/0x70 [ 6320.532418] ret_from_fork+0x1f/0x30 [ 6340.054028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6340.059239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6340.090255] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6340.097283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6340.102702] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6340.105008] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6340.121924] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6340.136523] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6340.139136] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6340.157385] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6340.164446] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6340.182500] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6340.213289] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6340.235886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6340.248780] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6340.258598] Call Trace: [ 6340.267495] [ 6340.268980] ? show_regs.cold.9+0x22/0x2f [ 6340.276516] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6340.284395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6340.290832] ? watchdog+0x40/0x40 [ 6340.296479] ? hrtimer_interrupt+0x128/0x2c0 [ 6340.303022] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6340.307437] ? apic_timer_interrupt+0xf/0x20 [ 6340.314473] [ 6340.315320] ? native_safe_halt+0x2e/0x50 [ 6340.316798] kvm_wait+0xcd/0xe0 [ 6340.333042] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6340.348261] do_raw_spin_lock+0x177/0x190 [ 6340.351475] _raw_spin_lock+0x21/0x30 [ 6340.358011] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6340.369007] ? do_raw_spin_unlock+0x75/0x190 [ 6340.375951] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6340.379182] ? do_wait_intr+0xf0/0xf0 [ 6340.380499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6340.390264] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6340.395328] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6340.404065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6340.412893] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6340.419285] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6340.425486] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6340.430139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6340.433410] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6340.441244] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6340.447221] lmv_intent_open+0x338/0x1150 [lmv] [ 6340.448893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6340.463999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6340.470450] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6340.473927] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6340.480389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6340.486277] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6340.490524] ? do_raw_spin_unlock+0x75/0x190 [ 6340.492086] ? _raw_spin_unlock+0x12/0x30 [ 6340.496087] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6340.500000] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6340.504228] ll_atomic_open+0x36a/0xd70 [lustre] [ 6340.508333] lookup_open+0x356/0xb40 [ 6340.510276] do_last+0x313/0xfc0 [ 6340.514269] ? nd_jump_root+0xe5/0x160 [ 6340.519341] ? path_init+0x437/0x520 [ 6340.524102] path_openat+0xf7/0x500 [ 6340.527113] ? do_raw_spin_unlock+0x75/0x190 [ 6340.537340] do_filp_open+0x99/0x140 [ 6340.538661] ? getname_flags+0x6e/0x330 [ 6340.543764] ? __check_object_size+0xff/0x256 [ 6340.551481] ? do_raw_spin_unlock+0x75/0x190 [ 6340.563028] ? _raw_spin_unlock+0x12/0x30 [ 6340.569822] do_sys_openat2+0x2b4/0x410 [ 6340.572959] do_sys_open+0x73/0xa0 [ 6340.577052] __x64_sys_openat+0x24/0x30 [ 6340.579520] do_syscall_64+0xc1/0x440 [ 6340.580633] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6340.584425] RIP: 0033:0x7fa749552332 [ 6340.587281] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6340.612567] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6340.616273] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6340.625979] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6340.630944] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6340.635452] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6340.641941] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6343.137031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4425s! [ 6343.140647] Showing busy workqueues and worker pools: [ 6343.143011] workqueue events: flags=0x0 [ 6343.144799] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6343.147936] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6343.154848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6343.160774] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6343.170565] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.177172] pending: kfree_rcu_monitor [ 6343.179056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6343.187073] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6343.202594] workqueue events_long: flags=0x0 [ 6343.206980] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.226110] pending: lu_objects_flush [obdclass] [ 6343.228882] workqueue events_unbound: flags=0x2 [ 6343.233081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6343.240044] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6343.246936] workqueue events_power_efficient: flags=0x80 [ 6343.248965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.251108] pending: check_lifetime [ 6343.252423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.254758] pending: neigh_periodic_work [ 6343.256211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.258317] pending: neigh_periodic_work [ 6343.259500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.262078] pending: do_cache_clean [ 6343.263122] workqueue mm_percpu_wq: flags=0x8 [ 6343.264485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.268729] pending: vmstat_update [ 6343.269888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.273130] pending: vmstat_update [ 6343.274644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6343.276915] pending: vmstat_update [ 6343.278440] workqueue writeback: flags=0x4a [ 6343.279814] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6343.282142] pending: wb_workfn, wb_workfn [ 6343.285389] workqueue xprtiod: flags=0xa [ 6343.288353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6343.290390] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6343.292874] workqueue ipv6_addrconf: flags=0x40008 [ 6343.294385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6343.296750] pending: addrconf_verify_work [ 6343.298076] workqueue knbd0-recv: flags=0x1a [ 6343.299403] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6343.301165] in-flight: 651:recv_work [ 6343.302386] workqueue ptlrpc_pinger: flags=0x2 [ 6343.303745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6343.305392] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6343.307391] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4426s workers=4 idle: 37 31846 28897 [ 6343.309705] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6333s workers=2 idle: 653 [ 6344.059221] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6344.064925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6344.098663] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6344.106338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6344.119316] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6344.124414] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6344.134303] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6344.140475] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6344.156930] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6344.159385] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6344.166813] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6344.172433] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6344.176532] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6344.180934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6344.187417] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6344.191346] Call Trace: [ 6344.193108] [ 6344.194263] ? show_regs.cold.9+0x22/0x2f [ 6344.203352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6344.204925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6344.211921] ? watchdog+0x40/0x40 [ 6344.214354] ? hrtimer_interrupt+0x128/0x2c0 [ 6344.216519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6344.219380] ? apic_timer_interrupt+0xf/0x20 [ 6344.222128] [ 6344.223347] ? native_safe_halt+0x2e/0x50 [ 6344.225241] kvm_wait+0xcd/0xe0 [ 6344.226018] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6344.227438] do_raw_spin_lock+0x177/0x190 [ 6344.234930] _raw_spin_lock+0x21/0x30 [ 6344.237567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6344.245167] ? do_raw_spin_unlock+0x75/0x190 [ 6344.248358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6344.251066] ? do_wait_intr+0xf0/0xf0 [ 6344.253462] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6344.256514] mdc_close+0x53b/0xd70 [mdc] [ 6344.257988] lmv_close+0x18d/0x490 [lmv] [ 6344.265424] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6344.272457] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6344.277930] ll_md_close+0x274/0x800 [lustre] [ 6344.285305] ll_file_release+0x128/0xc50 [lustre] [ 6344.292102] __fput+0x10d/0x440 [ 6344.296089] ____fput+0x12/0x20 [ 6344.300819] task_work_run+0xc6/0x110 [ 6344.303402] exit_to_usermode_loop+0x1dd/0x1f0 [ 6344.305917] do_syscall_64+0x3d6/0x440 [ 6344.310417] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6344.313451] RIP: 0033:0x7fa749552cd8 [ 6344.315904] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6344.338900] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6344.343211] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6344.347781] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6344.356545] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6344.361724] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6344.373808] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6348.059417] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6348.062165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.063409] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6348.107476] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.107501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.107508] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6348.108321] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6348.108334] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6348.108339] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6348.108343] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6348.108346] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6348.108348] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6348.108350] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6348.108353] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6348.108356] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6348.108359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.108362] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6348.108371] Call Trace: [ 6348.108378] [ 6348.108383] ? show_regs.cold.9+0x22/0x2f [ 6348.108392] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.108400] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.108404] ? watchdog+0x40/0x40 [ 6348.108409] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.108414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.108421] ? apic_timer_interrupt+0xf/0x20 [ 6348.108426] [ 6348.108430] ? native_safe_halt+0x2e/0x50 [ 6348.108435] kvm_wait+0xcd/0xe0 [ 6348.108442] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6348.108451] do_raw_spin_lock+0x177/0x190 [ 6348.108456] _raw_spin_lock+0x21/0x30 [ 6348.108461] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6348.108711] process_one_work+0x2c8/0x700 [ 6348.108719] worker_thread+0x296/0x6e0 [ 6348.108724] ? rescuer_thread+0x570/0x570 [ 6348.108728] kthread+0x1d1/0x200 [ 6348.108733] ? set_kthread_struct+0x70/0x70 [ 6348.108736] ret_from_fork+0x1f/0x30 [ 6348.317424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.388902] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.399199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.411798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6348.418032] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6348.442025] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6348.452168] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6348.459455] RDX: 0000000000003136 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6348.474995] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6348.486711] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6348.499903] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6348.509905] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6348.522874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.524694] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6348.535689] Call Trace: [ 6348.541723] [ 6348.542492] ? show_regs.cold.9+0x22/0x2f [ 6348.543792] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.552371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.559183] ? watchdog+0x40/0x40 [ 6348.560985] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.563960] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.569374] ? apic_timer_interrupt+0xf/0x20 [ 6348.579964] [ 6348.592951] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6348.595859] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6348.603067] do_raw_spin_lock+0x177/0x190 [ 6348.607984] _raw_spin_lock+0x21/0x30 [ 6348.612856] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6348.617081] obd_mod_free+0x46/0x140 [ptlrpc] [ 6348.625735] mdc_commit_open+0x91/0xb0 [mdc] [ 6348.629117] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6348.636521] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6348.639918] after_reply+0xe2c/0x1780 [ptlrpc] [ 6348.643824] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6348.647016] ? __next_timer_interrupt+0x160/0x160 [ 6348.649590] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6348.653430] ? do_wait_intr+0xf0/0xf0 [ 6348.657303] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6348.664814] kthread+0x1d1/0x200 [ 6348.665848] ? set_kthread_struct+0x70/0x70 [ 6348.667033] ret_from_fork+0x1f/0x30 [ 6368.054065] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6368.056700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6368.105496] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6368.112086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6368.125872] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6368.127392] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6368.162737] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6368.172026] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6368.176368] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6368.192189] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6368.198798] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6368.204105] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6368.219411] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6368.233797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6368.252873] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6368.265959] Call Trace: [ 6368.266905] [ 6368.285619] ? show_regs.cold.9+0x22/0x2f [ 6368.286885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6368.295202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6368.300161] ? watchdog+0x40/0x40 [ 6368.307609] ? hrtimer_interrupt+0x128/0x2c0 [ 6368.314323] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6368.316144] ? apic_timer_interrupt+0xf/0x20 [ 6368.327418] [ 6368.328809] ? native_safe_halt+0x2e/0x50 [ 6368.334730] kvm_wait+0xcd/0xe0 [ 6368.339787] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6368.343189] do_raw_spin_lock+0x177/0x190 [ 6368.347847] _raw_spin_lock+0x21/0x30 [ 6368.351366] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6368.356638] ? do_raw_spin_unlock+0x75/0x190 [ 6368.363013] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6368.371031] ? do_wait_intr+0xf0/0xf0 [ 6368.375371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6368.381879] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6368.391063] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6368.397098] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6368.400276] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6368.408406] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6368.414771] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6368.425362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6368.434310] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6368.436504] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6368.458578] lmv_intent_open+0x338/0x1150 [lmv] [ 6368.460267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6368.472772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6368.478765] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6368.483294] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6368.486798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6368.506903] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6368.516859] ? do_raw_spin_unlock+0x75/0x190 [ 6368.519333] ? _raw_spin_unlock+0x12/0x30 [ 6368.529896] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6368.540693] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6368.551809] ll_atomic_open+0x36a/0xd70 [lustre] [ 6368.561482] lookup_open+0x356/0xb40 [ 6368.565748] do_last+0x313/0xfc0 [ 6368.583406] ? nd_jump_root+0xe5/0x160 [ 6368.588107] ? path_init+0x437/0x520 [ 6368.594886] path_openat+0xf7/0x500 [ 6368.600454] ? do_raw_spin_unlock+0x75/0x190 [ 6368.610513] do_filp_open+0x99/0x140 [ 6368.611656] ? getname_flags+0x6e/0x330 [ 6368.612932] ? __check_object_size+0xff/0x256 [ 6368.633073] ? do_raw_spin_unlock+0x75/0x190 [ 6368.634568] ? _raw_spin_unlock+0x12/0x30 [ 6368.653583] do_sys_openat2+0x2b4/0x410 [ 6368.665524] do_sys_open+0x73/0xa0 [ 6368.670722] __x64_sys_openat+0x24/0x30 [ 6368.684116] do_syscall_64+0xc1/0x440 [ 6368.691449] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6368.713772] RIP: 0033:0x7fa749552332 [ 6368.714991] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6368.743645] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6368.760812] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6368.771767] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6368.791166] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6368.808837] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6368.821695] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6372.059011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6372.064469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6372.089533] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6372.102255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6372.117223] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6372.126204] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6372.144467] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6372.154083] RAX: 0000000000001e66 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6372.164268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6372.172787] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6372.180061] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6372.185542] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6372.190393] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6372.194500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6372.197722] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6372.205245] Call Trace: [ 6372.213959] [ 6372.214870] ? show_regs.cold.9+0x22/0x2f [ 6372.221515] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6372.226876] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6372.229770] ? watchdog+0x40/0x40 [ 6372.234459] ? hrtimer_interrupt+0x128/0x2c0 [ 6372.242358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6372.246866] ? apic_timer_interrupt+0xf/0x20 [ 6372.252530] [ 6372.254864] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6372.266238] do_raw_spin_lock+0x177/0x190 [ 6372.269725] _raw_spin_lock+0x21/0x30 [ 6372.279165] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6372.287671] ? do_raw_spin_unlock+0x75/0x190 [ 6372.289702] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6372.300734] ? do_wait_intr+0xf0/0xf0 [ 6372.307661] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6372.313886] mdc_close+0x53b/0xd70 [mdc] [ 6372.319186] lmv_close+0x18d/0x490 [lmv] [ 6372.321820] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6372.331045] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6372.345599] ll_md_close+0x274/0x800 [lustre] [ 6372.347943] ll_file_release+0x128/0xc50 [lustre] [ 6372.352542] __fput+0x10d/0x440 [ 6372.353613] ____fput+0x12/0x20 [ 6372.356448] task_work_run+0xc6/0x110 [ 6372.358596] exit_to_usermode_loop+0x1dd/0x1f0 [ 6372.360965] do_syscall_64+0x3d6/0x440 [ 6372.362823] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6372.371114] RIP: 0033:0x7fa749552cd8 [ 6372.373472] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6372.387262] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6372.393781] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6372.398059] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6372.403026] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6372.408867] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6372.416482] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6373.354964] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4456s! [ 6373.358808] Showing busy workqueues and worker pools: [ 6373.361478] workqueue events: flags=0x0 [ 6373.363172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6373.365861] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6373.375790] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6373.382014] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6373.388991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.393173] pending: kfree_rcu_monitor [ 6373.398627] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6373.401058] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6373.405400] workqueue events_long: flags=0x0 [ 6373.414240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.419411] pending: lu_objects_flush [obdclass] [ 6373.421875] workqueue events_unbound: flags=0x2 [ 6373.425139] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6373.427841] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6373.437904] workqueue events_power_efficient: flags=0x80 [ 6373.447755] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.453319] pending: check_lifetime [ 6373.457668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.462993] pending: neigh_periodic_work [ 6373.467532] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.471108] pending: neigh_periodic_work [ 6373.473137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.478007] pending: do_cache_clean [ 6373.482699] workqueue mm_percpu_wq: flags=0x8 [ 6373.486648] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.490146] pending: vmstat_update [ 6373.494107] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.503296] pending: vmstat_update [ 6373.504983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6373.517867] pending: vmstat_update [ 6373.521416] workqueue writeback: flags=0x4a [ 6373.525562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6373.530734] pending: wb_workfn, wb_workfn [ 6373.533119] workqueue xprtiod: flags=0xa [ 6373.536369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6373.544194] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6373.547206] workqueue ipv6_addrconf: flags=0x40008 [ 6373.554657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6373.560327] pending: addrconf_verify_work [ 6373.562483] workqueue knbd0-recv: flags=0x1a [ 6373.567090] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6373.569787] in-flight: 651:recv_work [ 6373.574243] workqueue ptlrpc_pinger: flags=0x2 [ 6373.576982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6373.590275] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6373.603117] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4456s workers=4 idle: 37 31846 28897 [ 6373.619178] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6363s workers=2 idle: 653 [ 6376.058070] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6376.062867] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6376.069550] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6376.069696] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6376.069703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6376.069710] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6376.070066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.070078] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6376.070082] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6376.070087] RAX: 0000000000007859 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6376.070090] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6376.070092] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6376.070094] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6376.070097] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6376.070100] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6376.070103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6376.070105] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6376.070114] Call Trace: [ 6376.070120] [ 6376.070125] ? show_regs.cold.9+0x22/0x2f [ 6376.070136] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6376.070142] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6376.070146] ? watchdog+0x40/0x40 [ 6376.070151] ? hrtimer_interrupt+0x128/0x2c0 [ 6376.070155] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6376.070163] ? apic_timer_interrupt+0xf/0x20 [ 6376.070167] [ 6376.070170] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.070174] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6376.070180] do_raw_spin_lock+0x177/0x190 [ 6376.070184] _raw_spin_lock+0x21/0x30 [ 6376.070189] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6376.070399] process_one_work+0x2c8/0x700 [ 6376.070407] worker_thread+0x296/0x6e0 [ 6376.070412] ? rescuer_thread+0x570/0x570 [ 6376.070416] kthread+0x1d1/0x200 [ 6376.070421] ? set_kthread_struct+0x70/0x70 [ 6376.070424] ret_from_fork+0x1f/0x30 [ 6376.281003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6376.346933] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6376.366014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6376.374219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6376.378039] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6376.391614] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6376.399594] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6376.401792] RDX: 0000000000007387 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6376.407807] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6376.410925] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6376.415168] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6376.423817] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6376.427401] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6376.428878] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6376.437206] Call Trace: [ 6376.441101] [ 6376.441723] ? show_regs.cold.9+0x22/0x2f [ 6376.442958] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6376.446343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6376.447803] ? watchdog+0x40/0x40 [ 6376.448917] ? hrtimer_interrupt+0x128/0x2c0 [ 6376.455143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6376.463889] ? apic_timer_interrupt+0xf/0x20 [ 6376.465405] [ 6376.465986] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6376.469299] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6376.470915] do_raw_spin_lock+0x177/0x190 [ 6376.472681] _raw_spin_lock+0x21/0x30 [ 6376.474186] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6376.477255] obd_mod_free+0x46/0x140 [ptlrpc] [ 6376.488834] mdc_commit_open+0x91/0xb0 [mdc] [ 6376.495806] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6376.497677] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6376.516254] after_reply+0xe2c/0x1780 [ptlrpc] [ 6376.520216] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6376.522727] ? __next_timer_interrupt+0x160/0x160 [ 6376.524164] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6376.525415] ? do_wait_intr+0xf0/0xf0 [ 6376.527955] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6376.534510] kthread+0x1d1/0x200 [ 6376.535525] ? set_kthread_struct+0x70/0x70 [ 6376.541333] ret_from_fork+0x1f/0x30 [ 6400.054020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6400.057598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6400.059020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6400.077351] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6400.077366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6400.077369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6400.077432] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6400.077437] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6400.077442] RAX: 00000000000047df RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6400.077444] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6400.077447] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6400.077449] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6400.077451] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6400.077455] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6400.077458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6400.077460] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6400.077468] Call Trace: [ 6400.077484] [ 6400.077493] ? show_regs.cold.9+0x22/0x2f [ 6400.077503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6400.077509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6400.077514] ? watchdog+0x40/0x40 [ 6400.077518] ? hrtimer_interrupt+0x128/0x2c0 [ 6400.077523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6400.077531] ? apic_timer_interrupt+0xf/0x20 [ 6400.077536] [ 6400.077539] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6400.077547] do_raw_spin_lock+0x177/0x190 [ 6400.077552] _raw_spin_lock+0x21/0x30 [ 6400.077556] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6400.077902] ? do_raw_spin_unlock+0x75/0x190 [ 6400.077908] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6400.078139] ? do_wait_intr+0xf0/0xf0 [ 6400.078148] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6400.078440] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6400.078770] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6400.079024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6400.079149] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6400.079186] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6400.079216] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6400.079243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6400.079331] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6400.083506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6400.083587] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6400.083594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6400.083597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6400.083611] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6400.083616] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6400.083620] RAX: 00000000000068d6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6400.083622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6400.083624] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6400.083626] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6400.083628] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6400.083632] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6400.083634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6400.083637] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6400.083646] Call Trace: [ 6400.083651] [ 6400.083657] ? show_regs.cold.9+0x22/0x2f [ 6400.083668] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6400.083676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6400.083680] ? watchdog+0x40/0x40 [ 6400.083684] ? hrtimer_interrupt+0x128/0x2c0 [ 6400.083689] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6400.083697] ? apic_timer_interrupt+0xf/0x20 [ 6400.083702] [ 6400.083705] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6400.083711] do_raw_spin_lock+0x177/0x190 [ 6400.083716] _raw_spin_lock+0x21/0x30 [ 6400.083722] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6400.084019] ? do_raw_spin_unlock+0x75/0x190 [ 6400.084026] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6400.084228] ? do_wait_intr+0xf0/0xf0 [ 6400.084236] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6400.084427] mdc_close+0x53b/0xd70 [mdc] [ 6400.084468] lmv_close+0x18d/0x490 [lmv] [ 6400.084498] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6400.084606] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6400.084688] ll_md_close+0x274/0x800 [lustre] [ 6400.084771] ll_file_release+0x128/0xc50 [lustre] [ 6400.084853] __fput+0x10d/0x440 [ 6400.086610] ____fput+0x12/0x20 [ 6400.089433] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6400.089494] lmv_intent_open+0x338/0x1150 [lmv] [ 6400.089530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6400.089708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6400.089799] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6400.089822] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6400.089921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6400.090006] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6400.090092] ? do_raw_spin_unlock+0x75/0x190 [ 6400.090100] ? _raw_spin_unlock+0x12/0x30 [ 6400.090107] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6400.090138] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6400.090159] ll_atomic_open+0x36a/0xd70 [lustre] [ 6400.090246] lookup_open+0x356/0xb40 [ 6400.090258] do_last+0x313/0xfc0 [ 6400.090263] ? nd_jump_root+0xe5/0x160 [ 6400.090268] ? path_init+0x437/0x520 [ 6400.090273] path_openat+0xf7/0x500 [ 6400.090278] ? do_raw_spin_unlock+0x75/0x190 [ 6400.090283] do_filp_open+0x99/0x140 [ 6400.090291] ? getname_flags+0x6e/0x330 [ 6400.090295] ? __check_object_size+0xff/0x256 [ 6400.090302] ? do_raw_spin_unlock+0x75/0x190 [ 6400.090306] ? _raw_spin_unlock+0x12/0x30 [ 6400.090311] do_sys_openat2+0x2b4/0x410 [ 6400.090316] do_sys_open+0x73/0xa0 [ 6400.090320] __x64_sys_openat+0x24/0x30 [ 6400.090325] do_syscall_64+0xc1/0x440 [ 6400.090331] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6400.090338] RIP: 0033:0x7fa749552332 [ 6400.090462] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6400.090468] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6400.090473] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6400.090476] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6400.090478] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6400.090480] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6400.090482] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6400.411452] task_work_run+0xc6/0x110 [ 6400.412934] exit_to_usermode_loop+0x1dd/0x1f0 [ 6400.414482] do_syscall_64+0x3d6/0x440 [ 6400.421015] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6400.424564] RIP: 0033:0x7fa749552cd8 [ 6400.425685] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6400.436923] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6400.439531] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6400.448458] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6400.457829] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6400.463194] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6400.468085] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6404.059914] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6404.062979] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6404.066552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6404.066627] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6404.066633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6404.066639] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6404.075690] Modules linked in: lustre(O) osp(O) [ 6404.115492] [ 6404.115501] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6404.115667] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6404.115671] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6404.115676] RAX: 0000000000002235 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6404.115679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6404.115681] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6404.115684] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6404.115686] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6404.115689] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6404.115692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6404.115694] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6404.115702] Call Trace: [ 6404.115707] [ 6404.115712] ? show_regs.cold.9+0x22/0x2f [ 6404.115721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6404.115727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6404.115731] ? watchdog+0x40/0x40 [ 6404.115735] ? hrtimer_interrupt+0x128/0x2c0 [ 6404.115739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6404.115747] ? apic_timer_interrupt+0xf/0x20 [ 6404.115752] [ 6404.115755] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6404.115760] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6404.115767] do_raw_spin_lock+0x177/0x190 [ 6404.115771] _raw_spin_lock+0x21/0x30 [ 6404.115776] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6404.116079] process_one_work+0x2c8/0x700 [ 6404.116085] worker_thread+0x296/0x6e0 [ 6404.116088] ? rescuer_thread+0x570/0x570 [ 6404.116091] kthread+0x1d1/0x200 [ 6404.116096] ? set_kthread_struct+0x70/0x70 [ 6404.116100] ret_from_fork+0x1f/0x30 [ 6404.241261] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6404.260862] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6404.270064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6404.274380] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6404.277006] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6404.286691] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6404.289702] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6404.298377] RDX: 000000000000707e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6404.301725] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6404.305404] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6404.307611] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6404.309813] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6404.318345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6404.324483] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6404.332615] Call Trace: [ 6404.333828] [ 6404.334669] ? show_regs.cold.9+0x22/0x2f [ 6404.336200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6404.338308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6404.340169] ? watchdog+0x40/0x40 [ 6404.341637] ? hrtimer_interrupt+0x128/0x2c0 [ 6404.343375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6404.345444] ? apic_timer_interrupt+0xf/0x20 [ 6404.347250] [ 6404.348302] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6404.350758] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6404.353033] do_raw_spin_lock+0x177/0x190 [ 6404.354800] _raw_spin_lock+0x21/0x30 [ 6404.356372] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6404.359536] obd_mod_free+0x46/0x140 [ptlrpc] [ 6404.362010] mdc_commit_open+0x91/0xb0 [mdc] [ 6404.364396] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6404.367331] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6404.370681] after_reply+0xe2c/0x1780 [ptlrpc] [ 6404.373334] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6404.375962] ? __next_timer_interrupt+0x160/0x160 [ 6404.378058] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6404.380527] ? do_wait_intr+0xf0/0xf0 [ 6404.382113] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6404.385012] kthread+0x1d1/0x200 [ 6404.386384] ? set_kthread_struct+0x70/0x70 [ 6404.388352] ret_from_fork+0x1f/0x30 [ 6404.391124] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4487s! [ 6404.396083] Showing busy workqueues and worker pools: [ 6404.398626] workqueue events: flags=0x0 [ 6404.400377] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6404.403098] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6404.408661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6404.411371] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6404.415017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.417448] pending: kfree_rcu_monitor [ 6404.419098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6404.422660] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6404.429415] workqueue events_long: flags=0x0 [ 6404.433492] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.438233] pending: lu_objects_flush [obdclass] [ 6404.449118] workqueue events_unbound: flags=0x2 [ 6404.452419] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6404.456215] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6404.469254] workqueue events_power_efficient: flags=0x80 [ 6404.471923] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.474799] pending: check_lifetime [ 6404.476282] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.479243] pending: neigh_periodic_work [ 6404.480781] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.483618] pending: neigh_periodic_work [ 6404.485290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.488131] pending: do_cache_clean [ 6404.489775] workqueue mm_percpu_wq: flags=0x8 [ 6404.491747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.494346] pending: vmstat_update [ 6404.497372] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.500982] pending: vmstat_update [ 6404.502375] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6404.505254] pending: vmstat_update [ 6404.506579] workqueue writeback: flags=0x4a [ 6404.507756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6404.510889] pending: wb_workfn, wb_workfn [ 6404.512586] workqueue xprtiod: flags=0xa [ 6404.514607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6404.518111] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6404.521911] workqueue ipv6_addrconf: flags=0x40008 [ 6404.523583] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6404.526296] pending: addrconf_verify_work [ 6404.528114] workqueue knbd0-recv: flags=0x1a [ 6404.530064] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6404.532326] in-flight: 651:recv_work [ 6404.533892] workqueue ptlrpc_pinger: flags=0x2 [ 6404.535422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6404.537821] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6404.540336] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4487s workers=4 idle: 37 31846 28897 [ 6404.543465] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6394s workers=2 idle: 653 [ 6428.060100] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6428.065819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6428.106325] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6428.114072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6428.120193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6428.122072] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6428.137930] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6428.144517] RAX: 0000000000007d20 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6428.153291] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6428.157520] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6428.164455] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6428.172432] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6428.175330] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6428.180316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6428.183059] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6428.186500] Call Trace: [ 6428.187617] [ 6428.188615] ? show_regs.cold.9+0x22/0x2f [ 6428.191045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6428.193473] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6428.196020] ? watchdog+0x40/0x40 [ 6428.198181] ? hrtimer_interrupt+0x128/0x2c0 [ 6428.200289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6428.202536] ? apic_timer_interrupt+0xf/0x20 [ 6428.204164] [ 6428.205019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6428.207050] do_raw_spin_lock+0x177/0x190 [ 6428.208965] _raw_spin_lock+0x21/0x30 [ 6428.210114] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6428.212251] ? do_raw_spin_unlock+0x75/0x190 [ 6428.214332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6428.216803] ? do_wait_intr+0xf0/0xf0 [ 6428.218815] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6428.220826] mdc_close+0x53b/0xd70 [mdc] [ 6428.222526] lmv_close+0x18d/0x490 [lmv] [ 6428.224010] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6428.225926] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6428.227986] ll_md_close+0x274/0x800 [lustre] [ 6428.229843] ll_file_release+0x128/0xc50 [lustre] [ 6428.231829] __fput+0x10d/0x440 [ 6428.232795] ____fput+0x12/0x20 [ 6428.233869] task_work_run+0xc6/0x110 [ 6428.235322] exit_to_usermode_loop+0x1dd/0x1f0 [ 6428.236860] do_syscall_64+0x3d6/0x440 [ 6428.238133] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6428.240040] RIP: 0033:0x7fa749552cd8 [ 6428.241291] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6428.247642] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6428.250274] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6428.252261] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6428.254500] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6428.257042] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6428.259937] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6432.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6432.057466] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6432.057908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6432.057991] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6432.057997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6432.057999] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.058014] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6432.058079] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6432.058083] RAX: 000000000000283f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6432.058086] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6432.058089] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6432.058091] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6432.058093] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6432.058096] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6432.058099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6432.058102] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6432.058110] Call Trace: [ 6432.058116] [ 6432.058122] ? show_regs.cold.9+0x22/0x2f [ 6432.058133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6432.058140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6432.058144] ? watchdog+0x40/0x40 [ 6432.058149] ? hrtimer_interrupt+0x128/0x2c0 [ 6432.058153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6432.058160] ? apic_timer_interrupt+0xf/0x20 [ 6432.058165] [ 6432.058167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.058173] do_raw_spin_lock+0x177/0x190 [ 6432.058178] _raw_spin_lock+0x21/0x30 [ 6432.058182] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6432.058693] ? do_raw_spin_unlock+0x75/0x190 [ 6432.058704] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6432.058911] ? do_wait_intr+0xf0/0xf0 [ 6432.058930] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6432.059242] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6432.059606] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6432.059822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.060098] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6432.060156] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6432.060191] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6432.060545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.060634] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6432.060863] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6432.061092] lmv_intent_open+0x338/0x1150 [lmv] [ 6432.061148] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6432.061156] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 6432.061124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.061170] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6432.061235] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6432.061214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.061297] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6432.061649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6432.061652] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6432.061663] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6432.061667] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6432.061672] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6432.061675] RDX: 00000000000007cf RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6432.061677] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6432.061679] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6432.061682] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6432.061685] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6432.061688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6432.061690] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6432.061698] Call Trace: [ 6432.061702] [ 6432.061705] ? show_regs.cold.9+0x22/0x2f [ 6432.061714] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6432.061720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6432.061724] ? watchdog+0x40/0x40 [ 6432.061728] ? hrtimer_interrupt+0x128/0x2c0 [ 6432.061732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6432.061740] ? apic_timer_interrupt+0xf/0x20 [ 6432.061744] [ 6432.061747] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6432.061751] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6432.061756] do_raw_spin_lock+0x177/0x190 [ 6432.061761] _raw_spin_lock+0x21/0x30 [ 6432.061766] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6432.062007] obd_mod_free+0x46/0x140 [ptlrpc] [ 6432.063732] mdc_commit_open+0x91/0xb0 [mdc] [ 6432.063770] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6432.063967] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6432.065728] after_reply+0xe2c/0x1780 [ptlrpc] [ 6432.065928] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6432.067757] ? __next_timer_interrupt+0x160/0x160 [ 6432.067770] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6432.067988] ? do_wait_intr+0xf0/0xf0 [ 6432.067997] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6432.068874] kthread+0x1d1/0x200 [ 6432.068881] ? set_kthread_struct+0x70/0x70 [ 6432.068885] ret_from_fork+0x1f/0x30 [ 6432.071995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel [ 6432.095432] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6432.095627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.095709] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6432.101603] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6432.101629] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6432.101636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6432.101643] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6432.105225] ? do_raw_spin_unlock+0x75/0x190 [ 6432.108483] [ 6432.108492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.108512] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6432.108516] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6432.108521] RAX: 0000000000002f98 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6432.108524] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6432.108526] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6432.108528] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6432.108531] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6432.108534] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6432.108537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6432.108539] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6432.108548] Call Trace: [ 6432.108554] [ 6432.108560] ? show_regs.cold.9+0x22/0x2f [ 6432.108570] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6432.108576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6432.108581] ? watchdog+0x40/0x40 [ 6432.108585] ? hrtimer_interrupt+0x128/0x2c0 [ 6432.108589] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6432.108598] ? apic_timer_interrupt+0xf/0x20 [ 6432.108604] [ 6432.108607] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.115417] ? _raw_spin_unlock+0x12/0x30 [ 6432.118745] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6432.118774] do_raw_spin_lock+0x177/0x190 [ 6432.121685] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6432.124691] _raw_spin_lock+0x21/0x30 [ 6432.127565] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6432.132656] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6432.139210] ll_atomic_open+0x36a/0xd70 [lustre] [ 6432.139364] lookup_open+0x356/0xb40 [ 6432.139376] do_last+0x313/0xfc0 [ 6432.139381] ? nd_jump_root+0xe5/0x160 [ 6432.139387] ? path_init+0x437/0x520 [ 6432.139391] path_openat+0xf7/0x500 [ 6432.139397] ? do_raw_spin_unlock+0x75/0x190 [ 6432.139404] do_filp_open+0x99/0x140 [ 6432.139411] ? getname_flags+0x6e/0x330 [ 6432.139416] ? __check_object_size+0xff/0x256 [ 6432.139424] ? do_raw_spin_unlock+0x75/0x190 [ 6432.139428] ? _raw_spin_unlock+0x12/0x30 [ 6432.139435] do_sys_openat2+0x2b4/0x410 [ 6432.139440] do_sys_open+0x73/0xa0 [ 6432.139444] __x64_sys_openat+0x24/0x30 [ 6432.139447] do_syscall_64+0xc1/0x440 [ 6432.139453] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6432.139460] RIP: 0033:0x7fa749552332 [ 6432.139468] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6432.139471] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6432.139634] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6432.139637] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6432.139638] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6432.139640] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6432.139641] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6432.753975] process_one_work+0x2c8/0x700 [ 6432.757590] worker_thread+0x296/0x6e0 [ 6432.762237] ? rescuer_thread+0x570/0x570 [ 6432.763781] kthread+0x1d1/0x200 [ 6432.766956] ? set_kthread_struct+0x70/0x70 [ 6432.770885] ret_from_fork+0x1f/0x30 [ 6434.785270] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4517s! [ 6434.791193] Showing busy workqueues and worker pools: [ 6434.796737] workqueue events: flags=0x0 [ 6434.804828] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6434.812067] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6434.832912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6434.843860] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6434.853344] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.858575] pending: kfree_rcu_monitor [ 6434.863099] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6434.866749] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6434.872535] workqueue events_long: flags=0x0 [ 6434.874650] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.877865] pending: lu_objects_flush [obdclass] [ 6434.882205] workqueue events_unbound: flags=0x2 [ 6434.885599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6434.888668] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6434.893099] workqueue events_power_efficient: flags=0x80 [ 6434.896177] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.899715] pending: check_lifetime [ 6434.901786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.904645] pending: neigh_periodic_work [ 6434.906569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.909663] pending: neigh_periodic_work [ 6434.911699] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.920604] pending: do_cache_clean [ 6434.922185] workqueue mm_percpu_wq: flags=0x8 [ 6434.924236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.927215] pending: vmstat_update [ 6434.929314] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.932613] pending: vmstat_update [ 6434.934496] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6434.938454] pending: vmstat_update [ 6434.940632] workqueue writeback: flags=0x4a [ 6434.942585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6434.945740] pending: wb_workfn, wb_workfn [ 6434.947762] workqueue xprtiod: flags=0xa [ 6434.949241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6434.952271] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6434.955880] workqueue ipv6_addrconf: flags=0x40008 [ 6434.958174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6434.960521] pending: addrconf_verify_work [ 6434.962229] workqueue knbd0-recv: flags=0x1a [ 6434.964161] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6434.966726] in-flight: 651:recv_work [ 6434.968276] workqueue ptlrpc_pinger: flags=0x2 [ 6434.969952] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6434.972428] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6434.974366] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4517s workers=4 idle: 37 31846 28897 [ 6434.976677] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6424s workers=2 idle: 653 [ 6456.059271] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6456.063957] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6456.109200] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6456.126661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6456.132467] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6456.139007] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6456.160141] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6456.167590] RAX: 000000000000254e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6456.174770] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6456.181710] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6456.191685] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6456.198310] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6456.203492] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6456.217369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6456.220356] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6456.223780] Call Trace: [ 6456.224850] [ 6456.226053] ? show_regs.cold.9+0x22/0x2f [ 6456.227544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6456.230159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6456.232877] ? watchdog+0x40/0x40 [ 6456.234659] ? hrtimer_interrupt+0x128/0x2c0 [ 6456.236698] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6456.239379] ? apic_timer_interrupt+0xf/0x20 [ 6456.241538] [ 6456.242507] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6456.245088] do_raw_spin_lock+0x177/0x190 [ 6456.247171] _raw_spin_lock+0x21/0x30 [ 6456.248821] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6456.251694] ? do_raw_spin_unlock+0x75/0x190 [ 6456.257389] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6456.260163] ? do_wait_intr+0xf0/0xf0 [ 6456.261605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6456.263926] mdc_close+0x53b/0xd70 [mdc] [ 6456.265525] lmv_close+0x18d/0x490 [lmv] [ 6456.267149] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6456.269843] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6456.272375] ll_md_close+0x274/0x800 [lustre] [ 6456.274462] ll_file_release+0x128/0xc50 [lustre] [ 6456.276787] __fput+0x10d/0x440 [ 6456.278177] ____fput+0x12/0x20 [ 6456.279682] task_work_run+0xc6/0x110 [ 6456.281484] exit_to_usermode_loop+0x1dd/0x1f0 [ 6456.283173] do_syscall_64+0x3d6/0x440 [ 6456.285153] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6456.287465] RIP: 0033:0x7fa749552cd8 [ 6456.289512] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6456.298327] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6456.301997] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6456.306055] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6456.309746] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6456.313161] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6456.323250] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6460.059400] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6460.061197] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6460.059122] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6460.061236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6460.061307] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6460.061312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6460.061315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.061329] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6460.061334] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6460.061345] RAX: 0000000000001708 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6460.061347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6460.061350] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6460.061351] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6460.061354] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6460.061357] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6460.061361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6460.061363] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6460.061372] Call Trace: [ 6460.061377] [ 6460.061382] ? show_regs.cold.9+0x22/0x2f [ 6460.061391] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6460.061397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6460.061402] ? watchdog+0x40/0x40 [ 6460.061406] ? hrtimer_interrupt+0x128/0x2c0 [ 6460.061410] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6460.061418] ? apic_timer_interrupt+0xf/0x20 [ 6460.061423] [ 6460.061426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.061433] do_raw_spin_lock+0x177/0x190 [ 6460.061437] _raw_spin_lock+0x21/0x30 [ 6460.061442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6460.061753] ? do_raw_spin_unlock+0x75/0x190 [ 6460.061759] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6460.061979] ? do_wait_intr+0xf0/0xf0 [ 6460.061987] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6460.062197] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6460.062426] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6460.062657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.063563] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6460.063601] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6460.063638] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6460.063666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.063750] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6460.065055] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6460.065104] lmv_intent_open+0x338/0x1150 [lmv] [ 6460.065137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.065250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.065352] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6460.065379] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6460.066304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.066392] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6460.066622] ? do_raw_spin_unlock+0x75/0x190 [ 6460.066630] ? _raw_spin_unlock+0x12/0x30 [ 6460.066637] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6460.066667] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6460.066684] ll_atomic_open+0x36a/0xd70 [lustre] [ 6460.066766] lookup_open+0x356/0xb40 [ 6460.067101] do_last+0x313/0xfc0 [ 6460.067108] ? nd_jump_root+0xe5/0x160 [ 6460.067113] ? path_init+0x437/0x520 [ 6460.067117] path_openat+0xf7/0x500 [ 6460.067122] ? do_raw_spin_unlock+0x75/0x190 [ 6460.067128] do_filp_open+0x99/0x140 [ 6460.067135] ? getname_flags+0x6e/0x330 [ 6460.067139] ? __check_object_size+0xff/0x256 [ 6460.067147] ? do_raw_spin_unlock+0x75/0x190 [ 6460.067151] ? _raw_spin_unlock+0x12/0x30 [ 6460.067155] do_sys_openat2+0x2b4/0x410 [ 6460.067160] do_sys_open+0x73/0xa0 [ 6460.067164] __x64_sys_openat+0x24/0x30 [ 6460.067167] do_syscall_64+0xc1/0x440 [ 6460.067174] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6460.067180] RIP: 0033:0x7fa749552332 [ 6460.067188] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6460.067192] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6460.070338] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6460.070342] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6460.070345] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6460.070347] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6460.070350] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6460.071150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6460.071296] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6460.071303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6460.071376] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6460.071801] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.071821] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6460.071826] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6460.071831] RAX: 0000000000004f0e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6460.071834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6460.071837] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6460.071839] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6460.071841] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6460.071844] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6460.071847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6460.071849] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6460.071856] Call Trace: [ 6460.071862] [ 6460.071866] ? show_regs.cold.9+0x22/0x2f [ 6460.071875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6460.071881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6460.071884] ? watchdog+0x40/0x40 [ 6460.071888] ? hrtimer_interrupt+0x128/0x2c0 [ 6460.071892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6460.071899] ? apic_timer_interrupt+0xf/0x20 [ 6460.071903] [ 6460.071906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.071910] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6460.071914] do_raw_spin_lock+0x177/0x190 [ 6460.071918] _raw_spin_lock+0x21/0x30 [ 6460.071922] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6460.072296] process_one_work+0x2c8/0x700 [ 6460.072303] worker_thread+0x296/0x6e0 [ 6460.072307] ? rescuer_thread+0x570/0x570 [ 6460.081339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6460.081423] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6460.081430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6460.081433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6460.081447] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6460.081451] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6460.081456] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6460.081458] RDX: 0000000000003baf RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6460.081461] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6460.081463] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6460.081465] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6460.081469] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6460.081472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6460.081758] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6460.081767] Call Trace: [ 6460.081773] [ 6460.086739] kthread+0x1d1/0x200 [ 6460.086760] ? set_kthread_struct+0x70/0x70 [ 6460.086765] ret_from_fork+0x1f/0x30 [ 6460.900732] ? show_regs.cold.9+0x22/0x2f [ 6460.902945] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6460.905328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6460.908131] ? watchdog+0x40/0x40 [ 6460.909861] ? hrtimer_interrupt+0x128/0x2c0 [ 6460.911941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6460.913988] ? apic_timer_interrupt+0xf/0x20 [ 6460.915881] [ 6460.916755] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6460.918584] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6460.920429] do_raw_spin_lock+0x177/0x190 [ 6460.922343] _raw_spin_lock+0x21/0x30 [ 6460.923769] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6460.926959] obd_mod_free+0x46/0x140 [ptlrpc] [ 6460.935998] mdc_commit_open+0x91/0xb0 [mdc] [ 6460.938972] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6460.941364] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6460.944235] after_reply+0xe2c/0x1780 [ptlrpc] [ 6460.947250] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6460.949601] ? __next_timer_interrupt+0x160/0x160 [ 6460.951331] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6460.953062] ? do_wait_intr+0xf0/0xf0 [ 6460.955612] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6460.958644] kthread+0x1d1/0x200 [ 6460.960058] ? set_kthread_struct+0x70/0x70 [ 6460.961635] ret_from_fork+0x1f/0x30 [ 6464.993742] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4547s! [ 6464.998673] Showing busy workqueues and worker pools: [ 6465.003160] workqueue events: flags=0x0 [ 6465.005586] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6465.009541] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6465.016365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6465.021951] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6465.028940] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.034554] pending: kfree_rcu_monitor [ 6465.037281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6465.042580] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6465.047823] workqueue events_long: flags=0x0 [ 6465.052499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.055813] pending: lu_objects_flush [obdclass] [ 6465.059874] workqueue events_unbound: flags=0x2 [ 6465.063780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6465.068327] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6465.074269] workqueue events_power_efficient: flags=0x80 [ 6465.077647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.082010] pending: check_lifetime [ 6465.083958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.090281] pending: neigh_periodic_work [ 6465.094579] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.099860] pending: neigh_periodic_work [ 6465.104188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.109828] pending: do_cache_clean [ 6465.112038] workqueue mm_percpu_wq: flags=0x8 [ 6465.115966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.120129] pending: vmstat_update [ 6465.122330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.126272] pending: vmstat_update [ 6465.129523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6465.134620] pending: vmstat_update [ 6465.137047] workqueue writeback: flags=0x4a [ 6465.139262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6465.144734] pending: wb_workfn, wb_workfn [ 6465.147767] workqueue xprtiod: flags=0xa [ 6465.149802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6465.154393] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6465.161634] workqueue ipv6_addrconf: flags=0x40008 [ 6465.165895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6465.171269] pending: addrconf_verify_work [ 6465.174982] workqueue knbd0-recv: flags=0x1a [ 6465.177921] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6465.184926] in-flight: 651:recv_work [ 6465.191158] workqueue ptlrpc_pinger: flags=0x2 [ 6465.193872] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6465.197420] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6465.200876] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4548s workers=4 idle: 37 31846 28897 [ 6465.207991] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6455s workers=2 idle: 653 [ 6473.791065] rcu: INFO: rcu_sched self-detected stall on CPU [ 6473.793386] rcu: 2-...!: (2499887 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 6473.806056] (t=4560102 jiffies g=258701 q=2822) [ 6473.817096] rcu: rcu_sched kthread starved for 4556897 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6473.837404] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6473.845903] rcu: RCU grace-period kthread stack dump: [ 6473.850752] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6473.858407] Call Trace: [ 6473.860978] __schedule+0x351/0xcb0 [ 6473.863083] schedule+0xc0/0x180 [ 6473.864876] schedule_timeout+0xb4/0x190 [ 6473.867215] ? do_raw_spin_unlock+0x75/0x190 [ 6473.869467] ? __next_timer_interrupt+0x160/0x160 [ 6473.871629] ? prepare_to_swait_event+0xf3/0x140 [ 6473.873961] rcu_gp_kthread+0x8dd/0x15b0 [ 6473.875820] ? force_qs_rnp+0x2c0/0x2c0 [ 6473.877499] kthread+0x1d1/0x200 [ 6473.878964] ? set_kthread_struct+0x70/0x70 [ 6473.881460] ret_from_fork+0x1f/0x30 [ 6473.884494] rcu: Stack dump where RCU GP kthread last ran: [ 6473.887538] Sending NMI from CPU 2 to CPUs 1: [ 6473.889838] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 6473.892842] NMI backtrace for cpu 2 [ 6473.894818] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6473.902412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6473.907906] Call Trace: [ 6473.908875] [ 6473.910066] ? dump_stack+0xbb/0x10e [ 6473.912899] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6473.915535] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6473.934519] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6473.941805] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6473.952981] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6473.959133] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6473.965749] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6473.967626] ? update_process_times+0x6d/0xc0 [ 6473.975652] ? tick_sched_handle.isra.18+0x27/0x90 [ 6473.983789] ? tick_sched_timer+0x53/0xb0 [ 6473.990756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6473.996698] ? tick_sched_handle.isra.18+0x90/0x90 [ 6474.001666] ? hrtimer_interrupt+0x128/0x2c0 [ 6474.006414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6474.011262] ? apic_timer_interrupt+0xf/0x20 [ 6474.016151] [ 6474.016924] ? native_safe_halt+0x2e/0x50 [ 6474.019578] ? kvm_wait+0xcd/0xe0 [ 6474.022561] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6474.026170] ? do_raw_spin_lock+0x177/0x190 [ 6474.028710] ? _raw_spin_lock+0x21/0x30 [ 6474.033898] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6474.040654] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6474.046657] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6474.052102] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6474.057637] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6474.064203] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6474.070420] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6474.077854] ? __next_timer_interrupt+0x160/0x160 [ 6474.084921] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6474.091888] ? do_wait_intr+0xf0/0xf0 [ 6474.096314] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6474.103423] ? kthread+0x1d1/0x200 [ 6474.107274] ? set_kthread_struct+0x70/0x70 [ 6474.112638] ? ret_from_fork+0x1f/0x30 [ 6484.059194] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6484.065296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6484.098036] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6484.105006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6484.111019] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6484.115478] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6484.127878] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6484.132934] RAX: 0000000000007d86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6484.136356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6484.140080] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6484.144913] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6484.149129] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6484.153609] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6484.160140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6484.166670] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6484.176434] Call Trace: [ 6484.177900] [ 6484.179471] ? show_regs.cold.9+0x22/0x2f [ 6484.182496] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6484.186878] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6484.189293] ? watchdog+0x40/0x40 [ 6484.191569] ? hrtimer_interrupt+0x128/0x2c0 [ 6484.196416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6484.202048] ? apic_timer_interrupt+0xf/0x20 [ 6484.208803] [ 6484.210955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6484.217725] do_raw_spin_lock+0x177/0x190 [ 6484.221852] _raw_spin_lock+0x21/0x30 [ 6484.226655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6484.234170] ? do_raw_spin_unlock+0x75/0x190 [ 6484.240068] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6484.244019] ? do_wait_intr+0xf0/0xf0 [ 6484.247877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6484.249876] mdc_close+0x53b/0xd70 [mdc] [ 6484.251445] lmv_close+0x18d/0x490 [lmv] [ 6484.253838] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6484.257043] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6484.260599] ll_md_close+0x274/0x800 [lustre] [ 6484.263282] ll_file_release+0x128/0xc50 [lustre] [ 6484.266615] __fput+0x10d/0x440 [ 6484.268530] ____fput+0x12/0x20 [ 6484.270379] task_work_run+0xc6/0x110 [ 6484.272180] exit_to_usermode_loop+0x1dd/0x1f0 [ 6484.276824] do_syscall_64+0x3d6/0x440 [ 6484.278209] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6484.286442] RIP: 0033:0x7fa749552cd8 [ 6484.289447] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6484.300957] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6484.306072] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6484.308863] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6484.312459] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6484.316527] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6484.320996] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6492.057288] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6492.061737] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6492.084612] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6492.090160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6492.094470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6492.096861] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6492.104426] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6492.107637] RAX: 0000000000000324 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6492.110836] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6492.114457] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6492.118334] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6492.121730] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6492.126317] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6492.129843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6492.131848] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6492.135756] Call Trace: [ 6492.137353] [ 6492.138013] ? show_regs.cold.9+0x22/0x2f [ 6492.139987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6492.142666] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6492.144694] ? watchdog+0x40/0x40 [ 6492.146412] ? hrtimer_interrupt+0x128/0x2c0 [ 6492.149017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6492.151935] ? apic_timer_interrupt+0xf/0x20 [ 6492.153995] [ 6492.155147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6492.158004] do_raw_spin_lock+0x177/0x190 [ 6492.160019] _raw_spin_lock+0x21/0x30 [ 6492.161540] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6492.165158] ? do_raw_spin_unlock+0x75/0x190 [ 6492.167352] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6492.170767] ? do_wait_intr+0xf0/0xf0 [ 6492.172634] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6492.175343] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6492.178753] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6492.182313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.185076] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6492.189409] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6492.192858] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6492.196031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.200470] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6492.203652] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6492.206604] lmv_intent_open+0x338/0x1150 [lmv] [ 6492.208793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.211861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.215157] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6492.216864] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6492.219223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.221764] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6492.223813] ? do_raw_spin_unlock+0x75/0x190 [ 6492.225849] ? _raw_spin_unlock+0x12/0x30 [ 6492.228492] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6492.230631] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6492.233217] ll_atomic_open+0x36a/0xd70 [lustre] [ 6492.237341] lookup_open+0x356/0xb40 [ 6492.238598] do_last+0x313/0xfc0 [ 6492.240497] ? nd_jump_root+0xe5/0x160 [ 6492.242524] ? path_init+0x437/0x520 [ 6492.244723] path_openat+0xf7/0x500 [ 6492.246788] ? do_raw_spin_unlock+0x75/0x190 [ 6492.249623] do_filp_open+0x99/0x140 [ 6492.251165] ? getname_flags+0x6e/0x330 [ 6492.256451] ? __check_object_size+0xff/0x256 [ 6492.258699] ? do_raw_spin_unlock+0x75/0x190 [ 6492.261338] ? _raw_spin_unlock+0x12/0x30 [ 6492.265362] do_sys_openat2+0x2b4/0x410 [ 6492.267733] do_sys_open+0x73/0xa0 [ 6492.270245] __x64_sys_openat+0x24/0x30 [ 6492.273278] do_syscall_64+0xc1/0x440 [ 6492.275861] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6492.279040] RIP: 0033:0x7fa749552332 [ 6492.281708] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6492.292002] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6492.296383] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6492.299935] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6492.303155] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6492.307489] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6492.312810] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6495.715242] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4578s! [ 6495.721051] Showing busy workqueues and worker pools: [ 6495.724783] workqueue events: flags=0x0 [ 6495.731012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6495.735756] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6495.747889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6495.752045] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6495.759549] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.763557] pending: kfree_rcu_monitor [ 6495.767405] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6495.770223] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6495.773049] workqueue events_long: flags=0x0 [ 6495.774579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.776216] pending: lu_objects_flush [obdclass] [ 6495.779652] workqueue events_unbound: flags=0x2 [ 6495.781488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6495.784102] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6495.787862] workqueue events_power_efficient: flags=0x80 [ 6495.790328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.793218] pending: check_lifetime [ 6495.795018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.797724] pending: neigh_periodic_work [ 6495.799499] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.802699] pending: neigh_periodic_work [ 6495.804723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.810004] pending: do_cache_clean [ 6495.815445] workqueue mm_percpu_wq: flags=0x8 [ 6495.816908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.821128] pending: vmstat_update [ 6495.823143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.827658] pending: vmstat_update [ 6495.829545] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6495.832657] pending: vmstat_update [ 6495.835334] workqueue writeback: flags=0x4a [ 6495.837347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6495.840446] pending: wb_workfn, wb_workfn [ 6495.842818] workqueue xprtiod: flags=0xa [ 6495.844830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6495.847563] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6495.850562] workqueue ipv6_addrconf: flags=0x40008 [ 6495.852591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6495.855693] pending: addrconf_verify_work [ 6495.857805] workqueue knbd0-recv: flags=0x1a [ 6495.859612] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6495.862652] in-flight: 651:recv_work [ 6495.864304] workqueue ptlrpc_pinger: flags=0x2 [ 6495.866555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6495.869298] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6495.872513] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4578s workers=4 idle: 37 31846 28897 [ 6495.876532] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6485s workers=2 idle: 653 [ 6500.058350] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6500.058735] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6500.072675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6500.072754] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6500.072761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6500.072767] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6500.074269] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6500.074281] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6500.074286] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6500.074290] RAX: 00000000000068ec RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6500.074292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6500.074295] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6500.074296] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6500.074299] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6500.074302] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6500.074304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6500.074307] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6500.074477] Call Trace: [ 6500.074654] [ 6500.074661] ? show_regs.cold.9+0x22/0x2f [ 6500.075462] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6500.075469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6500.075474] ? watchdog+0x40/0x40 [ 6500.075480] ? hrtimer_interrupt+0x128/0x2c0 [ 6500.075486] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6500.075494] ? apic_timer_interrupt+0xf/0x20 [ 6500.075498] [ 6500.075502] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6500.075507] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6500.075515] do_raw_spin_lock+0x177/0x190 [ 6500.075520] _raw_spin_lock+0x21/0x30 [ 6500.075524] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6500.075807] process_one_work+0x2c8/0x700 [ 6500.075814] worker_thread+0x296/0x6e0 [ 6500.075818] ? rescuer_thread+0x570/0x570 [ 6500.075822] kthread+0x1d1/0x200 [ 6500.075902] ? set_kthread_struct+0x70/0x70 [ 6500.075909] ret_from_fork+0x1f/0x30 [ 6500.282507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6500.316523] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6500.323312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6500.330621] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6500.334650] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6500.350002] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6500.355560] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6500.361798] RDX: 0000000000002565 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6500.369629] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6500.377294] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6500.382397] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6500.386218] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6500.391004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6500.393705] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6500.402284] Call Trace: [ 6500.403604] [ 6500.404570] ? show_regs.cold.9+0x22/0x2f [ 6500.408631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6500.411580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6500.413628] ? watchdog+0x40/0x40 [ 6500.415402] ? hrtimer_interrupt+0x128/0x2c0 [ 6500.417292] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6500.422977] ? apic_timer_interrupt+0xf/0x20 [ 6500.424688] [ 6500.426466] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6500.435707] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6500.442569] do_raw_spin_lock+0x177/0x190 [ 6500.449066] _raw_spin_lock+0x21/0x30 [ 6500.450395] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6500.456088] obd_mod_free+0x46/0x140 [ptlrpc] [ 6500.458555] mdc_commit_open+0x91/0xb0 [mdc] [ 6500.460633] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6500.463326] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6500.466343] after_reply+0xe2c/0x1780 [ptlrpc] [ 6500.472760] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6500.477669] ? __next_timer_interrupt+0x160/0x160 [ 6500.481117] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6500.487275] ? do_wait_intr+0xf0/0xf0 [ 6500.488655] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6500.491431] kthread+0x1d1/0x200 [ 6500.492947] ? set_kthread_struct+0x70/0x70 [ 6500.493933] ret_from_fork+0x1f/0x30 [ 6512.059405] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6512.104402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6512.143402] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6512.149183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6512.153144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6512.159493] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6512.174473] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6512.177688] RAX: 00000000000059e7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6512.182265] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6512.185515] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6512.189923] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6512.193915] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6512.209180] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6512.211532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6512.215261] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6512.218361] Call Trace: [ 6512.219388] [ 6512.228241] ? show_regs.cold.9+0x22/0x2f [ 6512.230059] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6512.236313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6512.238268] ? watchdog+0x40/0x40 [ 6512.239518] ? hrtimer_interrupt+0x128/0x2c0 [ 6512.241603] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6512.243967] ? apic_timer_interrupt+0xf/0x20 [ 6512.248246] [ 6512.249208] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6512.252097] do_raw_spin_lock+0x177/0x190 [ 6512.253198] _raw_spin_lock+0x21/0x30 [ 6512.254486] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6512.260445] ? do_raw_spin_unlock+0x75/0x190 [ 6512.262243] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6512.268919] ? do_wait_intr+0xf0/0xf0 [ 6512.270532] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6512.276689] mdc_close+0x53b/0xd70 [mdc] [ 6512.278584] lmv_close+0x18d/0x490 [lmv] [ 6512.290731] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6512.292345] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6512.293521] ll_md_close+0x274/0x800 [lustre] [ 6512.304230] ll_file_release+0x128/0xc50 [lustre] [ 6512.309074] __fput+0x10d/0x440 [ 6512.310529] ____fput+0x12/0x20 [ 6512.311503] task_work_run+0xc6/0x110 [ 6512.314217] exit_to_usermode_loop+0x1dd/0x1f0 [ 6512.315878] do_syscall_64+0x3d6/0x440 [ 6512.317288] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6512.320522] RIP: 0033:0x7fa749552cd8 [ 6512.322770] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6512.331478] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6512.342970] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6512.356366] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6512.358315] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6512.369671] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6512.373051] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6520.054251] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6520.056735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6520.072428] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6520.076362] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6520.079472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6520.081789] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6520.087522] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6520.089829] RAX: 000000000000126e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6520.092532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6520.094875] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6520.097440] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6520.099540] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6520.101445] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6520.104385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6520.105975] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6520.108074] Call Trace: [ 6520.108832] [ 6520.109325] ? show_regs.cold.9+0x22/0x2f [ 6520.110416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6520.112030] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6520.113846] ? watchdog+0x40/0x40 [ 6520.114805] ? hrtimer_interrupt+0x128/0x2c0 [ 6520.116060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6520.117522] ? apic_timer_interrupt+0xf/0x20 [ 6520.118723] [ 6520.119359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6520.121241] do_raw_spin_lock+0x177/0x190 [ 6520.122449] _raw_spin_lock+0x21/0x30 [ 6520.123665] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6520.125366] ? do_raw_spin_unlock+0x75/0x190 [ 6520.126691] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6520.128468] ? do_wait_intr+0xf0/0xf0 [ 6520.129699] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6520.131244] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6520.132957] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6520.134858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.136413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6520.138133] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6520.139442] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6520.141297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.142854] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6520.144872] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6520.146710] lmv_intent_open+0x338/0x1150 [lmv] [ 6520.148227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.151959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.155412] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6520.156858] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6520.159414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.161027] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6520.162460] ? do_raw_spin_unlock+0x75/0x190 [ 6520.164060] ? _raw_spin_unlock+0x12/0x30 [ 6520.165295] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6520.166848] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6520.168532] ll_atomic_open+0x36a/0xd70 [lustre] [ 6520.170025] lookup_open+0x356/0xb40 [ 6520.171080] do_last+0x313/0xfc0 [ 6520.172045] ? nd_jump_root+0xe5/0x160 [ 6520.173142] ? path_init+0x437/0x520 [ 6520.176257] path_openat+0xf7/0x500 [ 6520.178717] ? do_raw_spin_unlock+0x75/0x190 [ 6520.180083] do_filp_open+0x99/0x140 [ 6520.181803] ? getname_flags+0x6e/0x330 [ 6520.183048] ? __check_object_size+0xff/0x256 [ 6520.184854] ? do_raw_spin_unlock+0x75/0x190 [ 6520.186461] ? _raw_spin_unlock+0x12/0x30 [ 6520.187722] do_sys_openat2+0x2b4/0x410 [ 6520.188815] do_sys_open+0x73/0xa0 [ 6520.189886] __x64_sys_openat+0x24/0x30 [ 6520.191446] do_syscall_64+0xc1/0x440 [ 6520.192606] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6520.194313] RIP: 0033:0x7fa749552332 [ 6520.195320] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6520.200863] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6520.205462] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6520.209814] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6520.214702] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6520.219060] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6520.222688] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6525.939565] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4608s! [ 6525.965767] Showing busy workqueues and worker pools: [ 6525.971754] workqueue events: flags=0x0 [ 6525.983187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6525.995178] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6526.014177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6526.022548] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6526.038783] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.046893] pending: kfree_rcu_monitor [ 6526.049651] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6526.054354] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6526.066620] workqueue events_long: flags=0x0 [ 6526.069834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.075709] pending: lu_objects_flush [obdclass] [ 6526.082156] workqueue events_unbound: flags=0x2 [ 6526.086319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6526.097547] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6526.108864] workqueue events_power_efficient: flags=0x80 [ 6526.115246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.127219] pending: check_lifetime [ 6526.128641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.141669] pending: neigh_periodic_work [ 6526.149735] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.164740] pending: neigh_periodic_work [ 6526.174313] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.182880] pending: do_cache_clean [ 6526.186982] workqueue mm_percpu_wq: flags=0x8 [ 6526.188547] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.201697] pending: vmstat_update [ 6526.205869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.212905] pending: vmstat_update [ 6526.216307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6526.222636] pending: vmstat_update [ 6526.225711] workqueue writeback: flags=0x4a [ 6526.229055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6526.238518] pending: wb_workfn, wb_workfn [ 6526.240136] workqueue xprtiod: flags=0xa [ 6526.253331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6526.262788] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6526.280647] workqueue ipv6_addrconf: flags=0x40008 [ 6526.307025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6526.309454] pending: addrconf_verify_work [ 6526.321826] workqueue knbd0-recv: flags=0x1a [ 6526.333163] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6526.341253] in-flight: 651:recv_work [ 6526.353325] workqueue ptlrpc_pinger: flags=0x2 [ 6526.366488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6526.377837] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6526.391605] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4609s workers=4 idle: 37 31846 28897 [ 6526.405824] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6516s workers=2 idle: 653 [ 6528.058668] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6528.062004] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6528.062201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6528.062274] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6528.062279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6528.062286] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6528.062693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6528.062705] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6528.062709] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6528.062715] RAX: 0000000000007d91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6528.062717] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6528.062792] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6528.062794] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6528.062797] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6528.062800] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6528.062803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6528.062805] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6528.062813] Call Trace: [ 6528.062819] [ 6528.062824] ? show_regs.cold.9+0x22/0x2f [ 6528.062834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6528.062840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6528.062844] ? watchdog+0x40/0x40 [ 6528.062848] ? hrtimer_interrupt+0x128/0x2c0 [ 6528.062851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6528.062859] ? apic_timer_interrupt+0xf/0x20 [ 6528.062864] [ 6528.062867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6528.062871] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6528.062877] do_raw_spin_lock+0x177/0x190 [ 6528.062882] _raw_spin_lock+0x21/0x30 [ 6528.062886] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6528.063106] process_one_work+0x2c8/0x700 [ 6528.063114] worker_thread+0x296/0x6e0 [ 6528.063194] ? rescuer_thread+0x570/0x570 [ 6528.063199] kthread+0x1d1/0x200 [ 6528.063202] ? set_kthread_struct+0x70/0x70 [ 6528.063206] ret_from_fork+0x1f/0x30 [ 6528.245502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6528.291112] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6528.301041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6528.311948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6528.315604] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6528.335377] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6528.344249] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6528.352073] RDX: 00000000000035fb RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6528.362462] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6528.367291] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6528.374298] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6528.381547] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6528.388694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6528.395422] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6528.402541] Call Trace: [ 6528.404221] [ 6528.406421] ? show_regs.cold.9+0x22/0x2f [ 6528.415248] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6528.419462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6528.423626] ? watchdog+0x40/0x40 [ 6528.427025] ? hrtimer_interrupt+0x128/0x2c0 [ 6528.430914] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6528.435653] ? apic_timer_interrupt+0xf/0x20 [ 6528.439562] [ 6528.441865] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6528.448387] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6528.454437] do_raw_spin_lock+0x177/0x190 [ 6528.459143] _raw_spin_lock+0x21/0x30 [ 6528.462654] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6528.469993] obd_mod_free+0x46/0x140 [ptlrpc] [ 6528.474903] mdc_commit_open+0x91/0xb0 [mdc] [ 6528.477543] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6528.481994] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6528.484754] after_reply+0xe2c/0x1780 [ptlrpc] [ 6528.489200] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6528.494688] ? __next_timer_interrupt+0x160/0x160 [ 6528.497516] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6528.503816] ? do_wait_intr+0xf0/0xf0 [ 6528.507547] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6528.513099] kthread+0x1d1/0x200 [ 6528.517072] ? set_kthread_struct+0x70/0x70 [ 6528.521017] ret_from_fork+0x1f/0x30 [ 6540.060113] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6540.062120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.106491] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.116077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.129269] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6540.132797] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6540.168280] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6540.192836] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6540.213093] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6540.234022] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6540.248848] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6540.263903] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6540.276476] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6540.281368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.286511] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6540.296476] Call Trace: [ 6540.298622] [ 6540.301262] ? show_regs.cold.9+0x22/0x2f [ 6540.303675] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.307442] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.311115] ? watchdog+0x40/0x40 [ 6540.312456] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.315570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.318834] ? apic_timer_interrupt+0xf/0x20 [ 6540.321489] [ 6540.322085] ? native_safe_halt+0x2e/0x50 [ 6540.324950] kvm_wait+0xcd/0xe0 [ 6540.328637] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6540.333810] do_raw_spin_lock+0x177/0x190 [ 6540.336755] _raw_spin_lock+0x21/0x30 [ 6540.339270] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6540.343518] ? do_raw_spin_unlock+0x75/0x190 [ 6540.351641] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6540.356708] ? do_wait_intr+0xf0/0xf0 [ 6540.358218] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6540.360452] mdc_close+0x53b/0xd70 [mdc] [ 6540.361739] lmv_close+0x18d/0x490 [lmv] [ 6540.364574] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6540.367279] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6540.370540] ll_md_close+0x274/0x800 [lustre] [ 6540.374728] ll_file_release+0x128/0xc50 [lustre] [ 6540.382650] __fput+0x10d/0x440 [ 6540.384784] ____fput+0x12/0x20 [ 6540.385781] task_work_run+0xc6/0x110 [ 6540.387561] exit_to_usermode_loop+0x1dd/0x1f0 [ 6540.389268] do_syscall_64+0x3d6/0x440 [ 6540.394443] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6540.396425] RIP: 0033:0x7fa749552cd8 [ 6540.397188] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6540.405029] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6540.408024] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6540.412510] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6540.418038] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6540.422822] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6540.424586] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6552.054020] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6552.056934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6552.075451] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6552.086156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6552.098677] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6552.103362] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6552.123688] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6552.132327] RAX: 0000000000002be6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6552.137365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6552.143133] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6552.147333] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6552.155332] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6552.160488] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6552.168543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6552.170481] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6552.180369] Call Trace: [ 6552.183844] [ 6552.184863] ? show_regs.cold.9+0x22/0x2f [ 6552.189136] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6552.192854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6552.195365] ? watchdog+0x40/0x40 [ 6552.197834] ? hrtimer_interrupt+0x128/0x2c0 [ 6552.202951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6552.207065] ? apic_timer_interrupt+0xf/0x20 [ 6552.212363] [ 6552.213334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6552.217971] do_raw_spin_lock+0x177/0x190 [ 6552.220290] _raw_spin_lock+0x21/0x30 [ 6552.222295] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6552.224147] ? do_raw_spin_unlock+0x75/0x190 [ 6552.226027] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6552.229460] ? do_wait_intr+0xf0/0xf0 [ 6552.234156] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6552.239479] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6552.244326] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6552.249321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.253966] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6552.257734] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6552.259340] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6552.261450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.267708] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6552.272454] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6552.277076] lmv_intent_open+0x338/0x1150 [lmv] [ 6552.282351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.289338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.293368] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6552.299791] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6552.304369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.307550] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6552.313491] ? do_raw_spin_unlock+0x75/0x190 [ 6552.316613] ? _raw_spin_unlock+0x12/0x30 [ 6552.318071] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6552.324107] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6552.328643] ll_atomic_open+0x36a/0xd70 [lustre] [ 6552.334904] lookup_open+0x356/0xb40 [ 6552.338033] do_last+0x313/0xfc0 [ 6552.340464] ? nd_jump_root+0xe5/0x160 [ 6552.343769] ? path_init+0x437/0x520 [ 6552.347450] path_openat+0xf7/0x500 [ 6552.349587] ? do_raw_spin_unlock+0x75/0x190 [ 6552.351128] do_filp_open+0x99/0x140 [ 6552.356173] ? getname_flags+0x6e/0x330 [ 6552.360479] ? __check_object_size+0xff/0x256 [ 6552.365613] ? do_raw_spin_unlock+0x75/0x190 [ 6552.366989] ? _raw_spin_unlock+0x12/0x30 [ 6552.371583] do_sys_openat2+0x2b4/0x410 [ 6552.376125] do_sys_open+0x73/0xa0 [ 6552.381346] __x64_sys_openat+0x24/0x30 [ 6552.383792] do_syscall_64+0xc1/0x440 [ 6552.385005] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6552.387857] RIP: 0033:0x7fa749552332 [ 6552.389009] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6552.401809] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6552.412328] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6552.418533] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6552.422077] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6552.426225] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6552.431523] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6556.057153] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6556.058389] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6556.062487] Modules linked in: [ 6556.066892] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6556.066970] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6556.066976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6556.066980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6556.066993] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6556.066997] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6556.067001] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6556.067004] RDX: 0000000000007840 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6556.067006] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6556.067008] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6556.067010] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6556.067013] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6556.067016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6556.067018] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6556.067026] Call Trace: [ 6556.067031] [ 6556.067037] ? show_regs.cold.9+0x22/0x2f [ 6556.067047] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6556.067054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6556.067058] ? watchdog+0x40/0x40 [ 6556.067062] ? hrtimer_interrupt+0x128/0x2c0 [ 6556.067065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6556.067073] ? apic_timer_interrupt+0xf/0x20 [ 6556.067077] [ 6556.067080] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6556.067084] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6556.067089] do_raw_spin_lock+0x177/0x190 [ 6556.067093] _raw_spin_lock+0x21/0x30 [ 6556.067099] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6556.067374] obd_mod_free+0x46/0x140 [ptlrpc] [ 6556.067774] mdc_commit_open+0x91/0xb0 [mdc] [ 6556.067811] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6556.068004] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6556.068270] after_reply+0xe2c/0x1780 [ptlrpc] [ 6556.068521] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6556.068869] ? __next_timer_interrupt+0x160/0x160 [ 6556.068879] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6556.070734] lustre(O) [ 6556.090809] ? do_wait_intr+0xf0/0xf0 [ 6556.090832] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6556.110512] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6556.110587] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6556.110594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6556.110600] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6556.110858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6556.110945] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6556.110949] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6556.110953] RAX: 000000000000780d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6556.110955] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6556.110957] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6556.110959] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6556.110961] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6556.110964] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6556.110967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6556.110969] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6556.110977] Call Trace: [ 6556.110983] [ 6556.110988] ? show_regs.cold.9+0x22/0x2f [ 6556.110996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6556.111003] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6556.111007] ? watchdog+0x40/0x40 [ 6556.111012] ? hrtimer_interrupt+0x128/0x2c0 [ 6556.111017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6556.111025] ? apic_timer_interrupt+0xf/0x20 [ 6556.111030] [ 6556.111033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6556.111038] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6556.111043] do_raw_spin_lock+0x177/0x190 [ 6556.111048] _raw_spin_lock+0x21/0x30 [ 6556.111052] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6556.111272] process_one_work+0x2c8/0x700 [ 6556.111281] worker_thread+0x296/0x6e0 [ 6556.111286] ? rescuer_thread+0x570/0x570 [ 6556.111291] kthread+0x1d1/0x200 [ 6556.111295] ? set_kthread_struct+0x70/0x70 [ 6556.111299] ret_from_fork+0x1f/0x30 [ 6556.419481] kthread+0x1d1/0x200 [ 6556.420669] ? set_kthread_struct+0x70/0x70 [ 6556.422058] ret_from_fork+0x1f/0x30 [ 6556.642116] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4639s! [ 6556.646860] Showing busy workqueues and worker pools: [ 6556.649859] workqueue events: flags=0x0 [ 6556.652975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6556.656710] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6556.661919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6556.665628] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6556.669059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.671462] pending: kfree_rcu_monitor [ 6556.673031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6556.674772] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6556.677906] workqueue events_long: flags=0x0 [ 6556.679444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.681910] pending: lu_objects_flush [obdclass] [ 6556.683924] workqueue events_unbound: flags=0x2 [ 6556.685420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6556.687370] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6556.690788] workqueue events_power_efficient: flags=0x80 [ 6556.692902] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.696294] pending: check_lifetime [ 6556.697610] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.700670] pending: neigh_periodic_work [ 6556.703418] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.707997] pending: neigh_periodic_work [ 6556.709799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.712639] pending: do_cache_clean [ 6556.714180] workqueue mm_percpu_wq: flags=0x8 [ 6556.715935] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.718444] pending: vmstat_update [ 6556.720147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.722697] pending: vmstat_update [ 6556.724220] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6556.726793] pending: vmstat_update [ 6556.728331] workqueue writeback: flags=0x4a [ 6556.729915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6556.732310] pending: wb_workfn, wb_workfn [ 6556.734059] workqueue xprtiod: flags=0xa [ 6556.735544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6556.737993] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6556.741087] workqueue ipv6_addrconf: flags=0x40008 [ 6556.742803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6556.745319] pending: addrconf_verify_work [ 6556.746969] workqueue knbd0-recv: flags=0x1a [ 6556.748613] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6556.751393] in-flight: 651:recv_work [ 6556.753274] workqueue ptlrpc_pinger: flags=0x2 [ 6556.755067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6556.757543] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6556.760113] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4639s workers=4 idle: 37 31846 28897 [ 6556.763669] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6546s workers=2 idle: 653 [ 6568.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6568.063265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.103885] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.113984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.118466] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6568.123370] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6568.136226] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6568.143370] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6568.145904] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6568.150519] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6568.153146] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6568.161117] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6568.165945] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6568.172757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.176299] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6568.180748] Call Trace: [ 6568.182540] [ 6568.183453] ? show_regs.cold.9+0x22/0x2f [ 6568.185596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.188317] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.191003] ? watchdog+0x40/0x40 [ 6568.195119] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.196574] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.199052] ? apic_timer_interrupt+0xf/0x20 [ 6568.202433] [ 6568.203319] ? native_safe_halt+0x2e/0x50 [ 6568.205761] kvm_wait+0xcd/0xe0 [ 6568.211513] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6568.214017] do_raw_spin_lock+0x177/0x190 [ 6568.217023] _raw_spin_lock+0x21/0x30 [ 6568.218312] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6568.221287] ? do_raw_spin_unlock+0x75/0x190 [ 6568.223958] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6568.228525] ? do_wait_intr+0xf0/0xf0 [ 6568.230291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6568.234164] mdc_close+0x53b/0xd70 [mdc] [ 6568.235407] lmv_close+0x18d/0x490 [lmv] [ 6568.236803] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6568.240597] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6568.243172] ll_md_close+0x274/0x800 [lustre] [ 6568.247780] ll_file_release+0x128/0xc50 [lustre] [ 6568.250609] __fput+0x10d/0x440 [ 6568.252431] ____fput+0x12/0x20 [ 6568.254562] task_work_run+0xc6/0x110 [ 6568.256363] exit_to_usermode_loop+0x1dd/0x1f0 [ 6568.258590] do_syscall_64+0x3d6/0x440 [ 6568.264708] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6568.268026] RIP: 0033:0x7fa749552cd8 [ 6568.270446] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6568.280618] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6568.285243] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6568.289412] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6568.293322] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6568.297712] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6568.301810] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6584.055804] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6584.057861] Modules linked in: lustre(O) [ 6584.057901] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6584.058308] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6584.058336] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6584.058482] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6584.058488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6584.058491] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6584.058504] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6584.058508] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6584.058512] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6584.058515] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6584.058517] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6584.058520] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6584.058522] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6584.058525] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6584.058528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6584.058530] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6584.058540] Call Trace: [ 6584.058546] [ 6584.058551] ? show_regs.cold.9+0x22/0x2f [ 6584.058559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6584.058566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6584.058571] ? watchdog+0x40/0x40 [ 6584.058575] ? hrtimer_interrupt+0x128/0x2c0 [ 6584.058580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6584.058587] ? apic_timer_interrupt+0xf/0x20 [ 6584.058592] [ 6584.058594] ? native_safe_halt+0x2e/0x50 [ 6584.058598] kvm_wait+0xcd/0xe0 [ 6584.058606] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6584.058615] do_raw_spin_lock+0x177/0x190 [ 6584.058620] _raw_spin_lock+0x21/0x30 [ 6584.058624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6584.058893] ? do_raw_spin_unlock+0x75/0x190 [ 6584.058900] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6584.059097] ? do_wait_intr+0xf0/0xf0 [ 6584.059168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 6584.059108] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6584.059201] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6584.059234] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6584.059239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6584.059244] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6584.059556] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6584.059563] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6584.059566] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6584.059571] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6584.059573] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6584.059576] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6584.059578] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6584.059580] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6584.059583] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6584.059587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6584.059597] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6584.059603] Call Trace: [ 6584.059607] [ 6584.059610] ? show_regs.cold.9+0x22/0x2f [ 6584.059618] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6584.059623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6584.059628] ? watchdog+0x40/0x40 [ 6584.059632] ? hrtimer_interrupt+0x128/0x2c0 [ 6584.059636] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6584.059657] ? apic_timer_interrupt+0xf/0x20 [ 6584.059662] [ 6584.059666] ? native_safe_halt+0x2e/0x50 [ 6584.059670] kvm_wait+0xcd/0xe0 [ 6584.059678] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6584.059684] do_raw_spin_lock+0x177/0x190 [ 6584.059688] _raw_spin_lock+0x21/0x30 [ 6584.059692] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6584.059973] process_one_work+0x2c8/0x700 [ 6584.059981] worker_thread+0x296/0x6e0 [ 6584.059985] ? rescuer_thread+0x570/0x570 [ 6584.059988] kthread+0x1d1/0x200 [ 6584.059994] ? set_kthread_struct+0x70/0x70 [ 6584.059997] ret_from_fork+0x1f/0x30 [ 6584.061126] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6584.061436] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6584.062969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.063081] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6584.063119] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6584.063149] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6584.063177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.063269] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6584.063711] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6584.063742] lmv_intent_open+0x338/0x1150 [lmv] [ 6584.063773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.063863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.063951] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6584.063974] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6584.066727] Modules linked in: [ 6584.071419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.071540] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6584.071627] ? do_raw_spin_unlock+0x75/0x190 [ 6584.071635] ? _raw_spin_unlock+0x12/0x30 [ 6584.071666] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6584.071691] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6584.071711] ll_atomic_open+0x36a/0xd70 [lustre] [ 6584.071797] lookup_open+0x356/0xb40 [ 6584.071807] do_last+0x313/0xfc0 [ 6584.071812] ? nd_jump_root+0xe5/0x160 [ 6584.071816] ? path_init+0x437/0x520 [ 6584.071821] path_openat+0xf7/0x500 [ 6584.071825] ? do_raw_spin_unlock+0x75/0x190 [ 6584.071829] do_filp_open+0x99/0x140 [ 6584.071835] ? getname_flags+0x6e/0x330 [ 6584.071840] ? __check_object_size+0xff/0x256 [ 6584.071847] ? do_raw_spin_unlock+0x75/0x190 [ 6584.071851] ? _raw_spin_unlock+0x12/0x30 [ 6584.071855] do_sys_openat2+0x2b4/0x410 [ 6584.071859] do_sys_open+0x73/0xa0 [ 6584.071864] __x64_sys_openat+0x24/0x30 [ 6584.071868] do_syscall_64+0xc1/0x440 [ 6584.071874] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6584.071881] RIP: 0033:0x7fa749552332 [ 6584.071888] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6584.071893] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6584.071898] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6584.071902] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6584.071905] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6584.071909] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6584.106772] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6584.106851] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6584.106858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6584.106860] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6584.106874] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6584.106878] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6584.106882] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6584.106884] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6584.106886] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6584.106888] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6584.106891] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6584.106894] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6584.106896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6584.106898] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6584.106906] Call Trace: [ 6584.106912] [ 6584.106917] ? show_regs.cold.9+0x22/0x2f [ 6584.106927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6584.106933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6584.106937] ? watchdog+0x40/0x40 [ 6584.106941] ? hrtimer_interrupt+0x128/0x2c0 [ 6584.106945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6584.106952] ? apic_timer_interrupt+0xf/0x20 [ 6584.106957] [ 6584.106960] ? native_safe_halt+0x2e/0x50 [ 6584.106964] kvm_wait+0xcd/0xe0 [ 6584.106971] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6584.106978] do_raw_spin_lock+0x177/0x190 [ 6584.106982] _raw_spin_lock+0x21/0x30 [ 6584.106987] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6584.107254] obd_mod_free+0x46/0x140 [ptlrpc] [ 6584.116898] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6584.733383] mdc_commit_open+0x91/0xb0 [mdc] [ 6584.735912] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6584.738519] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6584.744675] after_reply+0xe2c/0x1780 [ptlrpc] [ 6584.751014] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6584.758311] ? __next_timer_interrupt+0x160/0x160 [ 6584.759938] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6584.762652] ? do_wait_intr+0xf0/0xf0 [ 6584.765654] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6584.768148] kthread+0x1d1/0x200 [ 6584.770887] ? set_kthread_struct+0x70/0x70 [ 6584.772427] ret_from_fork+0x1f/0x30 [ 6586.858205] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4669s! [ 6586.867791] Showing busy workqueues and worker pools: [ 6586.879436] workqueue events: flags=0x0 [ 6586.883466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6586.885842] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6586.906507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6586.912860] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6586.919023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6586.925463] pending: kfree_rcu_monitor [ 6586.928576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6586.937807] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6586.947224] workqueue events_long: flags=0x0 [ 6586.949401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6586.956012] pending: lu_objects_flush [obdclass] [ 6586.959392] workqueue events_unbound: flags=0x2 [ 6586.963346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6586.970439] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6586.976815] workqueue events_power_efficient: flags=0x80 [ 6586.984478] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6586.993091] pending: check_lifetime [ 6586.999467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6587.007190] pending: neigh_periodic_work [ 6587.011268] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6587.019502] pending: neigh_periodic_work [ 6587.025835] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6587.035260] pending: do_cache_clean [ 6587.038437] workqueue mm_percpu_wq: flags=0x8 [ 6587.041132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6587.048445] pending: vmstat_update [ 6587.055080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6587.061872] pending: vmstat_update [ 6587.063782] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6587.072655] pending: vmstat_update [ 6587.077183] workqueue writeback: flags=0x4a [ 6587.080537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6587.087559] pending: wb_workfn, wb_workfn [ 6587.092142] workqueue xprtiod: flags=0xa [ 6587.097160] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6587.104385] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6587.112090] workqueue ipv6_addrconf: flags=0x40008 [ 6587.121704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6587.129232] pending: addrconf_verify_work [ 6587.135077] workqueue knbd0-recv: flags=0x1a [ 6587.141595] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6587.145975] in-flight: 651:recv_work [ 6587.151449] workqueue ptlrpc_pinger: flags=0x2 [ 6587.155834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6587.160201] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6587.165122] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4670s workers=4 idle: 37 31846 28897 [ 6587.177410] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6577s workers=2 idle: 653 [ 6596.059526] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6596.066857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6596.110976] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6596.133501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6596.138259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6596.141973] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6596.165888] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6596.171179] RAX: 000000000000081e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6596.182664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6596.188311] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6596.195895] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6596.208584] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6596.218010] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6596.230799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6596.234766] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6596.241000] Call Trace: [ 6596.242410] [ 6596.247107] ? show_regs.cold.9+0x22/0x2f [ 6596.251537] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6596.257086] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6596.258831] ? watchdog+0x40/0x40 [ 6596.260016] ? hrtimer_interrupt+0x128/0x2c0 [ 6596.273932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6596.282392] ? apic_timer_interrupt+0xf/0x20 [ 6596.283855] [ 6596.284549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6596.295193] do_raw_spin_lock+0x177/0x190 [ 6596.309308] _raw_spin_lock+0x21/0x30 [ 6596.314319] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6596.321599] ? do_raw_spin_unlock+0x75/0x190 [ 6596.324680] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6596.331071] ? do_wait_intr+0xf0/0xf0 [ 6596.332672] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6596.335112] mdc_close+0x53b/0xd70 [mdc] [ 6596.337951] lmv_close+0x18d/0x490 [lmv] [ 6596.340449] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6596.344414] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6596.347474] ll_md_close+0x274/0x800 [lustre] [ 6596.349877] ll_file_release+0x128/0xc50 [lustre] [ 6596.352921] __fput+0x10d/0x440 [ 6596.354823] ____fput+0x12/0x20 [ 6596.356337] task_work_run+0xc6/0x110 [ 6596.357941] exit_to_usermode_loop+0x1dd/0x1f0 [ 6596.363894] do_syscall_64+0x3d6/0x440 [ 6596.366625] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6596.368916] RIP: 0033:0x7fa749552cd8 [ 6596.371330] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6596.389810] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6596.395293] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6596.400474] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6596.406710] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6596.412228] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6596.418998] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6612.056613] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6612.058195] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6612.061241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6612.061328] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6612.061439] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6612.061445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6612.061501] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6612.061505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6612.061512] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6612.063713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6612.063787] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6612.063793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6612.063797] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6612.063810] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6612.063815] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6612.063819] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6612.063822] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6612.063824] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6612.063826] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6612.063828] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6612.063832] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6612.063834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6612.063836] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6612.063845] Call Trace: [ 6612.063851] [ 6612.063857] ? show_regs.cold.9+0x22/0x2f [ 6612.063867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6612.063873] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6612.063878] ? watchdog+0x40/0x40 [ 6612.063881] ? hrtimer_interrupt+0x128/0x2c0 [ 6612.063884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6612.063893] ? apic_timer_interrupt+0xf/0x20 [ 6612.063897] [ 6612.063899] ? native_safe_halt+0x2e/0x50 [ 6612.063903] kvm_wait+0xcd/0xe0 [ 6612.063913] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6612.063920] do_raw_spin_lock+0x177/0x190 [ 6612.064215] _raw_spin_lock+0x21/0x30 [ 6612.064223] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6612.064642] obd_mod_free+0x46/0x140 [ptlrpc] [ 6612.064837] mdc_commit_open+0x91/0xb0 [mdc] [ 6612.064882] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6612.065418] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6612.065845] [ 6612.065849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6612.065824] after_reply+0xe2c/0x1780 [ptlrpc] [ 6612.066400] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6612.066407] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6612.066413] RAX: 0000000000004812 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6612.066416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6612.066419] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6612.066421] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6612.066424] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6612.066427] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6612.066431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6612.066434] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6612.066442] Call Trace: [ 6612.066447] [ 6612.066451] ? show_regs.cold.9+0x22/0x2f [ 6612.066461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6612.066467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6612.066471] ? watchdog+0x40/0x40 [ 6612.066475] ? hrtimer_interrupt+0x128/0x2c0 [ 6612.066480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6612.066489] ? apic_timer_interrupt+0xf/0x20 [ 6612.066494] [ 6612.066497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6612.066503] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6612.066509] do_raw_spin_lock+0x177/0x190 [ 6612.066515] _raw_spin_lock+0x21/0x30 [ 6612.066520] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6612.066729] process_one_work+0x2c8/0x700 [ 6612.066738] worker_thread+0x296/0x6e0 [ 6612.066801] ? rescuer_thread+0x570/0x570 [ 6612.066807] kthread+0x1d1/0x200 [ 6612.066813] ? set_kthread_struct+0x70/0x70 [ 6612.066818] ret_from_fork+0x1f/0x30 [ 6612.100852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6612.100863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6612.100878] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6612.100882] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6612.100886] RAX: 0000000000002f87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6612.100888] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6612.100890] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6612.100891] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6612.100893] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6612.100896] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6612.100897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6612.100899] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6612.100906] Call Trace: [ 6612.100910] [ 6612.100914] ? show_regs.cold.9+0x22/0x2f [ 6612.100920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6612.100925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6612.100927] ? watchdog+0x40/0x40 [ 6612.100930] ? hrtimer_interrupt+0x128/0x2c0 [ 6612.100933] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6612.100939] ? apic_timer_interrupt+0xf/0x20 [ 6612.100942] [ 6612.100944] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6612.100950] do_raw_spin_lock+0x177/0x190 [ 6612.100953] _raw_spin_lock+0x21/0x30 [ 6612.100956] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6612.109613] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6612.110697] ? __next_timer_interrupt+0x160/0x160 [ 6612.110713] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6612.111436] ? do_wait_intr+0xf0/0xf0 [ 6612.111446] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6612.111752] kthread+0x1d1/0x200 [ 6612.111760] ? set_kthread_struct+0x70/0x70 [ 6612.111764] ret_from_fork+0x1f/0x30 [ 6612.839401] ? do_raw_spin_unlock+0x75/0x190 [ 6612.840974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6612.857891] ? do_wait_intr+0xf0/0xf0 [ 6612.865203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6612.868044] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6612.878612] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6612.887551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.891787] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6612.899360] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6612.908317] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6612.911734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.920354] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6612.926578] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6612.935076] lmv_intent_open+0x338/0x1150 [lmv] [ 6612.936671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.946050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.948091] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6612.963184] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6612.971909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.983390] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6612.988130] ? do_raw_spin_unlock+0x75/0x190 [ 6612.989524] ? _raw_spin_unlock+0x12/0x30 [ 6613.002894] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6613.011357] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6613.020659] ll_atomic_open+0x36a/0xd70 [lustre] [ 6613.027551] lookup_open+0x356/0xb40 [ 6613.037485] do_last+0x313/0xfc0 [ 6613.042179] ? nd_jump_root+0xe5/0x160 [ 6613.050758] ? path_init+0x437/0x520 [ 6613.057376] path_openat+0xf7/0x500 [ 6613.061481] ? do_raw_spin_unlock+0x75/0x190 [ 6613.064656] do_filp_open+0x99/0x140 [ 6613.067796] ? getname_flags+0x6e/0x330 [ 6613.075660] ? __check_object_size+0xff/0x256 [ 6613.081957] ? do_raw_spin_unlock+0x75/0x190 [ 6613.090409] ? _raw_spin_unlock+0x12/0x30 [ 6613.093750] do_sys_openat2+0x2b4/0x410 [ 6613.099774] do_sys_open+0x73/0xa0 [ 6613.107514] __x64_sys_openat+0x24/0x30 [ 6613.112797] do_syscall_64+0xc1/0x440 [ 6613.118184] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6613.127586] RIP: 0033:0x7fa749552332 [ 6613.135414] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6613.160469] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6613.167663] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6613.177718] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6613.189417] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6613.201694] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6613.210569] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6617.573032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4700s! [ 6617.582541] Showing busy workqueues and worker pools: [ 6617.586109] workqueue events: flags=0x0 [ 6617.597037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6617.599030] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6617.628608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6617.634373] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6617.651728] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.672063] pending: kfree_rcu_monitor [ 6617.681151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6617.683592] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6617.693280] workqueue events_long: flags=0x0 [ 6617.703603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.708610] pending: lu_objects_flush [obdclass] [ 6617.713482] workqueue events_unbound: flags=0x2 [ 6617.722314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6617.731757] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6617.740291] workqueue events_power_efficient: flags=0x80 [ 6617.743583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.748344] pending: check_lifetime [ 6617.750563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.753942] pending: neigh_periodic_work [ 6617.758756] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.765778] pending: neigh_periodic_work [ 6617.768707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.772351] pending: do_cache_clean [ 6617.777838] workqueue mm_percpu_wq: flags=0x8 [ 6617.784208] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.795215] pending: vmstat_update [ 6617.800912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.807514] pending: vmstat_update [ 6617.808855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6617.816130] pending: vmstat_update [ 6617.820615] workqueue writeback: flags=0x4a [ 6617.822195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6617.832007] pending: wb_workfn, wb_workfn [ 6617.837198] workqueue xprtiod: flags=0xa [ 6617.844720] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6617.850427] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6617.862678] workqueue ipv6_addrconf: flags=0x40008 [ 6617.864273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6617.868346] pending: addrconf_verify_work [ 6617.877293] workqueue knbd0-recv: flags=0x1a [ 6617.879555] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6617.883902] in-flight: 651:recv_work [ 6617.885291] workqueue ptlrpc_pinger: flags=0x2 [ 6617.888428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6617.904525] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6617.915859] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4700s workers=4 idle: 37 31846 28897 [ 6617.924667] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6607s workers=2 idle: 653 [ 6624.061869] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6624.069135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6624.129977] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6624.139474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6624.146146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.149791] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6624.163748] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6624.168809] RAX: 0000000000004f0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6624.174120] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6624.186037] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6624.189626] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6624.202704] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6624.212918] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6624.218988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6624.225881] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6624.232131] Call Trace: [ 6624.233134] [ 6624.236645] ? show_regs.cold.9+0x22/0x2f [ 6624.242825] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6624.245659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6624.247219] ? watchdog+0x40/0x40 [ 6624.248571] ? hrtimer_interrupt+0x128/0x2c0 [ 6624.252426] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6624.257279] ? apic_timer_interrupt+0xf/0x20 [ 6624.261162] [ 6624.262325] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.266044] do_raw_spin_lock+0x177/0x190 [ 6624.267284] _raw_spin_lock+0x21/0x30 [ 6624.271575] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6624.276555] ? do_raw_spin_unlock+0x75/0x190 [ 6624.278392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6624.282226] ? do_wait_intr+0xf0/0xf0 [ 6624.284014] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6624.290031] mdc_close+0x53b/0xd70 [mdc] [ 6624.293187] lmv_close+0x18d/0x490 [lmv] [ 6624.295971] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6624.298375] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6624.301477] ll_md_close+0x274/0x800 [lustre] [ 6624.303369] ll_file_release+0x128/0xc50 [lustre] [ 6624.308861] __fput+0x10d/0x440 [ 6624.310125] ____fput+0x12/0x20 [ 6624.313360] task_work_run+0xc6/0x110 [ 6624.315593] exit_to_usermode_loop+0x1dd/0x1f0 [ 6624.319010] do_syscall_64+0x3d6/0x440 [ 6624.322978] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6624.325181] RIP: 0033:0x7fa749552cd8 [ 6624.326712] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6624.342352] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6624.348222] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6624.353725] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6624.359502] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6624.363782] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6624.369369] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6640.057024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6640.058005] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6640.063910] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6640.064006] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6640.064011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6640.064016] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6640.073751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6640.073830] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6640.073838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6640.073841] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6640.073856] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6640.073860] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6640.073866] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6640.073868] RDX: 00000000000030e5 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6640.073870] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6640.073872] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6640.073875] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6640.073878] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6640.073881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6640.073883] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6640.073891] Call Trace: [ 6640.073897] [ 6640.073902] ? show_regs.cold.9+0x22/0x2f [ 6640.073912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6640.073918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6640.073922] ? watchdog+0x40/0x40 [ 6640.073926] ? hrtimer_interrupt+0x128/0x2c0 [ 6640.073930] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6640.073938] ? apic_timer_interrupt+0xf/0x20 [ 6640.073943] [ 6640.073946] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6640.073951] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6640.073955] do_raw_spin_lock+0x177/0x190 [ 6640.073960] _raw_spin_lock+0x21/0x30 [ 6640.073965] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6640.074611] obd_mod_free+0x46/0x140 [ptlrpc] [ 6640.074816] mdc_commit_open+0x91/0xb0 [mdc] [ 6640.074849] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6640.075046] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6640.075640] after_reply+0xe2c/0x1780 [ptlrpc] [ 6640.075845] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6640.076032] ? __next_timer_interrupt+0x160/0x160 [ 6640.076042] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6640.076697] ? do_wait_intr+0xf0/0xf0 [ 6640.076707] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6640.076913] kthread+0x1d1/0x200 [ 6640.076920] ? set_kthread_struct+0x70/0x70 [ 6640.076924] ret_from_fork+0x1f/0x30 [ 6640.441559] [ 6640.442026] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6640.449713] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6640.461886] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6640.465241] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6640.473002] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6640.478019] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6640.483832] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6640.488267] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6640.493674] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6640.501229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6640.505317] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6640.514011] Call Trace: [ 6640.518037] [ 6640.520328] ? show_regs.cold.9+0x22/0x2f [ 6640.521991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6640.527211] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6640.532016] ? watchdog+0x40/0x40 [ 6640.537616] ? hrtimer_interrupt+0x128/0x2c0 [ 6640.542912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6640.552317] ? apic_timer_interrupt+0xf/0x20 [ 6640.558932] [ 6640.559733] ? native_safe_halt+0x2e/0x50 [ 6640.563632] kvm_wait+0xcd/0xe0 [ 6640.572022] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6640.582059] do_raw_spin_lock+0x177/0x190 [ 6640.586549] _raw_spin_lock+0x21/0x30 [ 6640.589130] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6640.593076] process_one_work+0x2c8/0x700 [ 6640.595447] worker_thread+0x296/0x6e0 [ 6640.598050] ? rescuer_thread+0x570/0x570 [ 6640.600505] kthread+0x1d1/0x200 [ 6640.604063] ? set_kthread_struct+0x70/0x70 [ 6640.607788] ret_from_fork+0x1f/0x30 [ 6644.054340] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6644.059751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6644.122235] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6644.130758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6644.134439] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6644.136537] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6644.146199] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6644.149276] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6644.153613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6644.156527] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6644.162499] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6644.168764] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6644.177031] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6644.182797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6644.189188] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6644.198037] Call Trace: [ 6644.199673] [ 6644.201783] ? show_regs.cold.9+0x22/0x2f [ 6644.203216] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6644.212927] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6644.216760] ? watchdog+0x40/0x40 [ 6644.219379] ? hrtimer_interrupt+0x128/0x2c0 [ 6644.225406] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6644.232258] ? apic_timer_interrupt+0xf/0x20 [ 6644.233889] [ 6644.235483] ? native_safe_halt+0x2e/0x50 [ 6644.241236] kvm_wait+0xcd/0xe0 [ 6644.244249] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6644.251000] do_raw_spin_lock+0x177/0x190 [ 6644.260025] _raw_spin_lock+0x21/0x30 [ 6644.261266] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6644.270265] ? do_raw_spin_unlock+0x75/0x190 [ 6644.271671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6644.275492] ? do_wait_intr+0xf0/0xf0 [ 6644.277430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6644.279463] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6644.281632] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6644.284808] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6644.287317] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6644.288735] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6644.291120] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6644.293229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6644.295467] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6644.298259] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6644.300664] lmv_intent_open+0x338/0x1150 [lmv] [ 6644.302728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6644.305068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6644.307650] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6644.309215] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6644.310970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6644.313162] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6644.314909] ? do_raw_spin_unlock+0x75/0x190 [ 6644.316574] ? _raw_spin_unlock+0x12/0x30 [ 6644.318130] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6644.320228] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6644.322469] ll_atomic_open+0x36a/0xd70 [lustre] [ 6644.324166] lookup_open+0x356/0xb40 [ 6644.325540] do_last+0x313/0xfc0 [ 6644.326779] ? nd_jump_root+0xe5/0x160 [ 6644.328228] ? path_init+0x437/0x520 [ 6644.329760] path_openat+0xf7/0x500 [ 6644.331130] ? do_raw_spin_unlock+0x75/0x190 [ 6644.332815] do_filp_open+0x99/0x140 [ 6644.334192] ? getname_flags+0x6e/0x330 [ 6644.335624] ? __check_object_size+0xff/0x256 [ 6644.337241] ? do_raw_spin_unlock+0x75/0x190 [ 6644.338893] ? _raw_spin_unlock+0x12/0x30 [ 6644.340440] do_sys_openat2+0x2b4/0x410 [ 6644.342070] do_sys_open+0x73/0xa0 [ 6644.343714] __x64_sys_openat+0x24/0x30 [ 6644.345199] do_syscall_64+0xc1/0x440 [ 6644.346559] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6644.350134] RIP: 0033:0x7fa749552332 [ 6644.351697] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6644.361652] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6644.365085] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6644.368019] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6644.371824] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6644.375122] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6644.379795] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6648.295439] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4731s! [ 6648.306377] Showing busy workqueues and worker pools: [ 6648.310632] workqueue events: flags=0x0 [ 6648.315046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6648.323705] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6648.335564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6648.338520] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6648.345114] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.351074] pending: kfree_rcu_monitor [ 6648.353844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6648.359837] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6648.367597] workqueue events_long: flags=0x0 [ 6648.370934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.374550] pending: lu_objects_flush [obdclass] [ 6648.379861] workqueue events_unbound: flags=0x2 [ 6648.385096] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6648.390146] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6648.399102] workqueue events_power_efficient: flags=0x80 [ 6648.403331] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.409123] pending: check_lifetime [ 6648.411838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.419285] pending: neigh_periodic_work [ 6648.422470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.427099] pending: neigh_periodic_work [ 6648.431187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.433347] pending: do_cache_clean [ 6648.443783] workqueue mm_percpu_wq: flags=0x8 [ 6648.456921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.467884] pending: vmstat_update [ 6648.469598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.481168] pending: vmstat_update [ 6648.483729] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6648.492196] pending: vmstat_update [ 6648.494448] workqueue writeback: flags=0x4a [ 6648.500058] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6648.509669] pending: wb_workfn, wb_workfn [ 6648.514067] workqueue xprtiod: flags=0xa [ 6648.518377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6648.522925] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6648.529207] workqueue ipv6_addrconf: flags=0x40008 [ 6648.534796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6648.541233] pending: addrconf_verify_work [ 6648.543119] workqueue knbd0-recv: flags=0x1a [ 6648.545191] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6648.547911] in-flight: 651:recv_work [ 6648.551986] workqueue ptlrpc_pinger: flags=0x2 [ 6648.553332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6648.558364] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6648.561561] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4731s workers=4 idle: 37 31846 28897 [ 6648.571662] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6638s workers=2 idle: 653 [ 6652.063226] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6652.074997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6652.132908] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6652.151971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6652.168017] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6652.171843] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6652.189929] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6652.205098] RAX: 0000000000006806 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6652.217105] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6652.222248] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6652.226457] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6652.230152] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6652.241464] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6652.255136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6652.267422] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6652.270862] Call Trace: [ 6652.271763] [ 6652.276701] ? show_regs.cold.9+0x22/0x2f [ 6652.280393] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6652.290414] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6652.300537] ? watchdog+0x40/0x40 [ 6652.304001] ? hrtimer_interrupt+0x128/0x2c0 [ 6652.308639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6652.312956] ? apic_timer_interrupt+0xf/0x20 [ 6652.320268] [ 6652.326969] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6652.330420] do_raw_spin_lock+0x177/0x190 [ 6652.335956] _raw_spin_lock+0x21/0x30 [ 6652.340433] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6652.353138] ? do_raw_spin_unlock+0x75/0x190 [ 6652.361913] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6652.371091] ? do_wait_intr+0xf0/0xf0 [ 6652.375947] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6652.379929] mdc_close+0x53b/0xd70 [mdc] [ 6652.385079] lmv_close+0x18d/0x490 [lmv] [ 6652.392984] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6652.413242] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6652.421801] ll_md_close+0x274/0x800 [lustre] [ 6652.423551] ll_file_release+0x128/0xc50 [lustre] [ 6652.432461] __fput+0x10d/0x440 [ 6652.433621] ____fput+0x12/0x20 [ 6652.440961] task_work_run+0xc6/0x110 [ 6652.442453] exit_to_usermode_loop+0x1dd/0x1f0 [ 6652.444290] do_syscall_64+0x3d6/0x440 [ 6652.445872] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6652.447997] RIP: 0033:0x7fa749552cd8 [ 6652.449411] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6652.456612] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6652.459468] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6652.461865] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6652.464457] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6652.467334] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6652.469960] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6653.795218] rcu: INFO: rcu_sched self-detected stall on CPU [ 6653.799414] rcu: 2-...!: (2607350 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 6653.814529] (t=4740106 jiffies g=258701 q=2873) [ 6653.816085] rcu: rcu_sched kthread starved for 4736901 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6653.835872] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6653.843033] rcu: RCU grace-period kthread stack dump: [ 6653.845939] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6653.852548] Call Trace: [ 6653.854610] __schedule+0x351/0xcb0 [ 6653.857252] schedule+0xc0/0x180 [ 6653.858578] schedule_timeout+0xb4/0x190 [ 6653.865944] ? do_raw_spin_unlock+0x75/0x190 [ 6653.867515] ? __next_timer_interrupt+0x160/0x160 [ 6653.869051] ? prepare_to_swait_event+0xf3/0x140 [ 6653.883438] rcu_gp_kthread+0x8dd/0x15b0 [ 6653.892582] ? force_qs_rnp+0x2c0/0x2c0 [ 6653.894670] kthread+0x1d1/0x200 [ 6653.896994] ? set_kthread_struct+0x70/0x70 [ 6653.900935] ret_from_fork+0x1f/0x30 [ 6653.902970] rcu: Stack dump where RCU GP kthread last ran: [ 6653.905948] NMI backtrace for cpu 2 [ 6653.907112] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6653.921469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6653.937997] Call Trace: [ 6653.938912] [ 6653.949311] ? dump_stack+0xbb/0x10e [ 6653.950589] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6653.952382] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6653.963592] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6653.970008] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6653.976585] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6653.980687] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 6653.982039] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 6653.989063] ? update_process_times+0x6d/0xc0 [ 6653.991632] ? tick_sched_handle.isra.18+0x27/0x90 [ 6653.993933] ? tick_sched_timer+0x53/0xb0 [ 6653.997912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6654.001091] ? tick_sched_handle.isra.18+0x90/0x90 [ 6654.005783] ? hrtimer_interrupt+0x128/0x2c0 [ 6654.007323] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6654.020689] ? apic_timer_interrupt+0xf/0x20 [ 6654.025597] [ 6654.028780] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6654.037886] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6654.041333] ? do_raw_spin_lock+0x177/0x190 [ 6654.044033] ? _raw_spin_lock+0x21/0x30 [ 6654.046654] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6654.060059] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6654.062890] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6654.071904] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6654.077563] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6654.079777] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6654.091794] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6654.095827] ? __next_timer_interrupt+0x160/0x160 [ 6654.100457] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6654.103163] ? do_wait_intr+0xf0/0xf0 [ 6654.104323] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6654.110476] ? kthread+0x1d1/0x200 [ 6654.112825] ? set_kthread_struct+0x70/0x70 [ 6654.124334] ? ret_from_fork+0x1f/0x30 [ 6654.125650] NMI backtrace for cpu 2 [ 6654.128795] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6654.137463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6654.141844] Call Trace: [ 6654.143706] [ 6654.144649] ? dump_stack+0xbb/0x10e [ 6654.146497] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6654.148887] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6654.153473] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6654.157200] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6654.161508] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6654.168446] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6654.171575] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6654.180238] ? update_process_times+0x6d/0xc0 [ 6654.183741] ? tick_sched_handle.isra.18+0x27/0x90 [ 6654.185679] ? tick_sched_timer+0x53/0xb0 [ 6654.189780] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6654.192791] ? tick_sched_handle.isra.18+0x90/0x90 [ 6654.194364] ? hrtimer_interrupt+0x128/0x2c0 [ 6654.201590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6654.206077] ? apic_timer_interrupt+0xf/0x20 [ 6654.209148] [ 6654.209868] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6654.212092] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6654.221637] ? do_raw_spin_lock+0x177/0x190 [ 6654.224567] ? _raw_spin_lock+0x21/0x30 [ 6654.225898] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6654.229974] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6654.232663] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6654.234670] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6654.237672] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6654.239836] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6654.242023] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6654.243938] ? __next_timer_interrupt+0x160/0x160 [ 6654.246141] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6654.247874] ? do_wait_intr+0xf0/0xf0 [ 6654.259992] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6654.262006] ? kthread+0x1d1/0x200 [ 6654.264389] ? set_kthread_struct+0x70/0x70 [ 6654.271424] ? ret_from_fork+0x1f/0x30 [ 6668.059153] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6668.063431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6668.080082] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6668.082756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6668.084783] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6668.086717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6668.088702] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6668.093129] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6668.094645] RAX: 0000000000003bd8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6668.096124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6668.098992] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6668.101654] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6668.104524] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6668.107496] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6668.110515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6668.112760] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6668.116541] Call Trace: [ 6668.117451] [ 6668.125647] ? show_regs.cold.9+0x22/0x2f [ 6668.129820] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6668.137426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6668.140561] ? watchdog+0x40/0x40 [ 6668.142378] ? hrtimer_interrupt+0x128/0x2c0 [ 6668.144637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6668.150870] ? apic_timer_interrupt+0xf/0x20 [ 6668.153439] [ 6668.154379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6668.161322] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6668.172317] do_raw_spin_lock+0x177/0x190 [ 6668.176826] _raw_spin_lock+0x21/0x30 [ 6668.180430] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6668.183716] process_one_work+0x2c8/0x700 [ 6668.186463] worker_thread+0x296/0x6e0 [ 6668.189529] ? rescuer_thread+0x570/0x570 [ 6668.193413] kthread+0x1d1/0x200 [ 6668.198505] ? set_kthread_struct+0x70/0x70 [ 6668.200549] ret_from_fork+0x1f/0x30 [ 6676.054167] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6676.058964] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6676.080371] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6676.089926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6676.094193] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6676.096422] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6676.112479] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6676.117374] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6676.121230] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6676.124708] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6676.128434] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6676.132169] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6676.135608] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6676.139503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6676.141210] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6676.144372] Call Trace: [ 6676.145445] [ 6676.146529] ? show_regs.cold.9+0x22/0x2f [ 6676.149394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6676.152016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6676.154967] ? watchdog+0x40/0x40 [ 6676.157554] ? hrtimer_interrupt+0x128/0x2c0 [ 6676.160092] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6676.162611] ? apic_timer_interrupt+0xf/0x20 [ 6676.164917] [ 6676.169084] ? native_safe_halt+0x2e/0x50 [ 6676.171302] kvm_wait+0xcd/0xe0 [ 6676.172905] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6676.175819] do_raw_spin_lock+0x177/0x190 [ 6676.181441] _raw_spin_lock+0x21/0x30 [ 6676.185130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6676.188236] ? do_raw_spin_unlock+0x75/0x190 [ 6676.191557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6676.195093] ? do_wait_intr+0xf0/0xf0 [ 6676.199130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6676.202525] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6676.205223] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6676.214900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6676.217983] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6676.224940] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6676.227839] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6676.229875] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6676.232939] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6676.237586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6676.241047] lmv_intent_open+0x338/0x1150 [lmv] [ 6676.243669] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6676.246920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6676.251156] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6676.256202] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6676.259585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6676.263067] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6676.265578] ? do_raw_spin_unlock+0x75/0x190 [ 6676.268194] ? _raw_spin_unlock+0x12/0x30 [ 6676.270642] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6676.273632] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6676.276404] ll_atomic_open+0x36a/0xd70 [lustre] [ 6676.282150] lookup_open+0x356/0xb40 [ 6676.285804] do_last+0x313/0xfc0 [ 6676.287465] ? nd_jump_root+0xe5/0x160 [ 6676.289625] ? path_init+0x437/0x520 [ 6676.291900] path_openat+0xf7/0x500 [ 6676.293928] ? do_raw_spin_unlock+0x75/0x190 [ 6676.297192] do_filp_open+0x99/0x140 [ 6676.299460] ? getname_flags+0x6e/0x330 [ 6676.306535] ? __check_object_size+0xff/0x256 [ 6676.308574] ? do_raw_spin_unlock+0x75/0x190 [ 6676.315603] ? _raw_spin_unlock+0x12/0x30 [ 6676.320433] do_sys_openat2+0x2b4/0x410 [ 6676.323342] do_sys_open+0x73/0xa0 [ 6676.326123] __x64_sys_openat+0x24/0x30 [ 6676.328920] do_syscall_64+0xc1/0x440 [ 6676.331668] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6676.335399] RIP: 0033:0x7fa749552332 [ 6676.337226] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6676.349656] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6676.354752] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6676.358784] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6676.362560] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6676.366590] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6676.370848] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6679.011050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4761s! [ 6679.017462] Showing busy workqueues and worker pools: [ 6679.021233] workqueue events: flags=0x0 [ 6679.025786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6679.030642] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6679.044779] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6679.069302] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6679.085530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.095862] pending: kfree_rcu_monitor [ 6679.098237] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6679.104506] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6679.108029] workqueue events_long: flags=0x0 [ 6679.112083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.120204] pending: lu_objects_flush [obdclass] [ 6679.123930] workqueue events_unbound: flags=0x2 [ 6679.126268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6679.132982] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6679.137995] workqueue events_power_efficient: flags=0x80 [ 6679.146087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.150429] pending: check_lifetime [ 6679.154337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.157187] pending: neigh_periodic_work [ 6679.159010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.162623] pending: neigh_periodic_work [ 6679.166057] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.176375] pending: do_cache_clean [ 6679.182077] workqueue mm_percpu_wq: flags=0x8 [ 6679.187185] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.192799] pending: vmstat_update [ 6679.195788] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.204775] pending: vmstat_update [ 6679.206121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6679.216344] pending: vmstat_update [ 6679.217722] workqueue writeback: flags=0x4a [ 6679.221931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6679.225203] pending: wb_workfn, wb_workfn [ 6679.228179] workqueue xprtiod: flags=0xa [ 6679.230475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6679.233807] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6679.237769] workqueue ipv6_addrconf: flags=0x40008 [ 6679.239736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6679.244488] pending: addrconf_verify_work [ 6679.248152] workqueue knbd0-recv: flags=0x1a [ 6679.250128] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6679.254816] in-flight: 651:recv_work [ 6679.256238] workqueue ptlrpc_pinger: flags=0x2 [ 6679.257641] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6679.261044] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6679.270268] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4762s workers=4 idle: 37 31846 28897 [ 6679.280674] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6669s workers=2 idle: 653 [ 6680.058018] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6680.059382] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6680.063295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel [ 6680.069021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.069104] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.069117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.069121] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6680.069139] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6680.069144] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6680.069149] RAX: 000000000000740b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6680.069152] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6680.069154] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6680.098114] ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.098138] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.098146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.098149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6680.098176] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6680.098181] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6680.098186] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6680.098188] RDX: 000000000000532b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6680.098191] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6680.098193] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6680.098196] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6680.098199] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6680.098202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.098205] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6680.098212] Call Trace: [ 6680.098218] [ 6680.098223] ? show_regs.cold.9+0x22/0x2f [ 6680.098232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.098238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.098243] ? watchdog+0x40/0x40 [ 6680.098248] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.098252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.098259] ? apic_timer_interrupt+0xf/0x20 [ 6680.098263] [ 6680.098266] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6680.098271] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6680.098276] do_raw_spin_lock+0x177/0x190 [ 6680.098281] _raw_spin_lock+0x21/0x30 [ 6680.098288] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6680.098620] obd_mod_free+0x46/0x140 [ptlrpc] [ 6680.099065] mdc_commit_open+0x91/0xb0 [mdc] [ 6680.099100] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6680.099419] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6680.099758] after_reply+0xe2c/0x1780 [ptlrpc] [ 6680.100112] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6680.100373] ? __next_timer_interrupt+0x160/0x160 [ 6680.100383] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6680.129996] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6680.130008] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6680.130012] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6680.130016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.130019] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6680.130028] Call Trace: [ 6680.130035] [ 6680.130040] ? show_regs.cold.9+0x22/0x2f [ 6680.130052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.130058] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.130063] ? watchdog+0x40/0x40 [ 6680.130068] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.130071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.130079] ? apic_timer_interrupt+0xf/0x20 [ 6680.130083] [ 6680.130086] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6680.130095] do_raw_spin_lock+0x177/0x190 [ 6680.130099] _raw_spin_lock+0x21/0x30 [ 6680.130103] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6680.130517] ? do_raw_spin_unlock+0x75/0x190 [ 6680.130523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6680.131010] ? do_wait_intr+0xf0/0xf0 [ 6680.131020] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6680.131432] mdc_close+0x53b/0xd70 [mdc] [ 6680.131470] lmv_close+0x18d/0x490 [lmv] [ 6680.131498] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6680.131631] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6680.131783] ll_md_close+0x274/0x800 [lustre] [ 6680.131968] ll_file_release+0x128/0xc50 [lustre] [ 6680.132060] __fput+0x10d/0x440 [ 6680.132072] ____fput+0x12/0x20 [ 6680.132076] task_work_run+0xc6/0x110 [ 6680.132084] exit_to_usermode_loop+0x1dd/0x1f0 [ 6680.132091] do_syscall_64+0x3d6/0x440 [ 6680.132095] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6680.132104] RIP: 0033:0x7fa749552cd8 [ 6680.132111] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6680.132115] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6680.132121] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6680.132124] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6680.132126] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6680.132128] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6680.132131] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6680.389815] ? do_wait_intr+0xf0/0xf0 [ 6680.391561] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6680.394043] kthread+0x1d1/0x200 [ 6680.395947] ? set_kthread_struct+0x70/0x70 [ 6680.397747] ret_from_fork+0x1f/0x30 [ 6696.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6696.058914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6696.091668] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6696.096240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6696.099754] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6696.102699] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6696.104851] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6696.112805] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6696.117422] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6696.124790] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6696.129070] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6696.137446] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6696.142067] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6696.148525] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6696.150504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6696.152573] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6696.158005] Call Trace: [ 6696.161237] [ 6696.161750] ? show_regs.cold.9+0x22/0x2f [ 6696.162838] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6696.166004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6696.172738] ? watchdog+0x40/0x40 [ 6696.174959] ? hrtimer_interrupt+0x128/0x2c0 [ 6696.177281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6696.183672] ? apic_timer_interrupt+0xf/0x20 [ 6696.186168] [ 6696.187038] ? native_safe_halt+0x2e/0x50 [ 6696.188340] kvm_wait+0xcd/0xe0 [ 6696.189204] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6696.191091] do_raw_spin_lock+0x177/0x190 [ 6696.192365] _raw_spin_lock+0x21/0x30 [ 6696.193325] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6696.195563] process_one_work+0x2c8/0x700 [ 6696.197018] worker_thread+0x296/0x6e0 [ 6696.198607] ? rescuer_thread+0x570/0x570 [ 6696.199645] kthread+0x1d1/0x200 [ 6696.200537] ? set_kthread_struct+0x70/0x70 [ 6696.201724] ret_from_fork+0x1f/0x30 [ 6704.054754] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6704.059342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6704.094696] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6704.100086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6704.102783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6704.105439] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6704.112962] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6704.115706] RAX: 0000000000007e1a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6704.118909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6704.122072] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6704.125643] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6704.128481] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6704.131914] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6704.136292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6704.140889] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6704.149190] Call Trace: [ 6704.151482] [ 6704.159042] ? show_regs.cold.9+0x22/0x2f [ 6704.160595] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6704.165358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6704.169347] ? watchdog+0x40/0x40 [ 6704.172352] ? hrtimer_interrupt+0x128/0x2c0 [ 6704.176491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6704.181974] ? apic_timer_interrupt+0xf/0x20 [ 6704.191361] [ 6704.192199] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6704.199390] do_raw_spin_lock+0x177/0x190 [ 6704.200805] _raw_spin_lock+0x21/0x30 [ 6704.208333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6704.211408] ? do_raw_spin_unlock+0x75/0x190 [ 6704.213336] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6704.217750] ? do_wait_intr+0xf0/0xf0 [ 6704.219121] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6704.224312] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6704.227502] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6704.234165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6704.240001] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6704.244132] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6704.248142] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6704.250777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6704.256552] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6704.265643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6704.272011] lmv_intent_open+0x338/0x1150 [lmv] [ 6704.275151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6704.281057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6704.283071] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6704.286924] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6704.292512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6704.296195] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6704.299589] ? do_raw_spin_unlock+0x75/0x190 [ 6704.304825] ? _raw_spin_unlock+0x12/0x30 [ 6704.309956] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6704.315379] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6704.323487] ll_atomic_open+0x36a/0xd70 [lustre] [ 6704.330412] lookup_open+0x356/0xb40 [ 6704.336039] do_last+0x313/0xfc0 [ 6704.339770] ? nd_jump_root+0xe5/0x160 [ 6704.343051] ? path_init+0x437/0x520 [ 6704.344782] path_openat+0xf7/0x500 [ 6704.346444] ? do_raw_spin_unlock+0x75/0x190 [ 6704.351846] do_filp_open+0x99/0x140 [ 6704.353318] ? getname_flags+0x6e/0x330 [ 6704.354906] ? __check_object_size+0xff/0x256 [ 6704.358400] ? do_raw_spin_unlock+0x75/0x190 [ 6704.361011] ? _raw_spin_unlock+0x12/0x30 [ 6704.365372] do_sys_openat2+0x2b4/0x410 [ 6704.369168] do_sys_open+0x73/0xa0 [ 6704.370423] __x64_sys_openat+0x24/0x30 [ 6704.373726] do_syscall_64+0xc1/0x440 [ 6704.375730] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6704.377987] RIP: 0033:0x7fa749552332 [ 6704.379766] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6704.388921] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6704.391451] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6704.393826] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6704.397756] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6704.401474] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6704.404495] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6708.059118] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6708.062408] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6708.068677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6708.068762] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6708.068770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6708.068773] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6708.068786] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6708.068791] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6708.068795] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6708.068798] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6708.068800] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6708.068803] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6708.068806] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6708.068809] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6708.068812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6708.068815] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6708.068823] Call Trace: [ 6708.068829] [ 6708.068835] ? show_regs.cold.9+0x22/0x2f [ 6708.068844] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6708.068850] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6708.068855] ? watchdog+0x40/0x40 [ 6708.068860] ? hrtimer_interrupt+0x128/0x2c0 [ 6708.068865] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6708.068871] ? apic_timer_interrupt+0xf/0x20 [ 6708.068876] [ 6708.068879] ? native_safe_halt+0x2e/0x50 [ 6708.068884] kvm_wait+0xcd/0xe0 [ 6708.068891] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6708.068900] do_raw_spin_lock+0x177/0x190 [ 6708.068904] _raw_spin_lock+0x21/0x30 [ 6708.068909] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6708.079765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6708.079846] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6708.079853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6708.079856] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6708.079870] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6708.079874] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6708.079879] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6708.079881] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6708.079884] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6708.079886] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6708.079889] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6708.079892] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6708.079895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6708.079897] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6708.079905] Call Trace: [ 6708.079910] [ 6708.079915] ? show_regs.cold.9+0x22/0x2f [ 6708.079924] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6708.079930] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6708.079934] ? watchdog+0x40/0x40 [ 6708.079938] ? hrtimer_interrupt+0x128/0x2c0 [ 6708.079942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6708.079947] ? apic_timer_interrupt+0xf/0x20 [ 6708.079952] [ 6708.079955] ? native_safe_halt+0x2e/0x50 [ 6708.079959] kvm_wait+0xcd/0xe0 [ 6708.079967] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6708.079974] do_raw_spin_lock+0x177/0x190 [ 6708.079979] _raw_spin_lock+0x21/0x30 [ 6708.079984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6708.123448] ? do_raw_spin_unlock+0x75/0x190 [ 6708.123478] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6708.123759] ? do_wait_intr+0xf0/0xf0 [ 6708.123767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6708.123993] mdc_close+0x53b/0xd70 [mdc] [ 6708.132538] obd_mod_free+0x46/0x140 [ptlrpc] [ 6708.132819] mdc_commit_open+0x91/0xb0 [mdc] [ 6708.132856] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6708.138508] lmv_close+0x18d/0x490 [lmv] [ 6708.138550] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6708.138663] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6708.138753] ll_md_close+0x274/0x800 [lustre] [ 6708.138844] ll_file_release+0x128/0xc50 [lustre] [ 6708.138934] __fput+0x10d/0x440 [ 6708.138944] ____fput+0x12/0x20 [ 6708.138949] task_work_run+0xc6/0x110 [ 6708.138957] exit_to_usermode_loop+0x1dd/0x1f0 [ 6708.138965] do_syscall_64+0x3d6/0x440 [ 6708.138970] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6708.138979] RIP: 0033:0x7fa749552cd8 [ 6708.138987] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6708.138991] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6708.139216] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6708.139219] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6708.139222] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6708.139224] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6708.139227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6708.673657] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6708.682366] after_reply+0xe2c/0x1780 [ptlrpc] [ 6708.687833] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6708.691884] ? __next_timer_interrupt+0x160/0x160 [ 6708.695547] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6708.700113] ? do_wait_intr+0xf0/0xf0 [ 6708.706338] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6708.709605] kthread+0x1d1/0x200 [ 6708.711595] ? set_kthread_struct+0x70/0x70 [ 6708.716296] ret_from_fork+0x1f/0x30 [ 6709.732686] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4792s! [ 6709.741262] Showing busy workqueues and worker pools: [ 6709.748776] workqueue events: flags=0x0 [ 6709.752947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6709.762841] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6709.781507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6709.790619] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6709.803446] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.811813] pending: kfree_rcu_monitor [ 6709.814480] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6709.817802] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6709.822802] workqueue events_long: flags=0x0 [ 6709.824468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.827023] pending: lu_objects_flush [obdclass] [ 6709.833365] workqueue events_unbound: flags=0x2 [ 6709.835448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6709.838235] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6709.842624] workqueue events_power_efficient: flags=0x80 [ 6709.844819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.851479] pending: check_lifetime [ 6709.855661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.860701] pending: neigh_periodic_work [ 6709.863672] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.872112] pending: neigh_periodic_work [ 6709.874667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.878396] pending: do_cache_clean [ 6709.880117] workqueue mm_percpu_wq: flags=0x8 [ 6709.883227] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.886774] pending: vmstat_update [ 6709.888796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.892626] pending: vmstat_update [ 6709.894640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6709.898349] pending: vmstat_update [ 6709.900579] workqueue writeback: flags=0x4a [ 6709.905496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6709.908803] pending: wb_workfn, wb_workfn [ 6709.912328] workqueue xprtiod: flags=0xa [ 6709.919757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6709.929915] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6709.939250] workqueue ipv6_addrconf: flags=0x40008 [ 6709.941957] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6709.949351] pending: addrconf_verify_work [ 6709.952743] workqueue knbd0-recv: flags=0x1a [ 6709.963232] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6709.968635] in-flight: 651:recv_work [ 6709.970071] workqueue ptlrpc_pinger: flags=0x2 [ 6709.972626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6709.975636] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6709.981826] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4792s workers=4 idle: 37 31846 28897 [ 6709.992888] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6699s workers=2 idle: 653 [ 6724.058678] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6724.068694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6724.122677] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6724.140851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6724.147440] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6724.156295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6724.157978] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6724.181889] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6724.190329] RAX: 00000000000058f1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6724.202039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6724.208631] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6724.216754] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6724.221355] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6724.227790] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6724.237247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6724.245589] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6724.247637] Call Trace: [ 6724.248334] [ 6724.248910] ? show_regs.cold.9+0x22/0x2f [ 6724.256686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6724.258276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6724.268414] ? watchdog+0x40/0x40 [ 6724.269600] ? hrtimer_interrupt+0x128/0x2c0 [ 6724.271076] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6724.282291] ? apic_timer_interrupt+0xf/0x20 [ 6724.285181] [ 6724.286363] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6724.294340] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6724.297678] do_raw_spin_lock+0x177/0x190 [ 6724.300703] _raw_spin_lock+0x21/0x30 [ 6724.306067] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6724.310655] process_one_work+0x2c8/0x700 [ 6724.315922] worker_thread+0x296/0x6e0 [ 6724.319142] ? rescuer_thread+0x570/0x570 [ 6724.328018] kthread+0x1d1/0x200 [ 6724.331333] ? set_kthread_struct+0x70/0x70 [ 6724.336631] ret_from_fork+0x1f/0x30 [ 6736.054027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6736.056592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6736.058052] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6736.059025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6736.059034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6736.059120] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6736.059125] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6736.059128] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6736.059141] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6736.059145] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6736.059149] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6736.059152] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6736.059154] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6736.059157] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6736.059159] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6736.059162] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6736.059165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6736.059167] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6736.059176] Call Trace: [ 6736.059182] [ 6736.059189] ? show_regs.cold.9+0x22/0x2f [ 6736.059200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6736.059205] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6736.059210] ? watchdog+0x40/0x40 [ 6736.059214] ? hrtimer_interrupt+0x128/0x2c0 [ 6736.059218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6736.059227] ? apic_timer_interrupt+0xf/0x20 [ 6736.059231] [ 6736.059235] ? native_safe_halt+0x2e/0x50 [ 6736.059239] kvm_wait+0xcd/0xe0 [ 6736.059247] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6736.059256] do_raw_spin_lock+0x177/0x190 [ 6736.059261] _raw_spin_lock+0x21/0x30 [ 6736.059265] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6736.059560] ? do_raw_spin_unlock+0x75/0x190 [ 6736.059567] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6736.059777] ? do_wait_intr+0xf0/0xf0 [ 6736.059785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6736.060006] mdc_close+0x53b/0xd70 [mdc] [ 6736.060045] lmv_close+0x18d/0x490 [lmv] [ 6736.060077] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6736.060186] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6736.060270] ll_md_close+0x274/0x800 [lustre] [ 6736.060357] ll_file_release+0x128/0xc50 [lustre] [ 6736.060440] __fput+0x10d/0x440 [ 6736.060449] ____fput+0x12/0x20 [ 6736.060454] task_work_run+0xc6/0x110 [ 6736.060461] exit_to_usermode_loop+0x1dd/0x1f0 [ 6736.060468] do_syscall_64+0x3d6/0x440 [ 6736.060473] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6736.060480] RIP: 0033:0x7fa749552cd8 [ 6736.060486] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6736.060490] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6736.060496] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6736.060498] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6736.060501] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6736.060503] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6736.060506] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6736.113587] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6736.115795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6736.115850] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6736.115857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6736.115859] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6736.115873] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6736.115877] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6736.115881] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6736.115884] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6736.115886] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6736.115888] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6736.115890] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6736.115893] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6736.115896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6736.115898] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6736.115908] Call Trace: [ 6736.115914] [ 6736.115919] ? show_regs.cold.9+0x22/0x2f [ 6736.115928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6736.115932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6736.115935] ? watchdog+0x40/0x40 [ 6736.115938] ? hrtimer_interrupt+0x128/0x2c0 [ 6736.115940] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6736.115947] ? apic_timer_interrupt+0xf/0x20 [ 6736.120812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6736.131518] [ 6736.136762] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6736.136781] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6736.136785] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6736.136789] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6736.136791] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6736.136793] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6736.136794] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6736.136796] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6736.136799] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6736.136801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6736.136802] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6736.136808] Call Trace: [ 6736.136812] [ 6736.136816] ? show_regs.cold.9+0x22/0x2f [ 6736.136824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6736.136829] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6736.136831] ? watchdog+0x40/0x40 [ 6736.136834] ? hrtimer_interrupt+0x128/0x2c0 [ 6736.136837] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6736.136841] ? apic_timer_interrupt+0xf/0x20 [ 6736.136845] [ 6736.136846] ? native_safe_halt+0x2e/0x50 [ 6736.136849] kvm_wait+0xcd/0xe0 [ 6736.136856] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6736.136866] do_raw_spin_lock+0x177/0x190 [ 6736.136870] _raw_spin_lock+0x21/0x30 [ 6736.136874] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6736.137169] ? do_raw_spin_unlock+0x75/0x190 [ 6736.137175] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6736.137378] ? do_wait_intr+0xf0/0xf0 [ 6736.137387] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6736.137589] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6736.143668] ? native_safe_halt+0x2e/0x50 [ 6736.147488] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6736.147718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.147815] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6736.147848] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6736.147875] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6736.147901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.147985] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6736.148138] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6736.148157] lmv_intent_open+0x338/0x1150 [lmv] [ 6736.148176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.148233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.148287] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6736.148302] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6736.148358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.148412] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6736.148467] ? do_raw_spin_unlock+0x75/0x190 [ 6736.148473] ? _raw_spin_unlock+0x12/0x30 [ 6736.148478] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6736.148498] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6736.148511] ll_atomic_open+0x36a/0xd70 [lustre] [ 6736.148567] lookup_open+0x356/0xb40 [ 6736.148574] do_last+0x313/0xfc0 [ 6736.148578] ? nd_jump_root+0xe5/0x160 [ 6736.148581] ? path_init+0x437/0x520 [ 6736.148584] path_openat+0xf7/0x500 [ 6736.148587] ? do_raw_spin_unlock+0x75/0x190 [ 6736.148591] do_filp_open+0x99/0x140 [ 6736.148595] ? getname_flags+0x6e/0x330 [ 6736.148598] ? __check_object_size+0xff/0x256 [ 6736.148603] ? do_raw_spin_unlock+0x75/0x190 [ 6736.148606] ? _raw_spin_unlock+0x12/0x30 [ 6736.148609] do_sys_openat2+0x2b4/0x410 [ 6736.148612] do_sys_open+0x73/0xa0 [ 6736.148614] __x64_sys_openat+0x24/0x30 [ 6736.148616] do_syscall_64+0xc1/0x440 [ 6736.148621] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6736.148625] RIP: 0033:0x7fa749552332 [ 6736.148632] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6736.148635] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6736.148638] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6736.148640] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6736.148641] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6736.148642] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6736.148644] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6736.461100] kvm_wait+0xcd/0xe0 [ 6736.462112] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6736.463554] do_raw_spin_lock+0x177/0x190 [ 6736.465325] _raw_spin_lock+0x21/0x30 [ 6736.466983] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6736.469255] obd_mod_free+0x46/0x140 [ptlrpc] [ 6736.472163] mdc_commit_open+0x91/0xb0 [mdc] [ 6736.474234] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6736.477143] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6736.479301] after_reply+0xe2c/0x1780 [ptlrpc] [ 6736.482122] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6736.485063] ? __next_timer_interrupt+0x160/0x160 [ 6736.487496] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6736.491227] ? do_wait_intr+0xf0/0xf0 [ 6736.493213] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6736.496096] kthread+0x1d1/0x200 [ 6736.497963] ? set_kthread_struct+0x70/0x70 [ 6736.499996] ret_from_fork+0x1f/0x30 [ 6740.448805] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4823s! [ 6740.455153] Showing busy workqueues and worker pools: [ 6740.458651] workqueue events: flags=0x0 [ 6740.461313] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6740.466360] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6740.481628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6740.490594] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6740.506248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.510464] pending: kfree_rcu_monitor [ 6740.515134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6740.523027] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6740.546591] workqueue events_long: flags=0x0 [ 6740.556938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.564292] pending: lu_objects_flush [obdclass] [ 6740.568266] workqueue events_unbound: flags=0x2 [ 6740.578320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6740.583337] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6740.594256] workqueue events_power_efficient: flags=0x80 [ 6740.599154] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.603662] pending: check_lifetime [ 6740.607147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.617350] pending: neigh_periodic_work [ 6740.622595] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.627371] pending: neigh_periodic_work [ 6740.629185] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.632586] pending: do_cache_clean [ 6740.634444] workqueue mm_percpu_wq: flags=0x8 [ 6740.636430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.639427] pending: vmstat_update [ 6740.641126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.644350] pending: vmstat_update [ 6740.646860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6740.650488] pending: vmstat_update [ 6740.652469] workqueue writeback: flags=0x4a [ 6740.655873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6740.668231] pending: wb_workfn, wb_workfn [ 6740.672034] workqueue xprtiod: flags=0xa [ 6740.673571] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6740.681452] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6740.688043] workqueue ipv6_addrconf: flags=0x40008 [ 6740.691911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6740.700305] pending: addrconf_verify_work [ 6740.703062] workqueue knbd0-recv: flags=0x1a [ 6740.709066] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6740.717086] in-flight: 651:recv_work [ 6740.718219] workqueue ptlrpc_pinger: flags=0x2 [ 6740.726418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6740.733100] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6740.739947] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4823s workers=4 idle: 37 31846 28897 [ 6740.748011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6730s workers=2 idle: 653 [ 6752.058493] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6752.066103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6752.116165] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6752.133372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6752.138192] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6752.145937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6752.148732] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6752.164560] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6752.169481] RAX: 0000000000007f0a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6752.173013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6752.176908] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6752.180502] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6752.186178] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6752.194588] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6752.202237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6752.206075] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6752.221657] Call Trace: [ 6752.224226] [ 6752.227574] ? show_regs.cold.9+0x22/0x2f [ 6752.234107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6752.238203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6752.241071] ? watchdog+0x40/0x40 [ 6752.243855] ? hrtimer_interrupt+0x128/0x2c0 [ 6752.246900] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6752.251035] ? apic_timer_interrupt+0xf/0x20 [ 6752.253276] [ 6752.254390] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6752.257907] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6752.260120] do_raw_spin_lock+0x177/0x190 [ 6752.265129] _raw_spin_lock+0x21/0x30 [ 6752.267585] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6752.270740] process_one_work+0x2c8/0x700 [ 6752.272911] worker_thread+0x296/0x6e0 [ 6752.274899] ? rescuer_thread+0x570/0x570 [ 6752.277011] kthread+0x1d1/0x200 [ 6752.278588] ? set_kthread_struct+0x70/0x70 [ 6752.280839] ret_from_fork+0x1f/0x30 [ 6764.063243] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6764.066431] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6764.068754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6764.068832] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6764.068840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6764.068843] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6764.068856] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6764.068861] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6764.068865] RAX: 0000000000007106 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6764.068868] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6764.068870] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6764.068872] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6764.068875] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6764.068878] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6764.068881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6764.068883] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6764.068892] Call Trace: [ 6764.068897] [ 6764.068902] ? show_regs.cold.9+0x22/0x2f [ 6764.068911] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6764.068918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6764.068922] ? watchdog+0x40/0x40 [ 6764.068926] ? hrtimer_interrupt+0x128/0x2c0 [ 6764.068930] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6764.068938] ? apic_timer_interrupt+0xf/0x20 [ 6764.068942] [ 6764.068945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6764.068952] do_raw_spin_lock+0x177/0x190 [ 6764.068957] _raw_spin_lock+0x21/0x30 [ 6764.068962] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6764.070671] ? do_raw_spin_unlock+0x75/0x190 [ 6764.070679] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6764.070895] ? do_wait_intr+0xf0/0xf0 [ 6764.070904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6764.075937] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6764.076023] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6764.076030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6764.076033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6764.076046] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6764.076051] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6764.076055] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6764.076058] RDX: 000000000000539c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6764.076060] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6764.076063] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6764.076065] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6764.076068] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6764.076071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6764.076073] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6764.076084] Call Trace: [ 6764.076089] [ 6764.076300] ? show_regs.cold.9+0x22/0x2f [ 6764.076311] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6764.076317] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6764.076322] ? watchdog+0x40/0x40 [ 6764.076326] ? hrtimer_interrupt+0x128/0x2c0 [ 6764.076330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6764.076339] ? apic_timer_interrupt+0xf/0x20 [ 6764.076344] [ 6764.076347] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6764.076351] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6764.076356] do_raw_spin_lock+0x177/0x190 [ 6764.076361] _raw_spin_lock+0x21/0x30 [ 6764.076366] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6764.076664] obd_mod_free+0x46/0x140 [ptlrpc] [ 6764.076882] mdc_commit_open+0x91/0xb0 [mdc] [ 6764.076923] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6764.077149] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6764.077422] after_reply+0xe2c/0x1780 [ptlrpc] [ 6764.077651] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6764.077865] ? __next_timer_interrupt+0x160/0x160 [ 6764.077876] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6764.078091] ? do_wait_intr+0xf0/0xf0 [ 6764.078100] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6764.078677] kthread+0x1d1/0x200 [ 6764.078683] ? set_kthread_struct+0x70/0x70 [ 6764.078688] ret_from_fork+0x1f/0x30 [ 6764.428306] mdc_close+0x53b/0xd70 [mdc] [ 6764.432461] lmv_close+0x18d/0x490 [lmv] [ 6764.437657] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6764.440817] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6764.445713] ll_md_close+0x274/0x800 [lustre] [ 6764.448572] ll_file_release+0x128/0xc50 [lustre] [ 6764.450866] __fput+0x10d/0x440 [ 6764.452708] ____fput+0x12/0x20 [ 6764.454917] task_work_run+0xc6/0x110 [ 6764.457793] exit_to_usermode_loop+0x1dd/0x1f0 [ 6764.460281] do_syscall_64+0x3d6/0x440 [ 6764.465185] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6764.468554] RIP: 0033:0x7fa749552cd8 [ 6764.471564] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6764.482575] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6764.487539] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6764.492540] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6764.496656] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6764.499761] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6764.507500] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6768.055071] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6768.059126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6768.085370] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6768.090446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6768.094754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6768.098033] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6768.109348] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6768.113262] RAX: 0000000000001a83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6768.117839] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6768.123646] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6768.130042] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6768.137125] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6768.142135] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6768.149093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6768.159340] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6768.168460] Call Trace: [ 6768.170941] [ 6768.171923] ? show_regs.cold.9+0x22/0x2f [ 6768.173449] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6768.178400] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6768.181090] ? watchdog+0x40/0x40 [ 6768.185045] ? hrtimer_interrupt+0x128/0x2c0 [ 6768.187120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6768.189001] ? apic_timer_interrupt+0xf/0x20 [ 6768.193113] [ 6768.195774] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6768.201104] do_raw_spin_lock+0x177/0x190 [ 6768.202960] _raw_spin_lock+0x21/0x30 [ 6768.207480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6768.211495] ? do_raw_spin_unlock+0x75/0x190 [ 6768.219259] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6768.221401] ? do_wait_intr+0xf0/0xf0 [ 6768.223474] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6768.226979] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6768.233328] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6768.237167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6768.242449] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6768.252214] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6768.254264] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6768.266213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6768.268681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6768.274932] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6768.278462] lmv_intent_open+0x338/0x1150 [lmv] [ 6768.281303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6768.284484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6768.288353] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6768.290478] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6768.293647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6768.296927] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6768.299461] ? do_raw_spin_unlock+0x75/0x190 [ 6768.301667] ? _raw_spin_unlock+0x12/0x30 [ 6768.304884] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6768.309130] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6768.313037] ll_atomic_open+0x36a/0xd70 [lustre] [ 6768.315899] lookup_open+0x356/0xb40 [ 6768.319330] do_last+0x313/0xfc0 [ 6768.321462] ? nd_jump_root+0xe5/0x160 [ 6768.323671] ? path_init+0x437/0x520 [ 6768.325912] path_openat+0xf7/0x500 [ 6768.332213] ? do_raw_spin_unlock+0x75/0x190 [ 6768.335375] do_filp_open+0x99/0x140 [ 6768.337927] ? getname_flags+0x6e/0x330 [ 6768.341021] ? __check_object_size+0xff/0x256 [ 6768.343138] ? do_raw_spin_unlock+0x75/0x190 [ 6768.345960] ? _raw_spin_unlock+0x12/0x30 [ 6768.347955] do_sys_openat2+0x2b4/0x410 [ 6768.350318] do_sys_open+0x73/0xa0 [ 6768.355374] __x64_sys_openat+0x24/0x30 [ 6768.358771] do_syscall_64+0xc1/0x440 [ 6768.361852] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6768.369106] RIP: 0033:0x7fa749552332 [ 6768.372547] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6768.395493] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6768.404087] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6768.408955] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6768.413082] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6768.422176] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6768.428042] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6771.172271] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4854s! [ 6771.185319] Showing busy workqueues and worker pools: [ 6771.191867] workqueue events: flags=0x0 [ 6771.199030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6771.204315] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6771.217983] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6771.234432] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6771.249252] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.266163] pending: kfree_rcu_monitor [ 6771.267713] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6771.274129] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6771.280237] workqueue events_long: flags=0x0 [ 6771.282657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.286431] pending: lu_objects_flush [obdclass] [ 6771.299246] workqueue events_unbound: flags=0x2 [ 6771.314795] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6771.328211] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6771.343141] workqueue events_power_efficient: flags=0x80 [ 6771.345119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.359070] pending: check_lifetime [ 6771.362463] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.365612] pending: neigh_periodic_work [ 6771.367940] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.372603] pending: neigh_periodic_work [ 6771.379252] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.382990] pending: do_cache_clean [ 6771.384436] workqueue mm_percpu_wq: flags=0x8 [ 6771.389792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.393242] pending: vmstat_update [ 6771.395374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.401522] pending: vmstat_update [ 6771.403449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6771.409703] pending: vmstat_update [ 6771.411115] workqueue writeback: flags=0x4a [ 6771.413013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6771.418620] pending: wb_workfn, wb_workfn [ 6771.421676] workqueue xprtiod: flags=0xa [ 6771.423021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6771.425750] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6771.432936] workqueue ipv6_addrconf: flags=0x40008 [ 6771.436222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6771.439680] pending: addrconf_verify_work [ 6771.445247] workqueue knbd0-recv: flags=0x1a [ 6771.447952] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6771.453859] in-flight: 651:recv_work [ 6771.455547] workqueue ptlrpc_pinger: flags=0x2 [ 6771.461733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6771.464622] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6771.467017] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4854s workers=4 idle: 37 31846 28897 [ 6771.470137] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6761s workers=2 idle: 653 [ 6780.057242] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6780.061292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6780.110730] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6780.118809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6780.124930] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6780.129442] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6780.133290] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6780.150150] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6780.154693] RAX: 0000000000002387 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6780.160755] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6780.165615] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6780.171153] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6780.175318] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6780.177468] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6780.181196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6780.186846] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6780.199113] Call Trace: [ 6780.201090] [ 6780.202135] ? show_regs.cold.9+0x22/0x2f [ 6780.204011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6780.206307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6780.208059] ? watchdog+0x40/0x40 [ 6780.210099] ? hrtimer_interrupt+0x128/0x2c0 [ 6780.212597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6780.220107] ? apic_timer_interrupt+0xf/0x20 [ 6780.223258] [ 6780.225484] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6780.229495] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6780.233369] do_raw_spin_lock+0x177/0x190 [ 6780.236307] _raw_spin_lock+0x21/0x30 [ 6780.239288] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6780.246364] process_one_work+0x2c8/0x700 [ 6780.249315] worker_thread+0x296/0x6e0 [ 6780.252175] ? rescuer_thread+0x570/0x570 [ 6780.255052] kthread+0x1d1/0x200 [ 6780.257136] ? set_kthread_struct+0x70/0x70 [ 6780.259306] ret_from_fork+0x1f/0x30 [ 6792.058958] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6792.059187] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6792.064568] Modules linked in: lustre(O) [ 6792.069735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6792.069892] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6792.069899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6792.069902] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6792.069929] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6792.069933] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6792.069938] RAX: 0000000000005ba1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6792.069940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6792.069942] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6792.069944] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6792.069946] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6792.069949] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6792.069951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6792.069954] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6792.069962] Call Trace: [ 6792.069977] [ 6792.069983] ? show_regs.cold.9+0x22/0x2f [ 6792.070003] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6792.070010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6792.070014] ? watchdog+0x40/0x40 [ 6792.070019] ? hrtimer_interrupt+0x128/0x2c0 [ 6792.070023] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6792.070042] ? apic_timer_interrupt+0xf/0x20 [ 6792.070046] [ 6792.070049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6792.070056] do_raw_spin_lock+0x177/0x190 [ 6792.070061] _raw_spin_lock+0x21/0x30 [ 6792.070067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6792.070351] ? do_raw_spin_unlock+0x75/0x190 [ 6792.070358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6792.070556] ? do_wait_intr+0xf0/0xf0 [ 6792.070564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6792.077285] osp(O) [ 6792.098759] mdc_close+0x53b/0xd70 [mdc] [ 6792.098815] lmv_close+0x18d/0x490 [lmv] [ 6792.098849] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6792.099222] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6792.099306] ll_md_close+0x274/0x800 [lustre] [ 6792.099391] ll_file_release+0x128/0xc50 [lustre] [ 6792.099481] __fput+0x10d/0x440 [ 6792.099491] ____fput+0x12/0x20 [ 6792.099495] task_work_run+0xc6/0x110 [ 6792.099504] exit_to_usermode_loop+0x1dd/0x1f0 [ 6792.099511] do_syscall_64+0x3d6/0x440 [ 6792.099515] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6792.099522] RIP: 0033:0x7fa749552cd8 [ 6792.099569] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6792.099573] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6792.099578] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6792.099581] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6792.099583] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6792.099586] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6792.099588] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6792.266271] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6792.284532] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6792.290781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6792.294397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6792.300264] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6792.314404] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6792.321503] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6792.325753] RDX: 0000000000001f77 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6792.331855] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6792.336701] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6792.338409] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6792.341804] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6792.345831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6792.348537] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6792.354317] Call Trace: [ 6792.355454] [ 6792.356436] ? show_regs.cold.9+0x22/0x2f [ 6792.358267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6792.361463] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6792.363514] ? watchdog+0x40/0x40 [ 6792.365152] ? hrtimer_interrupt+0x128/0x2c0 [ 6792.366893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6792.369274] ? apic_timer_interrupt+0xf/0x20 [ 6792.371159] [ 6792.372091] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6792.375283] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6792.377881] do_raw_spin_lock+0x177/0x190 [ 6792.379501] _raw_spin_lock+0x21/0x30 [ 6792.381827] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6792.384221] obd_mod_free+0x46/0x140 [ptlrpc] [ 6792.386435] mdc_commit_open+0x91/0xb0 [mdc] [ 6792.388423] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6792.391076] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6792.393333] after_reply+0xe2c/0x1780 [ptlrpc] [ 6792.395206] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6792.397346] ? __next_timer_interrupt+0x160/0x160 [ 6792.399362] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6792.401473] ? do_wait_intr+0xf0/0xf0 [ 6792.403309] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6792.405710] kthread+0x1d1/0x200 [ 6792.407157] ? set_kthread_struct+0x70/0x70 [ 6792.408860] ret_from_fork+0x1f/0x30 [ 6796.056943] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6796.059921] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6796.083815] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6796.087416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6796.091041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6796.092651] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6796.102675] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6796.106542] RAX: 0000000000007085 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6796.110150] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6796.113295] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6796.116440] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6796.119412] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6796.123491] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6796.127467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6796.129040] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6796.132379] Call Trace: [ 6796.133597] [ 6796.135034] ? show_regs.cold.9+0x22/0x2f [ 6796.139425] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6796.144393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6796.149894] ? watchdog+0x40/0x40 [ 6796.153154] ? hrtimer_interrupt+0x128/0x2c0 [ 6796.156376] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6796.159777] ? apic_timer_interrupt+0xf/0x20 [ 6796.168493] [ 6796.171124] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6796.179891] do_raw_spin_lock+0x177/0x190 [ 6796.184788] _raw_spin_lock+0x21/0x30 [ 6796.191238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6796.195350] ? do_raw_spin_unlock+0x75/0x190 [ 6796.197290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6796.214100] ? do_wait_intr+0xf0/0xf0 [ 6796.221099] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6796.225172] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6796.227959] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6796.237862] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6796.245796] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6796.248405] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6796.251933] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6796.255411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6796.265144] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6796.269158] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6796.271588] lmv_intent_open+0x338/0x1150 [lmv] [ 6796.274035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6796.277394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6796.288448] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6796.295126] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6796.306556] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6796.312928] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6796.325968] ? do_raw_spin_unlock+0x75/0x190 [ 6796.328972] ? _raw_spin_unlock+0x12/0x30 [ 6796.334342] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6796.337378] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6796.340615] ll_atomic_open+0x36a/0xd70 [lustre] [ 6796.343457] lookup_open+0x356/0xb40 [ 6796.345139] do_last+0x313/0xfc0 [ 6796.347436] ? nd_jump_root+0xe5/0x160 [ 6796.386447] ? path_init+0x437/0x520 [ 6796.388659] path_openat+0xf7/0x500 [ 6796.391642] ? do_raw_spin_unlock+0x75/0x190 [ 6796.394016] do_filp_open+0x99/0x140 [ 6796.412456] ? getname_flags+0x6e/0x330 [ 6796.414458] ? __check_object_size+0xff/0x256 [ 6796.418364] ? do_raw_spin_unlock+0x75/0x190 [ 6796.421532] ? _raw_spin_unlock+0x12/0x30 [ 6796.423582] do_sys_openat2+0x2b4/0x410 [ 6796.426154] do_sys_open+0x73/0xa0 [ 6796.428007] __x64_sys_openat+0x24/0x30 [ 6796.429752] do_syscall_64+0xc1/0x440 [ 6796.431308] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6796.433411] RIP: 0033:0x7fa749552332 [ 6796.434992] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6796.442853] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6796.446087] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6796.448989] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6796.452009] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6796.455008] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6796.458167] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6801.889068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4884s! [ 6801.893258] Showing busy workqueues and worker pools: [ 6801.896340] workqueue events: flags=0x0 [ 6801.898254] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6801.901331] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6801.907133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6801.910353] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6801.924149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6801.928158] pending: kfree_rcu_monitor [ 6801.929554] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6801.933987] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6801.938871] workqueue events_long: flags=0x0 [ 6801.941265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6801.945500] pending: lu_objects_flush [obdclass] [ 6801.950124] workqueue events_unbound: flags=0x2 [ 6801.956466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6801.964380] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6801.973190] workqueue events_power_efficient: flags=0x80 [ 6801.978188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6801.982219] pending: check_lifetime [ 6801.984316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6801.988196] pending: neigh_periodic_work [ 6801.991150] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6801.995144] pending: neigh_periodic_work [ 6801.997316] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6802.003361] pending: do_cache_clean [ 6802.007373] workqueue mm_percpu_wq: flags=0x8 [ 6802.013597] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6802.017954] pending: vmstat_update [ 6802.019736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6802.028075] pending: vmstat_update [ 6802.031334] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6802.039223] pending: vmstat_update [ 6802.043664] workqueue writeback: flags=0x4a [ 6802.047109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6802.054533] pending: wb_workfn, wb_workfn [ 6802.056500] workqueue xprtiod: flags=0xa [ 6802.058553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6802.067593] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6802.075116] workqueue ipv6_addrconf: flags=0x40008 [ 6802.082833] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6802.085800] pending: addrconf_verify_work [ 6802.094444] workqueue knbd0-recv: flags=0x1a [ 6802.100394] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6802.105401] in-flight: 651:recv_work [ 6802.110018] workqueue ptlrpc_pinger: flags=0x2 [ 6802.115816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6802.118851] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6802.126313] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4884s workers=4 idle: 37 31846 28897 [ 6802.130188] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6792s workers=2 idle: 653 [ 6808.065758] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6808.075393] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6808.104817] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6808.110619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6808.114740] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6808.117995] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6808.121311] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6808.132382] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6808.135214] RAX: 000000000000460d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6808.141224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6808.143641] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6808.149070] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6808.154614] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6808.158061] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6808.163085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6808.168230] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6808.173871] Call Trace: [ 6808.175940] [ 6808.178485] ? show_regs.cold.9+0x22/0x2f [ 6808.179598] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6808.181609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6808.186289] ? watchdog+0x40/0x40 [ 6808.190187] ? hrtimer_interrupt+0x128/0x2c0 [ 6808.197775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6808.199243] ? apic_timer_interrupt+0xf/0x20 [ 6808.205122] [ 6808.205898] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6808.210674] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6808.214240] do_raw_spin_lock+0x177/0x190 [ 6808.219541] _raw_spin_lock+0x21/0x30 [ 6808.222143] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6808.224899] process_one_work+0x2c8/0x700 [ 6808.227236] worker_thread+0x296/0x6e0 [ 6808.232194] ? rescuer_thread+0x570/0x570 [ 6808.233303] kthread+0x1d1/0x200 [ 6808.234935] ? set_kthread_struct+0x70/0x70 [ 6808.236948] ret_from_fork+0x1f/0x30 [ 6820.063601] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6820.063539] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6820.071780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6820.071857] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6820.071864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6820.071866] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6820.071879] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6820.071883] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6820.071888] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6820.071890] RDX: 00000000000061b7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6820.071892] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6820.071895] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6820.071897] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6820.071900] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6820.071903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6820.071905] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6820.071914] Call Trace: [ 6820.071919] [ 6820.071925] ? show_regs.cold.9+0x22/0x2f [ 6820.071933] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6820.071939] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6820.071943] ? watchdog+0x40/0x40 [ 6820.071947] ? hrtimer_interrupt+0x128/0x2c0 [ 6820.071951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6820.071959] ? apic_timer_interrupt+0xf/0x20 [ 6820.071964] [ 6820.071967] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6820.082011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6820.082087] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6820.082096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6820.082099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6820.082113] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6820.082117] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6820.082122] RAX: 000000000000621f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6820.082125] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6820.082127] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6820.082130] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6820.082132] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6820.082135] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6820.082137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6820.082139] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6820.082149] Call Trace: [ 6820.082156] [ 6820.082163] ? show_regs.cold.9+0x22/0x2f [ 6820.082174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6820.082181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6820.082185] ? watchdog+0x40/0x40 [ 6820.082189] ? hrtimer_interrupt+0x128/0x2c0 [ 6820.082192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6820.082201] ? apic_timer_interrupt+0xf/0x20 [ 6820.082205] [ 6820.082208] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6820.082214] do_raw_spin_lock+0x177/0x190 [ 6820.114413] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6820.114434] do_raw_spin_lock+0x177/0x190 [ 6820.114441] _raw_spin_lock+0x21/0x30 [ 6820.114450] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6820.115079] obd_mod_free+0x46/0x140 [ptlrpc] [ 6820.115676] mdc_commit_open+0x91/0xb0 [mdc] [ 6820.115713] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6820.115912] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6820.127438] _raw_spin_lock+0x21/0x30 [ 6820.127460] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6820.137494] after_reply+0xe2c/0x1780 [ptlrpc] [ 6820.138137] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6820.143627] ? do_raw_spin_unlock+0x75/0x190 [ 6820.143650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6820.144843] ? do_wait_intr+0xf0/0xf0 [ 6820.144854] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6820.145759] mdc_close+0x53b/0xd70 [mdc] [ 6820.145884] lmv_close+0x18d/0x490 [lmv] [ 6820.145928] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6820.146044] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6820.146136] ll_md_close+0x274/0x800 [lustre] [ 6820.146297] ll_file_release+0x128/0xc50 [lustre] [ 6820.146588] __fput+0x10d/0x440 [ 6820.146918] ____fput+0x12/0x20 [ 6820.147198] task_work_run+0xc6/0x110 [ 6820.147208] exit_to_usermode_loop+0x1dd/0x1f0 [ 6820.147309] do_syscall_64+0x3d6/0x440 [ 6820.147315] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6820.165059] ? __next_timer_interrupt+0x160/0x160 [ 6820.165081] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6820.165432] ? do_wait_intr+0xf0/0xf0 [ 6820.165443] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6820.165653] kthread+0x1d1/0x200 [ 6820.165659] ? set_kthread_struct+0x70/0x70 [ 6820.165664] ret_from_fork+0x1f/0x30 [ 6820.557605] RIP: 0033:0x7fa749552cd8 [ 6820.561908] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6820.579052] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6820.586373] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6820.592927] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6820.599539] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6820.607719] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6820.612102] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6828.055656] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6828.061640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6828.095761] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6828.105785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6828.109436] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6828.111539] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6828.120711] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6828.127302] RAX: 0000000000007e1d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6828.132061] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6828.138893] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6828.146929] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6828.152697] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6828.159252] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6828.163313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6828.168939] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6828.175992] Call Trace: [ 6828.179069] [ 6828.179883] ? show_regs.cold.9+0x22/0x2f [ 6828.181386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6828.183124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6828.187594] ? watchdog+0x40/0x40 [ 6828.194983] ? hrtimer_interrupt+0x128/0x2c0 [ 6828.199145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6828.201210] ? apic_timer_interrupt+0xf/0x20 [ 6828.202562] [ 6828.203193] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6828.206221] do_raw_spin_lock+0x177/0x190 [ 6828.208024] _raw_spin_lock+0x21/0x30 [ 6828.209446] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6828.212772] ? do_raw_spin_unlock+0x75/0x190 [ 6828.214382] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6828.218295] ? do_wait_intr+0xf0/0xf0 [ 6828.219817] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6828.222001] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6828.225376] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6828.229535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6828.235203] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6828.239456] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6828.241658] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6828.244956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6828.248463] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6828.251787] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6828.254202] lmv_intent_open+0x338/0x1150 [lmv] [ 6828.256061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6828.265005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6828.269539] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6828.271238] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6828.281308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6828.284675] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6828.291100] ? do_raw_spin_unlock+0x75/0x190 [ 6828.295135] ? _raw_spin_unlock+0x12/0x30 [ 6828.299034] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6828.301265] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6828.304898] ll_atomic_open+0x36a/0xd70 [lustre] [ 6828.308579] lookup_open+0x356/0xb40 [ 6828.313813] do_last+0x313/0xfc0 [ 6828.320017] ? nd_jump_root+0xe5/0x160 [ 6828.323617] ? path_init+0x437/0x520 [ 6828.324913] path_openat+0xf7/0x500 [ 6828.331532] ? do_raw_spin_unlock+0x75/0x190 [ 6828.334509] do_filp_open+0x99/0x140 [ 6828.338775] ? getname_flags+0x6e/0x330 [ 6828.340069] ? __check_object_size+0xff/0x256 [ 6828.344981] ? do_raw_spin_unlock+0x75/0x190 [ 6828.347449] ? _raw_spin_unlock+0x12/0x30 [ 6828.356661] do_sys_openat2+0x2b4/0x410 [ 6828.359188] do_sys_open+0x73/0xa0 [ 6828.362061] __x64_sys_openat+0x24/0x30 [ 6828.364581] do_syscall_64+0xc1/0x440 [ 6828.366370] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6828.368037] RIP: 0033:0x7fa749552332 [ 6828.374046] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6828.394140] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6828.401905] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6828.409823] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6828.416508] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6828.422532] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6828.426967] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6832.608063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4915s! [ 6832.611545] Showing busy workqueues and worker pools: [ 6832.613857] workqueue events: flags=0x0 [ 6832.615779] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6832.618793] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6832.623670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6832.626353] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6832.630197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.633086] pending: kfree_rcu_monitor [ 6832.634902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6832.638943] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6832.642924] workqueue events_long: flags=0x0 [ 6832.645076] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.648565] pending: lu_objects_flush [obdclass] [ 6832.651668] workqueue events_unbound: flags=0x2 [ 6832.655235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6832.658362] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6832.663641] workqueue events_power_efficient: flags=0x80 [ 6832.667617] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.671136] pending: check_lifetime [ 6832.673454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.676478] pending: neigh_periodic_work [ 6832.679183] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.682977] pending: neigh_periodic_work [ 6832.684642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.687471] pending: do_cache_clean [ 6832.689240] workqueue mm_percpu_wq: flags=0x8 [ 6832.690683] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.693471] pending: vmstat_update [ 6832.695191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.697902] pending: vmstat_update [ 6832.699369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6832.701993] pending: vmstat_update [ 6832.703510] workqueue writeback: flags=0x4a [ 6832.704994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6832.707499] pending: wb_workfn, wb_workfn [ 6832.709267] workqueue xprtiod: flags=0xa [ 6832.710940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6832.713302] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6832.716386] workqueue ipv6_addrconf: flags=0x40008 [ 6832.718659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6832.721352] pending: addrconf_verify_work [ 6832.722953] workqueue knbd0-recv: flags=0x1a [ 6832.724711] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6832.727357] in-flight: 651:recv_work [ 6832.728968] workqueue ptlrpc_pinger: flags=0x2 [ 6832.730812] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6832.733041] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6832.736924] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4915s workers=4 idle: 37 31846 28897 [ 6832.740968] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6822s workers=2 idle: 653 [ 6833.798025] rcu: INFO: rcu_sched self-detected stall on CPU [ 6833.800914] rcu: 2-...!: (2711944 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 6833.805243] (t=4920109 jiffies g=258701 q=2941) [ 6833.807217] rcu: rcu_sched kthread starved for 4916904 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6833.812277] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6833.816229] rcu: RCU grace-period kthread stack dump: [ 6833.818369] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6833.825787] Call Trace: [ 6833.828683] __schedule+0x351/0xcb0 [ 6833.830458] schedule+0xc0/0x180 [ 6833.833096] schedule_timeout+0xb4/0x190 [ 6833.835347] ? do_raw_spin_unlock+0x75/0x190 [ 6833.837444] ? __next_timer_interrupt+0x160/0x160 [ 6833.839939] ? prepare_to_swait_event+0xf3/0x140 [ 6833.842133] rcu_gp_kthread+0x8dd/0x15b0 [ 6833.844243] ? force_qs_rnp+0x2c0/0x2c0 [ 6833.846236] kthread+0x1d1/0x200 [ 6833.847755] ? set_kthread_struct+0x70/0x70 [ 6833.849765] ret_from_fork+0x1f/0x30 [ 6833.852203] rcu: Stack dump where RCU GP kthread last ran: [ 6833.855039] Sending NMI from CPU 2 to CPUs 1: [ 6833.857768] NMI backtrace for cpu 1 [ 6833.857774] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6833.858426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6833.858429] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6833.858432] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6833.858434] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6833.858436] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 [ 6833.858439] RAX: 0000000000007d85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6833.858440] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6833.858442] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6833.858444] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6833.858445] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6833.858447] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6833.858448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6833.858450] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6833.858451] Call Trace: [ 6833.858452] [ 6833.858453] ? show_regs.cold.9+0x22/0x2f [ 6833.858454] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6833.858455] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6833.858456] ? nmi_handle+0xbd/0x210 [ 6833.858457] ? default_do_nmi+0x4b/0x1b0 [ 6833.858458] ? do_nmi+0x280/0x380 [ 6833.858459] ? end_repeat_nmi+0x16/0x1f [ 6833.858460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6833.858461] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6833.858463] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6833.858464] [ 6833.858465] do_raw_spin_lock+0x177/0x190 [ 6833.858466] _raw_spin_lock+0x21/0x30 [ 6833.858467] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6833.858468] process_one_work+0x2c8/0x700 [ 6833.858469] worker_thread+0x296/0x6e0 [ 6833.858470] ? rescuer_thread+0x570/0x570 [ 6833.858471] kthread+0x1d1/0x200 [ 6833.858472] ? set_kthread_struct+0x70/0x70 [ 6833.858473] ret_from_fork+0x1f/0x30 [ 6833.858540] NMI backtrace for cpu 2 [ 6833.969645] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6833.975416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6833.980056] Call Trace: [ 6833.981206] [ 6833.982168] ? dump_stack+0xbb/0x10e [ 6833.984179] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6833.986768] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6833.989065] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6833.997197] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6834.002733] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6834.006503] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6834.008110] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6834.010929] ? update_process_times+0x6d/0xc0 [ 6834.012929] ? tick_sched_handle.isra.18+0x27/0x90 [ 6834.015544] ? tick_sched_timer+0x53/0xb0 [ 6834.017778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6834.019485] ? tick_sched_handle.isra.18+0x90/0x90 [ 6834.021098] ? hrtimer_interrupt+0x128/0x2c0 [ 6834.023288] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6834.025808] ? apic_timer_interrupt+0xf/0x20 [ 6834.027987] [ 6834.028878] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6834.031725] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6834.034630] ? do_raw_spin_lock+0x177/0x190 [ 6834.036652] ? _raw_spin_lock+0x21/0x30 [ 6834.038432] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6834.041273] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6834.043690] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6834.045311] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6834.056613] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6834.068940] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6834.070864] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6834.077825] ? __next_timer_interrupt+0x160/0x160 [ 6834.081434] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6834.084426] ? do_wait_intr+0xf0/0xf0 [ 6834.086629] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6834.089241] ? kthread+0x1d1/0x200 [ 6834.096936] ? set_kthread_struct+0x70/0x70 [ 6834.098069] ? ret_from_fork+0x1f/0x30 [ 6848.060607] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6848.074782] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6848.116531] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6848.124540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6848.128105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6848.130678] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6848.138328] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6848.141960] RAX: 0000000000007d85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6848.146784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6848.150554] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6848.155073] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6848.158255] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6848.162800] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6848.165192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6848.168581] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6848.171973] Call Trace: [ 6848.173612] [ 6848.174474] ? show_regs.cold.9+0x22/0x2f [ 6848.177562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6848.181469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6848.184965] ? watchdog+0x40/0x40 [ 6848.189107] ? hrtimer_interrupt+0x128/0x2c0 [ 6848.192222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6848.199440] ? apic_timer_interrupt+0xf/0x20 [ 6848.201635] [ 6848.204190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6848.207885] do_raw_spin_lock+0x177/0x190 [ 6848.220033] _raw_spin_lock+0x21/0x30 [ 6848.221789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6848.224217] ? do_raw_spin_unlock+0x75/0x190 [ 6848.226175] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6848.228810] ? do_wait_intr+0xf0/0xf0 [ 6848.230240] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6848.237261] mdc_close+0x53b/0xd70 [mdc] [ 6848.238639] lmv_close+0x18d/0x490 [lmv] [ 6848.246416] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6848.256275] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6848.261481] ll_md_close+0x274/0x800 [lustre] [ 6848.271750] ll_file_release+0x128/0xc50 [lustre] [ 6848.273637] __fput+0x10d/0x440 [ 6848.278058] ____fput+0x12/0x20 [ 6848.279230] task_work_run+0xc6/0x110 [ 6848.282789] exit_to_usermode_loop+0x1dd/0x1f0 [ 6848.286651] do_syscall_64+0x3d6/0x440 [ 6848.290954] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6848.292733] RIP: 0033:0x7fa749552cd8 [ 6848.295378] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6848.314145] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6848.318654] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6848.324033] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6848.331449] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6848.344447] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6848.352119] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6860.054134] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6860.058166] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 6860.058316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.058384] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.058390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.058393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.058420] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6860.058424] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6860.058428] RAX: 0000000000005e8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6860.058431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6860.058433] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6860.058435] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6860.058437] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6860.058441] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6860.058444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.058446] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6860.058454] Call Trace: [ 6860.058461] [ 6860.058467] ? show_regs.cold.9+0x22/0x2f [ 6860.058477] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.058484] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.058489] ? watchdog+0x40/0x40 [ 6860.058493] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.058496] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.058506] ? apic_timer_interrupt+0xf/0x20 [ 6860.058512] [ 6860.058514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.058534] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 6860.058537] do_raw_spin_lock+0x177/0x190 [ 6860.058544] Modules linked in: lustre(O) [ 6860.058546] _raw_spin_lock+0x21/0x30 [ 6860.058552] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror [ 6860.058557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6860.058865] ? do_raw_spin_unlock+0x75/0x190 [ 6860.059222] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6860.059438] ? do_wait_intr+0xf0/0xf0 [ 6860.059448] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6860.059653] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6860.059888] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6860.060337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.060583] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6860.060626] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6860.060657] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6860.060683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.060768] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6860.061222] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6860.061253] lmv_intent_open+0x338/0x1150 [lmv] [ 6860.061285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.061708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.061788] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6860.061823] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6860.061907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.062134] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6860.062220] ? do_raw_spin_unlock+0x75/0x190 [ 6860.062227] ? _raw_spin_unlock+0x12/0x30 [ 6860.062233] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6860.062266] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6860.062293] ll_atomic_open+0x36a/0xd70 [lustre] [ 6860.062380] lookup_open+0x356/0xb40 [ 6860.062388] do_last+0x313/0xfc0 [ 6860.062393] ? nd_jump_root+0xe5/0x160 [ 6860.063466] ? path_init+0x437/0x520 [ 6860.063471] path_openat+0xf7/0x500 [ 6860.063476] ? do_raw_spin_unlock+0x75/0x190 [ 6860.063483] do_filp_open+0x99/0x140 [ 6860.063489] ? getname_flags+0x6e/0x330 [ 6860.063493] ? __check_object_size+0xff/0x256 [ 6860.063501] ? do_raw_spin_unlock+0x75/0x190 [ 6860.063505] ? _raw_spin_unlock+0x12/0x30 [ 6860.063511] do_sys_openat2+0x2b4/0x410 [ 6860.064996] do_sys_open+0x73/0xa0 [ 6860.065022] __x64_sys_openat+0x24/0x30 [ 6860.065027] do_syscall_64+0xc1/0x440 [ 6860.065033] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6860.065041] RIP: 0033:0x7fa749552332 [ 6860.065050] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6860.065054] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6860.065060] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6860.065063] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6860.065066] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6860.065068] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6860.065070] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6860.065356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.065489] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.065494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.065499] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6860.066242] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.066251] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6860.066256] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6860.066260] RAX: 000000000000735f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6860.066263] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6860.066265] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6860.066267] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6860.066269] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6860.066273] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6860.066276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.066278] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6860.066287] Call Trace: [ 6860.066291] [ 6860.066296] ? show_regs.cold.9+0x22/0x2f [ 6860.066305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.066313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.066318] ? watchdog+0x40/0x40 [ 6860.066322] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.066326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.066333] ? apic_timer_interrupt+0xf/0x20 [ 6860.066338] [ 6860.066341] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.066347] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6860.066353] do_raw_spin_lock+0x177/0x190 [ 6860.066357] _raw_spin_lock+0x21/0x30 [ 6860.066362] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6860.066831] process_one_work+0x2c8/0x700 [ 6860.066839] worker_thread+0x296/0x6e0 [ 6860.066843] ? rescuer_thread+0x570/0x570 [ 6860.066847] kthread+0x1d1/0x200 [ 6860.066852] ? set_kthread_struct+0x70/0x70 [ 6860.066856] ret_from_fork+0x1f/0x30 [ 6860.820208] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.825399] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.835255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.843743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6860.848464] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6860.859644] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6860.863047] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6860.866366] RDX: 0000000000007c0b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6860.869676] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6860.877925] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6860.882326] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6860.888718] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6860.898133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.903824] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6860.908660] Call Trace: [ 6860.909528] [ 6860.912493] ? show_regs.cold.9+0x22/0x2f [ 6860.916250] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.920310] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.925939] ? watchdog+0x40/0x40 [ 6860.928280] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.933301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.938178] ? apic_timer_interrupt+0xf/0x20 [ 6860.939721] [ 6860.940420] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6860.949934] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6860.956039] do_raw_spin_lock+0x177/0x190 [ 6860.959276] _raw_spin_lock+0x21/0x30 [ 6860.964736] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6860.967650] obd_mod_free+0x46/0x140 [ptlrpc] [ 6860.972081] mdc_commit_open+0x91/0xb0 [mdc] [ 6860.976478] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6860.982019] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6860.985985] after_reply+0xe2c/0x1780 [ptlrpc] [ 6860.990331] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6860.993324] ? __next_timer_interrupt+0x160/0x160 [ 6860.996387] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6861.001575] ? do_wait_intr+0xf0/0xf0 [ 6861.003399] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6861.008808] kthread+0x1d1/0x200 [ 6861.011303] ? set_kthread_struct+0x70/0x70 [ 6861.014830] ret_from_fork+0x1f/0x30 [ 6862.817378] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4945s! [ 6862.823109] Showing busy workqueues and worker pools: [ 6862.825876] workqueue events: flags=0x0 [ 6862.827837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6862.831433] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6862.836549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6862.839645] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6862.848196] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6862.855145] pending: kfree_rcu_monitor [ 6862.863136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6862.867293] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6862.875898] workqueue events_long: flags=0x0 [ 6862.883018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6862.891845] pending: lu_objects_flush [obdclass] [ 6862.901631] workqueue events_unbound: flags=0x2 [ 6862.907031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6862.916392] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6862.933784] workqueue events_power_efficient: flags=0x80 [ 6862.939667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6862.947097] pending: check_lifetime [ 6862.953377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6862.969659] pending: neigh_periodic_work [ 6862.976387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6862.992168] pending: neigh_periodic_work [ 6863.004629] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6863.012331] pending: do_cache_clean [ 6863.015884] workqueue mm_percpu_wq: flags=0x8 [ 6863.023212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6863.031296] pending: vmstat_update [ 6863.034780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6863.048431] pending: vmstat_update [ 6863.055278] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6863.070738] pending: vmstat_update [ 6863.078460] workqueue writeback: flags=0x4a [ 6863.083157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6863.091648] pending: wb_workfn, wb_workfn [ 6863.095818] workqueue xprtiod: flags=0xa [ 6863.103897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6863.114003] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6863.126392] workqueue ipv6_addrconf: flags=0x40008 [ 6863.134792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6863.139688] pending: addrconf_verify_work [ 6863.145855] workqueue knbd0-recv: flags=0x1a [ 6863.151234] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6863.157918] in-flight: 651:recv_work [ 6863.161454] workqueue ptlrpc_pinger: flags=0x2 [ 6863.167676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6863.175025] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6863.179346] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4946s workers=4 idle: 37 31846 28897 [ 6863.193169] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6853s workers=2 idle: 653 [ 6876.059025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6876.061715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6876.081918] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6876.086766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6876.090200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6876.095777] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6876.104222] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6876.114824] RAX: 0000000000001195 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6876.118498] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6876.120857] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6876.123697] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6876.126597] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6876.129698] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6876.133207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6876.135720] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6876.138868] Call Trace: [ 6876.139879] [ 6876.140741] ? show_regs.cold.9+0x22/0x2f [ 6876.142406] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6876.144346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6876.146195] ? watchdog+0x40/0x40 [ 6876.147315] ? hrtimer_interrupt+0x128/0x2c0 [ 6876.148508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6876.150587] ? apic_timer_interrupt+0xf/0x20 [ 6876.152628] [ 6876.153590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6876.155560] do_raw_spin_lock+0x177/0x190 [ 6876.157237] _raw_spin_lock+0x21/0x30 [ 6876.158961] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6876.161578] ? do_raw_spin_unlock+0x75/0x190 [ 6876.163260] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6876.165687] ? do_wait_intr+0xf0/0xf0 [ 6876.167057] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6876.169323] mdc_close+0x53b/0xd70 [mdc] [ 6876.171084] lmv_close+0x18d/0x490 [lmv] [ 6876.172849] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6876.175132] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6876.177118] ll_md_close+0x274/0x800 [lustre] [ 6876.179416] ll_file_release+0x128/0xc50 [lustre] [ 6876.181914] __fput+0x10d/0x440 [ 6876.183293] ____fput+0x12/0x20 [ 6876.184630] task_work_run+0xc6/0x110 [ 6876.186069] exit_to_usermode_loop+0x1dd/0x1f0 [ 6876.188023] do_syscall_64+0x3d6/0x440 [ 6876.189704] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6876.191900] RIP: 0033:0x7fa749552cd8 [ 6876.195112] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6876.203925] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6876.206918] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6876.210317] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6876.213660] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6876.217183] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6876.219412] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6888.054271] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 6888.058033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6888.060502] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6888.060514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.060585] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.060590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.060595] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6888.060889] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.060898] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6888.060902] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6888.060907] RAX: 0000000000007885 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6888.060909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6888.060912] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6888.060914] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6888.060916] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6888.060920] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6888.060922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.060925] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6888.060934] Call Trace: [ 6888.060938] [ 6888.060943] ? show_regs.cold.9+0x22/0x2f [ 6888.060951] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.060956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.060960] ? watchdog+0x40/0x40 [ 6888.060964] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.060967] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.060975] ? apic_timer_interrupt+0xf/0x20 [ 6888.060979] [ 6888.060982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.060986] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6888.060991] do_raw_spin_lock+0x177/0x190 [ 6888.060996] _raw_spin_lock+0x21/0x30 [ 6888.061000] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6888.061219] process_one_work+0x2c8/0x700 [ 6888.061225] worker_thread+0x296/0x6e0 [ 6888.061229] ? rescuer_thread+0x570/0x570 [ 6888.061233] kthread+0x1d1/0x200 [ 6888.061237] ? set_kthread_struct+0x70/0x70 [ 6888.061241] ret_from_fork+0x1f/0x30 [ 6888.069904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.069986] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.069995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.069998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.070013] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6888.070017] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6888.070022] RAX: 0000000000004e83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6888.070025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6888.070027] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6888.070029] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6888.070032] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6888.070035] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6888.070038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.070040] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6888.070050] Call Trace: [ 6888.070056] [ 6888.070062] ? show_regs.cold.9+0x22/0x2f [ 6888.070073] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.070081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.070085] ? watchdog+0x40/0x40 [ 6888.070090] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.070094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.070104] ? apic_timer_interrupt+0xf/0x20 [ 6888.070109] [ 6888.070112] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.070119] do_raw_spin_lock+0x177/0x190 [ 6888.070124] _raw_spin_lock+0x21/0x30 [ 6888.070130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6888.074967] ? do_raw_spin_unlock+0x75/0x190 [ 6888.074979] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6888.076853] ? do_wait_intr+0xf0/0xf0 [ 6888.076868] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6888.085560] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.085644] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.085652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.085655] RIP: 0010:load_balance+0x725/0x1090 [ 6888.085668] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [ 6888.085673] RSP: 0018:ffffb893c0d58de0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6888.085678] RAX: 0000000000000006 RBX: ffff8d73820d80b0 RCX: ffff8d7382133440 [ 6888.085680] RDX: 0000000000000008 RSI: ffff8d73820d8000 RDI: 0000000000000202 [ 6888.085682] RBP: ffffb893c0d58e90 R08: ffff8d7359110000 R09: 0000000000000000 [ 6888.085685] R10: ffff8d73820d8098 R11: 000000000000af67 R12: ffff8d73820d8000 [ 6888.085687] R13: ffff8d73821b3e08 R14: 0000000000000001 R15: ffff8d73821b3380 [ 6888.085691] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6888.085693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.085696] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6888.085706] Call Trace: [ 6888.085711] [ 6888.085716] ? show_regs.cold.9+0x22/0x2f [ 6888.085725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.085730] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.085734] ? watchdog+0x40/0x40 [ 6888.085738] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.085742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.085751] ? apic_timer_interrupt+0xf/0x20 [ 6888.085756] ? apic_timer_interrupt+0xa/0x20 [ 6888.085761] ? load_balance+0x725/0x1090 [ 6888.085766] rebalance_domains+0x348/0x580 [ 6888.085771] ? kvm_clock_get_cycles+0x2c/0x50 [ 6888.085779] run_rebalance_domains+0x68/0xa0 [ 6888.085783] __do_softirq+0x134/0x4e8 [ 6888.085789] irq_exit_rcu+0x1c6/0x1e0 [ 6888.085795] irq_exit+0x12/0x30 [ 6888.085799] smp_apic_timer_interrupt+0xe0/0x270 [ 6888.085805] apic_timer_interrupt+0xf/0x20 [ 6888.085811] [ 6888.085813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6888.085820] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6888.085824] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6888.085828] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6888.085831] RDX: 0000000000000e1e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6888.085833] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6888.085835] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6888.085838] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6888.085843] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6888.085848] do_raw_spin_lock+0x177/0x190 [ 6888.097930] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6888.098738] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6888.099239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.100258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6888.100302] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6888.100343] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6888.100997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.101305] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6888.104904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6888.104949] lmv_intent_open+0x338/0x1150 [lmv] [ 6888.104984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.105124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.105204] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6888.105225] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6888.105323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.105418] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6888.105504] ? do_raw_spin_unlock+0x75/0x190 [ 6888.105512] ? _raw_spin_unlock+0x12/0x30 [ 6888.105519] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6888.105543] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6888.105564] ll_atomic_open+0x36a/0xd70 [lustre] [ 6888.105654] lookup_open+0x356/0xb40 [ 6888.105664] do_last+0x313/0xfc0 [ 6888.105670] ? nd_jump_root+0xe5/0x160 [ 6888.105675] ? path_init+0x437/0x520 [ 6888.105680] path_openat+0xf7/0x500 [ 6888.105685] ? do_raw_spin_unlock+0x75/0x190 [ 6888.105689] do_filp_open+0x99/0x140 [ 6888.105696] ? getname_flags+0x6e/0x330 [ 6888.105701] ? __check_object_size+0xff/0x256 [ 6888.105711] ? do_raw_spin_unlock+0x75/0x190 [ 6888.105715] ? _raw_spin_unlock+0x12/0x30 [ 6888.105721] do_sys_openat2+0x2b4/0x410 [ 6888.105726] do_sys_open+0x73/0xa0 [ 6888.105732] __x64_sys_openat+0x24/0x30 [ 6888.105736] do_syscall_64+0xc1/0x440 [ 6888.105744] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6888.105750] RIP: 0033:0x7fa749552332 [ 6888.105758] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6888.105762] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6888.105767] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6888.105769] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6888.105771] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6888.105773] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6888.105776] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6889.026798] _raw_spin_lock+0x21/0x30 [ 6889.027990] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6889.033049] obd_mod_free+0x46/0x140 [ptlrpc] [ 6889.035962] mdc_commit_open+0x91/0xb0 [mdc] [ 6889.039765] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6889.049691] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6889.058363] after_reply+0xe2c/0x1780 [ptlrpc] [ 6889.065309] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6889.070141] ? __next_timer_interrupt+0x160/0x160 [ 6889.078752] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6889.084664] ? do_wait_intr+0xf0/0xf0 [ 6889.088803] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6889.090826] kthread+0x1d1/0x200 [ 6889.095667] ? set_kthread_struct+0x70/0x70 [ 6889.101448] ret_from_fork+0x1f/0x30 [ 6893.542526] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4976s! [ 6893.563190] Showing busy workqueues and worker pools: [ 6893.573704] workqueue events: flags=0x0 [ 6893.576139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6893.592861] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6893.608021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6893.614186] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6893.622586] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.628285] pending: kfree_rcu_monitor [ 6893.634082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6893.640765] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6893.646769] workqueue events_long: flags=0x0 [ 6893.647990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.651801] pending: lu_objects_flush [obdclass] [ 6893.655763] workqueue events_unbound: flags=0x2 [ 6893.658665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6893.663310] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6893.674369] workqueue events_power_efficient: flags=0x80 [ 6893.681824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.695051] pending: check_lifetime [ 6893.701171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.708754] pending: neigh_periodic_work [ 6893.712759] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.718295] pending: neigh_periodic_work [ 6893.725117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.729731] pending: do_cache_clean [ 6893.732790] workqueue mm_percpu_wq: flags=0x8 [ 6893.735107] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.742461] pending: vmstat_update [ 6893.744973] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.749269] pending: vmstat_update [ 6893.753714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6893.764916] pending: vmstat_update [ 6893.771641] workqueue writeback: flags=0x4a [ 6893.773984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6893.781200] pending: wb_workfn, wb_workfn [ 6893.784565] workqueue xprtiod: flags=0xa [ 6893.786907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6893.796285] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6893.804023] workqueue ipv6_addrconf: flags=0x40008 [ 6893.808589] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6893.813026] pending: addrconf_verify_work [ 6893.817771] workqueue knbd0-recv: flags=0x1a [ 6893.822159] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6893.828562] in-flight: 651:recv_work [ 6893.835320] workqueue ptlrpc_pinger: flags=0x2 [ 6893.841341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6893.849403] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6893.863741] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4976s workers=4 idle: 37 31846 28897 [ 6893.876731] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6883s workers=2 idle: 653 [ 6904.065830] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6904.071529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6904.123561] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6904.152962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6904.164592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.167298] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6904.176662] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6904.190009] RAX: 00000000000003cb RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6904.196078] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6904.204954] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6904.210982] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6904.217716] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6904.220922] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6904.227256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6904.242189] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6904.249477] Call Trace: [ 6904.253164] [ 6904.258577] ? show_regs.cold.9+0x22/0x2f [ 6904.265722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6904.270549] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6904.282964] ? watchdog+0x40/0x40 [ 6904.286361] ? hrtimer_interrupt+0x128/0x2c0 [ 6904.290974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6904.299599] ? apic_timer_interrupt+0xf/0x20 [ 6904.304992] [ 6904.305845] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.314998] do_raw_spin_lock+0x177/0x190 [ 6904.325669] _raw_spin_lock+0x21/0x30 [ 6904.327545] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6904.330940] ? do_raw_spin_unlock+0x75/0x190 [ 6904.333558] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6904.340297] ? do_wait_intr+0xf0/0xf0 [ 6904.341605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6904.353962] mdc_close+0x53b/0xd70 [mdc] [ 6904.356160] lmv_close+0x18d/0x490 [lmv] [ 6904.360264] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6904.362405] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6904.364508] ll_md_close+0x274/0x800 [lustre] [ 6904.367887] ll_file_release+0x128/0xc50 [lustre] [ 6904.370272] __fput+0x10d/0x440 [ 6904.372460] ____fput+0x12/0x20 [ 6904.373923] task_work_run+0xc6/0x110 [ 6904.376224] exit_to_usermode_loop+0x1dd/0x1f0 [ 6904.377791] do_syscall_64+0x3d6/0x440 [ 6904.379626] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6904.381534] RIP: 0033:0x7fa749552cd8 [ 6904.383004] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6904.391078] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6904.394383] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6904.397196] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6904.399452] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6904.401465] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6904.404271] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6916.057115] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6916.059781] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6916.060459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 6916.064865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6916.065016] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6916.065024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6916.065026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6916.065041] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6916.065046] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6916.065051] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6916.065054] RDX: 0000000000007e81 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6916.065056] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6916.065058] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6916.065065] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6916.065069] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6916.065071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6916.065074] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6916.065084] Call Trace: [ 6916.065090] [ 6916.065095] ? show_regs.cold.9+0x22/0x2f [ 6916.065104] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6916.065111] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6916.065115] ? watchdog+0x40/0x40 [ 6916.065124] ? hrtimer_interrupt+0x128/0x2c0 [ 6916.065128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6916.065139] ? apic_timer_interrupt+0xf/0x20 [ 6916.065144] [ 6916.065148] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6916.065153] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6916.065157] do_raw_spin_lock+0x177/0x190 [ 6916.065166] _raw_spin_lock+0x21/0x30 [ 6916.065172] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6916.065660] obd_mod_free+0x46/0x140 [ptlrpc] [ 6916.066040] mdc_commit_open+0x91/0xb0 [mdc] [ 6916.066084] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6916.066328] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6916.066722] after_reply+0xe2c/0x1780 [ptlrpc] [ 6916.067044] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6916.067525] ? __next_timer_interrupt+0x160/0x160 [ 6916.067542] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6916.067951] ? do_wait_intr+0xf0/0xf0 [ 6916.067960] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6916.073807] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6916.073891] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6916.073902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6916.073909] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6916.074384] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6916.074399] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6916.074407] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6916.074412] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6916.074414] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6916.074417] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6916.074420] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6916.074422] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6916.074426] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6916.074436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6916.074439] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6916.074446] Call Trace: [ 6916.074452] [ 6916.074457] ? show_regs.cold.9+0x22/0x2f [ 6916.074468] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6916.074475] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6916.074480] ? watchdog+0x40/0x40 [ 6916.074484] ? hrtimer_interrupt+0x128/0x2c0 [ 6916.074494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6916.074501] ? apic_timer_interrupt+0xf/0x20 [ 6916.074510] [ 6916.074513] ? native_safe_halt+0x2e/0x50 [ 6916.074521] kvm_wait+0xcd/0xe0 [ 6916.074529] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6916.074662] do_raw_spin_lock+0x177/0x190 [ 6916.074667] _raw_spin_lock+0x21/0x30 [ 6916.074672] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6916.075011] process_one_work+0x2c8/0x700 [ 6916.075020] worker_thread+0x296/0x6e0 [ 6916.075025] ? rescuer_thread+0x570/0x570 [ 6916.075029] kthread+0x1d1/0x200 [ 6916.075033] ? set_kthread_struct+0x70/0x70 [ 6916.075037] ret_from_fork+0x1f/0x30 [ 6916.430185] kthread+0x1d1/0x200 [ 6916.432384] ? set_kthread_struct+0x70/0x70 [ 6916.433896] ret_from_fork+0x1f/0x30 [ 6920.055400] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6920.066301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6920.097869] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6920.109194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6920.113398] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6920.118603] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6920.132278] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6920.137175] RAX: 0000000000007f88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6920.141626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6920.144806] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6920.149210] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6920.159955] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6920.168301] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6920.172656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6920.176017] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6920.180565] Call Trace: [ 6920.181848] [ 6920.183164] ? show_regs.cold.9+0x22/0x2f [ 6920.185414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6920.187865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6920.190233] ? watchdog+0x40/0x40 [ 6920.192022] ? hrtimer_interrupt+0x128/0x2c0 [ 6920.195252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6920.197588] ? apic_timer_interrupt+0xf/0x20 [ 6920.199903] [ 6920.200459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6920.202833] do_raw_spin_lock+0x177/0x190 [ 6920.205238] _raw_spin_lock+0x21/0x30 [ 6920.206679] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6920.209500] ? do_raw_spin_unlock+0x75/0x190 [ 6920.211897] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6920.214401] ? do_wait_intr+0xf0/0xf0 [ 6920.216235] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6920.219572] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6920.222545] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6920.227594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6920.230473] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6920.232923] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6920.235866] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6920.239560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6920.243053] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6920.247030] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6920.251348] lmv_intent_open+0x338/0x1150 [lmv] [ 6920.253868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6920.256493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6920.260011] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6920.262291] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6920.265186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6920.268086] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6920.271257] ? do_raw_spin_unlock+0x75/0x190 [ 6920.273766] ? _raw_spin_unlock+0x12/0x30 [ 6920.275686] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6920.279006] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6920.281581] ll_atomic_open+0x36a/0xd70 [lustre] [ 6920.284809] lookup_open+0x356/0xb40 [ 6920.286705] do_last+0x313/0xfc0 [ 6920.288123] ? nd_jump_root+0xe5/0x160 [ 6920.289929] ? path_init+0x437/0x520 [ 6920.291865] path_openat+0xf7/0x500 [ 6920.293663] ? do_raw_spin_unlock+0x75/0x190 [ 6920.296291] do_filp_open+0x99/0x140 [ 6920.298064] ? getname_flags+0x6e/0x330 [ 6920.299952] ? __check_object_size+0xff/0x256 [ 6920.302047] ? do_raw_spin_unlock+0x75/0x190 [ 6920.305341] ? _raw_spin_unlock+0x12/0x30 [ 6920.307669] do_sys_openat2+0x2b4/0x410 [ 6920.309668] do_sys_open+0x73/0xa0 [ 6920.311472] __x64_sys_openat+0x24/0x30 [ 6920.313329] do_syscall_64+0xc1/0x440 [ 6920.315231] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6920.318873] RIP: 0033:0x7fa749552332 [ 6920.320924] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6920.331451] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6920.335981] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6920.339625] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6920.343670] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6920.347224] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6920.351111] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6924.257045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5007s! [ 6924.267220] Showing busy workqueues and worker pools: [ 6924.272840] workqueue events: flags=0x0 [ 6924.276665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6924.279750] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6924.287984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6924.292173] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6924.301283] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.305560] pending: kfree_rcu_monitor [ 6924.309269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6924.318861] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6924.325985] workqueue events_long: flags=0x0 [ 6924.329467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.331907] pending: lu_objects_flush [obdclass] [ 6924.333704] workqueue events_unbound: flags=0x2 [ 6924.335242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6924.337278] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6924.340586] workqueue events_power_efficient: flags=0x80 [ 6924.342094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.344652] pending: check_lifetime [ 6924.346143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.348634] pending: neigh_periodic_work [ 6924.350227] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.352787] pending: neigh_periodic_work [ 6924.354189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.356367] pending: do_cache_clean [ 6924.357584] workqueue mm_percpu_wq: flags=0x8 [ 6924.359024] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.361099] pending: vmstat_update [ 6924.362166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.364272] pending: vmstat_update [ 6924.365610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6924.371794] pending: vmstat_update [ 6924.374250] workqueue writeback: flags=0x4a [ 6924.377218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6924.383842] pending: wb_workfn, wb_workfn [ 6924.385148] workqueue xprtiod: flags=0xa [ 6924.386654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6924.390068] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6924.392799] workqueue ipv6_addrconf: flags=0x40008 [ 6924.394078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6924.399906] pending: addrconf_verify_work [ 6924.401641] workqueue knbd0-recv: flags=0x1a [ 6924.403014] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6924.411952] in-flight: 651:recv_work [ 6924.413975] workqueue ptlrpc_pinger: flags=0x2 [ 6924.421477] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6924.427408] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6924.433608] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5007s workers=4 idle: 37 31846 28897 [ 6924.437379] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6914s workers=2 idle: 653 [ 6932.059032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 6932.062469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6932.085676] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6932.098751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6932.104883] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6932.106946] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6932.118578] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6932.121740] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6932.124751] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6932.128538] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6932.131806] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6932.134970] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6932.137886] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6932.141412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6932.143930] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6932.148207] Call Trace: [ 6932.150160] [ 6932.151169] ? show_regs.cold.9+0x22/0x2f [ 6932.156131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6932.160140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6932.166604] ? watchdog+0x40/0x40 [ 6932.168199] ? hrtimer_interrupt+0x128/0x2c0 [ 6932.170508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6932.173062] ? apic_timer_interrupt+0xf/0x20 [ 6932.175068] [ 6932.176050] ? native_safe_halt+0x2e/0x50 [ 6932.177973] kvm_wait+0xcd/0xe0 [ 6932.179612] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6932.182442] do_raw_spin_lock+0x177/0x190 [ 6932.184259] _raw_spin_lock+0x21/0x30 [ 6932.185927] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6932.188769] ? do_raw_spin_unlock+0x75/0x190 [ 6932.190690] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6932.193032] ? do_wait_intr+0xf0/0xf0 [ 6932.194715] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6932.197463] mdc_close+0x53b/0xd70 [mdc] [ 6932.199959] lmv_close+0x18d/0x490 [lmv] [ 6932.201749] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6932.204484] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6932.206858] ll_md_close+0x274/0x800 [lustre] [ 6932.208960] ll_file_release+0x128/0xc50 [lustre] [ 6932.211277] __fput+0x10d/0x440 [ 6932.212857] ____fput+0x12/0x20 [ 6932.214381] task_work_run+0xc6/0x110 [ 6932.216021] exit_to_usermode_loop+0x1dd/0x1f0 [ 6932.218424] do_syscall_64+0x3d6/0x440 [ 6932.220212] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6932.222770] RIP: 0033:0x7fa749552cd8 [ 6932.225570] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6932.234251] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6932.238553] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6932.242737] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6932.246316] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6932.249203] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6932.253534] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6944.061339] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6944.067363] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6944.069756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6944.069924] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6944.069930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6944.069936] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6944.070206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6944.070231] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6944.070235] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6944.070240] RAX: 0000000000007f87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6944.070242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6944.070244] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6944.070246] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6944.070248] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6944.070251] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6944.070253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6944.070255] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6944.070263] Call Trace: [ 6944.070278] [ 6944.070284] ? show_regs.cold.9+0x22/0x2f [ 6944.070293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6944.070299] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6944.070303] ? watchdog+0x40/0x40 [ 6944.070307] ? hrtimer_interrupt+0x128/0x2c0 [ 6944.070311] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6944.070320] ? apic_timer_interrupt+0xf/0x20 [ 6944.070325] [ 6944.070328] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6944.071758] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6944.071766] do_raw_spin_lock+0x177/0x190 [ 6944.071771] _raw_spin_lock+0x21/0x30 [ 6944.071776] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6944.094555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6944.094636] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6944.094644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6944.094646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6944.096346] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6944.096353] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6944.096359] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6944.096361] RDX: 0000000000005b30 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6944.096363] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6944.096366] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6944.096368] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6944.096371] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6944.096374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6944.096377] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6944.096385] Call Trace: [ 6944.096391] [ 6944.096397] ? show_regs.cold.9+0x22/0x2f [ 6944.096407] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6944.096415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6944.096419] ? watchdog+0x40/0x40 [ 6944.096423] ? hrtimer_interrupt+0x128/0x2c0 [ 6944.096426] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6944.096434] ? apic_timer_interrupt+0xf/0x20 [ 6944.096439] [ 6944.096441] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6944.096445] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6944.096450] do_raw_spin_lock+0x177/0x190 [ 6944.096454] _raw_spin_lock+0x21/0x30 [ 6944.096461] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6944.100380] obd_mod_free+0x46/0x140 [ptlrpc] [ 6944.148888] process_one_work+0x2c8/0x700 [ 6944.148909] worker_thread+0x296/0x6e0 [ 6944.148914] ? rescuer_thread+0x570/0x570 [ 6944.148918] kthread+0x1d1/0x200 [ 6944.148923] ? set_kthread_struct+0x70/0x70 [ 6944.148926] ret_from_fork+0x1f/0x30 [ 6944.688261] mdc_commit_open+0x91/0xb0 [mdc] [ 6944.689889] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6944.700699] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6944.710691] after_reply+0xe2c/0x1780 [ptlrpc] [ 6944.712658] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6944.723482] ? __next_timer_interrupt+0x160/0x160 [ 6944.732337] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6944.735452] ? do_wait_intr+0xf0/0xf0 [ 6944.738305] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6944.746670] kthread+0x1d1/0x200 [ 6944.747980] ? set_kthread_struct+0x70/0x70 [ 6944.750744] ret_from_fork+0x1f/0x30 [ 6952.057884] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6952.066526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6952.116537] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6952.135153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6952.141468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6952.149558] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6952.177410] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6952.184607] RAX: 0000000000001385 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6952.195305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6952.203017] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6952.213710] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6952.228677] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6952.236898] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6952.244882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6952.257801] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6952.269142] Call Trace: [ 6952.272230] [ 6952.273169] ? show_regs.cold.9+0x22/0x2f [ 6952.274792] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6952.279166] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6952.281165] ? watchdog+0x40/0x40 [ 6952.286843] ? hrtimer_interrupt+0x128/0x2c0 [ 6952.288673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6952.295260] ? apic_timer_interrupt+0xf/0x20 [ 6952.301832] [ 6952.302613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6952.305319] do_raw_spin_lock+0x177/0x190 [ 6952.306856] _raw_spin_lock+0x21/0x30 [ 6952.311630] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6952.313896] ? do_raw_spin_unlock+0x75/0x190 [ 6952.316825] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6952.320230] ? do_wait_intr+0xf0/0xf0 [ 6952.322131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6952.325099] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6952.328202] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6952.331820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.334717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6952.337519] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6952.339739] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6952.349594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.355939] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6952.362549] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6952.366888] lmv_intent_open+0x338/0x1150 [lmv] [ 6952.372232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.375181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.380842] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6952.385266] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6952.391255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.398426] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6952.406108] ? do_raw_spin_unlock+0x75/0x190 [ 6952.410298] ? _raw_spin_unlock+0x12/0x30 [ 6952.416002] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6952.417721] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6952.423433] ll_atomic_open+0x36a/0xd70 [lustre] [ 6952.429562] lookup_open+0x356/0xb40 [ 6952.434251] do_last+0x313/0xfc0 [ 6952.439872] ? nd_jump_root+0xe5/0x160 [ 6952.443536] ? path_init+0x437/0x520 [ 6952.446716] path_openat+0xf7/0x500 [ 6952.449970] ? do_raw_spin_unlock+0x75/0x190 [ 6952.455567] do_filp_open+0x99/0x140 [ 6952.460000] ? getname_flags+0x6e/0x330 [ 6952.462429] ? __check_object_size+0xff/0x256 [ 6952.467421] ? do_raw_spin_unlock+0x75/0x190 [ 6952.470027] ? _raw_spin_unlock+0x12/0x30 [ 6952.473876] do_sys_openat2+0x2b4/0x410 [ 6952.477742] do_sys_open+0x73/0xa0 [ 6952.480041] __x64_sys_openat+0x24/0x30 [ 6952.487749] do_syscall_64+0xc1/0x440 [ 6952.489887] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6952.492853] RIP: 0033:0x7fa749552332 [ 6952.495631] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6952.522671] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6952.536096] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6952.547349] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6952.557476] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6952.568914] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6952.580234] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6954.473820] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5037s! [ 6954.478349] Showing busy workqueues and worker pools: [ 6954.485177] workqueue events: flags=0x0 [ 6954.486474] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6954.494132] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6954.500910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6954.506173] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6954.518781] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.528544] pending: kfree_rcu_monitor [ 6954.533503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6954.538002] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6954.544973] workqueue events_long: flags=0x0 [ 6954.546384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.555206] pending: lu_objects_flush [obdclass] [ 6954.565024] workqueue events_unbound: flags=0x2 [ 6954.571072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6954.574418] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6954.581550] workqueue events_power_efficient: flags=0x80 [ 6954.591505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.602651] pending: check_lifetime [ 6954.604012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.613996] pending: neigh_periodic_work [ 6954.615631] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.623229] pending: neigh_periodic_work [ 6954.628240] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.634738] pending: do_cache_clean [ 6954.637802] workqueue mm_percpu_wq: flags=0x8 [ 6954.641091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.646635] pending: vmstat_update [ 6954.649282] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.661015] pending: vmstat_update [ 6954.665868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6954.668975] pending: vmstat_update [ 6954.679172] workqueue writeback: flags=0x4a [ 6954.686972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6954.696717] pending: wb_workfn, wb_workfn [ 6954.702735] workqueue xprtiod: flags=0xa [ 6954.708712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6954.715419] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6954.729575] workqueue ipv6_addrconf: flags=0x40008 [ 6954.733573] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6954.742661] pending: addrconf_verify_work [ 6954.747063] workqueue knbd0-recv: flags=0x1a [ 6954.753159] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6954.762894] in-flight: 651:recv_work [ 6954.767823] workqueue ptlrpc_pinger: flags=0x2 [ 6954.771035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6954.775680] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6954.784092] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5037s workers=4 idle: 37 31846 28897 [ 6954.791950] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6944s workers=2 idle: 653 [ 6960.059047] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6960.069408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6960.109875] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6960.121409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6960.128345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6960.135183] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6960.148363] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6960.152081] RAX: 00000000000058db RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6960.158997] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6960.166991] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6960.171906] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6960.182480] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6960.187245] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6960.194111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6960.198384] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6960.201821] Call Trace: [ 6960.204040] [ 6960.204862] ? show_regs.cold.9+0x22/0x2f [ 6960.216074] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6960.220445] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6960.222256] ? watchdog+0x40/0x40 [ 6960.223505] ? hrtimer_interrupt+0x128/0x2c0 [ 6960.227392] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6960.234718] ? apic_timer_interrupt+0xf/0x20 [ 6960.237910] [ 6960.239033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6960.244070] do_raw_spin_lock+0x177/0x190 [ 6960.248669] _raw_spin_lock+0x21/0x30 [ 6960.253008] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6960.260304] ? do_raw_spin_unlock+0x75/0x190 [ 6960.263745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6960.267409] ? do_wait_intr+0xf0/0xf0 [ 6960.271702] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6960.278614] mdc_close+0x53b/0xd70 [mdc] [ 6960.281460] lmv_close+0x18d/0x490 [lmv] [ 6960.284863] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6960.288357] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6960.298224] ll_md_close+0x274/0x800 [lustre] [ 6960.303419] ll_file_release+0x128/0xc50 [lustre] [ 6960.310610] __fput+0x10d/0x440 [ 6960.311853] ____fput+0x12/0x20 [ 6960.314676] task_work_run+0xc6/0x110 [ 6960.316263] exit_to_usermode_loop+0x1dd/0x1f0 [ 6960.318185] do_syscall_64+0x3d6/0x440 [ 6960.320091] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6960.325199] RIP: 0033:0x7fa749552cd8 [ 6960.327105] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6960.339358] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6960.344109] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6960.347254] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6960.359443] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6960.364745] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6960.374910] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6972.057092] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 6972.058004] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 6972.063387] Modules linked in: [ 6972.067201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6972.067276] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6972.067283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6972.067285] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6972.067300] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6972.067305] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6972.067309] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 6972.067311] RDX: 00000000000049c2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 6972.067313] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 6972.067316] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 6972.067318] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6972.067321] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 6972.067324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6972.067326] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 6972.067335] Call Trace: [ 6972.067342] [ 6972.067348] ? show_regs.cold.9+0x22/0x2f [ 6972.067359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6972.067365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6972.067369] ? watchdog+0x40/0x40 [ 6972.067374] ? hrtimer_interrupt+0x128/0x2c0 [ 6972.067378] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6972.067386] ? apic_timer_interrupt+0xf/0x20 [ 6972.067391] [ 6972.067393] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6972.067397] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6972.067401] do_raw_spin_lock+0x177/0x190 [ 6972.067406] _raw_spin_lock+0x21/0x30 [ 6972.067412] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6972.067748] obd_mod_free+0x46/0x140 [ptlrpc] [ 6972.067965] mdc_commit_open+0x91/0xb0 [mdc] [ 6972.068002] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6972.075243] lustre(O) [ 6972.094328] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6972.095099] after_reply+0xe2c/0x1780 [ptlrpc] [ 6972.095405] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6972.096425] ? __next_timer_interrupt+0x160/0x160 [ 6972.096439] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6972.096815] ? do_wait_intr+0xf0/0xf0 [ 6972.096823] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6972.097143] kthread+0x1d1/0x200 [ 6972.097151] ? set_kthread_struct+0x70/0x70 [ 6972.097155] ret_from_fork+0x1f/0x30 [ 6972.314107] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6972.383328] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6972.403415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6972.415679] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6972.426890] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6972.438443] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6972.464039] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6972.470530] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6972.478046] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 6972.482734] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6972.486466] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 6972.490476] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 6972.494857] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 6972.499856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6972.504328] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 6972.512496] Call Trace: [ 6972.513519] [ 6972.514273] ? show_regs.cold.9+0x22/0x2f [ 6972.516271] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6972.518665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6972.524445] ? watchdog+0x40/0x40 [ 6972.529091] ? hrtimer_interrupt+0x128/0x2c0 [ 6972.532474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6972.535758] ? apic_timer_interrupt+0xf/0x20 [ 6972.538435] [ 6972.539431] ? native_safe_halt+0x2e/0x50 [ 6972.541689] kvm_wait+0xcd/0xe0 [ 6972.543658] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6972.546936] do_raw_spin_lock+0x177/0x190 [ 6972.549074] _raw_spin_lock+0x21/0x30 [ 6972.550922] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6972.555168] process_one_work+0x2c8/0x700 [ 6972.557639] worker_thread+0x296/0x6e0 [ 6972.559375] ? rescuer_thread+0x570/0x570 [ 6972.561386] kthread+0x1d1/0x200 [ 6972.563201] ? set_kthread_struct+0x70/0x70 [ 6972.565272] ret_from_fork+0x1f/0x30 [ 6980.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 6980.058525] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6980.119494] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6980.136852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6980.148695] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6980.154862] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6980.178729] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6980.192069] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6980.205379] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 6980.207572] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 6980.219014] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 6980.222289] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 6980.227961] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 6980.241614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6980.249580] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 6980.251561] Call Trace: [ 6980.266496] [ 6980.273252] ? show_regs.cold.9+0x22/0x2f [ 6980.276942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6980.280677] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6980.282265] ? watchdog+0x40/0x40 [ 6980.287950] ? hrtimer_interrupt+0x128/0x2c0 [ 6980.297033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6980.306131] ? apic_timer_interrupt+0xf/0x20 [ 6980.313067] [ 6980.314136] ? native_safe_halt+0x2e/0x50 [ 6980.321721] kvm_wait+0xcd/0xe0 [ 6980.327950] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6980.333648] do_raw_spin_lock+0x177/0x190 [ 6980.341253] _raw_spin_lock+0x21/0x30 [ 6980.347913] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6980.359282] ? do_raw_spin_unlock+0x75/0x190 [ 6980.362119] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6980.367203] ? do_wait_intr+0xf0/0xf0 [ 6980.371819] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6980.376859] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6980.386946] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6980.400240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.406229] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6980.416177] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6980.429049] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6980.433832] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.443768] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6980.453958] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6980.458593] lmv_intent_open+0x338/0x1150 [lmv] [ 6980.465944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.474113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.481277] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6980.493777] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6980.501903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.510196] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6980.518036] ? do_raw_spin_unlock+0x75/0x190 [ 6980.523119] ? _raw_spin_unlock+0x12/0x30 [ 6980.530528] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6980.536150] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6980.546557] ll_atomic_open+0x36a/0xd70 [lustre] [ 6980.556887] lookup_open+0x356/0xb40 [ 6980.564545] do_last+0x313/0xfc0 [ 6980.567670] ? nd_jump_root+0xe5/0x160 [ 6980.572897] ? path_init+0x437/0x520 [ 6980.577058] path_openat+0xf7/0x500 [ 6980.587019] ? do_raw_spin_unlock+0x75/0x190 [ 6980.592045] do_filp_open+0x99/0x140 [ 6980.601395] ? getname_flags+0x6e/0x330 [ 6980.603370] ? __check_object_size+0xff/0x256 [ 6980.608000] ? do_raw_spin_unlock+0x75/0x190 [ 6980.609224] ? _raw_spin_unlock+0x12/0x30 [ 6980.610485] do_sys_openat2+0x2b4/0x410 [ 6980.617494] do_sys_open+0x73/0xa0 [ 6980.623801] __x64_sys_openat+0x24/0x30 [ 6980.625956] do_syscall_64+0xc1/0x440 [ 6980.628108] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6980.634081] RIP: 0033:0x7fa749552332 [ 6980.637614] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6980.654336] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6980.660004] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 6980.673251] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 6980.685159] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 6980.697432] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 6980.705003] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 6985.189187] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5068s! [ 6985.199762] Showing busy workqueues and worker pools: [ 6985.206276] workqueue events: flags=0x0 [ 6985.211634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6985.224318] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 6985.238921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6985.252260] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6985.262265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.273985] pending: kfree_rcu_monitor [ 6985.276719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6985.283813] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 6985.296151] workqueue events_long: flags=0x0 [ 6985.298661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.306425] pending: lu_objects_flush [obdclass] [ 6985.310747] workqueue events_unbound: flags=0x2 [ 6985.319181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 6985.333853] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6985.350230] workqueue events_power_efficient: flags=0x80 [ 6985.359957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.366404] pending: check_lifetime [ 6985.372775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.381013] pending: neigh_periodic_work [ 6985.389105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.398006] pending: neigh_periodic_work [ 6985.401745] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.410672] pending: do_cache_clean [ 6985.414597] workqueue mm_percpu_wq: flags=0x8 [ 6985.417183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.423345] pending: vmstat_update [ 6985.425940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.431562] pending: vmstat_update [ 6985.433820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6985.436805] pending: vmstat_update [ 6985.438538] workqueue writeback: flags=0x4a [ 6985.441145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6985.444302] pending: wb_workfn, wb_workfn [ 6985.447067] workqueue xprtiod: flags=0xa [ 6985.449688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6985.453574] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6985.461188] workqueue ipv6_addrconf: flags=0x40008 [ 6985.464115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6985.470353] pending: addrconf_verify_work [ 6985.473245] workqueue knbd0-recv: flags=0x1a [ 6985.475415] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6985.484927] in-flight: 651:recv_work [ 6985.495267] workqueue ptlrpc_pinger: flags=0x2 [ 6985.496714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6985.506254] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6985.514298] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5068s workers=4 idle: 37 31846 28897 [ 6985.519878] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6975s workers=2 idle: 653 [ 6988.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 6988.062972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6988.083942] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6988.087324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6988.091457] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6988.093601] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6988.111929] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6988.114902] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 6988.119022] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 6988.121106] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6988.124487] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 6988.128598] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 6988.134548] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 6988.138334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6988.141387] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 6988.144934] Call Trace: [ 6988.147066] [ 6988.147712] ? show_regs.cold.9+0x22/0x2f [ 6988.152111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6988.154264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6988.156391] ? watchdog+0x40/0x40 [ 6988.158349] ? hrtimer_interrupt+0x128/0x2c0 [ 6988.159941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6988.162152] ? apic_timer_interrupt+0xf/0x20 [ 6988.163355] [ 6988.164433] ? native_safe_halt+0x2e/0x50 [ 6988.166242] kvm_wait+0xcd/0xe0 [ 6988.167057] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6988.169190] do_raw_spin_lock+0x177/0x190 [ 6988.170766] _raw_spin_lock+0x21/0x30 [ 6988.172413] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6988.174730] ? do_raw_spin_unlock+0x75/0x190 [ 6988.176651] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6988.178980] ? do_wait_intr+0xf0/0xf0 [ 6988.180473] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6988.183028] mdc_close+0x53b/0xd70 [mdc] [ 6988.184527] lmv_close+0x18d/0x490 [lmv] [ 6988.186231] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6988.188497] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6988.190396] ll_md_close+0x274/0x800 [lustre] [ 6988.192230] ll_file_release+0x128/0xc50 [lustre] [ 6988.194332] __fput+0x10d/0x440 [ 6988.195594] ____fput+0x12/0x20 [ 6988.196926] task_work_run+0xc6/0x110 [ 6988.198773] exit_to_usermode_loop+0x1dd/0x1f0 [ 6988.200589] do_syscall_64+0x3d6/0x440 [ 6988.202084] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6988.204059] RIP: 0033:0x7fa749552cd8 [ 6988.205507] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6988.212404] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6988.215489] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 6988.218135] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 6988.220863] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 6988.223486] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 6988.227228] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7000.057048] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 7000.058120] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7000.059956] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7000.060035] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7000.060043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7000.060049] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7000.060427] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7000.060439] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7000.061142] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7000.061148] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7000.061150] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7000.061153] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7000.061155] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7000.061157] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7000.061160] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7000.061162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7000.061164] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7000.061173] Call Trace: [ 7000.061179] [ 7000.061184] ? show_regs.cold.9+0x22/0x2f [ 7000.061193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7000.061199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7000.061204] ? watchdog+0x40/0x40 [ 7000.061209] ? hrtimer_interrupt+0x128/0x2c0 [ 7000.061213] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7000.061221] ? apic_timer_interrupt+0xf/0x20 [ 7000.061225] [ 7000.061228] ? native_safe_halt+0x2e/0x50 [ 7000.061232] kvm_wait+0xcd/0xe0 [ 7000.061240] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7000.061250] do_raw_spin_lock+0x177/0x190 [ 7000.061254] _raw_spin_lock+0x21/0x30 [ 7000.061257] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7000.064959] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7000.065038] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7000.065045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7000.065048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7000.065063] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7000.065067] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7000.065072] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7000.065074] RDX: 0000000000006636 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7000.065077] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7000.065079] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7000.065082] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7000.065085] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7000.065088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7000.065091] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7000.065100] Call Trace: [ 7000.065105] [ 7000.065111] ? show_regs.cold.9+0x22/0x2f [ 7000.065121] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7000.065128] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7000.065133] ? watchdog+0x40/0x40 [ 7000.065630] ? hrtimer_interrupt+0x128/0x2c0 [ 7000.065639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7000.065647] ? apic_timer_interrupt+0xf/0x20 [ 7000.065653] [ 7000.065656] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7000.065661] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7000.065666] do_raw_spin_lock+0x177/0x190 [ 7000.065671] _raw_spin_lock+0x21/0x30 [ 7000.065677] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7000.065950] obd_mod_free+0x46/0x140 [ptlrpc] [ 7000.067524] mdc_commit_open+0x91/0xb0 [mdc] [ 7000.067563] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7000.067803] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7000.067999] after_reply+0xe2c/0x1780 [ptlrpc] [ 7000.068209] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7000.068412] ? __next_timer_interrupt+0x160/0x160 [ 7000.068423] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7000.070082] ? do_wait_intr+0xf0/0xf0 [ 7000.070093] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7000.070810] kthread+0x1d1/0x200 [ 7000.070818] ? set_kthread_struct+0x70/0x70 [ 7000.070822] ret_from_fork+0x1f/0x30 [ 7000.459626] process_one_work+0x2c8/0x700 [ 7000.460920] worker_thread+0x296/0x6e0 [ 7000.462145] ? rescuer_thread+0x570/0x570 [ 7000.472547] kthread+0x1d1/0x200 [ 7000.475810] ? set_kthread_struct+0x70/0x70 [ 7000.482687] ret_from_fork+0x1f/0x30 [ 7012.054010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7012.059806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7012.100108] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7012.107336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7012.116015] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7012.127300] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7012.141611] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7012.149617] RAX: 0000000000000c0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7012.158480] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7012.172628] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7012.179119] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7012.181713] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7012.187664] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7012.191791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7012.195733] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7012.199695] Call Trace: [ 7012.200741] [ 7012.201426] ? show_regs.cold.9+0x22/0x2f [ 7012.205519] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7012.210304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7012.214426] ? watchdog+0x40/0x40 [ 7012.216709] ? hrtimer_interrupt+0x128/0x2c0 [ 7012.219150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7012.223406] ? apic_timer_interrupt+0xf/0x20 [ 7012.226929] [ 7012.227845] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7012.233329] do_raw_spin_lock+0x177/0x190 [ 7012.238377] _raw_spin_lock+0x21/0x30 [ 7012.240660] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7012.244418] ? do_raw_spin_unlock+0x75/0x190 [ 7012.246704] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7012.251388] ? do_wait_intr+0xf0/0xf0 [ 7012.253360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7012.257583] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7012.268159] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7012.275158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7012.281447] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7012.283879] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7012.288460] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7012.293216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7012.296296] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7012.303185] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7012.308969] lmv_intent_open+0x338/0x1150 [lmv] [ 7012.310762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7012.314075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7012.320708] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7012.322751] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7012.325483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7012.328203] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7012.332120] ? do_raw_spin_unlock+0x75/0x190 [ 7012.334691] ? _raw_spin_unlock+0x12/0x30 [ 7012.338809] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7012.341096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7012.343706] ll_atomic_open+0x36a/0xd70 [lustre] [ 7012.346105] lookup_open+0x356/0xb40 [ 7012.348008] do_last+0x313/0xfc0 [ 7012.349416] ? nd_jump_root+0xe5/0x160 [ 7012.351746] ? path_init+0x437/0x520 [ 7012.354715] path_openat+0xf7/0x500 [ 7012.356987] ? do_raw_spin_unlock+0x75/0x190 [ 7012.361025] do_filp_open+0x99/0x140 [ 7012.362660] ? getname_flags+0x6e/0x330 [ 7012.369300] ? __check_object_size+0xff/0x256 [ 7012.371743] ? do_raw_spin_unlock+0x75/0x190 [ 7012.376893] ? _raw_spin_unlock+0x12/0x30 [ 7012.382729] do_sys_openat2+0x2b4/0x410 [ 7012.384420] do_sys_open+0x73/0xa0 [ 7012.390107] __x64_sys_openat+0x24/0x30 [ 7012.392606] do_syscall_64+0xc1/0x440 [ 7012.400834] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7012.404838] RIP: 0033:0x7fa749552332 [ 7012.406621] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7012.426693] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7012.435483] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7012.442388] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7012.449032] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7012.455172] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7012.461559] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7013.801013] rcu: INFO: rcu_sched self-detected stall on CPU [ 7013.804893] rcu: 2-...!: (2814708 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 7013.814083] (t=5100112 jiffies g=258701 q=2970) [ 7013.816572] rcu: rcu_sched kthread starved for 5096907 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7013.823360] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7013.838039] rcu: RCU grace-period kthread stack dump: [ 7013.841567] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7013.849812] Call Trace: [ 7013.851001] __schedule+0x351/0xcb0 [ 7013.853188] schedule+0xc0/0x180 [ 7013.854815] schedule_timeout+0xb4/0x190 [ 7013.857593] ? do_raw_spin_unlock+0x75/0x190 [ 7013.862890] ? __next_timer_interrupt+0x160/0x160 [ 7013.865430] ? prepare_to_swait_event+0xf3/0x140 [ 7013.868534] rcu_gp_kthread+0x8dd/0x15b0 [ 7013.869894] ? force_qs_rnp+0x2c0/0x2c0 [ 7013.876714] kthread+0x1d1/0x200 [ 7013.877818] ? set_kthread_struct+0x70/0x70 [ 7013.880753] ret_from_fork+0x1f/0x30 [ 7013.884760] rcu: Stack dump where RCU GP kthread last ran: [ 7013.888390] Sending NMI from CPU 2 to CPUs 1: [ 7013.893655] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 7013.894764] NMI backtrace for cpu 2 [ 7013.899616] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7013.904525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7013.908242] Call Trace: [ 7013.912253] [ 7013.913328] ? dump_stack+0xbb/0x10e [ 7013.924556] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7013.930620] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7013.932028] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7013.938943] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7013.940403] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7013.946005] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7013.949147] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7013.950979] ? update_process_times+0x6d/0xc0 [ 7013.953162] ? tick_sched_handle.isra.18+0x27/0x90 [ 7013.954731] ? tick_sched_timer+0x53/0xb0 [ 7013.956925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7013.958802] ? tick_sched_handle.isra.18+0x90/0x90 [ 7013.961023] ? hrtimer_interrupt+0x128/0x2c0 [ 7013.962420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7013.967122] ? apic_timer_interrupt+0xf/0x20 [ 7013.968887] [ 7013.971241] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7013.972678] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7013.974155] ? do_raw_spin_lock+0x177/0x190 [ 7013.979509] ? _raw_spin_lock+0x21/0x30 [ 7013.981736] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7013.986060] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7013.987800] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7013.990223] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7013.995620] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7013.998527] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7014.001180] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7014.003504] ? __next_timer_interrupt+0x160/0x160 [ 7014.005376] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7014.006822] ? do_wait_intr+0xf0/0xf0 [ 7014.009002] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7014.010832] ? kthread+0x1d1/0x200 [ 7014.012043] ? set_kthread_struct+0x70/0x70 [ 7014.013886] ? ret_from_fork+0x1f/0x30 [ 7015.907933] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5098s! [ 7015.913134] Showing busy workqueues and worker pools: [ 7015.916714] workqueue events: flags=0x0 [ 7015.919404] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7015.923495] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7015.931369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7015.933921] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7015.940247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7015.944012] pending: kfree_rcu_monitor [ 7015.946661] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7015.951878] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7015.958413] workqueue events_long: flags=0x0 [ 7015.961453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7015.967803] pending: lu_objects_flush [obdclass] [ 7015.971063] workqueue events_unbound: flags=0x2 [ 7015.973901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7015.977216] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7015.981297] workqueue events_power_efficient: flags=0x80 [ 7015.983411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7015.987986] pending: check_lifetime [ 7015.990188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7015.994361] pending: neigh_periodic_work [ 7015.998159] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7016.003050] pending: neigh_periodic_work [ 7016.006172] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7016.011027] pending: do_cache_clean [ 7016.012431] workqueue mm_percpu_wq: flags=0x8 [ 7016.013841] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7016.018878] pending: vmstat_update [ 7016.021633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7016.026431] pending: vmstat_update [ 7016.029013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7016.035063] pending: vmstat_update [ 7016.038463] workqueue writeback: flags=0x4a [ 7016.042724] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7016.046995] pending: wb_workfn, wb_workfn [ 7016.050737] workqueue xprtiod: flags=0xa [ 7016.052275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7016.059017] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7016.063472] pending: [ 7016.069592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7016.069652] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7016.069658] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7016.069661] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7016.070057] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7016.070062] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7016.070067] RAX: 0000000000006c0a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7016.070079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7016.070081] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7016.070083] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7016.070085] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7016.070089] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7016.070092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7016.070094] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7016.070102] Call Trace: [ 7016.070108] [ 7016.070114] ? show_regs.cold.9+0x22/0x2f [ 7016.070124] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7016.070129] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7016.070133] ? watchdog+0x40/0x40 [ 7016.070137] ? hrtimer_interrupt+0x128/0x2c0 [ 7016.070140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7016.070149] ? apic_timer_interrupt+0xf/0x20 [ 7016.070154] [ 7016.070156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7016.070163] do_raw_spin_lock+0x177/0x190 [ 7016.070168] _raw_spin_lock+0x21/0x30 [ 7016.070173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7016.070470] ? do_raw_spin_unlock+0x75/0x190 [ 7016.070477] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7016.070989] ? do_wait_intr+0xf0/0xf0 [ 7016.070997] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7016.072464] xs_stream_data_receive_workfn, xs_error_handle [ 7016.102598] mdc_close+0x53b/0xd70 [mdc] [ 7016.109043] , xprt_autoclose [ 7016.109111] workqueue ipv6_addrconf: flags=0x40008 [ 7016.117765] lmv_close+0x18d/0x490 [lmv] [ 7016.117813] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7016.119695] pwq 4: [ 7016.139811] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7016.143259] cpus=2 [ 7016.150698] ll_md_close+0x274/0x800 [lustre] [ 7016.158556] node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7016.158577] pending: addrconf_verify_work [ 7016.158622] workqueue knbd0-recv: flags=0x1a [ 7016.158627] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7016.158637] in-flight: 651:recv_work [ 7016.158649] workqueue ptlrpc_pinger: flags=0x2 [ 7016.158652] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7016.158659] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7016.167726] ll_file_release+0x128/0xc50 [lustre] [ 7016.168265] __fput+0x10d/0x440 [ 7016.171693] ____fput+0x12/0x20 [ 7016.171717] task_work_run+0xc6/0x110 [ 7016.171726] exit_to_usermode_loop+0x1dd/0x1f0 [ 7016.171736] do_syscall_64+0x3d6/0x440 [ 7016.171740] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7016.171754] RIP: 0033:0x7fa749552cd8 [ 7016.171764] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7016.171770] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7016.171775] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7016.171777] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7016.171842] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7016.171845] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7016.171847] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7016.441969] [ 7016.446628] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5099s workers=4 idle: 37 31846 28897 [ 7016.461886] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7006s workers=2 idle: 653 [ 7040.057966] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7040.060027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7040.063878] Modules linked in: [ 7040.067530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7040.067608] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7040.067694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7040.067697] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7040.067709] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7040.067713] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7040.067717] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7040.067719] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7040.067722] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7040.067724] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7040.067727] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7040.067730] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7040.069153] lustre(O) [ 7040.089414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7040.096103] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7040.096173] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7040.096180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7040.096186] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7040.097035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.097050] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7040.097055] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7040.097059] RAX: 0000000000004ef0 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7040.097062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7040.097064] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7040.097066] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7040.097188] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7040.097195] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7040.097198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7040.097200] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7040.097209] Call Trace: [ 7040.097217] [ 7040.097223] ? show_regs.cold.9+0x22/0x2f [ 7040.097235] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7040.097241] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7040.097245] ? watchdog+0x40/0x40 [ 7040.097250] ? hrtimer_interrupt+0x128/0x2c0 [ 7040.097254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7040.097263] ? apic_timer_interrupt+0xf/0x20 [ 7040.097267] [ 7040.097270] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.097275] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7040.097280] do_raw_spin_lock+0x177/0x190 [ 7040.097285] _raw_spin_lock+0x21/0x30 [ 7040.097289] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7040.097599] process_one_work+0x2c8/0x700 [ 7040.097607] worker_thread+0x296/0x6e0 [ 7040.097611] ? rescuer_thread+0x570/0x570 [ 7040.097616] kthread+0x1d1/0x200 [ 7040.097621] ? set_kthread_struct+0x70/0x70 [ 7040.097625] ret_from_fork+0x1f/0x30 [ 7040.328365] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7040.341248] Call Trace: [ 7040.344481] [ 7040.346235] ? show_regs.cold.9+0x22/0x2f [ 7040.350175] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7040.352856] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7040.358540] ? watchdog+0x40/0x40 [ 7040.368773] ? hrtimer_interrupt+0x128/0x2c0 [ 7040.371468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7040.376994] ? apic_timer_interrupt+0xf/0x20 [ 7040.383575] [ 7040.385613] ? native_safe_halt+0x2e/0x50 [ 7040.389641] kvm_wait+0xcd/0xe0 [ 7040.391173] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7040.393459] do_raw_spin_lock+0x177/0x190 [ 7040.395144] _raw_spin_lock+0x21/0x30 [ 7040.396741] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7040.399268] obd_mod_free+0x46/0x140 [ptlrpc] [ 7040.401614] mdc_commit_open+0x91/0xb0 [mdc] [ 7040.403630] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7040.406636] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7040.409260] after_reply+0xe2c/0x1780 [ptlrpc] [ 7040.411607] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7040.414687] ? __next_timer_interrupt+0x160/0x160 [ 7040.418734] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7040.425233] ? do_wait_intr+0xf0/0xf0 [ 7040.426657] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7040.441349] kthread+0x1d1/0x200 [ 7040.444441] ? set_kthread_struct+0x70/0x70 [ 7040.449727] ret_from_fork+0x1f/0x30 [ 7044.054678] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7044.063070] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7044.063467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7044.063557] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7044.063563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7044.063566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7044.063578] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7044.063594] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7044.063599] RAX: 0000000000007d86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7044.063602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7044.063604] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7044.063606] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7044.063609] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7044.063612] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7044.063614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7044.063617] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7044.063626] Call Trace: [ 7044.063631] [ 7044.063637] ? show_regs.cold.9+0x22/0x2f [ 7044.063646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7044.063652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7044.063656] ? watchdog+0x40/0x40 [ 7044.063660] ? hrtimer_interrupt+0x128/0x2c0 [ 7044.063665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7044.063673] ? apic_timer_interrupt+0xf/0x20 [ 7044.063678] [ 7044.063681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7044.063686] do_raw_spin_lock+0x177/0x190 [ 7044.063691] _raw_spin_lock+0x21/0x30 [ 7044.063696] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7044.063955] ? do_raw_spin_unlock+0x75/0x190 [ 7044.063960] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7044.064829] ? do_wait_intr+0xf0/0xf0 [ 7044.064838] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7044.065805] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7044.067255] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7044.067809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7044.067926] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7044.067966] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7044.068502] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7044.068532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7044.068646] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7044.068896] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7044.068927] lmv_intent_open+0x338/0x1150 [lmv] [ 7044.068957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7044.079386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7044.079468] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7044.079475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7044.079478] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7044.079497] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7044.079501] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7044.079505] RAX: 000000000000158a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7044.079507] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7044.079509] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7044.079511] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7044.079513] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7044.079515] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7044.079518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7044.079520] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7044.079528] Call Trace: [ 7044.079533] [ 7044.079538] ? show_regs.cold.9+0x22/0x2f [ 7044.079546] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7044.079553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7044.079558] ? watchdog+0x40/0x40 [ 7044.079562] ? hrtimer_interrupt+0x128/0x2c0 [ 7044.079566] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7044.079573] ? apic_timer_interrupt+0xf/0x20 [ 7044.079593] [ 7044.079595] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7044.079601] do_raw_spin_lock+0x177/0x190 [ 7044.079605] _raw_spin_lock+0x21/0x30 [ 7044.079610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7044.079974] ? do_raw_spin_unlock+0x75/0x190 [ 7044.079982] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7044.080274] ? do_wait_intr+0xf0/0xf0 [ 7044.080283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7044.080638] mdc_close+0x53b/0xd70 [mdc] [ 7044.080675] lmv_close+0x18d/0x490 [lmv] [ 7044.080708] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7044.080819] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7044.080900] ll_md_close+0x274/0x800 [lustre] [ 7044.080985] ll_file_release+0x128/0xc50 [lustre] [ 7044.081422] __fput+0x10d/0x440 [ 7044.081437] ____fput+0x12/0x20 [ 7044.081442] task_work_run+0xc6/0x110 [ 7044.081450] exit_to_usermode_loop+0x1dd/0x1f0 [ 7044.081457] do_syscall_64+0x3d6/0x440 [ 7044.081462] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7044.081469] RIP: 0033:0x7fa749552cd8 [ 7044.081480] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7044.081484] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7044.081489] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7044.081491] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7044.081494] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7044.081496] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7044.081498] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7044.738453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7044.745612] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7044.757795] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7044.765994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7044.772196] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7044.775971] ? do_raw_spin_unlock+0x75/0x190 [ 7044.778908] ? _raw_spin_unlock+0x12/0x30 [ 7044.781417] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7044.789976] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7044.793075] ll_atomic_open+0x36a/0xd70 [lustre] [ 7044.797141] lookup_open+0x356/0xb40 [ 7044.800963] do_last+0x313/0xfc0 [ 7044.804160] ? nd_jump_root+0xe5/0x160 [ 7044.808153] ? path_init+0x437/0x520 [ 7044.810073] path_openat+0xf7/0x500 [ 7044.814261] ? do_raw_spin_unlock+0x75/0x190 [ 7044.815794] do_filp_open+0x99/0x140 [ 7044.823954] ? getname_flags+0x6e/0x330 [ 7044.826420] ? __check_object_size+0xff/0x256 [ 7044.829204] ? do_raw_spin_unlock+0x75/0x190 [ 7044.830739] ? _raw_spin_unlock+0x12/0x30 [ 7044.833971] do_sys_openat2+0x2b4/0x410 [ 7044.835830] do_sys_open+0x73/0xa0 [ 7044.837855] __x64_sys_openat+0x24/0x30 [ 7044.840661] do_syscall_64+0xc1/0x440 [ 7044.844159] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7044.851464] RIP: 0033:0x7fa749552332 [ 7044.855624] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7044.869098] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7044.872813] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7044.877879] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7044.880806] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7044.886874] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7044.891536] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7046.625430] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5129s! [ 7046.639785] Showing busy workqueues and worker pools: [ 7046.648370] workqueue events: flags=0x0 [ 7046.650698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7046.658486] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7046.674448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7046.679561] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7046.683802] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.687080] pending: kfree_rcu_monitor [ 7046.688960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7046.691931] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7046.701040] workqueue events_long: flags=0x0 [ 7046.704359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.713659] pending: lu_objects_flush [obdclass] [ 7046.719488] workqueue events_unbound: flags=0x2 [ 7046.721134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7046.727540] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7046.731853] workqueue events_power_efficient: flags=0x80 [ 7046.736164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.740604] pending: check_lifetime [ 7046.742561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.748485] pending: neigh_periodic_work [ 7046.751882] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.755566] pending: neigh_periodic_work [ 7046.758095] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.761548] pending: do_cache_clean [ 7046.765987] workqueue mm_percpu_wq: flags=0x8 [ 7046.769484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.773928] pending: vmstat_update [ 7046.777787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.785721] pending: vmstat_update [ 7046.788070] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7046.794156] pending: vmstat_update [ 7046.796149] workqueue writeback: flags=0x4a [ 7046.801311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7046.807950] pending: wb_workfn, wb_workfn [ 7046.809623] workqueue xprtiod: flags=0xa [ 7046.811109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7046.814938] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7046.818868] workqueue ipv6_addrconf: flags=0x40008 [ 7046.827088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7046.832256] pending: addrconf_verify_work [ 7046.834323] workqueue knbd0-recv: flags=0x1a [ 7046.836757] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7046.843313] in-flight: 651:recv_work [ 7046.846029] workqueue ptlrpc_pinger: flags=0x2 [ 7046.848487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7046.855724] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7046.858071] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5129s workers=4 idle: 37 31846 28897 [ 7046.864834] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7036s workers=2 idle: 653 [ 7068.057048] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7068.058300] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7068.062641] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 7068.067292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7068.067372] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7068.067380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7068.067383] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7068.067397] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7068.067401] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7068.067405] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7068.067407] RDX: 0000000000005765 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7068.067409] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7068.067411] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7068.067413] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7068.067416] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7068.067419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7068.067422] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7068.067431] Call Trace: [ 7068.067436] [ 7068.067442] ? show_regs.cold.9+0x22/0x2f [ 7068.067453] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7068.067459] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7068.067463] ? watchdog+0x40/0x40 [ 7068.067469] ? hrtimer_interrupt+0x128/0x2c0 [ 7068.067473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7068.067481] ? apic_timer_interrupt+0xf/0x20 [ 7068.067485] [ 7068.067488] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7068.067493] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7068.067497] do_raw_spin_lock+0x177/0x190 [ 7068.067502] _raw_spin_lock+0x21/0x30 [ 7068.067515] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7068.067846] obd_mod_free+0x46/0x140 [ptlrpc] [ 7068.068050] mdc_commit_open+0x91/0xb0 [mdc] [ 7068.068091] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7068.068298] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7068.068496] after_reply+0xe2c/0x1780 [ptlrpc] [ 7068.068697] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7068.074524] lquota(O) [ 7068.097362] ? __next_timer_interrupt+0x160/0x160 [ 7068.101764] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7068.101828] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7068.101836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7068.101843] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7068.102262] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7068.102273] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7068.102277] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7068.102281] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7068.102283] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7068.102286] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7068.102288] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7068.102290] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7068.102294] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7068.102296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7068.102299] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7068.102307] Call Trace: [ 7068.102313] [ 7068.102318] ? show_regs.cold.9+0x22/0x2f [ 7068.102326] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7068.102332] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7068.102337] ? watchdog+0x40/0x40 [ 7068.102341] ? hrtimer_interrupt+0x128/0x2c0 [ 7068.102345] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7068.102352] ? apic_timer_interrupt+0xf/0x20 [ 7068.102356] [ 7068.102358] ? native_safe_halt+0x2e/0x50 [ 7068.102362] kvm_wait+0xcd/0xe0 [ 7068.102369] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7068.102376] do_raw_spin_lock+0x177/0x190 [ 7068.102380] _raw_spin_lock+0x21/0x30 [ 7068.102384] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7068.108543] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7068.111143] process_one_work+0x2c8/0x700 [ 7068.111169] worker_thread+0x296/0x6e0 [ 7068.111175] ? rescuer_thread+0x570/0x570 [ 7068.111178] kthread+0x1d1/0x200 [ 7068.111183] ? set_kthread_struct+0x70/0x70 [ 7068.111186] ret_from_fork+0x1f/0x30 [ 7068.298719] ? do_wait_intr+0xf0/0xf0 [ 7068.300511] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7068.303387] kthread+0x1d1/0x200 [ 7068.305068] ? set_kthread_struct+0x70/0x70 [ 7068.307215] ret_from_fork+0x1f/0x30 [ 7072.055281] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7072.059167] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7072.063157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 7072.066314] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7072.066379] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7072.066385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7072.066387] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7072.066400] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7072.066404] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7072.066407] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7072.066408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7072.066410] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7072.066411] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7072.066413] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7072.066418] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7072.066420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7072.066421] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7072.066428] Call Trace: [ 7072.066433] [ 7072.066438] ? show_regs.cold.9+0x22/0x2f [ 7072.066448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7072.066453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7072.066456] ? watchdog+0x40/0x40 [ 7072.066459] ? hrtimer_interrupt+0x128/0x2c0 [ 7072.066462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7072.066467] ? apic_timer_interrupt+0xf/0x20 [ 7072.066470] [ 7072.066472] ? native_safe_halt+0x2e/0x50 [ 7072.066474] kvm_wait+0xcd/0xe0 [ 7072.066480] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7072.066487] do_raw_spin_lock+0x177/0x190 [ 7072.066489] _raw_spin_lock+0x21/0x30 [ 7072.066492] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7072.066834] ? do_raw_spin_unlock+0x75/0x190 [ 7072.066841] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7072.067161] ? do_wait_intr+0xf0/0xf0 [ 7072.067168] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7072.067322] mdc_close+0x53b/0xd70 [mdc] [ 7072.067350] lmv_close+0x18d/0x490 [lmv] [ 7072.067372] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7072.067534] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7072.067616] ll_md_close+0x274/0x800 [lustre] [ 7072.067835] ll_file_release+0x128/0xc50 [lustre] [ 7072.067938] __fput+0x10d/0x440 [ 7072.067946] ____fput+0x12/0x20 [ 7072.067949] task_work_run+0xc6/0x110 [ 7072.067954] exit_to_usermode_loop+0x1dd/0x1f0 [ 7072.067959] do_syscall_64+0x3d6/0x440 [ 7072.067963] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7072.067968] RIP: 0033:0x7fa749552cd8 [ 7072.067974] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7072.067976] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7072.067980] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7072.067981] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7072.067983] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7072.067984] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7072.067986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7072.259968] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7072.275482] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7072.280916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7072.285036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7072.287472] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7072.295576] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7072.307760] RAX: 0000000000005862 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7072.319735] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7072.329470] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7072.337567] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7072.346317] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7072.354138] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7072.359513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7072.362863] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7072.366295] Call Trace: [ 7072.367232] [ 7072.368158] ? show_regs.cold.9+0x22/0x2f [ 7072.369992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7072.371988] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7072.374175] ? watchdog+0x40/0x40 [ 7072.375866] ? hrtimer_interrupt+0x128/0x2c0 [ 7072.377850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7072.380335] ? apic_timer_interrupt+0xf/0x20 [ 7072.382612] [ 7072.383496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7072.386124] do_raw_spin_lock+0x177/0x190 [ 7072.388178] _raw_spin_lock+0x21/0x30 [ 7072.389937] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7072.392621] ? do_raw_spin_unlock+0x75/0x190 [ 7072.394926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7072.397550] ? do_wait_intr+0xf0/0xf0 [ 7072.400151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7072.402704] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7072.405893] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7072.408927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7072.411485] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7072.413790] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7072.416606] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7072.418543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7072.421492] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7072.424323] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7072.427101] lmv_intent_open+0x338/0x1150 [lmv] [ 7072.430306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7072.432997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7072.435660] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7072.437944] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7072.439729] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7072.442924] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7072.445808] ? do_raw_spin_unlock+0x75/0x190 [ 7072.448338] ? _raw_spin_unlock+0x12/0x30 [ 7072.450454] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7072.453441] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7072.456175] ll_atomic_open+0x36a/0xd70 [lustre] [ 7072.458343] lookup_open+0x356/0xb40 [ 7072.460564] do_last+0x313/0xfc0 [ 7072.462453] ? nd_jump_root+0xe5/0x160 [ 7072.464294] ? path_init+0x437/0x520 [ 7072.465793] path_openat+0xf7/0x500 [ 7072.467593] ? do_raw_spin_unlock+0x75/0x190 [ 7072.469762] do_filp_open+0x99/0x140 [ 7072.471625] ? getname_flags+0x6e/0x330 [ 7072.474248] ? __check_object_size+0xff/0x256 [ 7072.476485] ? do_raw_spin_unlock+0x75/0x190 [ 7072.478606] ? _raw_spin_unlock+0x12/0x30 [ 7072.481136] do_sys_openat2+0x2b4/0x410 [ 7072.482903] do_sys_open+0x73/0xa0 [ 7072.484364] __x64_sys_openat+0x24/0x30 [ 7072.486314] do_syscall_64+0xc1/0x440 [ 7072.488115] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7072.496149] RIP: 0033:0x7fa749552332 [ 7072.497276] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7072.517973] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7072.531118] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7072.533510] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7072.538802] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7072.543542] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7072.546978] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7077.353022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5160s! [ 7077.359965] Showing busy workqueues and worker pools: [ 7077.362296] workqueue events: flags=0x0 [ 7077.368320] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7077.376597] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7077.388246] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7077.392133] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7077.402216] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.410656] pending: kfree_rcu_monitor [ 7077.415594] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7077.423004] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7077.429472] workqueue events_long: flags=0x0 [ 7077.432930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.436432] pending: lu_objects_flush [obdclass] [ 7077.438745] workqueue events_unbound: flags=0x2 [ 7077.440939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7077.442668] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7077.450208] workqueue events_power_efficient: flags=0x80 [ 7077.451975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.457570] pending: check_lifetime [ 7077.463058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.466561] pending: neigh_periodic_work [ 7077.475200] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.477885] pending: neigh_periodic_work [ 7077.479667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.485752] pending: do_cache_clean [ 7077.491104] workqueue mm_percpu_wq: flags=0x8 [ 7077.492450] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.497367] pending: vmstat_update [ 7077.502205] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.508357] pending: vmstat_update [ 7077.511777] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.518811] pending: vmstat_update [ 7077.520576] workqueue writeback: flags=0x4a [ 7077.522468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7077.528600] pending: wb_workfn, wb_workfn [ 7077.531603] workqueue xprtiod: flags=0xa [ 7077.534174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7077.535815] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7077.544321] workqueue ipv6_addrconf: flags=0x40008 [ 7077.545830] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7077.548420] pending: addrconf_verify_work [ 7077.550451] workqueue knbd0-recv: flags=0x1a [ 7077.552247] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7077.557987] in-flight: 651:recv_work [ 7077.560681] workqueue ptlrpc_pinger: flags=0x2 [ 7077.563504] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7077.566438] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7077.569271] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5160s workers=4 idle: 37 31846 28897 [ 7077.573171] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7067s workers=2 idle: 653 [ 7096.057072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7096.058403] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7096.060787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [ 7096.067587] Modules linked in: [ 7096.067588] crc32c_intel ghash_clmulni_intel ata_piix serio_raw [ 7096.085723] lustre(O) [ 7096.086828] libata [ 7096.093885] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7096.094521] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7096.094528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7096.094531] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7096.094585] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7096.094589] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7096.094594] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7096.094596] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7096.094598] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7096.094600] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7096.094603] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7096.094606] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7096.094609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7096.094612] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7096.094620] Call Trace: [ 7096.094703] [ 7096.094711] ? show_regs.cold.9+0x22/0x2f [ 7096.094722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7096.094728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7096.094732] ? watchdog+0x40/0x40 [ 7096.094736] ? hrtimer_interrupt+0x128/0x2c0 [ 7096.094740] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7096.094747] ? apic_timer_interrupt+0xf/0x20 [ 7096.094751] [ 7096.094754] ? native_safe_halt+0x2e/0x50 [ 7096.094758] kvm_wait+0xcd/0xe0 [ 7096.094765] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7096.094773] do_raw_spin_lock+0x177/0x190 [ 7096.094778] _raw_spin_lock+0x21/0x30 [ 7096.094783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7096.096317] obd_mod_free+0x46/0x140 [ptlrpc] [ 7096.097152] mdc_commit_open+0x91/0xb0 [mdc] [ 7096.097185] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7096.098209] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7096.098223] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7096.098228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7096.098234] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7096.099107] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7096.099119] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7096.099123] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7096.099128] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7096.099130] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7096.099132] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7096.099135] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7096.099137] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7096.099140] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7096.099143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7096.099146] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7096.099155] Call Trace: [ 7096.099159] [ 7096.099163] ? show_regs.cold.9+0x22/0x2f [ 7096.099173] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7096.099180] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7096.099185] ? watchdog+0x40/0x40 [ 7096.099189] ? hrtimer_interrupt+0x128/0x2c0 [ 7096.099193] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7096.099200] ? apic_timer_interrupt+0xf/0x20 [ 7096.099205] [ 7096.099208] ? native_safe_halt+0x2e/0x50 [ 7096.099212] kvm_wait+0xcd/0xe0 [ 7096.099219] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7096.099227] do_raw_spin_lock+0x177/0x190 [ 7096.099231] _raw_spin_lock+0x21/0x30 [ 7096.099235] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7096.102841] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7096.131148] process_one_work+0x2c8/0x700 [ 7096.131170] worker_thread+0x296/0x6e0 [ 7096.131174] ? rescuer_thread+0x570/0x570 [ 7096.131178] kthread+0x1d1/0x200 [ 7096.131182] ? set_kthread_struct+0x70/0x70 [ 7096.131186] ret_from_fork+0x1f/0x30 [ 7096.296139] after_reply+0xe2c/0x1780 [ptlrpc] [ 7096.298004] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7096.300095] ? __next_timer_interrupt+0x160/0x160 [ 7096.301371] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7096.302873] ? do_wait_intr+0xf0/0xf0 [ 7096.304166] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7096.306309] kthread+0x1d1/0x200 [ 7096.309228] ? set_kthread_struct+0x70/0x70 [ 7096.310289] ret_from_fork+0x1f/0x30 [ 7100.060959] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7100.065322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7100.088197] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7100.092692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7100.095554] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7100.097661] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7100.104160] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7100.106683] RAX: 0000000000001d65 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7100.109401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7100.112446] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7100.115090] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7100.117850] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7100.120606] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7100.123789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7100.126124] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7100.128827] Call Trace: [ 7100.129823] [ 7100.130715] ? show_regs.cold.9+0x22/0x2f [ 7100.132396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7100.134409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7100.136496] ? watchdog+0x40/0x40 [ 7100.137872] ? hrtimer_interrupt+0x128/0x2c0 [ 7100.139522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7100.141731] ? apic_timer_interrupt+0xf/0x20 [ 7100.143510] [ 7100.144278] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7100.146364] do_raw_spin_lock+0x177/0x190 [ 7100.148128] _raw_spin_lock+0x21/0x30 [ 7100.149840] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7100.152517] ? do_raw_spin_unlock+0x75/0x190 [ 7100.154111] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7100.156179] ? do_wait_intr+0xf0/0xf0 [ 7100.157692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7100.159883] mdc_close+0x53b/0xd70 [mdc] [ 7100.161437] lmv_close+0x18d/0x490 [lmv] [ 7100.163715] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7100.166856] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7100.169454] ll_md_close+0x274/0x800 [lustre] [ 7100.171849] ll_file_release+0x128/0xc50 [lustre] [ 7100.174149] __fput+0x10d/0x440 [ 7100.175779] ____fput+0x12/0x20 [ 7100.177757] task_work_run+0xc6/0x110 [ 7100.179661] exit_to_usermode_loop+0x1dd/0x1f0 [ 7100.182475] do_syscall_64+0x3d6/0x440 [ 7100.184357] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7100.186452] RIP: 0033:0x7fa749552cd8 [ 7100.187741] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7100.194853] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7100.196926] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7100.198871] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7100.202958] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7100.205152] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7100.206931] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7104.054401] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7104.074942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7104.108724] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7104.120572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7104.125773] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7104.129713] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7104.141972] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7104.148535] RAX: 0000000000007a0d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7104.152739] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7104.156567] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7104.158281] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7104.162065] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7104.166168] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7104.170968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7104.174595] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7104.179215] Call Trace: [ 7104.181010] [ 7104.181964] ? show_regs.cold.9+0x22/0x2f [ 7104.184475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7104.186990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7104.190144] ? watchdog+0x40/0x40 [ 7104.191892] ? hrtimer_interrupt+0x128/0x2c0 [ 7104.194184] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7104.196583] ? apic_timer_interrupt+0xf/0x20 [ 7104.198737] [ 7104.199977] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7104.203008] do_raw_spin_lock+0x177/0x190 [ 7104.205289] _raw_spin_lock+0x21/0x30 [ 7104.207787] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7104.210725] ? do_raw_spin_unlock+0x75/0x190 [ 7104.213073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7104.215838] ? do_wait_intr+0xf0/0xf0 [ 7104.217576] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7104.220057] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7104.222769] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7104.225363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7104.227687] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7104.230603] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7104.232786] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7104.234984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7104.237835] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7104.240916] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7104.243956] lmv_intent_open+0x338/0x1150 [lmv] [ 7104.246589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7104.249049] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7104.251612] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7104.253806] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7104.255824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7104.258809] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7104.260698] ? do_raw_spin_unlock+0x75/0x190 [ 7104.262164] ? _raw_spin_unlock+0x12/0x30 [ 7104.264236] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7104.266636] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7104.268897] ll_atomic_open+0x36a/0xd70 [lustre] [ 7104.271655] lookup_open+0x356/0xb40 [ 7104.273656] do_last+0x313/0xfc0 [ 7104.275067] ? nd_jump_root+0xe5/0x160 [ 7104.276879] ? path_init+0x437/0x520 [ 7104.278482] path_openat+0xf7/0x500 [ 7104.280273] ? do_raw_spin_unlock+0x75/0x190 [ 7104.282178] do_filp_open+0x99/0x140 [ 7104.283847] ? getname_flags+0x6e/0x330 [ 7104.285567] ? __check_object_size+0xff/0x256 [ 7104.287506] ? do_raw_spin_unlock+0x75/0x190 [ 7104.289495] ? _raw_spin_unlock+0x12/0x30 [ 7104.291388] do_sys_openat2+0x2b4/0x410 [ 7104.293267] do_sys_open+0x73/0xa0 [ 7104.294723] __x64_sys_openat+0x24/0x30 [ 7104.296955] do_syscall_64+0xc1/0x440 [ 7104.298711] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7104.300842] RIP: 0033:0x7fa749552332 [ 7104.302651] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7104.315020] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7104.318194] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7104.321270] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7104.323966] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7104.330630] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7104.334057] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7108.067175] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5190s! [ 7108.072766] Showing busy workqueues and worker pools: [ 7108.076804] workqueue events: flags=0x0 [ 7108.078985] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7108.081859] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7108.087123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7108.090132] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7108.096553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.099750] pending: kfree_rcu_monitor [ 7108.101712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7108.108067] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7108.113417] workqueue events_long: flags=0x0 [ 7108.117110] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.122968] pending: lu_objects_flush [obdclass] [ 7108.126354] workqueue events_unbound: flags=0x2 [ 7108.128366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7108.134429] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7108.141607] workqueue events_power_efficient: flags=0x80 [ 7108.145110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.148667] pending: check_lifetime [ 7108.150734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.153229] pending: neigh_periodic_work [ 7108.156544] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.160660] pending: neigh_periodic_work [ 7108.163883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.170235] pending: do_cache_clean [ 7108.171967] workqueue mm_percpu_wq: flags=0x8 [ 7108.180273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.187641] pending: vmstat_update [ 7108.189446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.198749] pending: vmstat_update [ 7108.200553] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.204497] pending: vmstat_update [ 7108.206384] workqueue writeback: flags=0x4a [ 7108.209893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7108.213570] pending: wb_workfn, wb_workfn [ 7108.215895] workqueue xprtiod: flags=0xa [ 7108.217843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7108.220444] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7108.227735] workqueue ipv6_addrconf: flags=0x40008 [ 7108.234334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7108.241876] pending: addrconf_verify_work [ 7108.247903] workqueue knbd0-recv: flags=0x1a [ 7108.250207] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7108.255044] in-flight: 651:recv_work [ 7108.259041] workqueue ptlrpc_pinger: flags=0x2 [ 7108.262164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7108.267849] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7108.277736] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5191s workers=4 idle: 37 31846 28897 [ 7108.285093] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7098s workers=2 idle: 653 [ 7124.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7124.058024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7124.060010] Modules linked in: [ 7124.062463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7124.062541] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7124.062548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7124.062552] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7124.062567] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7124.062572] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7124.062577] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7124.062579] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7124.062582] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7124.062584] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7124.062587] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7124.062590] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7124.062593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7124.062595] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7124.062603] Call Trace: [ 7124.062608] [ 7124.062614] ? show_regs.cold.9+0x22/0x2f [ 7124.062625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7124.062633] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7124.062637] ? watchdog+0x40/0x40 [ 7124.062642] ? hrtimer_interrupt+0x128/0x2c0 [ 7124.062646] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7124.062653] ? apic_timer_interrupt+0xf/0x20 [ 7124.062657] [ 7124.062660] ? native_safe_halt+0x2e/0x50 [ 7124.062665] kvm_wait+0xcd/0xe0 [ 7124.062673] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7124.062680] do_raw_spin_lock+0x177/0x190 [ 7124.062684] _raw_spin_lock+0x21/0x30 [ 7124.062688] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7124.064405] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7124.064474] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7124.064481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7124.064488] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7124.064736] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7124.064745] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7124.064749] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7124.064754] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7124.064756] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7124.064759] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7124.064762] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7124.064764] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7124.064768] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7124.064771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7124.064773] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7124.064780] Call Trace: [ 7124.064785] [ 7124.064790] ? show_regs.cold.9+0x22/0x2f [ 7124.064874] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7124.064881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7124.064886] ? watchdog+0x40/0x40 [ 7124.064890] ? hrtimer_interrupt+0x128/0x2c0 [ 7124.064948] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7124.064953] ? apic_timer_interrupt+0xf/0x20 [ 7124.064957] [ 7124.064960] ? native_safe_halt+0x2e/0x50 [ 7124.064963] kvm_wait+0xcd/0xe0 [ 7124.064970] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7124.064977] do_raw_spin_lock+0x177/0x190 [ 7124.064981] _raw_spin_lock+0x21/0x30 [ 7124.064984] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7124.065204] process_one_work+0x2c8/0x700 [ 7124.065212] worker_thread+0x296/0x6e0 [ 7124.065217] ? rescuer_thread+0x570/0x570 [ 7124.065220] kthread+0x1d1/0x200 [ 7124.065225] ? set_kthread_struct+0x70/0x70 [ 7124.065228] ret_from_fork+0x1f/0x30 [ 7124.227360] obd_mod_free+0x46/0x140 [ptlrpc] [ 7124.228868] mdc_commit_open+0x91/0xb0 [mdc] [ 7124.230410] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7124.234531] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7124.238218] after_reply+0xe2c/0x1780 [ptlrpc] [ 7124.239671] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7124.241616] ? __next_timer_interrupt+0x160/0x160 [ 7124.243321] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7124.244993] ? do_wait_intr+0xf0/0xf0 [ 7124.246309] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7124.248242] kthread+0x1d1/0x200 [ 7124.249420] ? set_kthread_struct+0x70/0x70 [ 7124.250628] ret_from_fork+0x1f/0x30 [ 7128.060709] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7128.065382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7128.107638] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7128.122065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7128.142600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7128.144713] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7128.176696] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7128.183808] RAX: 0000000000000137 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7128.194640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7128.214052] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7128.230306] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7128.234447] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7128.241544] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7128.257795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7128.263294] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7128.270706] Call Trace: [ 7128.272065] [ 7128.272792] ? show_regs.cold.9+0x22/0x2f [ 7128.278968] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7128.285704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7128.289732] ? watchdog+0x40/0x40 [ 7128.294746] ? hrtimer_interrupt+0x128/0x2c0 [ 7128.300757] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7128.308758] ? apic_timer_interrupt+0xf/0x20 [ 7128.314588] [ 7128.317773] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7128.325789] do_raw_spin_lock+0x177/0x190 [ 7128.330435] _raw_spin_lock+0x21/0x30 [ 7128.331680] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7128.340589] ? do_raw_spin_unlock+0x75/0x190 [ 7128.345151] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7128.352282] ? do_wait_intr+0xf0/0xf0 [ 7128.357892] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7128.363292] mdc_close+0x53b/0xd70 [mdc] [ 7128.370420] lmv_close+0x18d/0x490 [lmv] [ 7128.371880] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7128.383966] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7128.391651] ll_md_close+0x274/0x800 [lustre] [ 7128.402565] ll_file_release+0x128/0xc50 [lustre] [ 7128.409514] __fput+0x10d/0x440 [ 7128.412793] ____fput+0x12/0x20 [ 7128.414141] task_work_run+0xc6/0x110 [ 7128.415558] exit_to_usermode_loop+0x1dd/0x1f0 [ 7128.422241] do_syscall_64+0x3d6/0x440 [ 7128.424476] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7128.426673] RIP: 0033:0x7fa749552cd8 [ 7128.428790] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7128.448522] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7128.455003] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7128.461243] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7128.468210] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7128.472791] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7128.479531] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7136.058723] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7136.071353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.122755] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.141215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.147903] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.157025] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7136.177124] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7136.183901] RAX: 0000000000004584 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7136.189605] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7136.199301] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7136.204126] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7136.208739] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7136.220672] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7136.235819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.246012] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7136.258214] Call Trace: [ 7136.259212] [ 7136.260016] ? show_regs.cold.9+0x22/0x2f [ 7136.268585] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.272908] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.275597] ? watchdog+0x40/0x40 [ 7136.276903] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.288658] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.291631] ? apic_timer_interrupt+0xf/0x20 [ 7136.297357] [ 7136.302145] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.306218] do_raw_spin_lock+0x177/0x190 [ 7136.312988] _raw_spin_lock+0x21/0x30 [ 7136.320471] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7136.333511] ? do_raw_spin_unlock+0x75/0x190 [ 7136.336455] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7136.338529] ? do_wait_intr+0xf0/0xf0 [ 7136.355961] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7136.364322] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7136.368940] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7136.383591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7136.388892] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7136.393740] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7136.400163] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7136.404910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7136.406531] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7136.414642] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7136.426215] lmv_intent_open+0x338/0x1150 [lmv] [ 7136.432224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7136.443098] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7136.460871] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7136.469393] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7136.479155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7136.488881] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7136.493628] ? do_raw_spin_unlock+0x75/0x190 [ 7136.502255] ? _raw_spin_unlock+0x12/0x30 [ 7136.503727] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7136.513111] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7136.518872] ll_atomic_open+0x36a/0xd70 [lustre] [ 7136.527056] lookup_open+0x356/0xb40 [ 7136.530058] do_last+0x313/0xfc0 [ 7136.532087] ? nd_jump_root+0xe5/0x160 [ 7136.536663] ? path_init+0x437/0x520 [ 7136.540695] path_openat+0xf7/0x500 [ 7136.542294] ? do_raw_spin_unlock+0x75/0x190 [ 7136.550081] do_filp_open+0x99/0x140 [ 7136.558134] ? getname_flags+0x6e/0x330 [ 7136.561923] ? __check_object_size+0xff/0x256 [ 7136.574090] ? do_raw_spin_unlock+0x75/0x190 [ 7136.583154] ? _raw_spin_unlock+0x12/0x30 [ 7136.594548] do_sys_openat2+0x2b4/0x410 [ 7136.601695] do_sys_open+0x73/0xa0 [ 7136.614654] __x64_sys_openat+0x24/0x30 [ 7136.619451] do_syscall_64+0xc1/0x440 [ 7136.626599] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7136.634586] RIP: 0033:0x7fa749552332 [ 7136.635805] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7136.674722] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7136.690087] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7136.707561] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7136.717166] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7136.725346] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7136.739742] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7138.786189] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5221s! [ 7138.792953] Showing busy workqueues and worker pools: [ 7138.800020] workqueue events: flags=0x0 [ 7138.803787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7138.813817] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7138.824508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7138.826647] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7138.836210] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.842323] pending: kfree_rcu_monitor [ 7138.845520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7138.853053] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7138.869270] workqueue events_long: flags=0x0 [ 7138.874533] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.878122] pending: lu_objects_flush [obdclass] [ 7138.879561] workqueue events_unbound: flags=0x2 [ 7138.889841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7138.891577] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7138.898952] workqueue events_power_efficient: flags=0x80 [ 7138.900888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.904230] pending: check_lifetime [ 7138.905564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.909694] pending: neigh_periodic_work [ 7138.911949] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.932885] pending: neigh_periodic_work [ 7138.942910] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.944843] pending: do_cache_clean [ 7138.945828] workqueue mm_percpu_wq: flags=0x8 [ 7138.956311] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.959601] pending: vmstat_update [ 7138.961431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.965868] pending: vmstat_update [ 7138.967673] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7138.970570] pending: vmstat_update [ 7138.972317] workqueue writeback: flags=0x4a [ 7138.976870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7138.982424] pending: wb_workfn, wb_workfn [ 7138.984802] workqueue xprtiod: flags=0xa [ 7138.986856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7138.990011] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7138.993782] workqueue ipv6_addrconf: flags=0x40008 [ 7138.996322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7138.999333] pending: addrconf_verify_work [ 7139.001443] workqueue knbd0-recv: flags=0x1a [ 7139.003439] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7139.006562] in-flight: 651:recv_work [ 7139.008481] workqueue ptlrpc_pinger: flags=0x2 [ 7139.010688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7139.013562] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7139.016402] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5221s workers=4 idle: 37 31846 28897 [ 7139.020226] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7128s workers=2 idle: 653 [ 7152.058054] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7152.062093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7152.064059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7152.094832] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7152.094847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7152.094850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7152.094877] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7152.094882] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7152.095098] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7152.095102] RDX: 00000000000044e1 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7152.095104] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7152.095106] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7152.095108] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7152.095111] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7152.095114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7152.095116] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7152.095125] Call Trace: [ 7152.095130] [ 7152.095136] ? show_regs.cold.9+0x22/0x2f [ 7152.095144] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7152.095151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7152.095215] ? watchdog+0x40/0x40 [ 7152.095220] ? hrtimer_interrupt+0x128/0x2c0 [ 7152.095224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7152.095233] ? apic_timer_interrupt+0xf/0x20 [ 7152.095238] [ 7152.095241] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7152.095246] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7152.095252] do_raw_spin_lock+0x177/0x190 [ 7152.095256] _raw_spin_lock+0x21/0x30 [ 7152.095264] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7152.095551] obd_mod_free+0x46/0x140 [ptlrpc] [ 7152.095848] mdc_commit_open+0x91/0xb0 [mdc] [ 7152.096334] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7152.096656] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7152.105003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 7152.112011] after_reply+0xe2c/0x1780 [ptlrpc] [ 7152.112476] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7152.112699] ? __next_timer_interrupt+0x160/0x160 [ 7152.112713] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7152.112921] ? do_wait_intr+0xf0/0xf0 [ 7152.112932] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7152.113157] kthread+0x1d1/0x200 [ 7152.113164] ? set_kthread_struct+0x70/0x70 [ 7152.113169] ret_from_fork+0x1f/0x30 [ 7152.288432] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7152.304959] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7152.309510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7152.315088] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7152.318381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7152.321474] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7152.339273] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7152.348242] RAX: 0000000000002c9b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7152.355607] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7152.361173] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7152.365065] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7152.368509] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7152.377536] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7152.384322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7152.386778] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7152.395031] Call Trace: [ 7152.397507] [ 7152.406301] ? show_regs.cold.9+0x22/0x2f [ 7152.410021] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7152.412942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7152.415984] ? watchdog+0x40/0x40 [ 7152.421078] ? hrtimer_interrupt+0x128/0x2c0 [ 7152.423691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7152.430171] ? apic_timer_interrupt+0xf/0x20 [ 7152.434764] [ 7152.435619] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7152.443023] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7152.447608] do_raw_spin_lock+0x177/0x190 [ 7152.453340] _raw_spin_lock+0x21/0x30 [ 7152.457122] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7152.464493] process_one_work+0x2c8/0x700 [ 7152.470456] worker_thread+0x296/0x6e0 [ 7152.476247] ? rescuer_thread+0x570/0x570 [ 7152.479910] kthread+0x1d1/0x200 [ 7152.481830] ? set_kthread_struct+0x70/0x70 [ 7152.487238] ret_from_fork+0x1f/0x30 [ 7156.062696] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7156.067574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7156.111029] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7156.118424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7156.126746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7156.129754] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7156.155210] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7156.160531] RAX: 0000000000003d14 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7156.164138] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7156.168249] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7156.173135] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7156.175080] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7156.182016] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7156.193110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7156.202088] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7156.210133] Call Trace: [ 7156.211239] [ 7156.213071] ? show_regs.cold.9+0x22/0x2f [ 7156.214761] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7156.222199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7156.226342] ? watchdog+0x40/0x40 [ 7156.228815] ? hrtimer_interrupt+0x128/0x2c0 [ 7156.232374] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7156.236837] ? apic_timer_interrupt+0xf/0x20 [ 7156.241606] [ 7156.245017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7156.250766] do_raw_spin_lock+0x177/0x190 [ 7156.255391] _raw_spin_lock+0x21/0x30 [ 7156.262218] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7156.264287] ? do_raw_spin_unlock+0x75/0x190 [ 7156.269116] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7156.282401] ? do_wait_intr+0xf0/0xf0 [ 7156.287492] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7156.301941] mdc_close+0x53b/0xd70 [mdc] [ 7156.308281] lmv_close+0x18d/0x490 [lmv] [ 7156.310690] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7156.320835] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7156.327319] ll_md_close+0x274/0x800 [lustre] [ 7156.332817] ll_file_release+0x128/0xc50 [lustre] [ 7156.334165] __fput+0x10d/0x440 [ 7156.339603] ____fput+0x12/0x20 [ 7156.340748] task_work_run+0xc6/0x110 [ 7156.341756] exit_to_usermode_loop+0x1dd/0x1f0 [ 7156.345770] do_syscall_64+0x3d6/0x440 [ 7156.349756] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7156.353228] RIP: 0033:0x7fa749552cd8 [ 7156.357786] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7156.379519] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7156.387924] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7156.392166] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7156.400118] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7156.403216] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7156.407230] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7164.067446] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7164.073657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.117413] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.131740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.138463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.147600] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7164.166655] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7164.175348] RAX: 0000000000005e6f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7164.178975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7164.182150] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7164.185743] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7164.189262] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7164.192649] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7164.196366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.198410] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7164.201881] Call Trace: [ 7164.203042] [ 7164.206649] ? show_regs.cold.9+0x22/0x2f [ 7164.212961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.217412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.222101] ? watchdog+0x40/0x40 [ 7164.223664] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.231585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.238008] ? apic_timer_interrupt+0xf/0x20 [ 7164.243347] [ 7164.244138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.251771] do_raw_spin_lock+0x177/0x190 [ 7164.256770] _raw_spin_lock+0x21/0x30 [ 7164.258817] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7164.265543] ? do_raw_spin_unlock+0x75/0x190 [ 7164.266773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7164.270106] ? do_wait_intr+0xf0/0xf0 [ 7164.273023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7164.276538] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7164.279519] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7164.282897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7164.286368] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7164.290429] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7164.292598] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7164.295366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7164.299098] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7164.302319] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7164.307524] lmv_intent_open+0x338/0x1150 [lmv] [ 7164.310769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7164.314691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7164.317534] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7164.319869] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7164.322424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7164.327247] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7164.329871] ? do_raw_spin_unlock+0x75/0x190 [ 7164.331763] ? _raw_spin_unlock+0x12/0x30 [ 7164.336868] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7164.339898] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7164.342443] ll_atomic_open+0x36a/0xd70 [lustre] [ 7164.345299] lookup_open+0x356/0xb40 [ 7164.347629] do_last+0x313/0xfc0 [ 7164.349981] ? nd_jump_root+0xe5/0x160 [ 7164.351517] ? path_init+0x437/0x520 [ 7164.352942] path_openat+0xf7/0x500 [ 7164.354791] ? do_raw_spin_unlock+0x75/0x190 [ 7164.356890] do_filp_open+0x99/0x140 [ 7164.358600] ? getname_flags+0x6e/0x330 [ 7164.360245] ? __check_object_size+0xff/0x256 [ 7164.364846] ? do_raw_spin_unlock+0x75/0x190 [ 7164.367953] ? _raw_spin_unlock+0x12/0x30 [ 7164.370825] do_sys_openat2+0x2b4/0x410 [ 7164.373584] do_sys_open+0x73/0xa0 [ 7164.375538] __x64_sys_openat+0x24/0x30 [ 7164.378819] do_syscall_64+0xc1/0x440 [ 7164.382050] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7164.385145] RIP: 0033:0x7fa749552332 [ 7164.387150] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7164.398440] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7164.405167] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7164.410879] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7164.415581] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7164.420965] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7164.428604] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7169.507941] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5252s! [ 7169.511881] Showing busy workqueues and worker pools: [ 7169.515413] workqueue events: flags=0x0 [ 7169.531951] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7169.534575] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7169.558164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7169.567171] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7169.580341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.588869] pending: kfree_rcu_monitor [ 7169.599898] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7169.606041] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7169.616723] workqueue events_long: flags=0x0 [ 7169.619015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.622754] pending: lu_objects_flush [obdclass] [ 7169.626190] workqueue events_unbound: flags=0x2 [ 7169.628651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7169.632009] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7169.636863] workqueue events_power_efficient: flags=0x80 [ 7169.639621] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.643663] pending: check_lifetime [ 7169.648258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.651990] pending: neigh_periodic_work [ 7169.657829] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.666076] pending: neigh_periodic_work [ 7169.673792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.681333] pending: do_cache_clean [ 7169.689686] workqueue mm_percpu_wq: flags=0x8 [ 7169.693657] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.701921] pending: vmstat_update [ 7169.713289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.718084] pending: vmstat_update [ 7169.722456] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7169.726390] pending: vmstat_update [ 7169.732175] workqueue writeback: flags=0x4a [ 7169.737596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7169.746746] pending: wb_workfn, wb_workfn [ 7169.751049] workqueue xprtiod: flags=0xa [ 7169.753937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7169.767852] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7169.771925] workqueue ipv6_addrconf: flags=0x40008 [ 7169.784045] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7169.789616] pending: addrconf_verify_work [ 7169.791711] workqueue knbd0-recv: flags=0x1a [ 7169.794249] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7169.799079] in-flight: 651:recv_work [ 7169.806199] workqueue ptlrpc_pinger: flags=0x2 [ 7169.808621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7169.813510] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7169.819021] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5252s workers=4 idle: 37 31846 28897 [ 7169.829042] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7159s workers=2 idle: 653 [ 7180.059040] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7180.061071] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7180.070672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7180.070751] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7180.070759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7180.070765] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7180.071249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7180.071261] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7180.071265] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7180.071270] RAX: 00000000000045ac RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7180.071272] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7180.071275] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7180.071277] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7180.071279] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7180.071282] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7180.071286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7180.071288] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7180.071296] Call Trace: [ 7180.071302] [ 7180.071308] ? show_regs.cold.9+0x22/0x2f [ 7180.071316] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7180.071322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7180.071326] ? watchdog+0x40/0x40 [ 7180.071331] ? hrtimer_interrupt+0x128/0x2c0 [ 7180.071335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7180.071343] ? apic_timer_interrupt+0xf/0x20 [ 7180.071348] [ 7180.071351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7180.071356] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7180.071362] do_raw_spin_lock+0x177/0x190 [ 7180.071367] _raw_spin_lock+0x21/0x30 [ 7180.071372] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7180.071593] process_one_work+0x2c8/0x700 [ 7180.071600] worker_thread+0x296/0x6e0 [ 7180.071605] ? rescuer_thread+0x570/0x570 [ 7180.071609] kthread+0x1d1/0x200 [ 7180.071613] ? set_kthread_struct+0x70/0x70 [ 7180.071617] ret_from_fork+0x1f/0x30 [ 7180.233593] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7180.276898] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7180.283623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7180.287700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7180.291118] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7180.298787] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7180.302095] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7180.305170] RDX: 0000000000005150 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7180.308295] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7180.311249] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7180.313737] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7180.316994] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7180.320437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7180.322820] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7180.326003] Call Trace: [ 7180.327185] [ 7180.327904] ? show_regs.cold.9+0x22/0x2f [ 7180.329464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7180.331553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7180.333467] ? watchdog+0x40/0x40 [ 7180.334786] ? hrtimer_interrupt+0x128/0x2c0 [ 7180.336815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7180.338858] ? apic_timer_interrupt+0xf/0x20 [ 7180.341750] [ 7180.342833] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7180.344691] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7180.347037] do_raw_spin_lock+0x177/0x190 [ 7180.348727] _raw_spin_lock+0x21/0x30 [ 7180.350088] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7180.352275] obd_mod_free+0x46/0x140 [ptlrpc] [ 7180.354793] mdc_commit_open+0x91/0xb0 [mdc] [ 7180.357218] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7180.359812] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7180.361926] after_reply+0xe2c/0x1780 [ptlrpc] [ 7180.364104] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7180.366692] ? __next_timer_interrupt+0x160/0x160 [ 7180.368894] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7180.370758] ? do_wait_intr+0xf0/0xf0 [ 7180.372506] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7180.374660] kthread+0x1d1/0x200 [ 7180.375815] ? set_kthread_struct+0x70/0x70 [ 7180.377755] ret_from_fork+0x1f/0x30 [ 7184.059354] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7184.066678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7184.110118] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7184.123639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7184.133407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7184.135471] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7184.157716] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7184.161111] RAX: 0000000000005b83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7184.175599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7184.183308] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7184.191214] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7184.196078] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7184.200287] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7184.205302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7184.208553] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7184.214266] Call Trace: [ 7184.215161] [ 7184.215870] ? show_regs.cold.9+0x22/0x2f [ 7184.218907] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7184.221240] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7184.225587] ? watchdog+0x40/0x40 [ 7184.227276] ? hrtimer_interrupt+0x128/0x2c0 [ 7184.229813] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7184.236674] ? apic_timer_interrupt+0xf/0x20 [ 7184.240399] [ 7184.241150] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7184.246595] do_raw_spin_lock+0x177/0x190 [ 7184.248700] _raw_spin_lock+0x21/0x30 [ 7184.252176] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7184.255158] ? do_raw_spin_unlock+0x75/0x190 [ 7184.259019] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7184.264597] ? do_wait_intr+0xf0/0xf0 [ 7184.269028] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7184.273715] mdc_close+0x53b/0xd70 [mdc] [ 7184.276378] lmv_close+0x18d/0x490 [lmv] [ 7184.278657] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7184.282119] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7184.286865] ll_md_close+0x274/0x800 [lustre] [ 7184.289232] ll_file_release+0x128/0xc50 [lustre] [ 7184.299111] __fput+0x10d/0x440 [ 7184.303174] ____fput+0x12/0x20 [ 7184.306906] task_work_run+0xc6/0x110 [ 7184.309001] exit_to_usermode_loop+0x1dd/0x1f0 [ 7184.313797] do_syscall_64+0x3d6/0x440 [ 7184.315314] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7184.321986] RIP: 0033:0x7fa749552cd8 [ 7184.323679] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7184.336576] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7184.340320] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7184.346101] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7184.350392] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7184.357141] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7184.361417] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7193.804713] rcu: INFO: rcu_sched self-detected stall on CPU [ 7193.807127] rcu: 2-...!: (2922214 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 7193.810589] (t=5280115 jiffies g=258701 q=2987) [ 7193.813054] rcu: rcu_sched kthread starved for 5276910 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7193.825822] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7193.830525] rcu: RCU grace-period kthread stack dump: [ 7193.839251] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7193.845244] Call Trace: [ 7193.853193] __schedule+0x351/0xcb0 [ 7193.854470] schedule+0xc0/0x180 [ 7193.855647] schedule_timeout+0xb4/0x190 [ 7193.861582] ? do_raw_spin_unlock+0x75/0x190 [ 7193.864823] ? __next_timer_interrupt+0x160/0x160 [ 7193.867638] ? prepare_to_swait_event+0xf3/0x140 [ 7193.869209] rcu_gp_kthread+0x8dd/0x15b0 [ 7193.876178] ? force_qs_rnp+0x2c0/0x2c0 [ 7193.881269] kthread+0x1d1/0x200 [ 7193.882458] ? set_kthread_struct+0x70/0x70 [ 7193.884297] ret_from_fork+0x1f/0x30 [ 7193.887806] rcu: Stack dump where RCU GP kthread last ran: [ 7193.892980] Sending NMI from CPU 2 to CPUs 1: [ 7193.899133] NMI backtrace for cpu 1 [ 7193.899141] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7193.899143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7193.899145] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7193.899148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7193.899150] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7193.899152] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 [ 7193.899154] RAX: 0000000000007086 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7193.899156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7193.899158] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7193.899159] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7193.899160] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7193.899162] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7193.899163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7193.899164] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7193.899166] Call Trace: [ 7193.899167] [ 7193.899168] ? show_regs.cold.9+0x22/0x2f [ 7193.899169] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7193.899170] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7193.899172] ? nmi_handle+0xbd/0x210 [ 7193.899173] ? default_do_nmi+0x4b/0x1b0 [ 7193.899174] ? do_nmi+0x280/0x380 [ 7193.899175] ? end_repeat_nmi+0x16/0x1f [ 7193.899176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7193.899177] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7193.899178] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7193.899179] [ 7193.899181] do_raw_spin_lock+0x177/0x190 [ 7193.899182] _raw_spin_lock+0x21/0x30 [ 7193.899183] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7193.899184] process_one_work+0x2c8/0x700 [ 7193.899185] worker_thread+0x296/0x6e0 [ 7193.899186] ? rescuer_thread+0x570/0x570 [ 7193.899187] kthread+0x1d1/0x200 [ 7193.899188] ? set_kthread_struct+0x70/0x70 [ 7193.899190] ret_from_fork+0x1f/0x30 [ 7193.901461] NMI backtrace for cpu 2 [ 7194.056256] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7194.060366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7194.064015] Call Trace: [ 7194.066461] [ 7194.068497] ? dump_stack+0xbb/0x10e [ 7194.071657] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7194.073416] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7194.075223] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7194.077256] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7194.083143] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7194.084885] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7194.088091] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7194.089644] ? update_process_times+0x6d/0xc0 [ 7194.096070] ? tick_sched_handle.isra.18+0x27/0x90 [ 7194.099861] ? tick_sched_timer+0x53/0xb0 [ 7194.101245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7194.103115] ? tick_sched_handle.isra.18+0x90/0x90 [ 7194.105166] ? hrtimer_interrupt+0x128/0x2c0 [ 7194.106620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7194.108516] ? apic_timer_interrupt+0xf/0x20 [ 7194.109938] [ 7194.110695] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7194.116315] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7194.120178] ? do_raw_spin_lock+0x177/0x190 [ 7194.122089] ? _raw_spin_lock+0x21/0x30 [ 7194.123280] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7194.125814] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7194.131918] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7194.133526] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7194.138582] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7194.141460] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7194.151491] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7194.156579] ? __next_timer_interrupt+0x160/0x160 [ 7194.161768] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7194.166189] ? do_wait_intr+0xf0/0xf0 [ 7194.171079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7194.178245] ? kthread+0x1d1/0x200 [ 7194.181525] ? set_kthread_struct+0x70/0x70 [ 7194.184446] ? ret_from_fork+0x1f/0x30 [ 7196.054054] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7196.059203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7196.100112] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7196.106295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7196.109965] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7196.111744] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7196.122576] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7196.125976] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7196.131975] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7196.135259] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7196.138378] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7196.140995] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7196.148165] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7196.152374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7196.160425] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7196.165041] Call Trace: [ 7196.170442] [ 7196.172865] ? show_regs.cold.9+0x22/0x2f [ 7196.173868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7196.176507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7196.181947] ? watchdog+0x40/0x40 [ 7196.183377] ? hrtimer_interrupt+0x128/0x2c0 [ 7196.185733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7196.188730] ? apic_timer_interrupt+0xf/0x20 [ 7196.192727] [ 7196.193815] ? native_safe_halt+0x2e/0x50 [ 7196.196651] kvm_wait+0xcd/0xe0 [ 7196.199026] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7196.202671] do_raw_spin_lock+0x177/0x190 [ 7196.204106] _raw_spin_lock+0x21/0x30 [ 7196.205417] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7196.211554] ? do_raw_spin_unlock+0x75/0x190 [ 7196.213488] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7196.217047] ? do_wait_intr+0xf0/0xf0 [ 7196.219928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7196.222326] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7196.224694] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7196.230487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.233722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7196.241018] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7196.242716] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7196.244243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.250800] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7196.257092] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7196.261649] lmv_intent_open+0x338/0x1150 [lmv] [ 7196.263715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.268866] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.270954] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7196.276746] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7196.278899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.281740] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7196.286967] ? do_raw_spin_unlock+0x75/0x190 [ 7196.288466] ? _raw_spin_unlock+0x12/0x30 [ 7196.289784] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7196.296984] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7196.299481] ll_atomic_open+0x36a/0xd70 [lustre] [ 7196.302734] lookup_open+0x356/0xb40 [ 7196.304707] do_last+0x313/0xfc0 [ 7196.306026] ? nd_jump_root+0xe5/0x160 [ 7196.309793] ? path_init+0x437/0x520 [ 7196.311058] path_openat+0xf7/0x500 [ 7196.314597] ? do_raw_spin_unlock+0x75/0x190 [ 7196.317337] do_filp_open+0x99/0x140 [ 7196.319868] ? getname_flags+0x6e/0x330 [ 7196.325524] ? __check_object_size+0xff/0x256 [ 7196.328417] ? do_raw_spin_unlock+0x75/0x190 [ 7196.329951] ? _raw_spin_unlock+0x12/0x30 [ 7196.332367] do_sys_openat2+0x2b4/0x410 [ 7196.338301] do_sys_open+0x73/0xa0 [ 7196.340388] __x64_sys_openat+0x24/0x30 [ 7196.341747] do_syscall_64+0xc1/0x440 [ 7196.346436] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7196.349701] RIP: 0033:0x7fa749552332 [ 7196.351063] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7196.361246] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7196.363211] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7196.365995] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7196.369855] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7196.375523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7196.378983] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7200.226698] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5283s! [ 7200.232377] Showing busy workqueues and worker pools: [ 7200.234992] workqueue events: flags=0x0 [ 7200.237127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7200.245220] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7200.252862] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7200.257266] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7200.272675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.277117] pending: kfree_rcu_monitor [ 7200.283589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7200.289349] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7200.300177] workqueue events_long: flags=0x0 [ 7200.305226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.308384] pending: lu_objects_flush [obdclass] [ 7200.310866] workqueue events_unbound: flags=0x2 [ 7200.315033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7200.317337] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7200.325452] workqueue events_power_efficient: flags=0x80 [ 7200.334785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.341370] pending: check_lifetime [ 7200.343389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.347621] pending: neigh_periodic_work [ 7200.350172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.353017] pending: neigh_periodic_work [ 7200.354604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.358131] pending: do_cache_clean [ 7200.360325] workqueue mm_percpu_wq: flags=0x8 [ 7200.366616] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.373700] pending: vmstat_update [ 7200.381765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.385854] pending: vmstat_update [ 7200.389009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.395640] pending: vmstat_update [ 7200.396729] workqueue writeback: flags=0x4a [ 7200.398675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7200.404233] pending: wb_workfn, wb_workfn [ 7200.406234] workqueue xprtiod: flags=0xa [ 7200.407511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7200.414442] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7200.425217] workqueue ipv6_addrconf: flags=0x40008 [ 7200.431101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7200.435227] pending: addrconf_verify_work [ 7200.439188] workqueue knbd0-recv: flags=0x1a [ 7200.441351] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7200.445499] in-flight: 651:recv_work [ 7200.449991] workqueue ptlrpc_pinger: flags=0x2 [ 7200.451921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7200.457098] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7200.460751] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5283s workers=4 idle: 37 31846 28897 [ 7200.468129] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7190s workers=2 idle: 653 [ 7212.059107] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7212.062538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7212.087257] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7212.092129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7212.098872] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7212.101143] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7212.110153] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7212.113491] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7212.120863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7212.126441] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7212.139349] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7212.145692] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7212.149836] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7212.152597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7212.155248] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7212.166970] Call Trace: [ 7212.170934] [ 7212.172512] ? show_regs.cold.9+0x22/0x2f [ 7212.178221] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7212.180546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7212.182903] ? watchdog+0x40/0x40 [ 7212.184277] ? hrtimer_interrupt+0x128/0x2c0 [ 7212.186193] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7212.193131] ? apic_timer_interrupt+0xf/0x20 [ 7212.196545] [ 7212.197551] ? native_safe_halt+0x2e/0x50 [ 7212.201083] kvm_wait+0xcd/0xe0 [ 7212.202303] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7212.206443] do_raw_spin_lock+0x177/0x190 [ 7212.207868] _raw_spin_lock+0x21/0x30 [ 7212.209792] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7212.212435] ? do_raw_spin_unlock+0x75/0x190 [ 7212.215785] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7212.220829] ? do_wait_intr+0xf0/0xf0 [ 7212.222332] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7212.225150] mdc_close+0x53b/0xd70 [mdc] [ 7212.227213] lmv_close+0x18d/0x490 [lmv] [ 7212.228996] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7212.231921] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7212.234112] ll_md_close+0x274/0x800 [lustre] [ 7212.236155] ll_file_release+0x128/0xc50 [lustre] [ 7212.237991] __fput+0x10d/0x440 [ 7212.239231] ____fput+0x12/0x20 [ 7212.243267] task_work_run+0xc6/0x110 [ 7212.245266] exit_to_usermode_loop+0x1dd/0x1f0 [ 7212.246904] do_syscall_64+0x3d6/0x440 [ 7212.248654] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7212.250220] RIP: 0033:0x7fa749552cd8 [ 7212.251734] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7212.260730] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7212.264866] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7212.268201] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7212.271338] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7212.276822] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7212.280200] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7220.057790] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 7220.065637] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7220.065954] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.066031] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.066037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.066044] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7220.066918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.066930] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7220.066934] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7220.066939] RAX: 0000000000002e18 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7220.066942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7220.066944] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7220.066946] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7220.066949] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7220.066952] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7220.066956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.066958] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7220.066967] Call Trace: [ 7220.066972] [ 7220.066977] ? show_regs.cold.9+0x22/0x2f [ 7220.066986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.066992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.066997] ? watchdog+0x40/0x40 [ 7220.067001] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.067006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.067014] ? apic_timer_interrupt+0xf/0x20 [ 7220.067029] [ 7220.067032] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.067037] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7220.067043] do_raw_spin_lock+0x177/0x190 [ 7220.067047] _raw_spin_lock+0x21/0x30 [ 7220.067052] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7220.067830] process_one_work+0x2c8/0x700 [ 7220.067838] worker_thread+0x296/0x6e0 [ 7220.067842] ? rescuer_thread+0x570/0x570 [ 7220.067846] kthread+0x1d1/0x200 [ 7220.067851] ? set_kthread_struct+0x70/0x70 [ 7220.067855] ret_from_fork+0x1f/0x30 [ 7220.357265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.428108] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.441609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.457282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7220.458905] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7220.503434] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7220.506060] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7220.527898] RDX: 0000000000001fca RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7220.541448] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7220.552898] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7220.557467] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7220.568003] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7220.579771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.585654] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7220.591993] Call Trace: [ 7220.593044] [ 7220.593844] ? show_regs.cold.9+0x22/0x2f [ 7220.596673] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.598479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.599868] ? watchdog+0x40/0x40 [ 7220.601757] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.604461] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.608160] ? apic_timer_interrupt+0xf/0x20 [ 7220.609522] [ 7220.610127] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7220.616031] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7220.621754] do_raw_spin_lock+0x177/0x190 [ 7220.625141] _raw_spin_lock+0x21/0x30 [ 7220.626839] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7220.630820] obd_mod_free+0x46/0x140 [ptlrpc] [ 7220.633485] mdc_commit_open+0x91/0xb0 [mdc] [ 7220.634821] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7220.640048] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7220.645740] after_reply+0xe2c/0x1780 [ptlrpc] [ 7220.649552] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7220.652921] ? __next_timer_interrupt+0x160/0x160 [ 7220.655105] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7220.657090] ? do_wait_intr+0xf0/0xf0 [ 7220.658806] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7220.660792] kthread+0x1d1/0x200 [ 7220.661732] ? set_kthread_struct+0x70/0x70 [ 7220.663583] ret_from_fork+0x1f/0x30 [ 7228.055184] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7228.069930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7228.128584] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7228.134014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7228.139435] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7228.141643] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7228.159752] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7228.166459] RAX: 0000000000006218 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7228.175649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7228.186078] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7228.199767] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7228.210307] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7228.220772] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7228.230405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7228.233706] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7228.239832] Call Trace: [ 7228.240901] [ 7228.242011] ? show_regs.cold.9+0x22/0x2f [ 7228.244212] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7228.247577] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7228.250908] ? watchdog+0x40/0x40 [ 7228.257553] ? hrtimer_interrupt+0x128/0x2c0 [ 7228.265755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7228.267365] ? apic_timer_interrupt+0xf/0x20 [ 7228.274850] [ 7228.275829] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7228.281811] do_raw_spin_lock+0x177/0x190 [ 7228.290648] _raw_spin_lock+0x21/0x30 [ 7228.292021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7228.309686] ? do_raw_spin_unlock+0x75/0x190 [ 7228.311125] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7228.315754] ? do_wait_intr+0xf0/0xf0 [ 7228.317937] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7228.322563] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7228.325491] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7228.329095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7228.331996] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7228.335038] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7228.338019] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7228.347712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7228.358659] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7228.361487] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7228.370785] lmv_intent_open+0x338/0x1150 [lmv] [ 7228.379422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7228.387817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7228.400990] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7228.408609] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7228.411654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7228.415054] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7228.425054] ? do_raw_spin_unlock+0x75/0x190 [ 7228.437454] ? _raw_spin_unlock+0x12/0x30 [ 7228.447263] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7228.451426] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7228.463044] ll_atomic_open+0x36a/0xd70 [lustre] [ 7228.464852] lookup_open+0x356/0xb40 [ 7228.466091] do_last+0x313/0xfc0 [ 7228.477845] ? nd_jump_root+0xe5/0x160 [ 7228.481168] ? path_init+0x437/0x520 [ 7228.482346] path_openat+0xf7/0x500 [ 7228.493409] ? do_raw_spin_unlock+0x75/0x190 [ 7228.494950] do_filp_open+0x99/0x140 [ 7228.505691] ? getname_flags+0x6e/0x330 [ 7228.508795] ? __check_object_size+0xff/0x256 [ 7228.514482] ? do_raw_spin_unlock+0x75/0x190 [ 7228.521289] ? _raw_spin_unlock+0x12/0x30 [ 7228.522745] do_sys_openat2+0x2b4/0x410 [ 7228.524118] do_sys_open+0x73/0xa0 [ 7228.539364] __x64_sys_openat+0x24/0x30 [ 7228.543090] do_syscall_64+0xc1/0x440 [ 7228.552532] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7228.554224] RIP: 0033:0x7fa749552332 [ 7228.578191] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7228.623737] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7228.626322] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7228.653716] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7228.674315] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7228.682730] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7228.693035] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7230.948540] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5313s! [ 7230.961270] Showing busy workqueues and worker pools: [ 7230.963079] workqueue events: flags=0x0 [ 7230.968285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7230.979038] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7230.989087] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7230.999656] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7231.017684] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.024695] pending: kfree_rcu_monitor [ 7231.033176] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7231.043445] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7231.054382] workqueue events_long: flags=0x0 [ 7231.055995] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.072316] pending: lu_objects_flush [obdclass] [ 7231.084356] workqueue events_unbound: flags=0x2 [ 7231.092937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7231.098825] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7231.113468] workqueue events_power_efficient: flags=0x80 [ 7231.133993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.149100] pending: check_lifetime [ 7231.159484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.165395] pending: neigh_periodic_work [ 7231.166692] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.177941] pending: neigh_periodic_work [ 7231.192070] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.194574] pending: do_cache_clean [ 7231.213381] workqueue mm_percpu_wq: flags=0x8 [ 7231.214890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.235925] pending: vmstat_update [ 7231.238609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.250021] pending: vmstat_update [ 7231.251299] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.270772] pending: vmstat_update [ 7231.278878] workqueue writeback: flags=0x4a [ 7231.288121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7231.298962] pending: wb_workfn, wb_workfn [ 7231.308161] workqueue xprtiod: flags=0xa [ 7231.309414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7231.317671] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7231.330403] workqueue ipv6_addrconf: flags=0x40008 [ 7231.332123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7231.340183] pending: addrconf_verify_work [ 7231.348031] workqueue knbd0-recv: flags=0x1a [ 7231.349480] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7231.362627] in-flight: 651:recv_work [ 7231.363969] workqueue ptlrpc_pinger: flags=0x2 [ 7231.375836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7231.383901] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7231.391189] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5314s workers=4 idle: 37 31846 28897 [ 7231.404862] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7221s workers=2 idle: 653 [ 7240.062935] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7240.069081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7240.126391] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7240.138321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7240.148417] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7240.153208] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7240.180056] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7240.199825] RAX: 0000000000007cd0 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7240.209455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7240.230932] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7240.239901] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7240.250396] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7240.261314] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7240.263939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7240.268661] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7240.275524] Call Trace: [ 7240.282113] [ 7240.283500] ? show_regs.cold.9+0x22/0x2f [ 7240.288712] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7240.299753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7240.302113] ? watchdog+0x40/0x40 [ 7240.306404] ? hrtimer_interrupt+0x128/0x2c0 [ 7240.312729] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7240.316503] ? apic_timer_interrupt+0xf/0x20 [ 7240.318047] [ 7240.321854] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7240.328386] do_raw_spin_lock+0x177/0x190 [ 7240.332584] _raw_spin_lock+0x21/0x30 [ 7240.338745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7240.344359] ? do_raw_spin_unlock+0x75/0x190 [ 7240.348555] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7240.352539] ? do_wait_intr+0xf0/0xf0 [ 7240.358448] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7240.367452] mdc_close+0x53b/0xd70 [mdc] [ 7240.374525] lmv_close+0x18d/0x490 [lmv] [ 7240.381229] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7240.388021] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7240.393013] ll_md_close+0x274/0x800 [lustre] [ 7240.397541] ll_file_release+0x128/0xc50 [lustre] [ 7240.402671] __fput+0x10d/0x440 [ 7240.406442] ____fput+0x12/0x20 [ 7240.410561] task_work_run+0xc6/0x110 [ 7240.412730] exit_to_usermode_loop+0x1dd/0x1f0 [ 7240.417576] do_syscall_64+0x3d6/0x440 [ 7240.421659] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7240.427694] RIP: 0033:0x7fa749552cd8 [ 7240.432531] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7240.454125] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7240.462599] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7240.465024] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7240.478097] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7240.483443] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7240.488403] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7248.058761] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7248.061045] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 7248.067258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.067335] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.067342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.067345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7248.067358] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7248.067363] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7248.067367] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7248.067369] RDX: 00000000000068b3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7248.067371] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7248.067374] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7248.067376] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7248.067379] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7248.067382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.067385] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7248.067393] Call Trace: [ 7248.067397] [ 7248.067402] ? show_regs.cold.9+0x22/0x2f [ 7248.067410] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.067416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.067420] ? watchdog+0x40/0x40 [ 7248.067424] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.067428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.067437] ? apic_timer_interrupt+0xf/0x20 [ 7248.067442] [ 7248.067445] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7248.067450] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7248.067455] do_raw_spin_lock+0x177/0x190 [ 7248.067459] _raw_spin_lock+0x21/0x30 [ 7248.067465] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7248.067731] obd_mod_free+0x46/0x140 [ptlrpc] [ 7248.067935] mdc_commit_open+0x91/0xb0 [mdc] [ 7248.067971] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7248.070557] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7248.073894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.073983] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.073991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.073999] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7248.074344] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.074359] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7248.074363] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7248.074368] RAX: 0000000000005d58 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7248.074371] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7248.074373] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7248.074376] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7248.074379] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7248.074382] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7248.074385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.074388] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7248.074398] Call Trace: [ 7248.074404] [ 7248.074410] ? show_regs.cold.9+0x22/0x2f [ 7248.074419] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.074425] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.074430] ? watchdog+0x40/0x40 [ 7248.074434] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.074438] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.074446] ? apic_timer_interrupt+0xf/0x20 [ 7248.074451] [ 7248.074454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.074459] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7248.074464] do_raw_spin_lock+0x177/0x190 [ 7248.074469] _raw_spin_lock+0x21/0x30 [ 7248.074473] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7248.074719] process_one_work+0x2c8/0x700 [ 7248.074726] worker_thread+0x296/0x6e0 [ 7248.074731] ? rescuer_thread+0x570/0x570 [ 7248.074734] kthread+0x1d1/0x200 [ 7248.074739] ? set_kthread_struct+0x70/0x70 [ 7248.074744] ret_from_fork+0x1f/0x30 [ 7248.560470] after_reply+0xe2c/0x1780 [ptlrpc] [ 7248.564737] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7248.569939] ? __next_timer_interrupt+0x160/0x160 [ 7248.573165] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7248.582950] ? do_wait_intr+0xf0/0xf0 [ 7248.586287] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7248.595203] kthread+0x1d1/0x200 [ 7248.600396] ? set_kthread_struct+0x70/0x70 [ 7248.604693] ret_from_fork+0x1f/0x30 [ 7256.054250] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7256.063105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7256.124385] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7256.131513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7256.139016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7256.147772] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7256.163131] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7256.175482] RAX: 0000000000004084 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7256.182736] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7256.189312] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7256.196365] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7256.204024] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7256.213124] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7256.217531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7256.221207] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7256.229014] Call Trace: [ 7256.230102] [ 7256.231876] ? show_regs.cold.9+0x22/0x2f [ 7256.233474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7256.241441] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7256.244473] ? watchdog+0x40/0x40 [ 7256.249479] ? hrtimer_interrupt+0x128/0x2c0 [ 7256.255861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7256.261492] ? apic_timer_interrupt+0xf/0x20 [ 7256.267418] [ 7256.271491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7256.281498] do_raw_spin_lock+0x177/0x190 [ 7256.289646] _raw_spin_lock+0x21/0x30 [ 7256.294608] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7256.301295] ? do_raw_spin_unlock+0x75/0x190 [ 7256.306361] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7256.317255] ? do_wait_intr+0xf0/0xf0 [ 7256.321230] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7256.327412] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7256.334498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7256.340057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7256.346444] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7256.354482] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7256.361039] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7256.362735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7256.373503] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7256.381097] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7256.383095] lmv_intent_open+0x338/0x1150 [lmv] [ 7256.390373] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7256.398608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7256.407266] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7256.411059] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7256.418388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7256.423273] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7256.426403] ? do_raw_spin_unlock+0x75/0x190 [ 7256.432615] ? _raw_spin_unlock+0x12/0x30 [ 7256.434336] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7256.441302] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7256.444544] ll_atomic_open+0x36a/0xd70 [lustre] [ 7256.449330] lookup_open+0x356/0xb40 [ 7256.450624] do_last+0x313/0xfc0 [ 7256.453684] ? nd_jump_root+0xe5/0x160 [ 7256.455710] ? path_init+0x437/0x520 [ 7256.457511] path_openat+0xf7/0x500 [ 7256.459952] ? do_raw_spin_unlock+0x75/0x190 [ 7256.467086] do_filp_open+0x99/0x140 [ 7256.470541] ? getname_flags+0x6e/0x330 [ 7256.477343] ? __check_object_size+0xff/0x256 [ 7256.482348] ? do_raw_spin_unlock+0x75/0x190 [ 7256.484245] ? _raw_spin_unlock+0x12/0x30 [ 7256.485501] do_sys_openat2+0x2b4/0x410 [ 7256.486831] do_sys_open+0x73/0xa0 [ 7256.495690] __x64_sys_openat+0x24/0x30 [ 7256.499676] do_syscall_64+0xc1/0x440 [ 7256.502171] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7256.504364] RIP: 0033:0x7fa749552332 [ 7256.506380] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7256.522923] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7256.534278] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7256.537553] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7256.541393] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7256.545497] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7256.549181] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7261.665043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5344s! [ 7261.669126] Showing busy workqueues and worker pools: [ 7261.675247] workqueue events: flags=0x0 [ 7261.678112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7261.684660] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7261.695269] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7261.699575] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7261.711679] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.719543] pending: kfree_rcu_monitor [ 7261.725613] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7261.734311] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7261.748090] workqueue events_long: flags=0x0 [ 7261.752819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.759835] pending: lu_objects_flush [obdclass] [ 7261.766069] workqueue events_unbound: flags=0x2 [ 7261.770065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7261.776743] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7261.785632] workqueue events_power_efficient: flags=0x80 [ 7261.788261] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.799394] pending: check_lifetime [ 7261.801311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.812575] pending: neigh_periodic_work [ 7261.819164] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.832323] pending: neigh_periodic_work [ 7261.839223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.843780] pending: do_cache_clean [ 7261.848760] workqueue mm_percpu_wq: flags=0x8 [ 7261.854736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.859057] pending: vmstat_update [ 7261.863624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.868648] pending: vmstat_update [ 7261.871026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.877366] pending: vmstat_update [ 7261.879602] workqueue writeback: flags=0x4a [ 7261.881675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7261.884677] pending: wb_workfn, wb_workfn [ 7261.896555] workqueue xprtiod: flags=0xa [ 7261.901232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7261.907124] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7261.913161] workqueue ipv6_addrconf: flags=0x40008 [ 7261.915712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7261.922318] pending: addrconf_verify_work [ 7261.925289] workqueue knbd0-recv: flags=0x1a [ 7261.930753] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7261.935201] in-flight: 651:recv_work [ 7261.937520] workqueue ptlrpc_pinger: flags=0x2 [ 7261.939995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7261.945353] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7261.950036] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5344s workers=4 idle: 37 31846 28897 [ 7261.956749] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7251s workers=2 idle: 653 [ 7268.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7268.065059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7268.089246] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7268.094792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7268.099219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7268.102333] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7268.112189] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7268.116671] RAX: 000000000000431b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7268.120934] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7268.125091] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7268.128263] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7268.131847] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7268.136066] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7268.139219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7268.143938] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7268.148014] Call Trace: [ 7268.149906] [ 7268.151019] ? show_regs.cold.9+0x22/0x2f [ 7268.153127] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7268.155596] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7268.157752] ? watchdog+0x40/0x40 [ 7268.159241] ? hrtimer_interrupt+0x128/0x2c0 [ 7268.161797] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7268.164164] ? apic_timer_interrupt+0xf/0x20 [ 7268.166108] [ 7268.167124] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7268.170080] do_raw_spin_lock+0x177/0x190 [ 7268.172028] _raw_spin_lock+0x21/0x30 [ 7268.173789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7268.177183] ? do_raw_spin_unlock+0x75/0x190 [ 7268.179163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7268.182175] ? do_wait_intr+0xf0/0xf0 [ 7268.184235] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7268.187638] mdc_close+0x53b/0xd70 [mdc] [ 7268.189990] lmv_close+0x18d/0x490 [lmv] [ 7268.192447] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7268.195459] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7268.197763] ll_md_close+0x274/0x800 [lustre] [ 7268.200310] ll_file_release+0x128/0xc50 [lustre] [ 7268.202777] __fput+0x10d/0x440 [ 7268.204481] ____fput+0x12/0x20 [ 7268.205979] task_work_run+0xc6/0x110 [ 7268.207766] exit_to_usermode_loop+0x1dd/0x1f0 [ 7268.210076] do_syscall_64+0x3d6/0x440 [ 7268.211708] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7268.213979] RIP: 0033:0x7fa749552cd8 [ 7268.215511] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7268.223750] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7268.226727] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7268.229363] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7268.232302] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7268.235535] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7268.238631] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7276.059218] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7276.060542] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7276.065381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.065472] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.065483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.065491] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7276.066057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.066069] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7276.066073] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7276.066078] RAX: 000000000000749a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7276.066080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7276.066082] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7276.066084] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7276.066086] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7276.066089] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7276.066092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.066094] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7276.066103] Call Trace: [ 7276.066113] [ 7276.066119] ? show_regs.cold.9+0x22/0x2f [ 7276.066128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.066134] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.066142] ? watchdog+0x40/0x40 [ 7276.066146] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.066150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.066157] ? apic_timer_interrupt+0xf/0x20 [ 7276.066162] [ 7276.066165] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.066169] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7276.066174] do_raw_spin_lock+0x177/0x190 [ 7276.066179] _raw_spin_lock+0x21/0x30 [ 7276.066183] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7276.068212] process_one_work+0x2c8/0x700 [ 7276.068552] worker_thread+0x296/0x6e0 [ 7276.068561] ? rescuer_thread+0x570/0x570 [ 7276.068565] kthread+0x1d1/0x200 [ 7276.068569] ? set_kthread_struct+0x70/0x70 [ 7276.068573] ret_from_fork+0x1f/0x30 [ 7276.478828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.546619] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.562361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.575521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7276.587886] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7276.609563] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7276.620579] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7276.629241] RDX: 00000000000074be RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7276.636280] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7276.648089] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7276.660633] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7276.670034] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7276.676371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.681826] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7276.690049] Call Trace: [ 7276.693769] [ 7276.694580] ? show_regs.cold.9+0x22/0x2f [ 7276.696708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.701151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.705375] ? watchdog+0x40/0x40 [ 7276.709719] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.712411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.718673] ? apic_timer_interrupt+0xf/0x20 [ 7276.724460] [ 7276.726286] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7276.744132] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7276.761206] do_raw_spin_lock+0x177/0x190 [ 7276.766028] _raw_spin_lock+0x21/0x30 [ 7276.771536] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7276.777986] obd_mod_free+0x46/0x140 [ptlrpc] [ 7276.780360] mdc_commit_open+0x91/0xb0 [mdc] [ 7276.787482] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7276.797958] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7276.803914] after_reply+0xe2c/0x1780 [ptlrpc] [ 7276.810165] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7276.818589] ? __next_timer_interrupt+0x160/0x160 [ 7276.821946] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7276.825585] ? do_wait_intr+0xf0/0xf0 [ 7276.837545] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7276.842152] kthread+0x1d1/0x200 [ 7276.844474] ? set_kthread_struct+0x70/0x70 [ 7276.846950] ret_from_fork+0x1f/0x30 [ 7288.054016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7288.056589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.129565] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.141641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.149213] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7288.154303] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7288.178812] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7288.187150] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7288.193060] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7288.201414] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7288.211511] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7288.215933] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7288.223687] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7288.228770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.235232] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7288.243571] Call Trace: [ 7288.246631] [ 7288.248108] ? show_regs.cold.9+0x22/0x2f [ 7288.254410] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.269603] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.278145] ? watchdog+0x40/0x40 [ 7288.280712] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.285209] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.287703] ? apic_timer_interrupt+0xf/0x20 [ 7288.292725] [ 7288.296519] ? native_safe_halt+0x2e/0x50 [ 7288.297927] kvm_wait+0xcd/0xe0 [ 7288.298992] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7288.302516] do_raw_spin_lock+0x177/0x190 [ 7288.306758] _raw_spin_lock+0x21/0x30 [ 7288.310635] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7288.318406] ? do_raw_spin_unlock+0x75/0x190 [ 7288.322916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7288.330366] ? do_wait_intr+0xf0/0xf0 [ 7288.338600] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7288.344845] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7288.349634] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7288.352929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.355712] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7288.358953] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7288.363603] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7288.367419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.373585] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7288.386823] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7288.392584] lmv_intent_open+0x338/0x1150 [lmv] [ 7288.394949] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.399496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.402709] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7288.405780] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7288.407925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.410752] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7288.412955] ? do_raw_spin_unlock+0x75/0x190 [ 7288.414983] ? _raw_spin_unlock+0x12/0x30 [ 7288.416962] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7288.419886] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7288.423775] ll_atomic_open+0x36a/0xd70 [lustre] [ 7288.428275] lookup_open+0x356/0xb40 [ 7288.431371] do_last+0x313/0xfc0 [ 7288.433658] ? nd_jump_root+0xe5/0x160 [ 7288.435423] ? path_init+0x437/0x520 [ 7288.436873] path_openat+0xf7/0x500 [ 7288.438608] ? do_raw_spin_unlock+0x75/0x190 [ 7288.440371] do_filp_open+0x99/0x140 [ 7288.441790] ? getname_flags+0x6e/0x330 [ 7288.443677] ? __check_object_size+0xff/0x256 [ 7288.445635] ? do_raw_spin_unlock+0x75/0x190 [ 7288.448636] ? _raw_spin_unlock+0x12/0x30 [ 7288.450502] do_sys_openat2+0x2b4/0x410 [ 7288.452414] do_sys_open+0x73/0xa0 [ 7288.454759] __x64_sys_openat+0x24/0x30 [ 7288.457149] do_syscall_64+0xc1/0x440 [ 7288.459534] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7288.462495] RIP: 0033:0x7fa749552332 [ 7288.464740] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7288.477926] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7288.485755] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7288.490525] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7288.494540] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7288.498671] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7288.501964] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7292.384742] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5375s! [ 7292.395387] Showing busy workqueues and worker pools: [ 7292.399850] workqueue events: flags=0x0 [ 7292.403620] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7292.413563] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7292.426771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7292.431157] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7292.441427] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.443788] pending: kfree_rcu_monitor [ 7292.456778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7292.459170] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7292.474758] workqueue events_long: flags=0x0 [ 7292.477755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.481891] pending: lu_objects_flush [obdclass] [ 7292.485625] workqueue events_unbound: flags=0x2 [ 7292.489083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7292.494660] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7292.501223] workqueue events_power_efficient: flags=0x80 [ 7292.510738] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.515821] pending: check_lifetime [ 7292.519717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.527443] pending: neigh_periodic_work [ 7292.529786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.535164] pending: neigh_periodic_work [ 7292.537219] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.539708] pending: do_cache_clean [ 7292.540804] workqueue mm_percpu_wq: flags=0x8 [ 7292.543101] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.548790] pending: vmstat_update [ 7292.555252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.559603] pending: vmstat_update [ 7292.562359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.568367] pending: vmstat_update [ 7292.570892] workqueue writeback: flags=0x4a [ 7292.575332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7292.577128] pending: wb_workfn, wb_workfn [ 7292.580273] workqueue xprtiod: flags=0xa [ 7292.582925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7292.587048] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7292.592400] workqueue ipv6_addrconf: flags=0x40008 [ 7292.594935] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7292.600635] pending: addrconf_verify_work [ 7292.604025] workqueue knbd0-recv: flags=0x1a [ 7292.605664] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7292.611589] in-flight: 651:recv_work [ 7292.613532] workqueue ptlrpc_pinger: flags=0x2 [ 7292.617544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7292.622682] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7292.625505] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5375s workers=4 idle: 37 31846 28897 [ 7292.632020] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7282s workers=2 idle: 653 [ 7296.059015] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7296.061980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7296.099338] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7296.110554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7296.115637] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7296.119387] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7296.132792] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7296.142772] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7296.151409] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7296.163714] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7296.166432] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7296.178060] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7296.184469] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7296.190796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7296.195952] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7296.199766] Call Trace: [ 7296.203323] [ 7296.205050] ? show_regs.cold.9+0x22/0x2f [ 7296.209432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7296.214451] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7296.219529] ? watchdog+0x40/0x40 [ 7296.220972] ? hrtimer_interrupt+0x128/0x2c0 [ 7296.225319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7296.229109] ? apic_timer_interrupt+0xf/0x20 [ 7296.234313] [ 7296.235157] ? native_safe_halt+0x2e/0x50 [ 7296.241919] kvm_wait+0xcd/0xe0 [ 7296.242884] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7296.248347] do_raw_spin_lock+0x177/0x190 [ 7296.251154] _raw_spin_lock+0x21/0x30 [ 7296.252834] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7296.255510] ? do_raw_spin_unlock+0x75/0x190 [ 7296.257657] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7296.269576] ? do_wait_intr+0xf0/0xf0 [ 7296.271656] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7296.277823] mdc_close+0x53b/0xd70 [mdc] [ 7296.280764] lmv_close+0x18d/0x490 [lmv] [ 7296.286189] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7296.291866] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7296.296791] ll_md_close+0x274/0x800 [lustre] [ 7296.301174] ll_file_release+0x128/0xc50 [lustre] [ 7296.303831] __fput+0x10d/0x440 [ 7296.305933] ____fput+0x12/0x20 [ 7296.307531] task_work_run+0xc6/0x110 [ 7296.309138] exit_to_usermode_loop+0x1dd/0x1f0 [ 7296.310799] do_syscall_64+0x3d6/0x440 [ 7296.312266] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7296.314267] RIP: 0033:0x7fa749552cd8 [ 7296.315684] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7296.321682] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7296.324193] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7296.326809] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7296.329806] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7296.333081] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7296.335671] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7304.057024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7304.058322] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7304.061598] Modules linked in: [ 7304.069812] Modules linked in: [ 7304.069815] lustre(O) osp(O) [ 7304.072697] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 7304.076488] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.076631] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.076639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.076645] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7304.080644] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.080692] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.080700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.080703] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7304.080717] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7304.080721] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7304.080726] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7304.080729] RDX: 00000000000079c9 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7304.080731] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7304.080734] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7304.080737] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7304.080741] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7304.080744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.080746] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7304.080755] Call Trace: [ 7304.080761] [ 7304.080767] ? show_regs.cold.9+0x22/0x2f [ 7304.080777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.080784] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.080789] ? watchdog+0x40/0x40 [ 7304.080794] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.080798] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.080807] ? apic_timer_interrupt+0xf/0x20 [ 7304.080812] [ 7304.080814] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7304.080819] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7304.080824] do_raw_spin_lock+0x177/0x190 [ 7304.080829] _raw_spin_lock+0x21/0x30 [ 7304.080836] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7304.081431] obd_mod_free+0x46/0x140 [ptlrpc] [ 7304.081688] mdc_commit_open+0x91/0xb0 [mdc] [ 7304.081725] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7304.083379] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7304.083642] after_reply+0xe2c/0x1780 [ptlrpc] [ 7304.083856] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7304.084415] ? __next_timer_interrupt+0x160/0x160 [ 7304.084426] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7304.084699] ? do_wait_intr+0xf0/0xf0 [ 7304.084708] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7304.100471] [ 7304.100479] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7304.100498] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7304.100502] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7304.100507] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7304.100509] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7304.100511] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7304.100513] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7304.100516] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7304.100518] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7304.100564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.100566] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7304.100574] Call Trace: [ 7304.100578] [ 7304.100584] ? show_regs.cold.9+0x22/0x2f [ 7304.100592] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.100599] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.136447] kthread+0x1d1/0x200 [ 7304.140779] ? watchdog+0x40/0x40 [ 7304.140799] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.140805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.140814] ? apic_timer_interrupt+0xf/0x20 [ 7304.140818] [ 7304.140823] ? native_safe_halt+0x2e/0x50 [ 7304.140828] kvm_wait+0xcd/0xe0 [ 7304.140836] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7304.140846] do_raw_spin_lock+0x177/0x190 [ 7304.140851] _raw_spin_lock+0x21/0x30 [ 7304.140856] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7304.141233] process_one_work+0x2c8/0x700 [ 7304.141239] worker_thread+0x296/0x6e0 [ 7304.141242] ? rescuer_thread+0x570/0x570 [ 7304.141245] kthread+0x1d1/0x200 [ 7304.141248] ? set_kthread_struct+0x70/0x70 [ 7304.141250] ret_from_fork+0x1f/0x30 [ 7304.541467] ? set_kthread_struct+0x70/0x70 [ 7304.543681] ret_from_fork+0x1f/0x30 [ 7320.054024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7320.059309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7320.082124] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7320.086255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7320.089709] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7320.091556] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7320.098253] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7320.101429] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7320.104393] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7320.107560] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7320.110681] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7320.113582] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7320.116306] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7320.119038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7320.121364] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7320.123953] Call Trace: [ 7320.124738] [ 7320.125663] ? show_regs.cold.9+0x22/0x2f [ 7320.128303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7320.130332] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7320.132124] ? watchdog+0x40/0x40 [ 7320.133345] ? hrtimer_interrupt+0x128/0x2c0 [ 7320.134911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7320.136923] ? apic_timer_interrupt+0xf/0x20 [ 7320.138247] [ 7320.138850] ? native_safe_halt+0x2e/0x50 [ 7320.140116] kvm_wait+0xcd/0xe0 [ 7320.141249] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7320.143239] do_raw_spin_lock+0x177/0x190 [ 7320.144630] _raw_spin_lock+0x21/0x30 [ 7320.145718] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7320.147347] ? do_raw_spin_unlock+0x75/0x190 [ 7320.148454] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7320.150283] ? do_wait_intr+0xf0/0xf0 [ 7320.151235] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7320.152993] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7320.154947] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7320.156838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7320.158873] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7320.160447] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7320.161929] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7320.163330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7320.165005] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7320.166851] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7320.168676] lmv_intent_open+0x338/0x1150 [lmv] [ 7320.170104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7320.172341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7320.174371] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7320.176007] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7320.177401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7320.179415] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7320.181152] ? do_raw_spin_unlock+0x75/0x190 [ 7320.182377] ? _raw_spin_unlock+0x12/0x30 [ 7320.183919] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7320.185990] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7320.188053] ll_atomic_open+0x36a/0xd70 [lustre] [ 7320.190009] lookup_open+0x356/0xb40 [ 7320.191528] do_last+0x313/0xfc0 [ 7320.192670] ? nd_jump_root+0xe5/0x160 [ 7320.194680] ? path_init+0x437/0x520 [ 7320.196106] path_openat+0xf7/0x500 [ 7320.197570] ? do_raw_spin_unlock+0x75/0x190 [ 7320.199263] do_filp_open+0x99/0x140 [ 7320.201026] ? getname_flags+0x6e/0x330 [ 7320.202548] ? __check_object_size+0xff/0x256 [ 7320.204515] ? do_raw_spin_unlock+0x75/0x190 [ 7320.206227] ? _raw_spin_unlock+0x12/0x30 [ 7320.207841] do_sys_openat2+0x2b4/0x410 [ 7320.209260] do_sys_open+0x73/0xa0 [ 7320.210416] __x64_sys_openat+0x24/0x30 [ 7320.211804] do_syscall_64+0xc1/0x440 [ 7320.213161] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7320.214944] RIP: 0033:0x7fa749552332 [ 7320.216123] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7320.232713] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7320.237578] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7320.245017] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7320.251901] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7320.258129] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7320.270390] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7323.112650] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5405s! [ 7323.120958] Showing busy workqueues and worker pools: [ 7323.126837] workqueue events: flags=0x0 [ 7323.128776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7323.135872] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7323.145507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7323.151871] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7323.164897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.169630] pending: kfree_rcu_monitor [ 7323.174727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7323.181107] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7323.190313] workqueue events_long: flags=0x0 [ 7323.192843] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.201446] pending: lu_objects_flush [obdclass] [ 7323.203677] workqueue events_unbound: flags=0x2 [ 7323.205496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7323.212563] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7323.217861] workqueue events_power_efficient: flags=0x80 [ 7323.220732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.223470] pending: check_lifetime [ 7323.225623] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.228676] pending: neigh_periodic_work [ 7323.231876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.238349] pending: neigh_periodic_work [ 7323.241136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.243406] pending: do_cache_clean [ 7323.244580] workqueue mm_percpu_wq: flags=0x8 [ 7323.248154] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.251947] pending: vmstat_update [ 7323.253916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.260984] pending: vmstat_update [ 7323.266358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7323.272096] pending: vmstat_update [ 7323.273508] workqueue writeback: flags=0x4a [ 7323.280923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7323.288501] pending: wb_workfn, wb_workfn [ 7323.291537] workqueue xprtiod: flags=0xa [ 7323.294021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7323.299540] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7323.308656] workqueue ipv6_addrconf: flags=0x40008 [ 7323.311131] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7323.319267] pending: addrconf_verify_work [ 7323.321446] workqueue knbd0-recv: flags=0x1a [ 7323.323988] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7323.327618] in-flight: 651:recv_work [ 7323.329857] workqueue ptlrpc_pinger: flags=0x2 [ 7323.332508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7323.335904] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7323.340073] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5406s workers=4 idle: 37 31846 28897 [ 7323.351363] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7313s workers=2 idle: 653 [ 7324.065947] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7324.072890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7324.098918] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7324.111178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7324.116149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7324.123875] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7324.137718] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7324.141496] RAX: 0000000000006984 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7324.145846] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7324.153616] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7324.162802] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7324.165388] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7324.169951] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7324.174809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7324.178150] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7324.182004] Call Trace: [ 7324.183300] [ 7324.184108] ? show_regs.cold.9+0x22/0x2f [ 7324.186023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7324.189189] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7324.190990] ? watchdog+0x40/0x40 [ 7324.193634] ? hrtimer_interrupt+0x128/0x2c0 [ 7324.200566] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7324.202128] ? apic_timer_interrupt+0xf/0x20 [ 7324.204618] [ 7324.205702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7324.209158] do_raw_spin_lock+0x177/0x190 [ 7324.211925] _raw_spin_lock+0x21/0x30 [ 7324.216813] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7324.223191] ? do_raw_spin_unlock+0x75/0x190 [ 7324.225142] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7324.228355] ? do_wait_intr+0xf0/0xf0 [ 7324.232360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7324.238184] mdc_close+0x53b/0xd70 [mdc] [ 7324.239885] lmv_close+0x18d/0x490 [lmv] [ 7324.241357] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7324.246632] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7324.249272] ll_md_close+0x274/0x800 [lustre] [ 7324.252624] ll_file_release+0x128/0xc50 [lustre] [ 7324.257366] __fput+0x10d/0x440 [ 7324.258695] ____fput+0x12/0x20 [ 7324.260176] task_work_run+0xc6/0x110 [ 7324.261874] exit_to_usermode_loop+0x1dd/0x1f0 [ 7324.264027] do_syscall_64+0x3d6/0x440 [ 7324.267048] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7324.271214] RIP: 0033:0x7fa749552cd8 [ 7324.272646] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7324.285353] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7324.288932] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7324.298051] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7324.300890] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7324.304182] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7324.315984] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7332.057016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7332.058295] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7332.065921] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 7332.071316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.071390] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.071398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.071400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7332.071413] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7332.071418] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7332.071423] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7332.071425] RDX: 0000000000006922 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7332.071428] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7332.071430] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7332.071432] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7332.071435] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7332.071438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.071441] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7332.071449] Call Trace: [ 7332.071453] [ 7332.071458] ? show_regs.cold.9+0x22/0x2f [ 7332.071467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.071473] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.071477] ? watchdog+0x40/0x40 [ 7332.071481] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.071486] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.071493] ? apic_timer_interrupt+0xf/0x20 [ 7332.071498] [ 7332.071500] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7332.071505] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7332.071509] do_raw_spin_lock+0x177/0x190 [ 7332.071514] _raw_spin_lock+0x21/0x30 [ 7332.071519] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7332.071817] obd_mod_free+0x46/0x140 [ptlrpc] [ 7332.072009] mdc_commit_open+0x91/0xb0 [mdc] [ 7332.072044] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7332.072229] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7332.072420] after_reply+0xe2c/0x1780 [ptlrpc] [ 7332.072627] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7332.072824] ? __next_timer_interrupt+0x160/0x160 [ 7332.072834] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7332.079216] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.079266] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.079271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.079277] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7332.103897] ? do_wait_intr+0xf0/0xf0 [ 7332.103920] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7332.104214] kthread+0x1d1/0x200 [ 7332.104220] ? set_kthread_struct+0x70/0x70 [ 7332.104223] ret_from_fork+0x1f/0x30 [ 7332.306500] [ 7332.307129] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.313132] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7332.320856] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7332.324075] RAX: 0000000000007475 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7332.327357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7332.330868] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7332.334870] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7332.340233] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7332.343977] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7332.349879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.353707] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7332.356151] Call Trace: [ 7332.357652] [ 7332.358319] ? show_regs.cold.9+0x22/0x2f [ 7332.359892] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.365069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.366544] ? watchdog+0x40/0x40 [ 7332.367433] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.369322] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.371914] ? apic_timer_interrupt+0xf/0x20 [ 7332.373974] [ 7332.374652] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.376134] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7332.382490] do_raw_spin_lock+0x177/0x190 [ 7332.384852] _raw_spin_lock+0x21/0x30 [ 7332.387225] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7332.389823] process_one_work+0x2c8/0x700 [ 7332.390909] worker_thread+0x296/0x6e0 [ 7332.392171] ? rescuer_thread+0x570/0x570 [ 7332.395354] kthread+0x1d1/0x200 [ 7332.397628] ? set_kthread_struct+0x70/0x70 [ 7332.400420] ret_from_fork+0x1f/0x30 [ 7348.056030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7348.063479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7348.097217] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7348.104102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7348.108092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7348.114973] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7348.126352] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7348.131863] RAX: 000000000000788b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7348.136174] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7348.140569] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7348.148061] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7348.151166] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7348.157752] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7348.161610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7348.166692] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7348.173244] Call Trace: [ 7348.175250] [ 7348.175910] ? show_regs.cold.9+0x22/0x2f [ 7348.182332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7348.185849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7348.188071] ? watchdog+0x40/0x40 [ 7348.189266] ? hrtimer_interrupt+0x128/0x2c0 [ 7348.191287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7348.192814] ? apic_timer_interrupt+0xf/0x20 [ 7348.194431] [ 7348.195030] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7348.196891] do_raw_spin_lock+0x177/0x190 [ 7348.198009] _raw_spin_lock+0x21/0x30 [ 7348.199368] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7348.201348] ? do_raw_spin_unlock+0x75/0x190 [ 7348.204775] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7348.212434] ? do_wait_intr+0xf0/0xf0 [ 7348.213964] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7348.220338] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7348.226211] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7348.230270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7348.232126] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7348.234837] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7348.236769] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7348.238580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7348.240660] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7348.243289] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7348.245604] lmv_intent_open+0x338/0x1150 [lmv] [ 7348.247275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7348.249859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7348.251719] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7348.253437] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7348.254911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7348.256960] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7348.258922] ? do_raw_spin_unlock+0x75/0x190 [ 7348.260388] ? _raw_spin_unlock+0x12/0x30 [ 7348.261894] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7348.264063] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7348.266248] ll_atomic_open+0x36a/0xd70 [lustre] [ 7348.268288] lookup_open+0x356/0xb40 [ 7348.269734] do_last+0x313/0xfc0 [ 7348.270923] ? nd_jump_root+0xe5/0x160 [ 7348.272632] ? path_init+0x437/0x520 [ 7348.274055] path_openat+0xf7/0x500 [ 7348.275560] ? do_raw_spin_unlock+0x75/0x190 [ 7348.276907] do_filp_open+0x99/0x140 [ 7348.278059] ? getname_flags+0x6e/0x330 [ 7348.279723] ? __check_object_size+0xff/0x256 [ 7348.281245] ? do_raw_spin_unlock+0x75/0x190 [ 7348.283402] ? _raw_spin_unlock+0x12/0x30 [ 7348.284911] do_sys_openat2+0x2b4/0x410 [ 7348.286344] do_sys_open+0x73/0xa0 [ 7348.287581] __x64_sys_openat+0x24/0x30 [ 7348.288756] do_syscall_64+0xc1/0x440 [ 7348.289834] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7348.291836] RIP: 0033:0x7fa749552332 [ 7348.293115] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7348.299800] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7348.304133] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7348.307093] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7348.309662] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7348.312484] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7348.315850] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7352.071275] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7352.076014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7352.114361] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7352.121843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7352.129606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7352.132577] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7352.142958] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7352.145696] RAX: 0000000000007a85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7352.152766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7352.157749] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7352.165245] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7352.168889] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7352.172017] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7352.175456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7352.177788] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7352.181289] Call Trace: [ 7352.184487] [ 7352.185466] ? show_regs.cold.9+0x22/0x2f [ 7352.187517] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7352.189529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7352.191531] ? watchdog+0x40/0x40 [ 7352.192875] ? hrtimer_interrupt+0x128/0x2c0 [ 7352.194711] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7352.197406] ? apic_timer_interrupt+0xf/0x20 [ 7352.200537] [ 7352.201481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7352.204044] do_raw_spin_lock+0x177/0x190 [ 7352.206483] _raw_spin_lock+0x21/0x30 [ 7352.209629] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7352.212422] ? do_raw_spin_unlock+0x75/0x190 [ 7352.214555] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7352.216779] ? do_wait_intr+0xf0/0xf0 [ 7352.218443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7352.223612] mdc_close+0x53b/0xd70 [mdc] [ 7352.225380] lmv_close+0x18d/0x490 [lmv] [ 7352.226995] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7352.232814] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7352.235514] ll_md_close+0x274/0x800 [lustre] [ 7352.237866] ll_file_release+0x128/0xc50 [lustre] [ 7352.240170] __fput+0x10d/0x440 [ 7352.241250] ____fput+0x12/0x20 [ 7352.242361] task_work_run+0xc6/0x110 [ 7352.244615] exit_to_usermode_loop+0x1dd/0x1f0 [ 7352.246655] do_syscall_64+0x3d6/0x440 [ 7352.250253] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7352.251901] RIP: 0033:0x7fa749552cd8 [ 7352.253975] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7352.264293] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7352.267654] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7352.271057] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7352.274472] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7352.280114] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7352.284099] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7353.833169] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5436s! [ 7353.838252] Showing busy workqueues and worker pools: [ 7353.842464] workqueue events: flags=0x0 [ 7353.844982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7353.849700] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7353.861280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7353.873213] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7353.883664] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7353.890462] pending: kfree_rcu_monitor [ 7353.896087] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7353.907227] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7353.921206] workqueue events_long: flags=0x0 [ 7353.923658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7353.929240] pending: lu_objects_flush [obdclass] [ 7353.932945] workqueue events_unbound: flags=0x2 [ 7353.937049] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7353.944664] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7353.955166] workqueue events_power_efficient: flags=0x80 [ 7353.962875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7353.970412] pending: check_lifetime [ 7353.976222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7353.984387] pending: neigh_periodic_work [ 7353.986828] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7353.995493] pending: neigh_periodic_work [ 7354.001118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7354.007330] pending: do_cache_clean [ 7354.014244] workqueue mm_percpu_wq: flags=0x8 [ 7354.023197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7354.033260] pending: vmstat_update [ 7354.040111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7354.050315] pending: vmstat_update [ 7354.056825] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7354.072205] pending: vmstat_update [ 7354.080962] workqueue writeback: flags=0x4a [ 7354.084871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7354.095815] pending: wb_workfn, wb_workfn [ 7354.100249] workqueue xprtiod: flags=0xa [ 7354.102729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7354.109300] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7354.127423] workqueue ipv6_addrconf: flags=0x40008 [ 7354.137017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7354.145665] pending: addrconf_verify_work [ 7354.152512] workqueue knbd0-recv: flags=0x1a [ 7354.154969] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7354.161169] in-flight: 651:recv_work [ 7354.163930] workqueue ptlrpc_pinger: flags=0x2 [ 7354.172018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7354.177982] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7354.181059] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5437s workers=4 idle: 37 31846 28897 [ 7354.185159] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7344s workers=2 idle: 653 [ 7360.059342] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7360.059545] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7360.063495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.063577] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.063585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.063588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7360.063602] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7360.063608] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7360.063613] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7360.063615] RDX: 0000000000007b3e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7360.063617] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7360.063619] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7360.063622] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7360.063625] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7360.063627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.063629] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7360.063854] Call Trace: [ 7360.063861] [ 7360.063867] ? show_regs.cold.9+0x22/0x2f [ 7360.063887] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.063893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.063897] ? watchdog+0x40/0x40 [ 7360.063901] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.063905] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.063913] ? apic_timer_interrupt+0xf/0x20 [ 7360.063918] [ 7360.063921] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7360.063926] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7360.063930] do_raw_spin_lock+0x177/0x190 [ 7360.063935] _raw_spin_lock+0x21/0x30 [ 7360.063941] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7360.064374] obd_mod_free+0x46/0x140 [ptlrpc] [ 7360.064572] mdc_commit_open+0x91/0xb0 [mdc] [ 7360.064612] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7360.065016] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7360.065310] after_reply+0xe2c/0x1780 [ptlrpc] [ 7360.065520] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7360.065713] ? __next_timer_interrupt+0x160/0x160 [ 7360.065724] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7360.066290] ? do_wait_intr+0xf0/0xf0 [ 7360.066302] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7360.066516] kthread+0x1d1/0x200 [ 7360.066522] ? set_kthread_struct+0x70/0x70 [ 7360.066525] ret_from_fork+0x1f/0x30 [ 7360.295498] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.339885] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.353448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.362480] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7360.366540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.372132] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7360.393710] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7360.402067] RAX: 0000000000005e37 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7360.407753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7360.416257] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7360.424215] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7360.431939] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7360.436097] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7360.446669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.449905] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7360.465493] Call Trace: [ 7360.467558] [ 7360.469596] ? show_regs.cold.9+0x22/0x2f [ 7360.473656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.477367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.482958] ? watchdog+0x40/0x40 [ 7360.487014] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.493051] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.502223] ? apic_timer_interrupt+0xf/0x20 [ 7360.506362] [ 7360.508921] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.518427] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7360.525636] do_raw_spin_lock+0x177/0x190 [ 7360.528438] _raw_spin_lock+0x21/0x30 [ 7360.531135] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7360.536931] process_one_work+0x2c8/0x700 [ 7360.541384] worker_thread+0x296/0x6e0 [ 7360.543047] ? rescuer_thread+0x570/0x570 [ 7360.548945] kthread+0x1d1/0x200 [ 7360.554345] ? set_kthread_struct+0x70/0x70 [ 7360.561587] ret_from_fork+0x1f/0x30 [ 7373.813449] rcu: INFO: rcu_sched self-detected stall on CPU [ 7373.817917] rcu: 2-...!: (3023043 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 7373.824023] (t=5460124 jiffies g=258701 q=3022) [ 7373.828598] rcu: rcu_sched kthread starved for 5456919 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7373.836175] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7373.842149] rcu: RCU grace-period kthread stack dump: [ 7373.849325] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7373.857209] Call Trace: [ 7373.861748] __schedule+0x351/0xcb0 [ 7373.863135] schedule+0xc0/0x180 [ 7373.864836] schedule_timeout+0xb4/0x190 [ 7373.871161] ? do_raw_spin_unlock+0x75/0x190 [ 7373.885527] ? __next_timer_interrupt+0x160/0x160 [ 7373.892701] ? prepare_to_swait_event+0xf3/0x140 [ 7373.899505] rcu_gp_kthread+0x8dd/0x15b0 [ 7373.901492] ? force_qs_rnp+0x2c0/0x2c0 [ 7373.903417] kthread+0x1d1/0x200 [ 7373.906398] ? set_kthread_struct+0x70/0x70 [ 7373.907801] ret_from_fork+0x1f/0x30 [ 7373.914332] rcu: Stack dump where RCU GP kthread last ran: [ 7373.920333] NMI backtrace for cpu 2 [ 7373.928196] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7373.940966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7373.959757] Call Trace: [ 7373.961360] [ 7373.962581] ? dump_stack+0xbb/0x10e [ 7373.967173] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7373.973489] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7373.978634] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7373.983470] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7373.992857] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7373.997670] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7374.005006] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7374.006626] ? update_process_times+0x6d/0xc0 [ 7374.014249] ? tick_sched_handle.isra.18+0x27/0x90 [ 7374.020492] ? tick_sched_timer+0x53/0xb0 [ 7374.026780] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7374.031243] ? tick_sched_handle.isra.18+0x90/0x90 [ 7374.036111] ? hrtimer_interrupt+0x128/0x2c0 [ 7374.042494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7374.047473] ? apic_timer_interrupt+0xf/0x20 [ 7374.049226] [ 7374.051791] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7374.062177] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7374.067577] ? do_raw_spin_lock+0x177/0x190 [ 7374.070877] ? _raw_spin_lock+0x21/0x30 [ 7374.072672] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7374.077923] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7374.082183] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7374.088306] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7374.097656] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7374.099792] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7374.115147] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7374.129735] ? __next_timer_interrupt+0x160/0x160 [ 7374.134747] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7374.139839] ? do_wait_intr+0xf0/0xf0 [ 7374.146456] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7374.151746] ? kthread+0x1d1/0x200 [ 7374.154741] ? set_kthread_struct+0x70/0x70 [ 7374.166296] ? ret_from_fork+0x1f/0x30 [ 7374.168463] NMI backtrace for cpu 2 [ 7374.173790] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7374.185394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7374.195302] Call Trace: [ 7374.199355] [ 7374.201949] ? dump_stack+0xbb/0x10e [ 7374.205634] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7374.208502] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7374.214773] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7374.221273] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7374.227196] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7374.230005] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7374.237177] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7374.242001] ? update_process_times+0x6d/0xc0 [ 7374.247181] ? tick_sched_handle.isra.18+0x27/0x90 [ 7374.249327] ? tick_sched_timer+0x53/0xb0 [ 7374.253612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7374.262366] ? tick_sched_handle.isra.18+0x90/0x90 [ 7374.266816] ? hrtimer_interrupt+0x128/0x2c0 [ 7374.273857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7374.278486] ? apic_timer_interrupt+0xf/0x20 [ 7374.284114] [ 7374.286248] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7374.289496] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7374.300619] ? do_raw_spin_lock+0x177/0x190 [ 7374.307744] ? _raw_spin_lock+0x21/0x30 [ 7374.312065] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7374.317456] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7374.323759] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7374.330979] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7374.336425] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7374.347727] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7374.357272] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7374.364011] ? __next_timer_interrupt+0x160/0x160 [ 7374.368584] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7374.373841] ? do_wait_intr+0xf0/0xf0 [ 7374.385233] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7374.391553] ? kthread+0x1d1/0x200 [ 7374.394311] ? set_kthread_struct+0x70/0x70 [ 7374.401139] ? ret_from_fork+0x1f/0x30 [ 7380.060537] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7380.067400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7380.070023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7380.114597] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7380.114613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7380.114616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7380.115319] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7380.115323] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7380.115329] RAX: 0000000000001241 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7380.115332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7380.115334] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7380.115336] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7380.115338] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7380.115342] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7380.115345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7380.115347] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7380.115356] Call Trace: [ 7380.115362] [ 7380.115368] ? show_regs.cold.9+0x22/0x2f [ 7380.115378] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7380.115385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7380.115389] ? watchdog+0x40/0x40 [ 7380.115394] ? hrtimer_interrupt+0x128/0x2c0 [ 7380.115398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7380.115420] ? apic_timer_interrupt+0xf/0x20 [ 7380.115424] [ 7380.115427] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7380.115435] do_raw_spin_lock+0x177/0x190 [ 7380.115440] _raw_spin_lock+0x21/0x30 [ 7380.115445] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7380.115776] ? do_raw_spin_unlock+0x75/0x190 [ 7380.115783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7380.115994] ? do_wait_intr+0xf0/0xf0 [ 7380.116002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7380.116331] mdc_close+0x53b/0xd70 [mdc] [ 7380.116369] lmv_close+0x18d/0x490 [lmv] [ 7380.116399] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7380.116566] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7380.117886] ll_md_close+0x274/0x800 [lustre] [ 7380.118529] ll_file_release+0x128/0xc50 [lustre] [ 7380.118609] __fput+0x10d/0x440 [ 7380.118618] ____fput+0x12/0x20 [ 7380.118622] task_work_run+0xc6/0x110 [ 7380.118629] exit_to_usermode_loop+0x1dd/0x1f0 [ 7380.118636] do_syscall_64+0x3d6/0x440 [ 7380.118917] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7380.118926] RIP: 0033:0x7fa749552cd8 [ 7380.118934] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7380.118939] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7380.118944] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7380.118947] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7380.118949] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7380.118951] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7380.118953] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7380.498754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7380.568000] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7380.577224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7380.583256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7380.587507] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7380.602075] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7380.614612] RAX: 000000000000308f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7380.623533] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7380.634153] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7380.645002] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7380.653437] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7380.667878] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7380.685982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7380.704561] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7380.724509] Call Trace: [ 7380.734390] [ 7380.736315] ? show_regs.cold.9+0x22/0x2f [ 7380.741742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7380.749934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7380.756769] ? watchdog+0x40/0x40 [ 7380.762338] ? hrtimer_interrupt+0x128/0x2c0 [ 7380.775840] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7380.781325] ? apic_timer_interrupt+0xf/0x20 [ 7380.789612] [ 7380.796748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7380.802872] do_raw_spin_lock+0x177/0x190 [ 7380.808962] _raw_spin_lock+0x21/0x30 [ 7380.810384] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7380.820340] ? do_raw_spin_unlock+0x75/0x190 [ 7380.825375] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7380.833933] ? do_wait_intr+0xf0/0xf0 [ 7380.838548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7380.843946] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7380.853948] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7380.861201] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7380.866831] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7380.874109] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7380.878362] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7380.885932] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7380.892941] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7380.899697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7380.907807] lmv_intent_open+0x338/0x1150 [lmv] [ 7380.912270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7380.920148] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7380.928258] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7380.935585] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7380.940983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7380.944929] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7380.950673] ? do_raw_spin_unlock+0x75/0x190 [ 7380.954057] ? _raw_spin_unlock+0x12/0x30 [ 7380.958826] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7380.960590] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7380.964147] ll_atomic_open+0x36a/0xd70 [lustre] [ 7380.970042] lookup_open+0x356/0xb40 [ 7380.974143] do_last+0x313/0xfc0 [ 7380.976224] ? nd_jump_root+0xe5/0x160 [ 7380.979379] ? path_init+0x437/0x520 [ 7380.983436] path_openat+0xf7/0x500 [ 7380.987268] ? do_raw_spin_unlock+0x75/0x190 [ 7380.991218] do_filp_open+0x99/0x140 [ 7380.995321] ? getname_flags+0x6e/0x330 [ 7380.999137] ? __check_object_size+0xff/0x256 [ 7381.002732] ? do_raw_spin_unlock+0x75/0x190 [ 7381.004154] ? _raw_spin_unlock+0x12/0x30 [ 7381.005523] do_sys_openat2+0x2b4/0x410 [ 7381.016718] do_sys_open+0x73/0xa0 [ 7381.021179] __x64_sys_openat+0x24/0x30 [ 7381.028072] do_syscall_64+0xc1/0x440 [ 7381.039178] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7381.045798] RIP: 0033:0x7fa749552332 [ 7381.053279] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7381.074214] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7381.086357] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7381.091861] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7381.100997] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7381.108817] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7381.115059] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7384.556458] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5467s! [ 7384.569572] Showing busy workqueues and worker pools: [ 7384.574518] workqueue events: flags=0x0 [ 7384.580930] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7384.588658] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7384.599566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7384.607254] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7384.620226] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.632879] pending: kfree_rcu_monitor [ 7384.643186] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7384.653090] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7384.669248] workqueue events_long: flags=0x0 [ 7384.679884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.682242] pending: lu_objects_flush [obdclass] [ 7384.694069] workqueue events_unbound: flags=0x2 [ 7384.702669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7384.707468] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7384.713081] workqueue events_power_efficient: flags=0x80 [ 7384.727706] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.739464] pending: check_lifetime [ 7384.742278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.751749] pending: neigh_periodic_work [ 7384.753299] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.763375] pending: neigh_periodic_work [ 7384.766289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.768609] pending: do_cache_clean [ 7384.775033] workqueue mm_percpu_wq: flags=0x8 [ 7384.777293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.785013] pending: vmstat_update [ 7384.790506] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.797639] pending: vmstat_update [ 7384.800385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7384.810052] pending: vmstat_update [ 7384.816213] workqueue writeback: flags=0x4a [ 7384.822907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7384.827169] pending: wb_workfn, wb_workfn [ 7384.829150] workqueue xprtiod: flags=0xa [ 7384.831919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7384.836216] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7384.845411] workqueue ipv6_addrconf: flags=0x40008 [ 7384.848142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7384.852176] pending: addrconf_verify_work [ 7384.854935] workqueue knbd0-recv: flags=0x1a [ 7384.858494] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7384.865087] in-flight: 651:recv_work [ 7384.867239] workqueue ptlrpc_pinger: flags=0x2 [ 7384.876377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7384.884150] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7384.893486] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5467s workers=4 idle: 37 31846 28897 [ 7384.904259] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7374s workers=2 idle: 653 [ 7388.082598] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7388.104449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7388.204693] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7388.221416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7388.237485] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7388.244343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7388.251762] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7388.285616] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7388.296144] RAX: 0000000000007e83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7388.301636] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7388.314730] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7388.329910] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7388.337161] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7388.346520] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7388.356367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7388.368222] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7388.377482] Call Trace: [ 7388.380611] [ 7388.381353] ? show_regs.cold.9+0x22/0x2f [ 7388.390250] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7388.393885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7388.397591] ? watchdog+0x40/0x40 [ 7388.402418] ? hrtimer_interrupt+0x128/0x2c0 [ 7388.405961] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7388.425824] ? apic_timer_interrupt+0xf/0x20 [ 7388.431228] [ 7388.437360] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7388.443428] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7388.449270] do_raw_spin_lock+0x177/0x190 [ 7388.450721] _raw_spin_lock+0x21/0x30 [ 7388.461556] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7388.470833] process_one_work+0x2c8/0x700 [ 7388.476644] worker_thread+0x296/0x6e0 [ 7388.483342] ? rescuer_thread+0x570/0x570 [ 7388.487153] kthread+0x1d1/0x200 [ 7388.488424] ? set_kthread_struct+0x70/0x70 [ 7388.491239] ret_from_fork+0x1f/0x30 [ 7400.058061] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7400.069847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7400.127070] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7400.134932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7400.143233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7400.151401] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7400.183501] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7400.190978] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7400.197848] RDX: 000000000000529f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7400.200310] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7400.204781] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7400.207272] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7400.212027] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7400.226317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7400.230876] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7400.237674] Call Trace: [ 7400.238607] [ 7400.239361] ? show_regs.cold.9+0x22/0x2f [ 7400.246275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7400.253906] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7400.264487] ? watchdog+0x40/0x40 [ 7400.267414] ? hrtimer_interrupt+0x128/0x2c0 [ 7400.273272] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7400.276978] ? apic_timer_interrupt+0xf/0x20 [ 7400.279317] [ 7400.283380] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7400.289404] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7400.300569] do_raw_spin_lock+0x177/0x190 [ 7400.304492] _raw_spin_lock+0x21/0x30 [ 7400.306492] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7400.309377] obd_mod_free+0x46/0x140 [ptlrpc] [ 7400.318577] mdc_commit_open+0x91/0xb0 [mdc] [ 7400.323303] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7400.333327] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7400.340705] after_reply+0xe2c/0x1780 [ptlrpc] [ 7400.345576] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7400.353044] ? __next_timer_interrupt+0x160/0x160 [ 7400.359181] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7400.361732] ? do_wait_intr+0xf0/0xf0 [ 7400.363888] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7400.368986] kthread+0x1d1/0x200 [ 7400.373908] ? set_kthread_struct+0x70/0x70 [ 7400.375805] ret_from_fork+0x1f/0x30 [ 7408.059060] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7408.061935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7408.100160] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7408.124144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7408.138902] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7408.147699] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7408.171083] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7408.180256] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7408.189354] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7408.194456] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7408.202561] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7408.208253] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7408.227895] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7408.238798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7408.247543] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7408.254724] Call Trace: [ 7408.256886] [ 7408.259919] ? show_regs.cold.9+0x22/0x2f [ 7408.266666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7408.271759] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7408.278587] ? watchdog+0x40/0x40 [ 7408.281763] ? hrtimer_interrupt+0x128/0x2c0 [ 7408.284277] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7408.288176] ? apic_timer_interrupt+0xf/0x20 [ 7408.293367] [ 7408.294528] ? native_safe_halt+0x2e/0x50 [ 7408.296657] kvm_wait+0xcd/0xe0 [ 7408.300360] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7408.304640] do_raw_spin_lock+0x177/0x190 [ 7408.310734] _raw_spin_lock+0x21/0x30 [ 7408.312852] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7408.317387] ? do_raw_spin_unlock+0x75/0x190 [ 7408.318807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7408.320832] ? do_wait_intr+0xf0/0xf0 [ 7408.321891] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7408.323504] mdc_close+0x53b/0xd70 [mdc] [ 7408.325592] lmv_close+0x18d/0x490 [lmv] [ 7408.327812] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7408.331428] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7408.335339] ll_md_close+0x274/0x800 [lustre] [ 7408.339291] ll_file_release+0x128/0xc50 [lustre] [ 7408.342172] __fput+0x10d/0x440 [ 7408.344076] ____fput+0x12/0x20 [ 7408.346098] task_work_run+0xc6/0x110 [ 7408.348069] exit_to_usermode_loop+0x1dd/0x1f0 [ 7408.350042] do_syscall_64+0x3d6/0x440 [ 7408.356989] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7408.369150] RIP: 0033:0x7fa749552cd8 [ 7408.372787] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7408.389457] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7408.392608] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7408.395100] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7408.398983] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7408.404101] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7408.408585] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7412.055547] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7412.063581] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7412.126405] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7412.142915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7412.159772] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7412.174303] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7412.196795] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7412.208337] RAX: 0000000000007c05 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7412.223599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7412.236410] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7412.240760] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7412.252346] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7412.259093] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7412.265557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7412.267707] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7412.276117] Call Trace: [ 7412.279519] [ 7412.280671] ? show_regs.cold.9+0x22/0x2f [ 7412.291386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7412.293222] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7412.303648] ? watchdog+0x40/0x40 [ 7412.306285] ? hrtimer_interrupt+0x128/0x2c0 [ 7412.309450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7412.315155] ? apic_timer_interrupt+0xf/0x20 [ 7412.319352] [ 7412.320421] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7412.325167] do_raw_spin_lock+0x177/0x190 [ 7412.332267] _raw_spin_lock+0x21/0x30 [ 7412.333688] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7412.345914] ? do_raw_spin_unlock+0x75/0x190 [ 7412.354475] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7412.357514] ? do_wait_intr+0xf0/0xf0 [ 7412.360708] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7412.365016] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7412.366501] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7412.373328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7412.376197] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7412.380834] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7412.383870] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7412.389724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7412.393057] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7412.395015] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7412.401154] lmv_intent_open+0x338/0x1150 [lmv] [ 7412.414305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7412.416335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7412.424254] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7412.433450] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7412.437171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7412.448425] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7412.456498] ? do_raw_spin_unlock+0x75/0x190 [ 7412.461487] ? _raw_spin_unlock+0x12/0x30 [ 7412.464102] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7412.467327] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7412.476064] ll_atomic_open+0x36a/0xd70 [lustre] [ 7412.479582] lookup_open+0x356/0xb40 [ 7412.482617] do_last+0x313/0xfc0 [ 7412.495272] ? nd_jump_root+0xe5/0x160 [ 7412.497436] ? path_init+0x437/0x520 [ 7412.501073] path_openat+0xf7/0x500 [ 7412.503307] ? do_raw_spin_unlock+0x75/0x190 [ 7412.507159] do_filp_open+0x99/0x140 [ 7412.509175] ? getname_flags+0x6e/0x330 [ 7412.511329] ? __check_object_size+0xff/0x256 [ 7412.517903] ? do_raw_spin_unlock+0x75/0x190 [ 7412.521749] ? _raw_spin_unlock+0x12/0x30 [ 7412.525280] do_sys_openat2+0x2b4/0x410 [ 7412.529684] do_sys_open+0x73/0xa0 [ 7412.533422] __x64_sys_openat+0x24/0x30 [ 7412.536566] do_syscall_64+0xc1/0x440 [ 7412.537921] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7412.542068] RIP: 0033:0x7fa749552332 [ 7412.544928] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7412.563327] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7412.569369] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7412.573863] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7412.584123] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7412.591191] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7412.598460] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7415.265068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5498s! [ 7415.268767] Showing busy workqueues and worker pools: [ 7415.272436] workqueue events: flags=0x0 [ 7415.274228] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7415.283040] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7415.289777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7415.293422] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7415.298054] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.300723] pending: kfree_rcu_monitor [ 7415.302982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7415.305412] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7415.309032] workqueue events_long: flags=0x0 [ 7415.310708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.313783] pending: lu_objects_flush [obdclass] [ 7415.317008] workqueue events_unbound: flags=0x2 [ 7415.318612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7415.320803] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7415.325224] workqueue events_power_efficient: flags=0x80 [ 7415.327669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.330124] pending: check_lifetime [ 7415.336980] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.342375] pending: neigh_periodic_work [ 7415.356058] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.361007] pending: neigh_periodic_work [ 7415.368186] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.372023] pending: do_cache_clean [ 7415.373783] workqueue mm_percpu_wq: flags=0x8 [ 7415.376585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.383759] pending: vmstat_update [ 7415.385330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.390846] pending: vmstat_update [ 7415.392349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7415.395641] pending: vmstat_update [ 7415.398162] workqueue writeback: flags=0x4a [ 7415.401084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7415.403648] pending: wb_workfn, wb_workfn [ 7415.410425] workqueue xprtiod: flags=0xa [ 7415.415425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7415.423461] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7415.427661] workqueue ipv6_addrconf: flags=0x40008 [ 7415.429027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7415.435184] pending: addrconf_verify_work [ 7415.438521] workqueue knbd0-recv: flags=0x1a [ 7415.444215] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7415.451457] in-flight: 651:recv_work [ 7415.453422] workqueue ptlrpc_pinger: flags=0x2 [ 7415.455812] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7415.458917] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7415.462129] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5498s workers=4 idle: 37 31846 28897 [ 7415.466419] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7405s workers=2 idle: 653 [ 7416.058512] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7416.066761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7416.101137] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7416.109415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7416.114670] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7416.120389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.125699] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7416.148377] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7416.153151] RAX: 0000000000007f87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7416.157633] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7416.165930] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7416.177410] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7416.188899] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7416.199662] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7416.203745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7416.207772] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7416.211120] Call Trace: [ 7416.212057] [ 7416.212623] ? show_regs.cold.9+0x22/0x2f [ 7416.215352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7416.216653] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7416.220594] ? watchdog+0x40/0x40 [ 7416.222807] ? hrtimer_interrupt+0x128/0x2c0 [ 7416.224703] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7416.228415] ? apic_timer_interrupt+0xf/0x20 [ 7416.230917] [ 7416.232027] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.236578] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7416.239120] do_raw_spin_lock+0x177/0x190 [ 7416.241221] _raw_spin_lock+0x21/0x30 [ 7416.242590] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7416.245401] process_one_work+0x2c8/0x700 [ 7416.248388] worker_thread+0x296/0x6e0 [ 7416.249750] ? rescuer_thread+0x570/0x570 [ 7416.255593] kthread+0x1d1/0x200 [ 7416.256767] ? set_kthread_struct+0x70/0x70 [ 7416.259517] ret_from_fork+0x1f/0x30 [ 7428.060042] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7428.072823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7428.117358] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7428.124092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7428.128386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7428.131010] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7428.138239] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7428.141683] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7428.145670] RDX: 00000000000033b4 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7428.149128] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7428.152827] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7428.155900] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7428.166548] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7428.172050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7428.181182] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7428.190954] Call Trace: [ 7428.191913] [ 7428.193393] ? show_regs.cold.9+0x22/0x2f [ 7428.199884] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7428.214074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7428.218238] ? watchdog+0x40/0x40 [ 7428.223213] ? hrtimer_interrupt+0x128/0x2c0 [ 7428.224745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7428.231248] ? apic_timer_interrupt+0xf/0x20 [ 7428.238705] [ 7428.241565] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7428.250319] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7428.258070] do_raw_spin_lock+0x177/0x190 [ 7428.265167] _raw_spin_lock+0x21/0x30 [ 7428.269563] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7428.278288] obd_mod_free+0x46/0x140 [ptlrpc] [ 7428.283699] mdc_commit_open+0x91/0xb0 [mdc] [ 7428.289286] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7428.294500] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7428.298181] after_reply+0xe2c/0x1780 [ptlrpc] [ 7428.301955] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7428.304944] ? __next_timer_interrupt+0x160/0x160 [ 7428.312320] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7428.314402] ? do_wait_intr+0xf0/0xf0 [ 7428.317273] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7428.320416] kthread+0x1d1/0x200 [ 7428.322945] ? set_kthread_struct+0x70/0x70 [ 7428.325040] ret_from_fork+0x1f/0x30 [ 7436.059389] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7436.069264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7436.171158] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7436.184334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7436.207698] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7436.223739] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7436.253348] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7436.255973] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7436.267734] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7436.281278] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7436.290918] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7436.295291] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7436.301590] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7436.306787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7436.310392] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7436.320430] Call Trace: [ 7436.321269] [ 7436.337976] ? show_regs.cold.9+0x22/0x2f [ 7436.341704] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7436.349047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7436.359871] ? watchdog+0x40/0x40 [ 7436.363939] ? hrtimer_interrupt+0x128/0x2c0 [ 7436.367937] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7436.375590] ? apic_timer_interrupt+0xf/0x20 [ 7436.382755] [ 7436.388003] ? native_safe_halt+0x2e/0x50 [ 7436.393624] kvm_wait+0xcd/0xe0 [ 7436.396007] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7436.399770] do_raw_spin_lock+0x177/0x190 [ 7436.403610] _raw_spin_lock+0x21/0x30 [ 7436.408408] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7436.412622] ? do_raw_spin_unlock+0x75/0x190 [ 7436.415605] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7436.421024] ? do_wait_intr+0xf0/0xf0 [ 7436.422970] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7436.426095] mdc_close+0x53b/0xd70 [mdc] [ 7436.427249] lmv_close+0x18d/0x490 [lmv] [ 7436.430616] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7436.441069] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7436.446185] ll_md_close+0x274/0x800 [lustre] [ 7436.450347] ll_file_release+0x128/0xc50 [lustre] [ 7436.456852] __fput+0x10d/0x440 [ 7436.458830] ____fput+0x12/0x20 [ 7436.461058] task_work_run+0xc6/0x110 [ 7436.467692] exit_to_usermode_loop+0x1dd/0x1f0 [ 7436.470810] do_syscall_64+0x3d6/0x440 [ 7436.472866] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7436.475663] RIP: 0033:0x7fa749552cd8 [ 7436.479212] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7436.501504] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7436.511645] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7436.515220] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7436.519186] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7436.522559] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7436.526066] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7440.057554] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7440.078025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7440.141785] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7440.153376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7440.162182] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7440.165143] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7440.175161] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7440.178548] RAX: 000000000000050c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7440.181301] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7440.183299] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7440.185552] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7440.188206] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7440.191114] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7440.194170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7440.195803] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7440.197497] Call Trace: [ 7440.198153] [ 7440.198617] ? show_regs.cold.9+0x22/0x2f [ 7440.200063] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7440.201180] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7440.202937] ? watchdog+0x40/0x40 [ 7440.203998] ? hrtimer_interrupt+0x128/0x2c0 [ 7440.205353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7440.207176] ? apic_timer_interrupt+0xf/0x20 [ 7440.208673] [ 7440.209525] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7440.211639] do_raw_spin_lock+0x177/0x190 [ 7440.213154] _raw_spin_lock+0x21/0x30 [ 7440.214708] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7440.217093] ? do_raw_spin_unlock+0x75/0x190 [ 7440.218764] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7440.220895] ? do_wait_intr+0xf0/0xf0 [ 7440.222464] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7440.224502] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7440.226059] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7440.228420] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7440.230223] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7440.233121] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7440.237822] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7440.241563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7440.252503] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7440.255027] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7440.261016] lmv_intent_open+0x338/0x1150 [lmv] [ 7440.266185] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7440.269701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7440.274337] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7440.276419] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7440.280701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7440.282963] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7440.285134] ? do_raw_spin_unlock+0x75/0x190 [ 7440.288584] ? _raw_spin_unlock+0x12/0x30 [ 7440.290212] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7440.296056] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7440.305542] ll_atomic_open+0x36a/0xd70 [lustre] [ 7440.307635] lookup_open+0x356/0xb40 [ 7440.310953] do_last+0x313/0xfc0 [ 7440.312282] ? nd_jump_root+0xe5/0x160 [ 7440.316133] ? path_init+0x437/0x520 [ 7440.323959] path_openat+0xf7/0x500 [ 7440.326710] ? do_raw_spin_unlock+0x75/0x190 [ 7440.330219] do_filp_open+0x99/0x140 [ 7440.334899] ? getname_flags+0x6e/0x330 [ 7440.336842] ? __check_object_size+0xff/0x256 [ 7440.338508] ? do_raw_spin_unlock+0x75/0x190 [ 7440.342766] ? _raw_spin_unlock+0x12/0x30 [ 7440.343952] do_sys_openat2+0x2b4/0x410 [ 7440.349495] do_sys_open+0x73/0xa0 [ 7440.352045] __x64_sys_openat+0x24/0x30 [ 7440.357794] do_syscall_64+0xc1/0x440 [ 7440.358739] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7440.360334] RIP: 0033:0x7fa749552332 [ 7440.368513] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7440.385209] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7440.392994] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7440.400223] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7440.406839] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7440.410905] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7440.420341] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7444.058657] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7444.076002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7444.130925] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7444.147592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7444.159828] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7444.170276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7444.177976] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7444.206550] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7444.217239] RAX: 0000000000004c05 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7444.234653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7444.246123] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7444.265891] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7444.267893] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7444.303538] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7444.320124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7444.347765] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7444.365983] Call Trace: [ 7444.372825] [ 7444.378882] ? show_regs.cold.9+0x22/0x2f [ 7444.382953] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7444.394785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7444.411314] ? watchdog+0x40/0x40 [ 7444.420072] ? hrtimer_interrupt+0x128/0x2c0 [ 7444.423586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7444.430982] ? apic_timer_interrupt+0xf/0x20 [ 7444.439238] [ 7444.440020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7444.441840] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7444.451537] do_raw_spin_lock+0x177/0x190 [ 7444.460009] _raw_spin_lock+0x21/0x30 [ 7444.467256] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7444.480281] process_one_work+0x2c8/0x700 [ 7444.483229] worker_thread+0x296/0x6e0 [ 7444.492053] ? rescuer_thread+0x570/0x570 [ 7444.501087] kthread+0x1d1/0x200 [ 7444.509655] ? set_kthread_struct+0x70/0x70 [ 7444.514053] ret_from_fork+0x1f/0x30 [ 7445.985044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5528s! [ 7445.988992] Showing busy workqueues and worker pools: [ 7445.991691] workqueue events: flags=0x0 [ 7445.993818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7445.996948] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7446.002215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7446.005223] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7446.009560] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.012452] pending: kfree_rcu_monitor [ 7446.014195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7446.017008] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7446.020817] workqueue events_long: flags=0x0 [ 7446.022837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.025794] pending: lu_objects_flush [obdclass] [ 7446.029726] workqueue events_unbound: flags=0x2 [ 7446.032589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7446.035560] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7446.039822] workqueue events_power_efficient: flags=0x80 [ 7446.042229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.045522] pending: check_lifetime [ 7446.047170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.050071] pending: neigh_periodic_work [ 7446.051928] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.054829] pending: neigh_periodic_work [ 7446.056621] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.059606] pending: do_cache_clean [ 7446.061418] workqueue mm_percpu_wq: flags=0x8 [ 7446.063405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.066194] pending: vmstat_update [ 7446.067949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.071006] pending: vmstat_update [ 7446.072497] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7446.075407] pending: vmstat_update [ 7446.077048] workqueue writeback: flags=0x4a [ 7446.078957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7446.081733] pending: wb_workfn, wb_workfn [ 7446.083527] workqueue xprtiod: flags=0xa [ 7446.085168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7446.087675] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7446.091379] workqueue ipv6_addrconf: flags=0x40008 [ 7446.093691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7446.096532] pending: addrconf_verify_work [ 7446.098469] workqueue knbd0-recv: flags=0x1a [ 7446.100131] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7446.102787] in-flight: 651:recv_work [ 7446.104398] workqueue ptlrpc_pinger: flags=0x2 [ 7446.106231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7446.108790] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7446.111449] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5528s workers=4 idle: 37 31846 28897 [ 7446.114763] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7436s workers=2 idle: 653 [ 7456.058010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7456.062321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7456.083925] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7456.089550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7456.094250] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7456.097762] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7456.110916] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7456.116714] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7456.121694] RDX: 00000000000030e1 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7456.125725] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7456.129969] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7456.132835] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7456.136057] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7456.139719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7456.142660] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7456.146811] Call Trace: [ 7456.147735] [ 7456.149039] ? show_regs.cold.9+0x22/0x2f [ 7456.153633] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7456.159238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7456.164387] ? watchdog+0x40/0x40 [ 7456.168702] ? hrtimer_interrupt+0x128/0x2c0 [ 7456.172042] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7456.175992] ? apic_timer_interrupt+0xf/0x20 [ 7456.181979] [ 7456.185650] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7456.190916] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7456.193011] do_raw_spin_lock+0x177/0x190 [ 7456.196976] _raw_spin_lock+0x21/0x30 [ 7456.198113] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7456.206755] obd_mod_free+0x46/0x140 [ptlrpc] [ 7456.210940] mdc_commit_open+0x91/0xb0 [mdc] [ 7456.214878] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7456.219460] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7456.222026] after_reply+0xe2c/0x1780 [ptlrpc] [ 7456.224181] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7456.226174] ? __next_timer_interrupt+0x160/0x160 [ 7456.228070] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7456.229799] ? do_wait_intr+0xf0/0xf0 [ 7456.236879] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7456.239889] kthread+0x1d1/0x200 [ 7456.241477] ? set_kthread_struct+0x70/0x70 [ 7456.242884] ret_from_fork+0x1f/0x30 [ 7464.059480] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7464.065694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7464.118646] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7464.132413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7464.138903] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7464.148412] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7464.165922] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7464.176491] RAX: 0000000000005aa1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7464.182343] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7464.186386] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7464.196290] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7464.206423] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7464.216279] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7464.229766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7464.238355] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7464.246126] Call Trace: [ 7464.248159] [ 7464.248905] ? show_regs.cold.9+0x22/0x2f [ 7464.258479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7464.262409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7464.267297] ? watchdog+0x40/0x40 [ 7464.271217] ? hrtimer_interrupt+0x128/0x2c0 [ 7464.272878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7464.283901] ? apic_timer_interrupt+0xf/0x20 [ 7464.287368] [ 7464.290645] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7464.306211] do_raw_spin_lock+0x177/0x190 [ 7464.307719] _raw_spin_lock+0x21/0x30 [ 7464.308947] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7464.322234] ? do_raw_spin_unlock+0x75/0x190 [ 7464.325916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7464.329088] ? do_wait_intr+0xf0/0xf0 [ 7464.330847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7464.334553] mdc_close+0x53b/0xd70 [mdc] [ 7464.338127] lmv_close+0x18d/0x490 [lmv] [ 7464.342473] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7464.349426] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7464.356324] ll_md_close+0x274/0x800 [lustre] [ 7464.361639] ll_file_release+0x128/0xc50 [lustre] [ 7464.367210] __fput+0x10d/0x440 [ 7464.370499] ____fput+0x12/0x20 [ 7464.375985] task_work_run+0xc6/0x110 [ 7464.377651] exit_to_usermode_loop+0x1dd/0x1f0 [ 7464.380557] do_syscall_64+0x3d6/0x440 [ 7464.381995] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7464.387194] RIP: 0033:0x7fa749552cd8 [ 7464.389684] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7464.415290] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7464.427424] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7464.431930] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7464.442303] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7464.445767] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7464.456888] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7472.056053] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7472.063662] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7472.069015] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7472.069123] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7472.074070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7472.074156] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7472.074163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7472.074169] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7472.074430] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.074441] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7472.074445] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7472.074449] RAX: 0000000000005b14 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7472.074452] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7472.074454] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7472.074456] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7472.074459] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7472.074462] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7472.074465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7472.074467] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7472.074474] Call Trace: [ 7472.074480] [ 7472.124496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7472.124521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.124545] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7472.124554] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7472.124559] RAX: 0000000000000a80 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7472.124565] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7472.124567] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7472.124569] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7472.124571] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7472.124576] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7472.124579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7472.124581] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7472.124592] Call Trace: [ 7472.124601] [ 7472.124610] ? show_regs.cold.9+0x22/0x2f [ 7472.124623] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7472.124633] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7472.124637] ? watchdog+0x40/0x40 [ 7472.124646] ? hrtimer_interrupt+0x128/0x2c0 [ 7472.124650] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7472.124667] ? apic_timer_interrupt+0xf/0x20 [ 7472.124672] [ 7472.124679] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.124690] do_raw_spin_lock+0x177/0x190 [ 7472.124695] _raw_spin_lock+0x21/0x30 [ 7472.124704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7472.125399] ? do_raw_spin_unlock+0x75/0x190 [ 7472.125409] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7472.133736] ? show_regs.cold.9+0x22/0x2f [ 7472.133757] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7472.133763] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7472.133767] ? watchdog+0x40/0x40 [ 7472.133771] ? hrtimer_interrupt+0x128/0x2c0 [ 7472.133775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7472.133782] ? apic_timer_interrupt+0xf/0x20 [ 7472.133787] [ 7472.133792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.133946] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7472.133954] do_raw_spin_lock+0x177/0x190 [ 7472.133960] _raw_spin_lock+0x21/0x30 [ 7472.133966] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7472.134228] process_one_work+0x2c8/0x700 [ 7472.134236] worker_thread+0x296/0x6e0 [ 7472.134240] ? rescuer_thread+0x570/0x570 [ 7472.134244] kthread+0x1d1/0x200 [ 7472.134248] ? set_kthread_struct+0x70/0x70 [ 7472.134251] ret_from_fork+0x1f/0x30 [ 7472.447725] ? do_wait_intr+0xf0/0xf0 [ 7472.452907] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7472.455323] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7472.458399] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7472.462650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7472.466649] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7472.469441] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7472.474109] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7472.475770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7472.478419] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7472.489575] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7472.494750] lmv_intent_open+0x338/0x1150 [lmv] [ 7472.499112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7472.502407] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7472.505775] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7472.511029] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7472.513039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7472.515911] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7472.518031] ? do_raw_spin_unlock+0x75/0x190 [ 7472.519954] ? _raw_spin_unlock+0x12/0x30 [ 7472.521458] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7472.524347] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7472.530869] ll_atomic_open+0x36a/0xd70 [lustre] [ 7472.536856] lookup_open+0x356/0xb40 [ 7472.543109] do_last+0x313/0xfc0 [ 7472.548039] ? nd_jump_root+0xe5/0x160 [ 7472.550370] ? path_init+0x437/0x520 [ 7472.554666] path_openat+0xf7/0x500 [ 7472.556736] ? do_raw_spin_unlock+0x75/0x190 [ 7472.560576] do_filp_open+0x99/0x140 [ 7472.562544] ? getname_flags+0x6e/0x330 [ 7472.567576] ? __check_object_size+0xff/0x256 [ 7472.577423] ? do_raw_spin_unlock+0x75/0x190 [ 7472.580902] ? _raw_spin_unlock+0x12/0x30 [ 7472.584046] do_sys_openat2+0x2b4/0x410 [ 7472.590432] do_sys_open+0x73/0xa0 [ 7472.594069] __x64_sys_openat+0x24/0x30 [ 7472.599226] do_syscall_64+0xc1/0x440 [ 7472.603503] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7472.607733] RIP: 0033:0x7fa749552332 [ 7472.609635] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7472.622691] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7472.635209] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7472.648855] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7472.663778] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7472.666482] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7472.673405] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7476.196993] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5559s! [ 7476.205254] Showing busy workqueues and worker pools: [ 7476.210559] workqueue events: flags=0x0 [ 7476.213307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7476.219368] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7476.230126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7476.235982] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7476.250377] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.261330] pending: kfree_rcu_monitor [ 7476.269292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7476.271665] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7476.278145] workqueue events_long: flags=0x0 [ 7476.283039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.286682] pending: lu_objects_flush [obdclass] [ 7476.289849] workqueue events_unbound: flags=0x2 [ 7476.292141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7476.294847] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7476.298702] workqueue events_power_efficient: flags=0x80 [ 7476.301169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.304091] pending: check_lifetime [ 7476.305609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.308893] pending: neigh_periodic_work [ 7476.310889] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.313327] pending: neigh_periodic_work [ 7476.314925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.316990] pending: do_cache_clean [ 7476.317977] workqueue mm_percpu_wq: flags=0x8 [ 7476.319108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.323860] pending: vmstat_update [ 7476.328418] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.330629] pending: vmstat_update [ 7476.335650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7476.347969] pending: vmstat_update [ 7476.349362] workqueue writeback: flags=0x4a [ 7476.350818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7476.358031] pending: wb_workfn, wb_workfn [ 7476.359573] workqueue xprtiod: flags=0xa [ 7476.363856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7476.367994] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7476.375520] workqueue ipv6_addrconf: flags=0x40008 [ 7476.378217] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7476.383275] pending: addrconf_verify_work [ 7476.388027] workqueue knbd0-recv: flags=0x1a [ 7476.391916] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7476.398112] in-flight: 651:recv_work [ 7476.400026] workqueue ptlrpc_pinger: flags=0x2 [ 7476.401921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7476.405480] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7476.411235] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5559s workers=4 idle: 37 31846 28897 [ 7476.415124] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7466s workers=2 idle: 653 [ 7484.058064] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7484.060959] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7484.082849] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7484.087426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7484.091989] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7484.100357] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7484.107467] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7484.110178] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7484.112568] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7484.115295] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7484.120409] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7484.129264] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7484.132397] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7484.136281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7484.139226] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7484.142744] Call Trace: [ 7484.143509] [ 7484.144278] ? show_regs.cold.9+0x22/0x2f [ 7484.145792] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7484.147541] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7484.149725] ? watchdog+0x40/0x40 [ 7484.151156] ? hrtimer_interrupt+0x128/0x2c0 [ 7484.153207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7484.156364] ? apic_timer_interrupt+0xf/0x20 [ 7484.158647] [ 7484.161751] ? native_safe_halt+0x2e/0x50 [ 7484.164054] kvm_wait+0xcd/0xe0 [ 7484.165682] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7484.167971] do_raw_spin_lock+0x177/0x190 [ 7484.169748] _raw_spin_lock+0x21/0x30 [ 7484.171582] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7484.178996] obd_mod_free+0x46/0x140 [ptlrpc] [ 7484.183125] mdc_commit_open+0x91/0xb0 [mdc] [ 7484.189220] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7484.198129] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7484.204516] after_reply+0xe2c/0x1780 [ptlrpc] [ 7484.213348] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7484.216080] ? __next_timer_interrupt+0x160/0x160 [ 7484.220661] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7484.224127] ? do_wait_intr+0xf0/0xf0 [ 7484.227236] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7484.230374] kthread+0x1d1/0x200 [ 7484.232369] ? set_kthread_struct+0x70/0x70 [ 7484.235159] ret_from_fork+0x1f/0x30 [ 7492.059018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7492.061990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7492.109608] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7492.120354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7492.126369] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7492.130302] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7492.139969] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7492.143427] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7492.148160] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7492.152879] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7492.156110] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7492.164781] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7492.170168] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7492.172109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7492.178206] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7492.184143] Call Trace: [ 7492.185217] [ 7492.186227] ? show_regs.cold.9+0x22/0x2f [ 7492.187908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7492.190123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7492.192037] ? watchdog+0x40/0x40 [ 7492.193390] ? hrtimer_interrupt+0x128/0x2c0 [ 7492.195082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7492.196966] ? apic_timer_interrupt+0xf/0x20 [ 7492.198683] [ 7492.199546] ? native_safe_halt+0x2e/0x50 [ 7492.201212] kvm_wait+0xcd/0xe0 [ 7492.202351] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7492.205210] do_raw_spin_lock+0x177/0x190 [ 7492.206942] _raw_spin_lock+0x21/0x30 [ 7492.211750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7492.215131] ? do_raw_spin_unlock+0x75/0x190 [ 7492.217219] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7492.220007] ? do_wait_intr+0xf0/0xf0 [ 7492.222264] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7492.224930] mdc_close+0x53b/0xd70 [mdc] [ 7492.226417] lmv_close+0x18d/0x490 [lmv] [ 7492.228288] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7492.231132] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7492.234082] ll_md_close+0x274/0x800 [lustre] [ 7492.236670] ll_file_release+0x128/0xc50 [lustre] [ 7492.238323] __fput+0x10d/0x440 [ 7492.239907] ____fput+0x12/0x20 [ 7492.241277] task_work_run+0xc6/0x110 [ 7492.243240] exit_to_usermode_loop+0x1dd/0x1f0 [ 7492.246429] do_syscall_64+0x3d6/0x440 [ 7492.251378] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7492.256072] RIP: 0033:0x7fa749552cd8 [ 7492.259682] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7492.274120] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7492.279198] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7492.283316] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7492.287036] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7492.289289] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7492.292916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7500.057015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7500.060226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7500.082611] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7500.086677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7500.090024] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7500.092667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7500.094847] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7500.102998] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7500.106119] RAX: 000000000000390f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7500.109249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7500.113095] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7500.116233] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7500.120039] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7500.123395] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7500.127315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7500.130240] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7500.134223] Call Trace: [ 7500.136348] [ 7500.137468] ? show_regs.cold.9+0x22/0x2f [ 7500.141612] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7500.144013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7500.146644] ? watchdog+0x40/0x40 [ 7500.147581] ? hrtimer_interrupt+0x128/0x2c0 [ 7500.148944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7500.150996] ? apic_timer_interrupt+0xf/0x20 [ 7500.153015] [ 7500.156787] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7500.159847] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7500.162447] do_raw_spin_lock+0x177/0x190 [ 7500.166091] _raw_spin_lock+0x21/0x30 [ 7500.169762] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7500.172100] process_one_work+0x2c8/0x700 [ 7500.173728] worker_thread+0x296/0x6e0 [ 7500.175097] ? rescuer_thread+0x570/0x570 [ 7500.176405] kthread+0x1d1/0x200 [ 7500.179701] ? set_kthread_struct+0x70/0x70 [ 7500.181959] ret_from_fork+0x1f/0x30 [ 7504.057570] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7504.068701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7504.128078] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7504.135311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7504.141210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7504.144029] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7504.155006] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7504.158580] RAX: 0000000000007884 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7504.168003] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7504.173204] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7504.178993] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7504.193483] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7504.199997] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7504.216132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7504.223349] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7504.228046] Call Trace: [ 7504.237567] [ 7504.244793] ? show_regs.cold.9+0x22/0x2f [ 7504.249625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7504.262909] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7504.269838] ? watchdog+0x40/0x40 [ 7504.277199] ? hrtimer_interrupt+0x128/0x2c0 [ 7504.284006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7504.291400] ? apic_timer_interrupt+0xf/0x20 [ 7504.299115] [ 7504.300050] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7504.305022] do_raw_spin_lock+0x177/0x190 [ 7504.308227] _raw_spin_lock+0x21/0x30 [ 7504.312668] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7504.317462] ? do_raw_spin_unlock+0x75/0x190 [ 7504.320658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7504.322664] ? do_wait_intr+0xf0/0xf0 [ 7504.327654] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7504.331052] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7504.335637] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7504.340618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.352043] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7504.354151] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7504.363992] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7504.379138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.389376] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7504.394436] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7504.403423] lmv_intent_open+0x338/0x1150 [lmv] [ 7504.410205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.412286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.423884] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7504.425440] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7504.439722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.444441] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7504.446207] ? do_raw_spin_unlock+0x75/0x190 [ 7504.450130] ? _raw_spin_unlock+0x12/0x30 [ 7504.452883] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7504.455185] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7504.459172] ll_atomic_open+0x36a/0xd70 [lustre] [ 7504.463841] lookup_open+0x356/0xb40 [ 7504.468392] do_last+0x313/0xfc0 [ 7504.471916] ? nd_jump_root+0xe5/0x160 [ 7504.475164] ? path_init+0x437/0x520 [ 7504.479992] path_openat+0xf7/0x500 [ 7504.484034] ? do_raw_spin_unlock+0x75/0x190 [ 7504.488062] do_filp_open+0x99/0x140 [ 7504.491527] ? getname_flags+0x6e/0x330 [ 7504.495618] ? __check_object_size+0xff/0x256 [ 7504.498933] ? do_raw_spin_unlock+0x75/0x190 [ 7504.501020] ? _raw_spin_unlock+0x12/0x30 [ 7504.503640] do_sys_openat2+0x2b4/0x410 [ 7504.505427] do_sys_open+0x73/0xa0 [ 7504.507668] __x64_sys_openat+0x24/0x30 [ 7504.510978] do_syscall_64+0xc1/0x440 [ 7504.515920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7504.517590] RIP: 0033:0x7fa749552332 [ 7504.521525] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7504.534136] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7504.538174] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7504.545658] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7504.552196] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7504.556182] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7504.565078] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7506.920741] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5589s! [ 7506.930729] Showing busy workqueues and worker pools: [ 7506.937337] workqueue events: flags=0x0 [ 7506.946273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7506.956549] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7506.971553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7506.980102] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7506.988361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.000233] pending: kfree_rcu_monitor [ 7507.002789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7507.014004] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7507.024714] workqueue events_long: flags=0x0 [ 7507.031144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.033240] pending: lu_objects_flush [obdclass] [ 7507.042937] workqueue events_unbound: flags=0x2 [ 7507.044015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7507.054979] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7507.062979] workqueue events_power_efficient: flags=0x80 [ 7507.069432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.074124] pending: check_lifetime [ 7507.078721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.090540] pending: neigh_periodic_work [ 7507.092758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.096547] pending: neigh_periodic_work [ 7507.101694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.105351] pending: do_cache_clean [ 7507.110737] workqueue mm_percpu_wq: flags=0x8 [ 7507.117176] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.129039] pending: vmstat_update [ 7507.132126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.140349] pending: vmstat_update [ 7507.145059] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7507.153629] pending: vmstat_update [ 7507.155043] workqueue writeback: flags=0x4a [ 7507.156345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7507.169300] pending: wb_workfn, wb_workfn [ 7507.174097] workqueue xprtiod: flags=0xa [ 7507.177086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7507.181336] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7507.186345] workqueue ipv6_addrconf: flags=0x40008 [ 7507.188791] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7507.193355] pending: addrconf_verify_work [ 7507.196985] workqueue knbd0-recv: flags=0x1a [ 7507.200061] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7507.206244] in-flight: 651:recv_work [ 7507.210996] workqueue ptlrpc_pinger: flags=0x2 [ 7507.219698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7507.228311] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7507.253385] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5590s workers=4 idle: 37 31846 28897 [ 7507.265373] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7497s workers=2 idle: 653 [ 7512.065516] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7512.070088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7512.150240] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7512.167242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7512.173002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7512.177256] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7512.195361] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7512.199601] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7512.204594] RDX: 0000000000000277 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7512.211888] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7512.217142] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7512.225702] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7512.231070] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7512.239078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7512.248656] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7512.257504] Call Trace: [ 7512.258871] [ 7512.259650] ? show_regs.cold.9+0x22/0x2f [ 7512.265992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7512.270044] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7512.275617] ? watchdog+0x40/0x40 [ 7512.280336] ? hrtimer_interrupt+0x128/0x2c0 [ 7512.284942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7512.293276] ? apic_timer_interrupt+0xf/0x20 [ 7512.295661] [ 7512.296909] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7512.299938] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7512.305533] do_raw_spin_lock+0x177/0x190 [ 7512.309638] _raw_spin_lock+0x21/0x30 [ 7512.311481] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7512.316469] obd_mod_free+0x46/0x140 [ptlrpc] [ 7512.324411] mdc_commit_open+0x91/0xb0 [mdc] [ 7512.326034] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7512.335917] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7512.349679] after_reply+0xe2c/0x1780 [ptlrpc] [ 7512.359833] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7512.364148] ? __next_timer_interrupt+0x160/0x160 [ 7512.372055] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7512.381068] ? do_wait_intr+0xf0/0xf0 [ 7512.386974] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7512.394008] kthread+0x1d1/0x200 [ 7512.401000] ? set_kthread_struct+0x70/0x70 [ 7512.405832] ret_from_fork+0x1f/0x30 [ 7520.059091] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7520.063275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7520.089638] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7520.094636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7520.099005] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7520.100892] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7520.110228] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7520.114627] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7520.118793] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7520.122083] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7520.125776] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7520.129574] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7520.133259] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7520.137036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7520.139449] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7520.142776] Call Trace: [ 7520.144054] [ 7520.145030] ? show_regs.cold.9+0x22/0x2f [ 7520.147003] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7520.149355] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7520.151923] ? watchdog+0x40/0x40 [ 7520.153639] ? hrtimer_interrupt+0x128/0x2c0 [ 7520.155407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7520.157905] ? apic_timer_interrupt+0xf/0x20 [ 7520.161687] [ 7520.162708] ? native_safe_halt+0x2e/0x50 [ 7520.164358] kvm_wait+0xcd/0xe0 [ 7520.166508] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7520.169333] do_raw_spin_lock+0x177/0x190 [ 7520.171535] _raw_spin_lock+0x21/0x30 [ 7520.172982] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7520.175620] ? do_raw_spin_unlock+0x75/0x190 [ 7520.178173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7520.180695] ? do_wait_intr+0xf0/0xf0 [ 7520.182552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7520.185269] mdc_close+0x53b/0xd70 [mdc] [ 7520.187129] lmv_close+0x18d/0x490 [lmv] [ 7520.189023] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7520.192452] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7520.194573] ll_md_close+0x274/0x800 [lustre] [ 7520.196758] ll_file_release+0x128/0xc50 [lustre] [ 7520.198989] __fput+0x10d/0x440 [ 7520.200722] ____fput+0x12/0x20 [ 7520.202897] task_work_run+0xc6/0x110 [ 7520.204610] exit_to_usermode_loop+0x1dd/0x1f0 [ 7520.206821] do_syscall_64+0x3d6/0x440 [ 7520.209633] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7520.211833] RIP: 0033:0x7fa749552cd8 [ 7520.213773] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7520.224539] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7520.229010] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7520.233678] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7520.238352] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7520.241890] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7520.245910] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7528.057367] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 7528.064773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7528.109224] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7528.114319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7528.117538] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7528.123343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7528.130585] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7528.141743] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7528.149182] RAX: 0000000000006686 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7528.154791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7528.159693] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7528.165657] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7528.172071] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7528.179257] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7528.185029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7528.188348] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7528.191075] Call Trace: [ 7528.192366] [ 7528.193920] ? show_regs.cold.9+0x22/0x2f [ 7528.195351] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7528.196607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7528.199514] ? watchdog+0x40/0x40 [ 7528.201347] ? hrtimer_interrupt+0x128/0x2c0 [ 7528.202964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7528.204791] ? apic_timer_interrupt+0xf/0x20 [ 7528.206349] [ 7528.207283] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7528.211028] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7528.214068] do_raw_spin_lock+0x177/0x190 [ 7528.215718] _raw_spin_lock+0x21/0x30 [ 7528.219625] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7528.223039] process_one_work+0x2c8/0x700 [ 7528.226202] worker_thread+0x296/0x6e0 [ 7528.227515] ? rescuer_thread+0x570/0x570 [ 7528.228477] kthread+0x1d1/0x200 [ 7528.230291] ? set_kthread_struct+0x70/0x70 [ 7528.232255] ret_from_fork+0x1f/0x30 [ 7532.054799] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7532.062927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7532.107224] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7532.111926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7532.114970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7532.116996] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7532.124885] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7532.127367] RAX: 0000000000007a90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7532.129732] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7532.132696] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7532.135609] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7532.137694] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7532.139592] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7532.143135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7532.145524] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7532.148229] Call Trace: [ 7532.149183] [ 7532.150028] ? show_regs.cold.9+0x22/0x2f [ 7532.151502] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7532.153942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7532.156389] ? watchdog+0x40/0x40 [ 7532.162756] ? hrtimer_interrupt+0x128/0x2c0 [ 7532.168647] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7532.171639] ? apic_timer_interrupt+0xf/0x20 [ 7532.177306] [ 7532.179290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7532.184892] do_raw_spin_lock+0x177/0x190 [ 7532.186946] _raw_spin_lock+0x21/0x30 [ 7532.188049] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7532.192744] ? do_raw_spin_unlock+0x75/0x190 [ 7532.193718] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7532.198498] ? do_wait_intr+0xf0/0xf0 [ 7532.200003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7532.202317] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7532.204878] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7532.207458] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.209460] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7532.213208] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7532.218059] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7532.222630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.228436] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7532.234506] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7532.237660] lmv_intent_open+0x338/0x1150 [lmv] [ 7532.240066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.242935] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.245796] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7532.248394] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7532.251175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.258768] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7532.260486] ? do_raw_spin_unlock+0x75/0x190 [ 7532.267412] ? _raw_spin_unlock+0x12/0x30 [ 7532.273843] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7532.276410] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7532.279273] ll_atomic_open+0x36a/0xd70 [lustre] [ 7532.282695] lookup_open+0x356/0xb40 [ 7532.286421] do_last+0x313/0xfc0 [ 7532.293549] ? nd_jump_root+0xe5/0x160 [ 7532.297460] ? path_init+0x437/0x520 [ 7532.303906] path_openat+0xf7/0x500 [ 7532.307201] ? do_raw_spin_unlock+0x75/0x190 [ 7532.309801] do_filp_open+0x99/0x140 [ 7532.311276] ? getname_flags+0x6e/0x330 [ 7532.312665] ? __check_object_size+0xff/0x256 [ 7532.314468] ? do_raw_spin_unlock+0x75/0x190 [ 7532.316473] ? _raw_spin_unlock+0x12/0x30 [ 7532.317954] do_sys_openat2+0x2b4/0x410 [ 7532.319561] do_sys_open+0x73/0xa0 [ 7532.320976] __x64_sys_openat+0x24/0x30 [ 7532.324341] do_syscall_64+0xc1/0x440 [ 7532.325979] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7532.328669] RIP: 0033:0x7fa749552332 [ 7532.329888] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7532.338882] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7532.341814] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7532.344819] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7532.348207] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7532.351405] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7532.354531] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7537.633054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5620s! [ 7537.636009] Showing busy workqueues and worker pools: [ 7537.640099] workqueue events: flags=0x0 [ 7537.641183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7537.646004] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7537.650973] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7537.655352] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7537.661979] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.666089] pending: kfree_rcu_monitor [ 7537.668212] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7537.672144] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7537.676514] workqueue events_long: flags=0x0 [ 7537.680105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.684780] pending: lu_objects_flush [obdclass] [ 7537.687388] workqueue events_unbound: flags=0x2 [ 7537.690587] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7537.693424] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7537.699084] workqueue events_power_efficient: flags=0x80 [ 7537.702997] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.707684] pending: check_lifetime [ 7537.709511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.717267] pending: neigh_periodic_work [ 7537.721762] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.728098] pending: neigh_periodic_work [ 7537.733162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.738337] pending: do_cache_clean [ 7537.739762] workqueue mm_percpu_wq: flags=0x8 [ 7537.743874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.747796] pending: vmstat_update [ 7537.749519] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.752944] pending: vmstat_update [ 7537.755209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.761460] pending: vmstat_update [ 7537.764107] workqueue writeback: flags=0x4a [ 7537.766304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7537.771320] pending: wb_workfn, wb_workfn [ 7537.777048] workqueue xprtiod: flags=0xa [ 7537.782781] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7537.788051] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7537.794703] workqueue ipv6_addrconf: flags=0x40008 [ 7537.796523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7537.805143] pending: addrconf_verify_work [ 7537.809087] workqueue knbd0-recv: flags=0x1a [ 7537.816184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7537.825073] in-flight: 651:recv_work [ 7537.829379] workqueue ptlrpc_pinger: flags=0x2 [ 7537.831120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7537.844786] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7537.848823] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5620s workers=4 idle: 37 31846 28897 [ 7537.855165] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7527s workers=2 idle: 653 [ 7540.058038] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7540.060858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7540.098283] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7540.104027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7540.110422] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7540.112533] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7540.121324] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7540.127014] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7540.134359] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7540.138304] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7540.142760] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7540.149126] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7540.155348] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7540.165543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7540.168327] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7540.171818] Call Trace: [ 7540.173152] [ 7540.174128] ? show_regs.cold.9+0x22/0x2f [ 7540.175861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7540.181949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7540.185440] ? watchdog+0x40/0x40 [ 7540.186832] ? hrtimer_interrupt+0x128/0x2c0 [ 7540.191710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7540.193756] ? apic_timer_interrupt+0xf/0x20 [ 7540.195549] [ 7540.196825] ? native_safe_halt+0x2e/0x50 [ 7540.198731] kvm_wait+0xcd/0xe0 [ 7540.200427] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7540.205467] do_raw_spin_lock+0x177/0x190 [ 7540.206763] _raw_spin_lock+0x21/0x30 [ 7540.208379] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7540.210599] obd_mod_free+0x46/0x140 [ptlrpc] [ 7540.213058] mdc_commit_open+0x91/0xb0 [mdc] [ 7540.214972] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7540.217303] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7540.219453] after_reply+0xe2c/0x1780 [ptlrpc] [ 7540.221782] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7540.224069] ? __next_timer_interrupt+0x160/0x160 [ 7540.226277] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7540.229175] ? do_wait_intr+0xf0/0xf0 [ 7540.230781] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7540.232640] kthread+0x1d1/0x200 [ 7540.233556] ? set_kthread_struct+0x70/0x70 [ 7540.235546] ret_from_fork+0x1f/0x30 [ 7548.059749] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7548.062742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7548.085667] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7548.094602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7548.098668] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7548.101502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7548.109574] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7548.112838] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7548.119764] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7548.125869] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7548.143145] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7548.145549] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7548.154149] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7548.158324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7548.161173] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7548.164668] Call Trace: [ 7548.165892] [ 7548.166873] ? show_regs.cold.9+0x22/0x2f [ 7548.171615] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7548.174158] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7548.176592] ? watchdog+0x40/0x40 [ 7548.181927] ? hrtimer_interrupt+0x128/0x2c0 [ 7548.188369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7548.190189] ? apic_timer_interrupt+0xf/0x20 [ 7548.200707] [ 7548.204459] ? native_safe_halt+0x2e/0x50 [ 7548.211353] kvm_wait+0xcd/0xe0 [ 7548.216061] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7548.226587] do_raw_spin_lock+0x177/0x190 [ 7548.230991] _raw_spin_lock+0x21/0x30 [ 7548.232536] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7548.237252] ? do_raw_spin_unlock+0x75/0x190 [ 7548.241258] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7548.246251] ? do_wait_intr+0xf0/0xf0 [ 7548.247503] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7548.251198] mdc_close+0x53b/0xd70 [mdc] [ 7548.253660] lmv_close+0x18d/0x490 [lmv] [ 7548.257345] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7548.266529] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7548.278707] ll_md_close+0x274/0x800 [lustre] [ 7548.282843] ll_file_release+0x128/0xc50 [lustre] [ 7548.284395] __fput+0x10d/0x440 [ 7548.285386] ____fput+0x12/0x20 [ 7548.292058] task_work_run+0xc6/0x110 [ 7548.295719] exit_to_usermode_loop+0x1dd/0x1f0 [ 7548.300051] do_syscall_64+0x3d6/0x440 [ 7548.301435] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7548.303188] RIP: 0033:0x7fa749552cd8 [ 7548.311022] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7548.328058] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7548.335973] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7548.340571] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7548.345700] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7548.351030] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7548.353498] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7553.816089] rcu: INFO: rcu_sched self-detected stall on CPU [ 7553.819145] rcu: 2-...!: (3123408 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 7553.824641] (t=5640127 jiffies g=258701 q=3089) [ 7553.826896] rcu: rcu_sched kthread starved for 5636922 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7553.831867] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7553.836202] rcu: RCU grace-period kthread stack dump: [ 7553.838711] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7553.843644] Call Trace: [ 7553.844840] __schedule+0x351/0xcb0 [ 7553.847124] schedule+0xc0/0x180 [ 7553.848769] schedule_timeout+0xb4/0x190 [ 7553.850780] ? do_raw_spin_unlock+0x75/0x190 [ 7553.852898] ? __next_timer_interrupt+0x160/0x160 [ 7553.855289] ? prepare_to_swait_event+0xf3/0x140 [ 7553.857734] rcu_gp_kthread+0x8dd/0x15b0 [ 7553.859309] ? force_qs_rnp+0x2c0/0x2c0 [ 7553.860972] kthread+0x1d1/0x200 [ 7553.862716] ? set_kthread_struct+0x70/0x70 [ 7553.864877] ret_from_fork+0x1f/0x30 [ 7553.866768] rcu: Stack dump where RCU GP kthread last ran: [ 7553.869329] Sending NMI from CPU 2 to CPUs 0: [ 7553.871762] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 7553.872062] NMI backtrace for cpu 2 [ 7553.877117] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7553.883872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7553.888717] Call Trace: [ 7553.890564] [ 7553.891555] ? dump_stack+0xbb/0x10e [ 7553.893808] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7553.896187] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7553.898629] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7553.900715] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7553.903106] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7553.905189] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7553.907109] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7553.909277] ? update_process_times+0x6d/0xc0 [ 7553.911249] ? tick_sched_handle.isra.18+0x27/0x90 [ 7553.913781] ? tick_sched_timer+0x53/0xb0 [ 7553.915451] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7553.916767] ? tick_sched_handle.isra.18+0x90/0x90 [ 7553.917868] ? hrtimer_interrupt+0x128/0x2c0 [ 7553.918803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7553.920871] ? apic_timer_interrupt+0xf/0x20 [ 7553.922773] [ 7553.923784] ? native_safe_halt+0x2e/0x50 [ 7553.925868] ? kvm_wait+0xcd/0xe0 [ 7553.928621] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7553.931254] ? do_raw_spin_lock+0x177/0x190 [ 7553.932432] ? _raw_spin_lock+0x21/0x30 [ 7553.933895] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7553.935854] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7553.937587] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7553.938834] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7553.941245] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7553.943828] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7553.947766] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7553.950358] ? __next_timer_interrupt+0x160/0x160 [ 7553.951767] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7553.953912] ? do_wait_intr+0xf0/0xf0 [ 7553.956611] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7553.958951] ? kthread+0x1d1/0x200 [ 7553.961058] ? set_kthread_struct+0x70/0x70 [ 7553.962145] ? ret_from_fork+0x1f/0x30 [ 7556.057025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 7556.060433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7556.086948] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7556.097836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7556.109040] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7556.113425] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7556.115536] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7556.126840] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7556.129794] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7556.133068] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7556.136749] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7556.139112] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7556.142478] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7556.145286] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7556.149080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7556.151322] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7556.154241] Call Trace: [ 7556.155221] [ 7556.156106] ? show_regs.cold.9+0x22/0x2f [ 7556.157749] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7556.159416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7556.162705] ? watchdog+0x40/0x40 [ 7556.164168] ? hrtimer_interrupt+0x128/0x2c0 [ 7556.166097] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7556.168444] ? apic_timer_interrupt+0xf/0x20 [ 7556.169485] [ 7556.170516] ? native_safe_halt+0x2e/0x50 [ 7556.172456] kvm_wait+0xcd/0xe0 [ 7556.174301] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7556.177743] do_raw_spin_lock+0x177/0x190 [ 7556.179642] _raw_spin_lock+0x21/0x30 [ 7556.181247] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7556.187969] process_one_work+0x2c8/0x700 [ 7556.197099] worker_thread+0x296/0x6e0 [ 7556.198362] ? rescuer_thread+0x570/0x570 [ 7556.203175] kthread+0x1d1/0x200 [ 7556.207921] ? set_kthread_struct+0x70/0x70 [ 7556.209840] ret_from_fork+0x1f/0x30 [ 7568.361717] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5651s! [ 7568.377919] Showing busy workqueues and worker pools: [ 7568.389888] workqueue events: flags=0x0 [ 7568.399851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7568.407793] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7568.419644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7568.424086] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7568.434498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.439320] pending: kfree_rcu_monitor [ 7568.442925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7568.450715] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7568.458864] workqueue events_long: flags=0x0 [ 7568.462364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.469299] pending: lu_objects_flush [obdclass] [ 7568.474159] workqueue events_unbound: flags=0x2 [ 7568.478679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7568.483104] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7568.491793] workqueue events_power_efficient: flags=0x80 [ 7568.496873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.505957] pending: check_lifetime [ 7568.507835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.512609] pending: neigh_periodic_work [ 7568.517630] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.521599] pending: neigh_periodic_work [ 7568.524010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.541830] pending: do_cache_clean [ 7568.543250] workqueue mm_percpu_wq: flags=0x8 [ 7568.550042] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.562842] pending: vmstat_update [ 7568.567881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.576917] pending: vmstat_update [ 7568.582626] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7568.587322] pending: vmstat_update [ 7568.589839] workqueue writeback: flags=0x4a [ 7568.592597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7568.596319] pending: wb_workfn, wb_workfn [ 7568.598865] workqueue xprtiod: flags=0xa [ 7568.601450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7568.603508] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7568.612552] workqueue ipv6_addrconf: flags=0x40008 [ 7568.618230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7568.621654] pending: addrconf_verify_work [ 7568.623306] workqueue knbd0-recv: flags=0x1a [ 7568.630902] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7568.643058] in-flight: 651:recv_work [ 7568.646895] workqueue ptlrpc_pinger: flags=0x2 [ 7568.653893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7568.661182] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7568.670014] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5651s workers=4 idle: 37 31846 28897 [ 7568.683842] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7558s workers=2 idle: 653 [ 7576.059036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7576.065369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7576.096253] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7576.107606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7576.119517] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7576.122164] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7576.136614] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7576.145323] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7576.149677] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7576.155643] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7576.158971] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7576.168284] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7576.175599] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7576.183066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7576.185790] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7576.192269] Call Trace: [ 7576.195457] [ 7576.198805] ? show_regs.cold.9+0x22/0x2f [ 7576.202885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7576.205803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7576.216444] ? watchdog+0x40/0x40 [ 7576.218759] ? hrtimer_interrupt+0x128/0x2c0 [ 7576.224073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7576.226642] ? apic_timer_interrupt+0xf/0x20 [ 7576.228530] [ 7576.230159] ? native_safe_halt+0x2e/0x50 [ 7576.232380] kvm_wait+0xcd/0xe0 [ 7576.234267] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7576.237244] do_raw_spin_lock+0x177/0x190 [ 7576.240305] _raw_spin_lock+0x21/0x30 [ 7576.243057] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7576.247579] ? do_raw_spin_unlock+0x75/0x190 [ 7576.250339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7576.255616] ? do_wait_intr+0xf0/0xf0 [ 7576.256917] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7576.261395] mdc_close+0x53b/0xd70 [mdc] [ 7576.267064] lmv_close+0x18d/0x490 [lmv] [ 7576.271962] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7576.274811] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7576.278633] ll_md_close+0x274/0x800 [lustre] [ 7576.282788] ll_file_release+0x128/0xc50 [lustre] [ 7576.285779] __fput+0x10d/0x440 [ 7576.288064] ____fput+0x12/0x20 [ 7576.292184] task_work_run+0xc6/0x110 [ 7576.300205] exit_to_usermode_loop+0x1dd/0x1f0 [ 7576.305376] do_syscall_64+0x3d6/0x440 [ 7576.310637] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7576.317302] RIP: 0033:0x7fa749552cd8 [ 7576.319388] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7576.340807] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7576.352695] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7576.360678] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7576.364323] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7576.371585] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7576.375915] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7580.058835] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7580.072980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7580.124044] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7580.141531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7580.152302] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7580.153902] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7580.180252] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7580.190819] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7580.199760] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7580.209881] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7580.222627] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7580.228608] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7580.236415] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7580.246237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7580.252724] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7580.262722] Call Trace: [ 7580.266225] [ 7580.272117] ? show_regs.cold.9+0x22/0x2f [ 7580.276348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7580.282797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7580.288748] ? watchdog+0x40/0x40 [ 7580.293812] ? hrtimer_interrupt+0x128/0x2c0 [ 7580.299693] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7580.304475] ? apic_timer_interrupt+0xf/0x20 [ 7580.309169] [ 7580.312203] ? native_safe_halt+0x2e/0x50 [ 7580.315587] kvm_wait+0xcd/0xe0 [ 7580.319209] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7580.323570] do_raw_spin_lock+0x177/0x190 [ 7580.329414] _raw_spin_lock+0x21/0x30 [ 7580.331025] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7580.336858] obd_mod_free+0x46/0x140 [ptlrpc] [ 7580.342886] mdc_commit_open+0x91/0xb0 [mdc] [ 7580.346545] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7580.355242] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7580.361724] after_reply+0xe2c/0x1780 [ptlrpc] [ 7580.367735] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7580.371527] ? __next_timer_interrupt+0x160/0x160 [ 7580.376461] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7580.389523] ? do_wait_intr+0xf0/0xf0 [ 7580.390970] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7580.397485] kthread+0x1d1/0x200 [ 7580.398896] ? set_kthread_struct+0x70/0x70 [ 7580.414269] ret_from_fork+0x1f/0x30 [ 7584.057076] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 7584.072778] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7584.146319] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7584.162421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7584.178598] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7584.191682] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7584.202988] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7584.231787] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7584.243715] RAX: 0000000000002d88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7584.259636] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7584.268236] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7584.282604] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7584.296264] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7584.309310] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7584.327078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7584.340633] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7584.352601] Call Trace: [ 7584.357501] [ 7584.360948] ? show_regs.cold.9+0x22/0x2f [ 7584.367410] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7584.372636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7584.381958] ? watchdog+0x40/0x40 [ 7584.389790] ? hrtimer_interrupt+0x128/0x2c0 [ 7584.399136] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7584.409954] ? apic_timer_interrupt+0xf/0x20 [ 7584.418745] [ 7584.422239] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7584.431709] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7584.433762] do_raw_spin_lock+0x177/0x190 [ 7584.440728] _raw_spin_lock+0x21/0x30 [ 7584.447309] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7584.452771] process_one_work+0x2c8/0x700 [ 7584.459176] worker_thread+0x296/0x6e0 [ 7584.467175] ? rescuer_thread+0x570/0x570 [ 7584.471802] kthread+0x1d1/0x200 [ 7584.481434] ? set_kthread_struct+0x70/0x70 [ 7584.489147] ret_from_fork+0x1f/0x30 [ 7596.054011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7596.056778] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7596.088078] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7596.094688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7596.098891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7596.101864] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7596.116396] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7596.118841] RAX: 0000000000004271 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7596.123046] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7596.126019] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7596.128983] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7596.132053] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7596.134901] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7596.138240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7596.140570] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7596.143580] Call Trace: [ 7596.144613] [ 7596.145453] ? show_regs.cold.9+0x22/0x2f [ 7596.147218] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7596.149715] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7596.151551] ? watchdog+0x40/0x40 [ 7596.153026] ? hrtimer_interrupt+0x128/0x2c0 [ 7596.154762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7596.156793] ? apic_timer_interrupt+0xf/0x20 [ 7596.158579] [ 7596.159402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7596.161505] do_raw_spin_lock+0x177/0x190 [ 7596.163366] _raw_spin_lock+0x21/0x30 [ 7596.164922] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7596.167379] ? do_raw_spin_unlock+0x75/0x190 [ 7596.169127] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7596.171337] ? do_wait_intr+0xf0/0xf0 [ 7596.172846] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7596.175815] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7596.178442] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7596.181543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.184012] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7596.186275] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7596.188253] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7596.190382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.192778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7596.195892] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7596.200577] lmv_intent_open+0x338/0x1150 [lmv] [ 7596.202976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.206486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.209708] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7596.213199] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7596.215556] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.220788] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7596.223014] ? do_raw_spin_unlock+0x75/0x190 [ 7596.225319] ? _raw_spin_unlock+0x12/0x30 [ 7596.227854] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7596.235851] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7596.239299] ll_atomic_open+0x36a/0xd70 [lustre] [ 7596.241799] lookup_open+0x356/0xb40 [ 7596.243388] do_last+0x313/0xfc0 [ 7596.245190] ? nd_jump_root+0xe5/0x160 [ 7596.248000] ? path_init+0x437/0x520 [ 7596.249551] path_openat+0xf7/0x500 [ 7596.251451] ? do_raw_spin_unlock+0x75/0x190 [ 7596.253262] do_filp_open+0x99/0x140 [ 7596.254765] ? getname_flags+0x6e/0x330 [ 7596.256171] ? __check_object_size+0xff/0x256 [ 7596.257902] ? do_raw_spin_unlock+0x75/0x190 [ 7596.260114] ? _raw_spin_unlock+0x12/0x30 [ 7596.263366] do_sys_openat2+0x2b4/0x410 [ 7596.267335] do_sys_open+0x73/0xa0 [ 7596.268617] __x64_sys_openat+0x24/0x30 [ 7596.270525] do_syscall_64+0xc1/0x440 [ 7596.271850] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7596.275127] RIP: 0033:0x7fa749552332 [ 7596.277490] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7596.289810] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7596.294092] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7596.296646] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7596.301529] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7596.309739] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7596.313923] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7599.073940] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5681s! [ 7599.078971] Showing busy workqueues and worker pools: [ 7599.082564] workqueue events: flags=0x0 [ 7599.086656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7599.092177] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7599.099743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7599.104321] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7599.113692] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.122278] pending: kfree_rcu_monitor [ 7599.125562] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7599.128537] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7599.133031] workqueue events_long: flags=0x0 [ 7599.135637] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.140365] pending: lu_objects_flush [obdclass] [ 7599.142905] workqueue events_unbound: flags=0x2 [ 7599.147260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7599.150394] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7599.155370] workqueue events_power_efficient: flags=0x80 [ 7599.157533] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.161340] pending: check_lifetime [ 7599.163819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.167638] pending: neigh_periodic_work [ 7599.170165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.180484] pending: neigh_periodic_work [ 7599.189432] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.201377] pending: do_cache_clean [ 7599.214711] workqueue mm_percpu_wq: flags=0x8 [ 7599.223957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.230468] pending: vmstat_update [ 7599.237505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.247161] pending: vmstat_update [ 7599.252009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7599.261542] pending: vmstat_update [ 7599.266557] workqueue writeback: flags=0x4a [ 7599.272255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7599.277143] pending: wb_workfn, wb_workfn [ 7599.278344] workqueue xprtiod: flags=0xa [ 7599.279734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7599.289290] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7599.298141] workqueue ipv6_addrconf: flags=0x40008 [ 7599.310135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7599.314988] pending: addrconf_verify_work [ 7599.317152] workqueue knbd0-recv: flags=0x1a [ 7599.327835] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7599.335305] in-flight: 651:recv_work [ 7599.339450] workqueue ptlrpc_pinger: flags=0x2 [ 7599.342495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7599.352477] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7599.358457] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5682s workers=4 idle: 37 31846 28897 [ 7599.365083] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7589s workers=2 idle: 653 [ 7604.059975] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7604.063213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7604.094492] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7604.101296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7604.107085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7604.114633] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7604.131917] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7604.140055] RAX: 00000000000077e0 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7604.147151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7604.149997] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7604.154813] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7604.157885] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7604.161616] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7604.166164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7604.168333] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7604.171721] Call Trace: [ 7604.174768] [ 7604.175687] ? show_regs.cold.9+0x22/0x2f [ 7604.177537] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7604.180125] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7604.182692] ? watchdog+0x40/0x40 [ 7604.184107] ? hrtimer_interrupt+0x128/0x2c0 [ 7604.185984] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7604.188030] ? apic_timer_interrupt+0xf/0x20 [ 7604.193926] [ 7604.195359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7604.198728] do_raw_spin_lock+0x177/0x190 [ 7604.200793] _raw_spin_lock+0x21/0x30 [ 7604.203812] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7604.207018] ? do_raw_spin_unlock+0x75/0x190 [ 7604.209900] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7604.216084] ? do_wait_intr+0xf0/0xf0 [ 7604.219391] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7604.224609] mdc_close+0x53b/0xd70 [mdc] [ 7604.226692] lmv_close+0x18d/0x490 [lmv] [ 7604.228410] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7604.231072] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7604.234823] ll_md_close+0x274/0x800 [lustre] [ 7604.244457] ll_file_release+0x128/0xc50 [lustre] [ 7604.248093] __fput+0x10d/0x440 [ 7604.251852] ____fput+0x12/0x20 [ 7604.254214] task_work_run+0xc6/0x110 [ 7604.255561] exit_to_usermode_loop+0x1dd/0x1f0 [ 7604.266310] do_syscall_64+0x3d6/0x440 [ 7604.268071] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7604.270838] RIP: 0033:0x7fa749552cd8 [ 7604.272641] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7604.284804] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7604.288937] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7604.296098] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7604.309560] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7604.311622] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7604.317295] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7608.058016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7608.061594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.089038] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.094082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.097448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7608.100360] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7608.105659] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7608.108819] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7608.112531] RDX: 000000000000685c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7608.115861] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7608.119226] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7608.122325] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7608.124825] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7608.127592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.129049] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7608.130849] Call Trace: [ 7608.131714] [ 7608.132311] ? show_regs.cold.9+0x22/0x2f [ 7608.133695] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.135606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.137076] ? watchdog+0x40/0x40 [ 7608.138236] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.139769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.141632] ? apic_timer_interrupt+0xf/0x20 [ 7608.143038] [ 7608.143731] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7608.145553] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7608.147233] do_raw_spin_lock+0x177/0x190 [ 7608.148672] _raw_spin_lock+0x21/0x30 [ 7608.149933] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7608.152073] obd_mod_free+0x46/0x140 [ptlrpc] [ 7608.153978] mdc_commit_open+0x91/0xb0 [mdc] [ 7608.155559] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7608.157499] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7608.159348] after_reply+0xe2c/0x1780 [ptlrpc] [ 7608.161045] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7608.163287] ? __next_timer_interrupt+0x160/0x160 [ 7608.165102] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7608.166940] ? do_wait_intr+0xf0/0xf0 [ 7608.168379] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7608.170518] kthread+0x1d1/0x200 [ 7608.171693] ? set_kthread_struct+0x70/0x70 [ 7608.173020] ret_from_fork+0x1f/0x30 [ 7612.057027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 7612.058933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7612.093990] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7612.096373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7612.107245] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7612.110716] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7612.111803] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7612.128998] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7612.133057] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7612.141809] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7612.146023] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7612.153388] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7612.156812] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7612.162579] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7612.171022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7612.174296] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7612.177882] Call Trace: [ 7612.181288] [ 7612.182209] ? show_regs.cold.9+0x22/0x2f [ 7612.183901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7612.186320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7612.188240] ? watchdog+0x40/0x40 [ 7612.192312] ? hrtimer_interrupt+0x128/0x2c0 [ 7612.194806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7612.197124] ? apic_timer_interrupt+0xf/0x20 [ 7612.199142] [ 7612.201043] ? native_safe_halt+0x2e/0x50 [ 7612.203165] kvm_wait+0xcd/0xe0 [ 7612.204811] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7612.207235] do_raw_spin_lock+0x177/0x190 [ 7612.208622] _raw_spin_lock+0x21/0x30 [ 7612.210665] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7612.213211] process_one_work+0x2c8/0x700 [ 7612.215237] worker_thread+0x296/0x6e0 [ 7612.216854] ? rescuer_thread+0x570/0x570 [ 7612.221273] kthread+0x1d1/0x200 [ 7612.224024] ? set_kthread_struct+0x70/0x70 [ 7612.228275] ret_from_fork+0x1f/0x30 [ 7624.055601] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7624.060105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7624.102086] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7624.107652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7624.112581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7624.116007] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7624.127107] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7624.131146] RAX: 0000000000007d2a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7624.134893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7624.138495] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7624.147978] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7624.153141] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7624.162888] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7624.170910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7624.173928] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7624.177495] Call Trace: [ 7624.178313] [ 7624.179073] ? show_regs.cold.9+0x22/0x2f [ 7624.180942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7624.183107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7624.185902] ? watchdog+0x40/0x40 [ 7624.187850] ? hrtimer_interrupt+0x128/0x2c0 [ 7624.190241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7624.198077] ? apic_timer_interrupt+0xf/0x20 [ 7624.199874] [ 7624.203465] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7624.207098] do_raw_spin_lock+0x177/0x190 [ 7624.209430] _raw_spin_lock+0x21/0x30 [ 7624.211111] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7624.216970] ? do_raw_spin_unlock+0x75/0x190 [ 7624.219819] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7624.224005] ? do_wait_intr+0xf0/0xf0 [ 7624.227444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7624.230108] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7624.235853] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7624.239010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.241997] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7624.244935] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7624.247330] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7624.250101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.255062] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7624.258669] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7624.261298] lmv_intent_open+0x338/0x1150 [lmv] [ 7624.264370] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.266410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.269604] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7624.272551] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7624.275568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.278569] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7624.281583] ? do_raw_spin_unlock+0x75/0x190 [ 7624.287100] ? _raw_spin_unlock+0x12/0x30 [ 7624.288826] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7624.291524] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7624.293579] ll_atomic_open+0x36a/0xd70 [lustre] [ 7624.296229] lookup_open+0x356/0xb40 [ 7624.298136] do_last+0x313/0xfc0 [ 7624.300496] ? nd_jump_root+0xe5/0x160 [ 7624.302053] ? path_init+0x437/0x520 [ 7624.304006] path_openat+0xf7/0x500 [ 7624.307484] ? do_raw_spin_unlock+0x75/0x190 [ 7624.309598] do_filp_open+0x99/0x140 [ 7624.311849] ? getname_flags+0x6e/0x330 [ 7624.313551] ? __check_object_size+0xff/0x256 [ 7624.315972] ? do_raw_spin_unlock+0x75/0x190 [ 7624.317965] ? _raw_spin_unlock+0x12/0x30 [ 7624.319910] do_sys_openat2+0x2b4/0x410 [ 7624.321773] do_sys_open+0x73/0xa0 [ 7624.323113] __x64_sys_openat+0x24/0x30 [ 7624.325284] do_syscall_64+0xc1/0x440 [ 7624.326924] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7624.330075] RIP: 0033:0x7fa749552332 [ 7624.331873] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7624.340599] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7624.344022] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7624.348592] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7624.351664] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7624.355978] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7624.359897] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7629.795776] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5712s! [ 7629.798592] Showing busy workqueues and worker pools: [ 7629.800091] workqueue events: flags=0x0 [ 7629.801610] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7629.804545] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7629.814630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7629.827657] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7629.835892] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.848628] pending: kfree_rcu_monitor [ 7629.852787] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7629.864259] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7629.876286] workqueue events_long: flags=0x0 [ 7629.882172] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.888852] pending: lu_objects_flush [obdclass] [ 7629.899308] workqueue events_unbound: flags=0x2 [ 7629.903641] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7629.912022] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7629.922097] workqueue events_power_efficient: flags=0x80 [ 7629.929582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.934150] pending: check_lifetime [ 7629.937724] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.946268] pending: neigh_periodic_work [ 7629.948541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.953814] pending: neigh_periodic_work [ 7629.956828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.962178] pending: do_cache_clean [ 7629.965239] workqueue mm_percpu_wq: flags=0x8 [ 7629.969995] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.973417] pending: vmstat_update [ 7629.975768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.978453] pending: vmstat_update [ 7629.979830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7629.982766] pending: vmstat_update [ 7629.984543] workqueue writeback: flags=0x4a [ 7629.986576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7629.989047] pending: wb_workfn, wb_workfn [ 7629.990795] workqueue xprtiod: flags=0xa [ 7629.992091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7629.994177] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7629.997545] workqueue ipv6_addrconf: flags=0x40008 [ 7629.999620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7630.002288] pending: addrconf_verify_work [ 7630.004166] workqueue knbd0-recv: flags=0x1a [ 7630.005715] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7630.008570] in-flight: 651:recv_work [ 7630.010021] workqueue ptlrpc_pinger: flags=0x2 [ 7630.012173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7630.015302] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7630.018367] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5712s workers=4 idle: 37 31846 28897 [ 7630.022027] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7619s workers=2 idle: 653 [ 7632.059032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7632.063299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7632.091099] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7632.097532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7632.101644] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7632.103447] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7632.114746] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7632.121725] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7632.131076] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7632.134417] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7632.140462] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7632.143840] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7632.147679] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7632.155184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7632.165659] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7632.174882] Call Trace: [ 7632.178305] [ 7632.179332] ? show_regs.cold.9+0x22/0x2f [ 7632.186048] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7632.191899] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7632.194809] ? watchdog+0x40/0x40 [ 7632.196392] ? hrtimer_interrupt+0x128/0x2c0 [ 7632.199209] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7632.202231] ? apic_timer_interrupt+0xf/0x20 [ 7632.206267] [ 7632.207936] ? native_safe_halt+0x2e/0x50 [ 7632.214411] kvm_wait+0xcd/0xe0 [ 7632.220901] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7632.228532] do_raw_spin_lock+0x177/0x190 [ 7632.234805] _raw_spin_lock+0x21/0x30 [ 7632.241149] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7632.245095] ? do_raw_spin_unlock+0x75/0x190 [ 7632.248040] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7632.251047] ? do_wait_intr+0xf0/0xf0 [ 7632.255454] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7632.261319] mdc_close+0x53b/0xd70 [mdc] [ 7632.263652] lmv_close+0x18d/0x490 [lmv] [ 7632.265342] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7632.272295] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7632.275500] ll_md_close+0x274/0x800 [lustre] [ 7632.277205] ll_file_release+0x128/0xc50 [lustre] [ 7632.280793] __fput+0x10d/0x440 [ 7632.284790] ____fput+0x12/0x20 [ 7632.287375] task_work_run+0xc6/0x110 [ 7632.291864] exit_to_usermode_loop+0x1dd/0x1f0 [ 7632.293895] do_syscall_64+0x3d6/0x440 [ 7632.295237] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7632.302153] RIP: 0033:0x7fa749552cd8 [ 7632.304308] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7632.317359] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7632.324388] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7632.328030] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7632.332418] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7632.339464] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7632.346053] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7636.060091] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7636.072288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7636.109041] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7636.116611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7636.124522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7636.128996] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7636.141081] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7636.149093] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7636.153950] RDX: 00000000000075a4 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7636.158344] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7636.167995] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7636.171692] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7636.174648] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7636.178669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7636.180611] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7636.186846] Call Trace: [ 7636.187741] [ 7636.188428] ? show_regs.cold.9+0x22/0x2f [ 7636.190492] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7636.193321] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7636.195863] ? watchdog+0x40/0x40 [ 7636.199439] ? hrtimer_interrupt+0x128/0x2c0 [ 7636.201597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7636.204103] ? apic_timer_interrupt+0xf/0x20 [ 7636.206325] [ 7636.208081] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7636.212742] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7636.216596] do_raw_spin_lock+0x177/0x190 [ 7636.217935] _raw_spin_lock+0x21/0x30 [ 7636.219181] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7636.223247] obd_mod_free+0x46/0x140 [ptlrpc] [ 7636.225355] mdc_commit_open+0x91/0xb0 [mdc] [ 7636.232694] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7636.239348] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7636.243342] after_reply+0xe2c/0x1780 [ptlrpc] [ 7636.245690] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7636.248039] ? __next_timer_interrupt+0x160/0x160 [ 7636.250085] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7636.252302] ? do_wait_intr+0xf0/0xf0 [ 7636.257395] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7636.261984] kthread+0x1d1/0x200 [ 7636.263429] ? set_kthread_struct+0x70/0x70 [ 7636.265219] ret_from_fork+0x1f/0x30 [ 7640.057062] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7640.060238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7640.114777] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7640.119621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7640.123501] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7640.127082] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7640.129584] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7640.138109] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7640.141148] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7640.143656] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7640.146871] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7640.150400] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7640.153693] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7640.156008] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7640.162465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7640.165643] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7640.168680] Call Trace: [ 7640.170697] [ 7640.171328] ? show_regs.cold.9+0x22/0x2f [ 7640.173129] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7640.175210] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7640.178315] ? watchdog+0x40/0x40 [ 7640.179956] ? hrtimer_interrupt+0x128/0x2c0 [ 7640.182671] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7640.184642] ? apic_timer_interrupt+0xf/0x20 [ 7640.187029] [ 7640.187852] ? native_safe_halt+0x2e/0x50 [ 7640.189839] kvm_wait+0xcd/0xe0 [ 7640.192941] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7640.195532] do_raw_spin_lock+0x177/0x190 [ 7640.197170] _raw_spin_lock+0x21/0x30 [ 7640.199115] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7640.201323] process_one_work+0x2c8/0x700 [ 7640.202585] worker_thread+0x296/0x6e0 [ 7640.205822] ? rescuer_thread+0x570/0x570 [ 7640.206872] kthread+0x1d1/0x200 [ 7640.207816] ? set_kthread_struct+0x70/0x70 [ 7640.209618] ret_from_fork+0x1f/0x30 [ 7656.054998] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7656.062111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7656.098415] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7656.104959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7656.108850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7656.111573] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7656.120224] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7656.123515] RAX: 0000000000002287 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7656.126970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7656.132357] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7656.135996] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7656.140003] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7656.143542] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7656.147359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7656.150398] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7656.154844] Call Trace: [ 7656.156065] [ 7656.156895] ? show_regs.cold.9+0x22/0x2f [ 7656.158967] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7656.162786] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7656.165349] ? watchdog+0x40/0x40 [ 7656.168104] ? hrtimer_interrupt+0x128/0x2c0 [ 7656.172818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7656.179140] ? apic_timer_interrupt+0xf/0x20 [ 7656.181228] [ 7656.182690] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7656.186326] do_raw_spin_lock+0x177/0x190 [ 7656.188252] _raw_spin_lock+0x21/0x30 [ 7656.189923] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7656.193403] ? do_raw_spin_unlock+0x75/0x190 [ 7656.195765] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7656.198949] ? do_wait_intr+0xf0/0xf0 [ 7656.201290] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7656.203387] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7656.205906] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7656.208797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7656.211470] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7656.213672] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7656.215125] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7656.217218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7656.220219] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7656.223187] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7656.225590] lmv_intent_open+0x338/0x1150 [lmv] [ 7656.228140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7656.230900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7656.235017] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7656.238170] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7656.240657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7656.242675] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7656.245950] ? do_raw_spin_unlock+0x75/0x190 [ 7656.247970] ? _raw_spin_unlock+0x12/0x30 [ 7656.250353] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7656.253189] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7656.257134] ll_atomic_open+0x36a/0xd70 [lustre] [ 7656.260370] lookup_open+0x356/0xb40 [ 7656.264444] do_last+0x313/0xfc0 [ 7656.266447] ? nd_jump_root+0xe5/0x160 [ 7656.270115] ? path_init+0x437/0x520 [ 7656.271821] path_openat+0xf7/0x500 [ 7656.273841] ? do_raw_spin_unlock+0x75/0x190 [ 7656.275953] do_filp_open+0x99/0x140 [ 7656.278462] ? getname_flags+0x6e/0x330 [ 7656.281550] ? __check_object_size+0xff/0x256 [ 7656.284011] ? do_raw_spin_unlock+0x75/0x190 [ 7656.285404] ? _raw_spin_unlock+0x12/0x30 [ 7656.287695] do_sys_openat2+0x2b4/0x410 [ 7656.288854] do_sys_open+0x73/0xa0 [ 7656.290982] __x64_sys_openat+0x24/0x30 [ 7656.293365] do_syscall_64+0xc1/0x440 [ 7656.294366] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7656.295668] RIP: 0033:0x7fa749552332 [ 7656.296699] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7656.314152] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7656.317437] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7656.322553] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7656.325333] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7656.327885] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7656.331939] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7660.059243] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7660.064749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7660.095216] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7660.101472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7660.112448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7660.114783] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7660.148804] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7660.154665] RAX: 0000000000003e88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7660.161046] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7660.164162] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7660.166170] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7660.168219] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7660.175627] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7660.183297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7660.187917] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7660.194661] Call Trace: [ 7660.198570] [ 7660.199666] ? show_regs.cold.9+0x22/0x2f [ 7660.202317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7660.208183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7660.212037] ? watchdog+0x40/0x40 [ 7660.219262] ? hrtimer_interrupt+0x128/0x2c0 [ 7660.223284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7660.230771] ? apic_timer_interrupt+0xf/0x20 [ 7660.237260] [ 7660.240405] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7660.245334] do_raw_spin_lock+0x177/0x190 [ 7660.249828] _raw_spin_lock+0x21/0x30 [ 7660.254115] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7660.260319] ? do_raw_spin_unlock+0x75/0x190 [ 7660.265288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7660.271812] ? do_wait_intr+0xf0/0xf0 [ 7660.277048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7660.283899] mdc_close+0x53b/0xd70 [mdc] [ 7660.288553] lmv_close+0x18d/0x490 [lmv] [ 7660.293296] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7660.299754] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7660.304905] ll_md_close+0x274/0x800 [lustre] [ 7660.311602] ll_file_release+0x128/0xc50 [lustre] [ 7660.316776] __fput+0x10d/0x440 [ 7660.319096] ____fput+0x12/0x20 [ 7660.323882] task_work_run+0xc6/0x110 [ 7660.325253] exit_to_usermode_loop+0x1dd/0x1f0 [ 7660.326470] do_syscall_64+0x3d6/0x440 [ 7660.333841] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7660.338959] RIP: 0033:0x7fa749552cd8 [ 7660.343420] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7660.356839] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7660.359871] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7660.363832] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7660.369768] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7660.375871] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7660.379954] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7660.513053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5743s! [ 7660.515397] Showing busy workqueues and worker pools: [ 7660.516647] workqueue events: flags=0x0 [ 7660.517667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7660.519431] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7660.525292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7660.531413] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7660.536304] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.540836] pending: kfree_rcu_monitor [ 7660.543543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7660.546471] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7660.550469] workqueue events_long: flags=0x0 [ 7660.552471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.555402] pending: lu_objects_flush [obdclass] [ 7660.558716] workqueue events_unbound: flags=0x2 [ 7660.560945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7660.564227] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7660.571222] workqueue events_power_efficient: flags=0x80 [ 7660.574915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.583634] pending: check_lifetime [ 7660.585861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.591609] pending: neigh_periodic_work [ 7660.595291] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.600480] pending: neigh_periodic_work [ 7660.602892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.607948] pending: do_cache_clean [ 7660.611542] workqueue mm_percpu_wq: flags=0x8 [ 7660.614660] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.619211] pending: vmstat_update [ 7660.621941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.626615] pending: vmstat_update [ 7660.630195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7660.634328] pending: vmstat_update [ 7660.638572] workqueue writeback: flags=0x4a [ 7660.642235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7660.647250] pending: wb_workfn, wb_workfn [ 7660.650297] workqueue xprtiod: flags=0xa [ 7660.653575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7660.658751] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7660.665467] workqueue ipv6_addrconf: flags=0x40008 [ 7660.666766] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7660.668517] pending: addrconf_verify_work [ 7660.674874] workqueue knbd0-recv: flags=0x1a [ 7660.678583] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7660.683253] in-flight: 651:recv_work [ 7660.685907] workqueue ptlrpc_pinger: flags=0x2 [ 7660.688810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7660.691260] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7660.696490] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5743s workers=4 idle: 37 31846 28897 [ 7660.702236] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7650s workers=2 idle: 653 [ 7664.058007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7664.062886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7664.094713] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7664.105603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7664.117106] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7664.126100] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7664.144145] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7664.155000] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7664.162570] RDX: 00000000000024ed RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7664.167757] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7664.172633] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7664.178068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7664.182402] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7664.190429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7664.200734] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7664.208454] Call Trace: [ 7664.209319] [ 7664.210030] ? show_regs.cold.9+0x22/0x2f [ 7664.215617] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7664.220910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7664.222604] ? watchdog+0x40/0x40 [ 7664.224649] ? hrtimer_interrupt+0x128/0x2c0 [ 7664.227760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7664.234124] ? apic_timer_interrupt+0xf/0x20 [ 7664.239059] [ 7664.243294] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7664.249019] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7664.252821] do_raw_spin_lock+0x177/0x190 [ 7664.256962] _raw_spin_lock+0x21/0x30 [ 7664.258604] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7664.263274] obd_mod_free+0x46/0x140 [ptlrpc] [ 7664.268824] mdc_commit_open+0x91/0xb0 [mdc] [ 7664.271977] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7664.278931] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7664.283407] after_reply+0xe2c/0x1780 [ptlrpc] [ 7664.285771] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7664.288538] ? __next_timer_interrupt+0x160/0x160 [ 7664.295602] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7664.297829] ? do_wait_intr+0xf0/0xf0 [ 7664.304139] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7664.307395] kthread+0x1d1/0x200 [ 7664.311628] ? set_kthread_struct+0x70/0x70 [ 7664.313113] ret_from_fork+0x1f/0x30 [ 7668.057423] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7668.062091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7668.080642] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7668.095626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7668.099091] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7668.103346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7668.112663] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7668.128205] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7668.131441] RAX: 0000000000004e8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7668.140222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7668.145176] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7668.150551] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7668.155972] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7668.162196] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7668.167555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7668.171457] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7668.176362] Call Trace: [ 7668.177091] [ 7668.177640] ? show_regs.cold.9+0x22/0x2f [ 7668.185960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7668.189274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7668.194931] ? watchdog+0x40/0x40 [ 7668.206294] ? hrtimer_interrupt+0x128/0x2c0 [ 7668.207829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7668.211604] ? apic_timer_interrupt+0xf/0x20 [ 7668.213499] [ 7668.214106] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7668.215904] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7668.223524] do_raw_spin_lock+0x177/0x190 [ 7668.225357] _raw_spin_lock+0x21/0x30 [ 7668.231693] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7668.239076] process_one_work+0x2c8/0x700 [ 7668.244026] worker_thread+0x296/0x6e0 [ 7668.247486] ? rescuer_thread+0x570/0x570 [ 7668.249801] kthread+0x1d1/0x200 [ 7668.253986] ? set_kthread_struct+0x70/0x70 [ 7668.257280] ret_from_fork+0x1f/0x30 [ 7688.056051] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7688.060184] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7688.065284] Modules linked in: [ 7688.069660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7688.069831] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7688.069839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7688.069842] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7688.069865] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7688.069870] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7688.069874] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7688.069877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7688.069880] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7688.069882] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7688.069884] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7688.069888] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7688.069891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7688.069894] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7688.069902] Call Trace: [ 7688.069918] [ 7688.069925] ? show_regs.cold.9+0x22/0x2f [ 7688.069937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7688.069943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7688.069947] ? watchdog+0x40/0x40 [ 7688.069951] ? hrtimer_interrupt+0x128/0x2c0 [ 7688.069955] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7688.069961] ? apic_timer_interrupt+0xf/0x20 [ 7688.069965] [ 7688.069968] ? native_safe_halt+0x2e/0x50 [ 7688.069972] kvm_wait+0xcd/0xe0 [ 7688.069979] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7688.069988] do_raw_spin_lock+0x177/0x190 [ 7688.069992] _raw_spin_lock+0x21/0x30 [ 7688.069996] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7688.070646] ? do_raw_spin_unlock+0x75/0x190 [ 7688.070653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7688.070848] ? do_wait_intr+0xf0/0xf0 [ 7688.070856] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7688.071312] mdc_close+0x53b/0xd70 [mdc] [ 7688.071358] lmv_close+0x18d/0x490 [lmv] [ 7688.071453] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7688.071565] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7688.071643] ll_md_close+0x274/0x800 [lustre] [ 7688.071725] ll_file_release+0x128/0xc50 [lustre] [ 7688.071863] __fput+0x10d/0x440 [ 7688.071875] ____fput+0x12/0x20 [ 7688.071879] task_work_run+0xc6/0x110 [ 7688.071891] exit_to_usermode_loop+0x1dd/0x1f0 [ 7688.071899] do_syscall_64+0x3d6/0x440 [ 7688.071904] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7688.071912] RIP: 0033:0x7fa749552cd8 [ 7688.071944] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7688.071948] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7688.071953] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7688.071955] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7688.071962] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7688.071965] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7688.071968] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7688.295396] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7688.317572] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7688.323368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7688.327942] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7688.330366] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7688.343288] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7688.347929] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7688.350470] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7688.354663] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7688.357736] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7688.361969] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7688.367420] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7688.374347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7688.379703] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7688.382876] Call Trace: [ 7688.384594] [ 7688.385628] ? show_regs.cold.9+0x22/0x2f [ 7688.387182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7688.389348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7688.392724] ? watchdog+0x40/0x40 [ 7688.394205] ? hrtimer_interrupt+0x128/0x2c0 [ 7688.396203] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7688.398204] ? apic_timer_interrupt+0xf/0x20 [ 7688.400952] [ 7688.401932] ? native_safe_halt+0x2e/0x50 [ 7688.404203] kvm_wait+0xcd/0xe0 [ 7688.406745] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7688.409873] do_raw_spin_lock+0x177/0x190 [ 7688.411687] _raw_spin_lock+0x21/0x30 [ 7688.414175] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7688.416072] ? do_raw_spin_unlock+0x75/0x190 [ 7688.418068] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7688.421234] ? do_wait_intr+0xf0/0xf0 [ 7688.423004] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7688.425821] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7688.430145] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7688.433958] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7688.436541] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7688.439877] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7688.443819] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7688.446064] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7688.449616] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7688.453105] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7688.455319] lmv_intent_open+0x338/0x1150 [lmv] [ 7688.456804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7688.463411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7688.466807] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7688.469287] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7688.471165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7688.474155] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7688.477004] ? do_raw_spin_unlock+0x75/0x190 [ 7688.478939] ? _raw_spin_unlock+0x12/0x30 [ 7688.483810] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7688.486378] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7688.489179] ll_atomic_open+0x36a/0xd70 [lustre] [ 7688.493410] lookup_open+0x356/0xb40 [ 7688.495800] do_last+0x313/0xfc0 [ 7688.496864] ? nd_jump_root+0xe5/0x160 [ 7688.499219] ? path_init+0x437/0x520 [ 7688.502374] path_openat+0xf7/0x500 [ 7688.504203] ? do_raw_spin_unlock+0x75/0x190 [ 7688.506591] do_filp_open+0x99/0x140 [ 7688.507963] ? getname_flags+0x6e/0x330 [ 7688.509673] ? __check_object_size+0xff/0x256 [ 7688.511267] ? do_raw_spin_unlock+0x75/0x190 [ 7688.513979] ? _raw_spin_unlock+0x12/0x30 [ 7688.515342] do_sys_openat2+0x2b4/0x410 [ 7688.518100] do_sys_open+0x73/0xa0 [ 7688.521888] __x64_sys_openat+0x24/0x30 [ 7688.524230] do_syscall_64+0xc1/0x440 [ 7688.525694] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7688.527842] RIP: 0033:0x7fa749552332 [ 7688.530185] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7688.540165] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7688.546178] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7688.549617] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7688.553465] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7688.557150] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7688.560620] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7690.720250] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5773s! [ 7690.722944] Showing busy workqueues and worker pools: [ 7690.725451] workqueue events: flags=0x0 [ 7690.727392] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7690.730636] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7690.735239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7690.738500] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7690.743964] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.747888] pending: kfree_rcu_monitor [ 7690.750111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7690.754824] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7690.757842] workqueue events_long: flags=0x0 [ 7690.759096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.761309] pending: lu_objects_flush [obdclass] [ 7690.763247] workqueue events_unbound: flags=0x2 [ 7690.764829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7690.766890] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7690.770122] workqueue events_power_efficient: flags=0x80 [ 7690.771682] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.773775] pending: check_lifetime [ 7690.774914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.777175] pending: neigh_periodic_work [ 7690.778601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.780946] pending: neigh_periodic_work [ 7690.782788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.785226] pending: do_cache_clean [ 7690.786679] workqueue mm_percpu_wq: flags=0x8 [ 7690.788169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.790767] pending: vmstat_update [ 7690.792555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.795746] pending: vmstat_update [ 7690.798537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7690.802453] pending: vmstat_update [ 7690.804671] workqueue writeback: flags=0x4a [ 7690.806411] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7690.808749] pending: wb_workfn, wb_workfn [ 7690.810861] workqueue xprtiod: flags=0xa [ 7690.812380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7690.816096] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7690.821621] workqueue ipv6_addrconf: flags=0x40008 [ 7690.823353] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7690.825740] pending: addrconf_verify_work [ 7690.828367] workqueue knbd0-recv: flags=0x1a [ 7690.830450] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7690.833599] in-flight: 651:recv_work [ 7690.835422] workqueue ptlrpc_pinger: flags=0x2 [ 7690.837693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7690.839976] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7690.841929] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5773s workers=4 idle: 37 31846 28897 [ 7690.845231] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7680s workers=2 idle: 653 [ 7692.058065] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7692.065000] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7692.103927] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7692.108735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7692.113588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7692.117440] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7692.125059] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7692.127789] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7692.130131] RDX: 00000000000002d5 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7692.134067] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7692.138616] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7692.140497] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7692.143090] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7692.147073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7692.150485] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7692.155130] Call Trace: [ 7692.157471] [ 7692.159824] ? show_regs.cold.9+0x22/0x2f [ 7692.161551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7692.163412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7692.165200] ? watchdog+0x40/0x40 [ 7692.166470] ? hrtimer_interrupt+0x128/0x2c0 [ 7692.169003] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7692.170467] ? apic_timer_interrupt+0xf/0x20 [ 7692.172172] [ 7692.174883] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7692.176640] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7692.181055] do_raw_spin_lock+0x177/0x190 [ 7692.183812] _raw_spin_lock+0x21/0x30 [ 7692.185058] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7692.187995] obd_mod_free+0x46/0x140 [ptlrpc] [ 7692.189537] mdc_commit_open+0x91/0xb0 [mdc] [ 7692.193215] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7692.194838] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7692.199166] after_reply+0xe2c/0x1780 [ptlrpc] [ 7692.201455] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7692.203407] ? __next_timer_interrupt+0x160/0x160 [ 7692.205057] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7692.206553] ? do_wait_intr+0xf0/0xf0 [ 7692.207613] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7692.209512] kthread+0x1d1/0x200 [ 7692.210570] ? set_kthread_struct+0x70/0x70 [ 7692.211811] ret_from_fork+0x1f/0x30 [ 7696.060490] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7696.065428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7696.110790] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7696.116776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7696.121129] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7696.126592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7696.129652] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7696.135720] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7696.140470] RAX: 0000000000007b01 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7696.143706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7696.149964] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7696.155962] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7696.157300] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7696.158988] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7696.160844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7696.162224] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7696.163761] Call Trace: [ 7696.164422] [ 7696.165049] ? show_regs.cold.9+0x22/0x2f [ 7696.165991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7696.169396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7696.170819] ? watchdog+0x40/0x40 [ 7696.171625] ? hrtimer_interrupt+0x128/0x2c0 [ 7696.172608] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7696.173683] ? apic_timer_interrupt+0xf/0x20 [ 7696.174676] [ 7696.175243] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7696.177754] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7696.179184] do_raw_spin_lock+0x177/0x190 [ 7696.180457] _raw_spin_lock+0x21/0x30 [ 7696.181414] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7696.185073] process_one_work+0x2c8/0x700 [ 7696.186149] worker_thread+0x296/0x6e0 [ 7696.187204] ? rescuer_thread+0x570/0x570 [ 7696.188226] kthread+0x1d1/0x200 [ 7696.188989] ? set_kthread_struct+0x70/0x70 [ 7696.191687] ret_from_fork+0x1f/0x30 [ 7716.054031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7716.057500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7716.062593] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7716.075197] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7716.075210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7716.075212] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7716.075233] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7716.075237] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7716.075242] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7716.075245] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7716.075247] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7716.075249] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7716.075252] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7716.075255] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7716.075257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7716.075259] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7716.075267] Call Trace: [ 7716.075272] [ 7716.075276] ? show_regs.cold.9+0x22/0x2f [ 7716.075284] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7716.075290] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7716.075298] ? watchdog+0x40/0x40 [ 7716.075302] ? hrtimer_interrupt+0x128/0x2c0 [ 7716.075305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7716.075311] ? apic_timer_interrupt+0xf/0x20 [ 7716.075316] [ 7716.075318] ? native_safe_halt+0x2e/0x50 [ 7716.075322] kvm_wait+0xcd/0xe0 [ 7716.075329] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7716.075337] do_raw_spin_lock+0x177/0x190 [ 7716.075341] _raw_spin_lock+0x21/0x30 [ 7716.075345] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7716.075884] ? do_raw_spin_unlock+0x75/0x190 [ 7716.075891] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7716.082487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 7716.087585] ? do_wait_intr+0xf0/0xf0 [ 7716.087610] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7716.088006] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7716.088276] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7716.088489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7716.088662] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7716.088703] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7716.088734] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7716.088839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7716.088970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7716.089170] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7716.089189] lmv_intent_open+0x338/0x1150 [lmv] [ 7716.089208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7716.089333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7716.089414] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7716.089437] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7716.089645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7716.089736] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7716.089913] ? do_raw_spin_unlock+0x75/0x190 [ 7716.089925] ? _raw_spin_unlock+0x12/0x30 [ 7716.089932] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7716.089964] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7716.089999] ll_atomic_open+0x36a/0xd70 [lustre] [ 7716.090141] lookup_open+0x356/0xb40 [ 7716.090150] do_last+0x313/0xfc0 [ 7716.090155] ? nd_jump_root+0xe5/0x160 [ 7716.090160] ? path_init+0x437/0x520 [ 7716.090164] path_openat+0xf7/0x500 [ 7716.090169] ? do_raw_spin_unlock+0x75/0x190 [ 7716.090174] do_filp_open+0x99/0x140 [ 7716.090181] ? getname_flags+0x6e/0x330 [ 7716.090186] ? __check_object_size+0xff/0x256 [ 7716.090193] ? do_raw_spin_unlock+0x75/0x190 [ 7716.090197] ? _raw_spin_unlock+0x12/0x30 [ 7716.090202] do_sys_openat2+0x2b4/0x410 [ 7716.090207] do_sys_open+0x73/0xa0 [ 7716.090211] __x64_sys_openat+0x24/0x30 [ 7716.090350] do_syscall_64+0xc1/0x440 [ 7716.090358] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7716.090365] RIP: 0033:0x7fa749552332 [ 7716.090373] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7716.090377] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7716.090381] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7716.090384] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7716.090386] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7716.090388] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7716.090390] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7716.290293] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7716.305877] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7716.310431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7716.313391] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7716.315165] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7716.321680] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7716.324319] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7716.326525] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7716.329213] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7716.332331] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7716.335503] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7716.338201] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7716.341837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7716.343943] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7716.348061] Call Trace: [ 7716.349155] [ 7716.350146] ? show_regs.cold.9+0x22/0x2f [ 7716.351582] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7716.353727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7716.355453] ? watchdog+0x40/0x40 [ 7716.356636] ? hrtimer_interrupt+0x128/0x2c0 [ 7716.358208] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7716.360145] ? apic_timer_interrupt+0xf/0x20 [ 7716.361676] [ 7716.362907] ? native_safe_halt+0x2e/0x50 [ 7716.364624] kvm_wait+0xcd/0xe0 [ 7716.366072] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7716.368161] do_raw_spin_lock+0x177/0x190 [ 7716.369858] _raw_spin_lock+0x21/0x30 [ 7716.371461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7716.374134] ? do_raw_spin_unlock+0x75/0x190 [ 7716.376161] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7716.378469] ? do_wait_intr+0xf0/0xf0 [ 7716.379914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7716.382075] mdc_close+0x53b/0xd70 [mdc] [ 7716.383836] lmv_close+0x18d/0x490 [lmv] [ 7716.385132] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7716.387159] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7716.388848] ll_md_close+0x274/0x800 [lustre] [ 7716.390668] ll_file_release+0x128/0xc50 [lustre] [ 7716.392306] __fput+0x10d/0x440 [ 7716.393656] ____fput+0x12/0x20 [ 7716.394908] task_work_run+0xc6/0x110 [ 7716.396029] exit_to_usermode_loop+0x1dd/0x1f0 [ 7716.397301] do_syscall_64+0x3d6/0x440 [ 7716.398494] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7716.400097] RIP: 0033:0x7fa749552cd8 [ 7716.401329] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7716.411211] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7716.415078] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7716.417652] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7716.419710] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7716.422497] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7716.426134] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7720.058052] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7720.062291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7720.086400] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7720.092989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7720.099658] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7720.102828] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7720.114112] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7720.119103] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7720.127681] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7720.137343] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7720.142981] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7720.161093] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7720.167185] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7720.178339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7720.187972] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7720.198208] Call Trace: [ 7720.200301] [ 7720.201884] ? show_regs.cold.9+0x22/0x2f [ 7720.211941] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7720.217485] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7720.224828] ? watchdog+0x40/0x40 [ 7720.226556] ? hrtimer_interrupt+0x128/0x2c0 [ 7720.229707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7720.232703] ? apic_timer_interrupt+0xf/0x20 [ 7720.235787] [ 7720.236476] ? native_safe_halt+0x2e/0x50 [ 7720.237657] kvm_wait+0xcd/0xe0 [ 7720.242148] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7720.247415] do_raw_spin_lock+0x177/0x190 [ 7720.254749] _raw_spin_lock+0x21/0x30 [ 7720.258879] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7720.263186] obd_mod_free+0x46/0x140 [ptlrpc] [ 7720.268249] mdc_commit_open+0x91/0xb0 [mdc] [ 7720.271076] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7720.279188] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7720.296763] after_reply+0xe2c/0x1780 [ptlrpc] [ 7720.305765] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7720.311991] ? __next_timer_interrupt+0x160/0x160 [ 7720.318007] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7720.321361] ? do_wait_intr+0xf0/0xf0 [ 7720.324279] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7720.329227] kthread+0x1d1/0x200 [ 7720.330452] ? set_kthread_struct+0x70/0x70 [ 7720.332557] ret_from_fork+0x1f/0x30 [ 7720.929152] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5803s! [ 7720.934337] Showing busy workqueues and worker pools: [ 7720.937315] workqueue events: flags=0x0 [ 7720.939502] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7720.942162] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7720.947593] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7720.950366] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7720.954248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7720.956884] pending: kfree_rcu_monitor [ 7720.958667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7720.961538] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7720.965050] workqueue events_long: flags=0x0 [ 7720.966209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7720.968693] pending: lu_objects_flush [obdclass] [ 7720.970986] workqueue events_unbound: flags=0x2 [ 7720.972975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7720.975553] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7720.979784] workqueue events_power_efficient: flags=0x80 [ 7720.982127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7720.985019] pending: check_lifetime [ 7720.986814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7720.989659] pending: neigh_periodic_work [ 7720.991621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7720.994579] pending: neigh_periodic_work [ 7720.996944] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7720.999566] pending: do_cache_clean [ 7721.000706] workqueue mm_percpu_wq: flags=0x8 [ 7721.002166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7721.004538] pending: vmstat_update [ 7721.006142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7721.008531] pending: vmstat_update [ 7721.010218] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7721.012974] pending: vmstat_update [ 7721.014591] workqueue writeback: flags=0x4a [ 7721.016262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7721.018243] pending: wb_workfn, wb_workfn [ 7721.020087] workqueue xprtiod: flags=0xa [ 7721.022125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7721.025036] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7721.028648] workqueue ipv6_addrconf: flags=0x40008 [ 7721.031846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7721.035839] pending: addrconf_verify_work [ 7721.039411] workqueue knbd0-recv: flags=0x1a [ 7721.042933] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7721.045899] in-flight: 651:recv_work [ 7721.048328] workqueue ptlrpc_pinger: flags=0x2 [ 7721.052787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7721.055387] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7721.059160] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5803s workers=4 idle: 37 31846 28897 [ 7721.064948] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7710s workers=2 idle: 653 [ 7724.057016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7724.060369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7724.084281] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7724.089775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7724.094135] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7724.097332] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7724.099420] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7724.108325] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7724.112034] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7724.115943] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7724.119458] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7724.123042] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7724.126649] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7724.129978] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7724.134540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7724.137273] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7724.141029] Call Trace: [ 7724.142268] [ 7724.143623] ? show_regs.cold.9+0x22/0x2f [ 7724.146016] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7724.148736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7724.151860] ? watchdog+0x40/0x40 [ 7724.153442] ? hrtimer_interrupt+0x128/0x2c0 [ 7724.155702] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7724.158463] ? apic_timer_interrupt+0xf/0x20 [ 7724.160626] [ 7724.161725] ? native_safe_halt+0x2e/0x50 [ 7724.163647] kvm_wait+0xcd/0xe0 [ 7724.165027] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7724.167290] do_raw_spin_lock+0x177/0x190 [ 7724.169151] _raw_spin_lock+0x21/0x30 [ 7724.170727] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7724.173308] process_one_work+0x2c8/0x700 [ 7724.174879] worker_thread+0x296/0x6e0 [ 7724.176166] ? rescuer_thread+0x570/0x570 [ 7724.177943] kthread+0x1d1/0x200 [ 7724.179115] ? set_kthread_struct+0x70/0x70 [ 7724.180916] ret_from_fork+0x1f/0x30 [ 7733.819028] rcu: INFO: rcu_sched self-detected stall on CPU [ 7733.821478] rcu: 2-...!: (3233904 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 7733.825509] (t=5820130 jiffies g=258701 q=3116) [ 7733.827547] rcu: rcu_sched kthread starved for 5816925 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7733.831742] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7733.835197] rcu: RCU grace-period kthread stack dump: [ 7733.837190] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7733.841324] Call Trace: [ 7733.842385] __schedule+0x351/0xcb0 [ 7733.843902] schedule+0xc0/0x180 [ 7733.845327] schedule_timeout+0xb4/0x190 [ 7733.846936] ? do_raw_spin_unlock+0x75/0x190 [ 7733.848680] ? __next_timer_interrupt+0x160/0x160 [ 7733.850694] ? prepare_to_swait_event+0xf3/0x140 [ 7733.852694] rcu_gp_kthread+0x8dd/0x15b0 [ 7733.854162] ? force_qs_rnp+0x2c0/0x2c0 [ 7733.855730] kthread+0x1d1/0x200 [ 7733.856933] ? set_kthread_struct+0x70/0x70 [ 7733.858641] ret_from_fork+0x1f/0x30 [ 7733.860045] rcu: Stack dump where RCU GP kthread last ran: [ 7733.862486] NMI backtrace for cpu 2 [ 7733.863856] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7733.868180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7733.871852] Call Trace: [ 7733.872810] [ 7733.873756] ? dump_stack+0xbb/0x10e [ 7733.874986] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7733.876458] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7733.877780] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7733.879845] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7733.881885] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7733.883859] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7733.886146] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7733.888337] ? update_process_times+0x6d/0xc0 [ 7733.890100] ? tick_sched_handle.isra.18+0x27/0x90 [ 7733.892000] ? tick_sched_timer+0x53/0xb0 [ 7733.893735] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7733.895377] ? tick_sched_handle.isra.18+0x90/0x90 [ 7733.897200] ? hrtimer_interrupt+0x128/0x2c0 [ 7733.899310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7733.901337] ? apic_timer_interrupt+0xf/0x20 [ 7733.903076] [ 7733.903971] ? native_safe_halt+0x2e/0x50 [ 7733.905435] ? kvm_wait+0xcd/0xe0 [ 7733.906899] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7733.909051] ? do_raw_spin_lock+0x177/0x190 [ 7733.910855] ? _raw_spin_lock+0x21/0x30 [ 7733.912648] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7733.915005] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7733.917097] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7733.919073] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7733.921824] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7733.924988] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7733.927332] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7733.930600] ? __next_timer_interrupt+0x160/0x160 [ 7733.932000] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7733.933673] ? do_wait_intr+0xf0/0xf0 [ 7733.936563] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7733.939666] ? kthread+0x1d1/0x200 [ 7733.941037] ? set_kthread_struct+0x70/0x70 [ 7733.943078] ? ret_from_fork+0x1f/0x30 [ 7733.944849] NMI backtrace for cpu 2 [ 7733.948482] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7733.955520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7733.959397] Call Trace: [ 7733.960584] [ 7733.961556] ? dump_stack+0xbb/0x10e [ 7733.963037] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7733.966204] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7733.970206] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7733.974688] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7733.978097] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7733.980620] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7733.982647] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7733.984977] ? update_process_times+0x6d/0xc0 [ 7733.986666] ? tick_sched_handle.isra.18+0x27/0x90 [ 7733.988294] ? tick_sched_timer+0x53/0xb0 [ 7733.989836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7733.991785] ? tick_sched_handle.isra.18+0x90/0x90 [ 7733.994364] ? hrtimer_interrupt+0x128/0x2c0 [ 7733.995880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7733.997938] ? apic_timer_interrupt+0xf/0x20 [ 7733.999654] [ 7734.000618] ? native_safe_halt+0x2e/0x50 [ 7734.002318] ? kvm_wait+0xcd/0xe0 [ 7734.003958] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7734.006373] ? do_raw_spin_lock+0x177/0x190 [ 7734.010916] ? _raw_spin_lock+0x21/0x30 [ 7734.011885] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7734.014287] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7734.016719] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7734.018824] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7734.021895] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7734.024566] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7734.027167] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7734.029466] ? __next_timer_interrupt+0x160/0x160 [ 7734.030871] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7734.032574] ? do_wait_intr+0xf0/0xf0 [ 7734.033689] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7734.035715] ? kthread+0x1d1/0x200 [ 7734.036843] ? set_kthread_struct+0x70/0x70 [ 7734.038033] ? ret_from_fork+0x1f/0x30 [ 7744.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7744.062919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7744.088674] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7744.097983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7744.102913] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7744.105623] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7744.115667] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7744.120085] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7744.124499] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7744.128336] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7744.132968] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7744.138586] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7744.142391] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7744.147865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7744.151434] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7744.155700] Call Trace: [ 7744.156961] [ 7744.158167] ? show_regs.cold.9+0x22/0x2f [ 7744.160464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7744.162963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7744.165504] ? watchdog+0x40/0x40 [ 7744.168513] ? hrtimer_interrupt+0x128/0x2c0 [ 7744.170981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7744.174592] ? apic_timer_interrupt+0xf/0x20 [ 7744.180668] [ 7744.181971] ? native_safe_halt+0x2e/0x50 [ 7744.187265] kvm_wait+0xcd/0xe0 [ 7744.188667] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7744.191734] do_raw_spin_lock+0x177/0x190 [ 7744.194129] _raw_spin_lock+0x21/0x30 [ 7744.196517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7744.199988] ? do_raw_spin_unlock+0x75/0x190 [ 7744.202065] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7744.204831] ? do_wait_intr+0xf0/0xf0 [ 7744.207300] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7744.210405] mdc_close+0x53b/0xd70 [mdc] [ 7744.212286] lmv_close+0x18d/0x490 [lmv] [ 7744.213882] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7744.216799] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7744.219238] ll_md_close+0x274/0x800 [lustre] [ 7744.221276] ll_file_release+0x128/0xc50 [lustre] [ 7744.223185] __fput+0x10d/0x440 [ 7744.224819] ____fput+0x12/0x20 [ 7744.226026] task_work_run+0xc6/0x110 [ 7744.227984] exit_to_usermode_loop+0x1dd/0x1f0 [ 7744.230305] do_syscall_64+0x3d6/0x440 [ 7744.232216] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7744.234679] RIP: 0033:0x7fa749552cd8 [ 7744.236627] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7744.245620] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7744.249192] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7744.252489] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7744.255845] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7744.258837] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7744.262182] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7748.054040] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7748.060625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7748.119606] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7748.135021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7748.166129] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7748.185932] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7748.209400] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7748.212916] RAX: 00000000000072ad RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7748.216181] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7748.219422] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7748.222561] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7748.225434] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7748.228478] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7748.231766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7748.234329] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7748.237422] Call Trace: [ 7748.238424] [ 7748.239239] ? show_regs.cold.9+0x22/0x2f [ 7748.244029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7748.248656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7748.252963] ? watchdog+0x40/0x40 [ 7748.255310] ? hrtimer_interrupt+0x128/0x2c0 [ 7748.257656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7748.261485] ? apic_timer_interrupt+0xf/0x20 [ 7748.268925] [ 7748.271634] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7748.280537] do_raw_spin_lock+0x177/0x190 [ 7748.288452] _raw_spin_lock+0x21/0x30 [ 7748.289684] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7748.293764] ? do_raw_spin_unlock+0x75/0x190 [ 7748.295404] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7748.297862] ? do_wait_intr+0xf0/0xf0 [ 7748.298952] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7748.302619] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7748.307952] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7748.313489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7748.315900] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7748.318345] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7748.319758] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7748.321533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7748.323531] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7748.325934] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7748.328671] lmv_intent_open+0x338/0x1150 [lmv] [ 7748.331690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7748.336615] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7748.341315] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7748.345644] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7748.347438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7748.349422] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7748.353429] ? do_raw_spin_unlock+0x75/0x190 [ 7748.356146] ? _raw_spin_unlock+0x12/0x30 [ 7748.357274] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7748.362858] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7748.364859] ll_atomic_open+0x36a/0xd70 [lustre] [ 7748.366843] lookup_open+0x356/0xb40 [ 7748.368055] do_last+0x313/0xfc0 [ 7748.370212] ? nd_jump_root+0xe5/0x160 [ 7748.375375] ? path_init+0x437/0x520 [ 7748.376590] path_openat+0xf7/0x500 [ 7748.379452] ? do_raw_spin_unlock+0x75/0x190 [ 7748.381476] do_filp_open+0x99/0x140 [ 7748.383704] ? getname_flags+0x6e/0x330 [ 7748.388332] ? __check_object_size+0xff/0x256 [ 7748.394413] ? do_raw_spin_unlock+0x75/0x190 [ 7748.395630] ? _raw_spin_unlock+0x12/0x30 [ 7748.396659] do_sys_openat2+0x2b4/0x410 [ 7748.401339] do_sys_open+0x73/0xa0 [ 7748.402712] __x64_sys_openat+0x24/0x30 [ 7748.405685] do_syscall_64+0xc1/0x440 [ 7748.413067] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7748.414370] RIP: 0033:0x7fa749552332 [ 7748.424168] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7748.441890] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7748.448914] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7748.453697] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7748.458379] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7748.463618] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7748.467563] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7751.140478] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5833s! [ 7751.144661] Showing busy workqueues and worker pools: [ 7751.150083] workqueue events: flags=0x0 [ 7751.153069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7751.160534] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7751.167770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7751.171308] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7751.175581] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.179128] pending: kfree_rcu_monitor [ 7751.180637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7751.184225] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7751.189022] workqueue events_long: flags=0x0 [ 7751.191209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.195070] pending: lu_objects_flush [obdclass] [ 7751.202250] workqueue events_unbound: flags=0x2 [ 7751.207540] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7751.213222] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7751.222209] workqueue events_power_efficient: flags=0x80 [ 7751.225235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.229177] pending: check_lifetime [ 7751.231458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.237665] pending: neigh_periodic_work [ 7751.242273] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.246577] pending: neigh_periodic_work [ 7751.249252] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.253641] pending: do_cache_clean [ 7751.256316] workqueue mm_percpu_wq: flags=0x8 [ 7751.263525] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.265970] pending: vmstat_update [ 7751.271729] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.278669] pending: vmstat_update [ 7751.283968] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7751.293161] pending: vmstat_update [ 7751.295532] workqueue writeback: flags=0x4a [ 7751.297748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7751.302477] pending: wb_workfn, wb_workfn [ 7751.304741] workqueue xprtiod: flags=0xa [ 7751.308412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7751.311580] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7751.317423] workqueue ipv6_addrconf: flags=0x40008 [ 7751.326036] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7751.331532] pending: addrconf_verify_work [ 7751.333888] workqueue knbd0-recv: flags=0x1a [ 7751.336613] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7751.341272] in-flight: 651:recv_work [ 7751.343393] workqueue ptlrpc_pinger: flags=0x2 [ 7751.349503] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7751.354711] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7751.363426] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5834s workers=4 idle: 37 31846 28897 [ 7751.367664] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7741s workers=2 idle: 653 [ 7752.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7752.059508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7752.095061] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7752.102120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7752.110235] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7752.115865] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7752.118122] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7752.133809] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7752.140693] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7752.143314] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7752.146028] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7752.149279] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7752.151991] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7752.159070] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7752.169083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7752.172670] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7752.180045] Call Trace: [ 7752.180663] [ 7752.182087] ? show_regs.cold.9+0x22/0x2f [ 7752.184474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7752.186468] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7752.188650] ? watchdog+0x40/0x40 [ 7752.190149] ? hrtimer_interrupt+0x128/0x2c0 [ 7752.193764] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7752.196096] ? apic_timer_interrupt+0xf/0x20 [ 7752.197753] [ 7752.200076] ? native_safe_halt+0x2e/0x50 [ 7752.202861] kvm_wait+0xcd/0xe0 [ 7752.204025] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7752.209387] do_raw_spin_lock+0x177/0x190 [ 7752.210503] _raw_spin_lock+0x21/0x30 [ 7752.211853] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7752.219539] process_one_work+0x2c8/0x700 [ 7752.220994] worker_thread+0x296/0x6e0 [ 7752.222988] ? rescuer_thread+0x570/0x570 [ 7752.225310] kthread+0x1d1/0x200 [ 7752.228537] ? set_kthread_struct+0x70/0x70 [ 7752.233766] ret_from_fork+0x1f/0x30 [ 7760.058877] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7760.065457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.094267] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.103166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.106987] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7760.113386] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7760.121993] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7760.123711] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7760.125702] RDX: 00000000000064b6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7760.129660] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7760.131874] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7760.139739] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7760.141442] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7760.144532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7760.146382] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7760.148256] Call Trace: [ 7760.151445] [ 7760.152229] ? show_regs.cold.9+0x22/0x2f [ 7760.153698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7760.155968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7760.158403] ? watchdog+0x40/0x40 [ 7760.159091] ? hrtimer_interrupt+0x128/0x2c0 [ 7760.160412] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7760.162481] ? apic_timer_interrupt+0xf/0x20 [ 7760.164518] [ 7760.165601] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7760.168102] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7760.170293] do_raw_spin_lock+0x177/0x190 [ 7760.172384] _raw_spin_lock+0x21/0x30 [ 7760.173940] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7760.176454] obd_mod_free+0x46/0x140 [ptlrpc] [ 7760.178948] mdc_commit_open+0x91/0xb0 [mdc] [ 7760.181029] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7760.183810] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7760.187429] after_reply+0xe2c/0x1780 [ptlrpc] [ 7760.189970] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7760.192696] ? __next_timer_interrupt+0x160/0x160 [ 7760.195031] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7760.197310] ? do_wait_intr+0xf0/0xf0 [ 7760.199122] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7760.201912] kthread+0x1d1/0x200 [ 7760.203302] ? set_kthread_struct+0x70/0x70 [ 7760.206393] ret_from_fork+0x1f/0x30 [ 7772.059055] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 7772.062117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7772.091415] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7772.098517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7772.103887] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7772.106866] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7772.117157] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7772.120973] RAX: 00000000000013fb RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7772.124026] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7772.128407] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7772.136720] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7772.141433] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7772.147170] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7772.157731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7772.164973] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7772.170335] Call Trace: [ 7772.174105] [ 7772.175915] ? show_regs.cold.9+0x22/0x2f [ 7772.178098] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7772.181810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7772.184658] ? watchdog+0x40/0x40 [ 7772.186404] ? hrtimer_interrupt+0x128/0x2c0 [ 7772.190215] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7772.195673] ? apic_timer_interrupt+0xf/0x20 [ 7772.200433] [ 7772.201766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7772.210917] do_raw_spin_lock+0x177/0x190 [ 7772.218848] _raw_spin_lock+0x21/0x30 [ 7772.231602] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7772.237361] ? do_raw_spin_unlock+0x75/0x190 [ 7772.243003] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7772.247006] ? do_wait_intr+0xf0/0xf0 [ 7772.250426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7772.255630] mdc_close+0x53b/0xd70 [mdc] [ 7772.267278] lmv_close+0x18d/0x490 [lmv] [ 7772.268718] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7772.281406] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7772.288980] ll_md_close+0x274/0x800 [lustre] [ 7772.293004] ll_file_release+0x128/0xc50 [lustre] [ 7772.298953] __fput+0x10d/0x440 [ 7772.301848] ____fput+0x12/0x20 [ 7772.303232] task_work_run+0xc6/0x110 [ 7772.306772] exit_to_usermode_loop+0x1dd/0x1f0 [ 7772.309943] do_syscall_64+0x3d6/0x440 [ 7772.313205] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7772.316950] RIP: 0033:0x7fa749552cd8 [ 7772.321917] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7772.349005] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7772.354863] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7772.368517] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7772.379879] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7772.384591] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7772.393077] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7776.054061] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7776.057124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7776.105782] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7776.112779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7776.119802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7776.123395] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7776.141026] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7776.148298] RAX: 0000000000004935 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7776.153215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7776.164236] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7776.171016] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7776.174726] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7776.179073] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7776.184965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7776.187938] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7776.194230] Call Trace: [ 7776.196127] [ 7776.196776] ? show_regs.cold.9+0x22/0x2f [ 7776.203775] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7776.205652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7776.208778] ? watchdog+0x40/0x40 [ 7776.210921] ? hrtimer_interrupt+0x128/0x2c0 [ 7776.214022] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7776.216919] ? apic_timer_interrupt+0xf/0x20 [ 7776.219235] [ 7776.220307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7776.223200] do_raw_spin_lock+0x177/0x190 [ 7776.225150] _raw_spin_lock+0x21/0x30 [ 7776.226894] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7776.229608] ? do_raw_spin_unlock+0x75/0x190 [ 7776.231877] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7776.234629] ? do_wait_intr+0xf0/0xf0 [ 7776.236353] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7776.238504] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7776.244741] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7776.250413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7776.252201] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7776.255889] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7776.261164] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7776.264347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7776.273066] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7776.282707] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7776.284700] lmv_intent_open+0x338/0x1150 [lmv] [ 7776.301414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7776.313397] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7776.324245] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7776.332493] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7776.343163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7776.350378] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7776.359206] ? do_raw_spin_unlock+0x75/0x190 [ 7776.364202] ? _raw_spin_unlock+0x12/0x30 [ 7776.370370] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7776.373770] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7776.376096] ll_atomic_open+0x36a/0xd70 [lustre] [ 7776.381152] lookup_open+0x356/0xb40 [ 7776.386302] do_last+0x313/0xfc0 [ 7776.390541] ? nd_jump_root+0xe5/0x160 [ 7776.392908] ? path_init+0x437/0x520 [ 7776.395435] path_openat+0xf7/0x500 [ 7776.398259] ? do_raw_spin_unlock+0x75/0x190 [ 7776.402926] do_filp_open+0x99/0x140 [ 7776.407653] ? getname_flags+0x6e/0x330 [ 7776.409808] ? __check_object_size+0xff/0x256 [ 7776.411386] ? do_raw_spin_unlock+0x75/0x190 [ 7776.414963] ? _raw_spin_unlock+0x12/0x30 [ 7776.419662] do_sys_openat2+0x2b4/0x410 [ 7776.421868] do_sys_open+0x73/0xa0 [ 7776.427771] __x64_sys_openat+0x24/0x30 [ 7776.430054] do_syscall_64+0xc1/0x440 [ 7776.432194] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7776.435339] RIP: 0033:0x7fa749552332 [ 7776.436410] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7776.445855] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7776.451074] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7776.453403] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7776.458512] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7776.464399] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7776.477405] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7780.060576] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7780.071555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7780.123565] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7780.136105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7780.141844] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7780.150184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7780.153288] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7780.175064] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7780.183038] RAX: 0000000000005c3c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7780.189735] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7780.193965] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7780.205010] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7780.214732] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7780.219038] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7780.229116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7780.235560] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7780.245653] Call Trace: [ 7780.246237] [ 7780.248269] ? show_regs.cold.9+0x22/0x2f [ 7780.251013] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7780.254766] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7780.256666] ? watchdog+0x40/0x40 [ 7780.257780] ? hrtimer_interrupt+0x128/0x2c0 [ 7780.264986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7780.267956] ? apic_timer_interrupt+0xf/0x20 [ 7780.273206] [ 7780.277698] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7780.284459] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7780.292633] do_raw_spin_lock+0x177/0x190 [ 7780.298100] _raw_spin_lock+0x21/0x30 [ 7780.301048] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7780.304671] process_one_work+0x2c8/0x700 [ 7780.308065] worker_thread+0x296/0x6e0 [ 7780.311754] ? rescuer_thread+0x570/0x570 [ 7780.313853] kthread+0x1d1/0x200 [ 7780.314934] ? set_kthread_struct+0x70/0x70 [ 7780.316585] ret_from_fork+0x1f/0x30 [ 7781.856931] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5864s! [ 7781.862759] Showing busy workqueues and worker pools: [ 7781.873567] workqueue events: flags=0x0 [ 7781.878768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7781.886714] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7781.911948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7781.920845] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7781.929150] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7781.937449] pending: kfree_rcu_monitor [ 7781.942902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7781.951490] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7781.960814] workqueue events_long: flags=0x0 [ 7781.971432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7781.976133] pending: lu_objects_flush [obdclass] [ 7781.984219] workqueue events_unbound: flags=0x2 [ 7781.993299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7781.996730] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7782.009023] workqueue events_power_efficient: flags=0x80 [ 7782.013680] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7782.023565] pending: check_lifetime [ 7782.028321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7782.037251] pending: neigh_periodic_work [ 7782.042280] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7782.053602] pending: neigh_periodic_work [ 7782.059608] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7782.068675] pending: do_cache_clean [ 7782.074148] workqueue mm_percpu_wq: flags=0x8 [ 7782.075405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7782.088688] pending: vmstat_update [ 7782.093292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7782.101249] pending: vmstat_update [ 7782.104282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7782.112034] pending: vmstat_update [ 7782.113087] workqueue writeback: flags=0x4a [ 7782.115855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7782.123059] pending: wb_workfn, wb_workfn [ 7782.127192] workqueue xprtiod: flags=0xa [ 7782.130128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7782.135059] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7782.143274] workqueue ipv6_addrconf: flags=0x40008 [ 7782.145197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7782.149766] pending: addrconf_verify_work [ 7782.156017] workqueue knbd0-recv: flags=0x1a [ 7782.157277] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7782.164344] in-flight: 651:recv_work [ 7782.167634] workqueue ptlrpc_pinger: flags=0x2 [ 7782.173066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7782.180504] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7782.188792] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5865s workers=4 idle: 37 31846 28897 [ 7782.198340] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7772s workers=2 idle: 653 [ 7788.058034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7788.060770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7788.092905] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7788.100070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7788.106770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7788.108778] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7788.115666] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7788.118706] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7788.121204] RDX: 000000000000729d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7788.125694] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7788.127827] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7788.131021] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7788.133753] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7788.137188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7788.139955] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7788.145175] Call Trace: [ 7788.146617] [ 7788.147776] ? show_regs.cold.9+0x22/0x2f [ 7788.149554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7788.155255] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7788.157941] ? watchdog+0x40/0x40 [ 7788.159218] ? hrtimer_interrupt+0x128/0x2c0 [ 7788.161330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7788.163578] ? apic_timer_interrupt+0xf/0x20 [ 7788.167692] [ 7788.168636] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7788.170909] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7788.173438] do_raw_spin_lock+0x177/0x190 [ 7788.175159] _raw_spin_lock+0x21/0x30 [ 7788.176922] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7788.179367] obd_mod_free+0x46/0x140 [ptlrpc] [ 7788.184454] mdc_commit_open+0x91/0xb0 [mdc] [ 7788.186181] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7788.188668] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7788.191477] after_reply+0xe2c/0x1780 [ptlrpc] [ 7788.195467] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7788.200476] ? __next_timer_interrupt+0x160/0x160 [ 7788.203226] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7788.205484] ? do_wait_intr+0xf0/0xf0 [ 7788.206992] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7788.209226] kthread+0x1d1/0x200 [ 7788.210824] ? set_kthread_struct+0x70/0x70 [ 7788.212768] ret_from_fork+0x1f/0x30 [ 7800.061501] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7800.074819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7800.107794] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7800.113288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7800.118108] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7800.123416] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7800.139804] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7800.144113] RAX: 0000000000007b88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7800.150016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7800.159980] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7800.171269] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7800.188302] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7800.196025] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7800.206546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7800.209538] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7800.214618] Call Trace: [ 7800.215878] [ 7800.216912] ? show_regs.cold.9+0x22/0x2f [ 7800.218857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7800.220904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7800.222864] ? watchdog+0x40/0x40 [ 7800.224013] ? hrtimer_interrupt+0x128/0x2c0 [ 7800.226179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7800.228019] ? apic_timer_interrupt+0xf/0x20 [ 7800.229946] [ 7800.230703] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7800.232901] do_raw_spin_lock+0x177/0x190 [ 7800.234780] _raw_spin_lock+0x21/0x30 [ 7800.236199] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7800.238485] ? do_raw_spin_unlock+0x75/0x190 [ 7800.239931] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7800.241955] ? do_wait_intr+0xf0/0xf0 [ 7800.243291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7800.245715] mdc_close+0x53b/0xd70 [mdc] [ 7800.247001] lmv_close+0x18d/0x490 [lmv] [ 7800.248291] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7800.250561] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7800.252229] ll_md_close+0x274/0x800 [lustre] [ 7800.253885] ll_file_release+0x128/0xc50 [lustre] [ 7800.255445] __fput+0x10d/0x440 [ 7800.256316] ____fput+0x12/0x20 [ 7800.257187] task_work_run+0xc6/0x110 [ 7800.258125] exit_to_usermode_loop+0x1dd/0x1f0 [ 7800.259570] do_syscall_64+0x3d6/0x440 [ 7800.260547] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7800.262107] RIP: 0033:0x7fa749552cd8 [ 7800.263510] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7800.271184] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7800.273669] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7800.276144] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7800.278905] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7800.282038] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7800.285676] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7808.054024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7808.056350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7808.057036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7808.084499] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7808.089211] Modules linked in: lustre(O) osp(O) [ 7808.095482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7808.095491] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7808.095505] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7808.095509] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7808.095514] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7808.095517] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7808.095519] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7808.095521] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7808.095523] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7808.095526] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7808.095529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7808.095531] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7808.095538] Call Trace: [ 7808.095543] [ 7808.095548] ? show_regs.cold.9+0x22/0x2f [ 7808.095557] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7808.095563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7808.095568] ? watchdog+0x40/0x40 [ 7808.095573] ? hrtimer_interrupt+0x128/0x2c0 [ 7808.095577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7808.095584] ? apic_timer_interrupt+0xf/0x20 [ 7808.095589] [ 7808.095592] ? native_safe_halt+0x2e/0x50 [ 7808.095597] kvm_wait+0xcd/0xe0 [ 7808.095604] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7808.095613] do_raw_spin_lock+0x177/0x190 [ 7808.095618] _raw_spin_lock+0x21/0x30 [ 7808.095622] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7808.098700] ? do_raw_spin_unlock+0x75/0x190 [ 7808.098775] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7808.099817] ? do_wait_intr+0xf0/0xf0 [ 7808.099830] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7808.100035] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7808.100366] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7808.100645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7808.100770] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7808.100805] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7808.100831] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7808.100950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7808.103601] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7808.106163] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7808.106171] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7808.106224] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7808.114615] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7808.114845] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7808.114876] lmv_intent_open+0x338/0x1150 [lmv] [ 7808.114899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7808.115019] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7808.115444] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7808.115466] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7808.115547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7808.115629] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7808.115719] ? do_raw_spin_unlock+0x75/0x190 [ 7808.115726] ? _raw_spin_unlock+0x12/0x30 [ 7808.115732] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7808.115749] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7808.115762] ll_atomic_open+0x36a/0xd70 [lustre] [ 7808.115817] lookup_open+0x356/0xb40 [ 7808.115824] do_last+0x313/0xfc0 [ 7808.115827] ? nd_jump_root+0xe5/0x160 [ 7808.115830] ? path_init+0x437/0x520 [ 7808.115833] path_openat+0xf7/0x500 [ 7808.115836] ? do_raw_spin_unlock+0x75/0x190 [ 7808.115839] do_filp_open+0x99/0x140 [ 7808.115844] ? getname_flags+0x6e/0x330 [ 7808.115847] ? __check_object_size+0xff/0x256 [ 7808.115853] ? do_raw_spin_unlock+0x75/0x190 [ 7808.115855] ? _raw_spin_unlock+0x12/0x30 [ 7808.115858] do_sys_openat2+0x2b4/0x410 [ 7808.115861] do_sys_open+0x73/0xa0 [ 7808.115864] __x64_sys_openat+0x24/0x30 [ 7808.115867] do_syscall_64+0xc1/0x440 [ 7808.115871] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7808.115876] RIP: 0033:0x7fa749552332 [ 7808.115882] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7808.115885] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7808.115888] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7808.115890] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7808.115892] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7808.115893] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7808.115895] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7808.391321] [ 7808.391675] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7808.393174] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7808.400259] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7808.403517] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7808.407130] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7808.410180] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7808.413590] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7808.416580] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7808.419638] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7808.423018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7808.425767] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7808.428514] Call Trace: [ 7808.429552] [ 7808.430441] ? show_regs.cold.9+0x22/0x2f [ 7808.433556] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7808.435391] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7808.437380] ? watchdog+0x40/0x40 [ 7808.438537] ? hrtimer_interrupt+0x128/0x2c0 [ 7808.440117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7808.441947] ? apic_timer_interrupt+0xf/0x20 [ 7808.443930] [ 7808.444807] ? native_safe_halt+0x2e/0x50 [ 7808.446393] kvm_wait+0xcd/0xe0 [ 7808.447686] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7808.450152] do_raw_spin_lock+0x177/0x190 [ 7808.451863] _raw_spin_lock+0x21/0x30 [ 7808.453351] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7808.455678] process_one_work+0x2c8/0x700 [ 7808.457391] worker_thread+0x296/0x6e0 [ 7808.458754] ? rescuer_thread+0x570/0x570 [ 7808.460560] kthread+0x1d1/0x200 [ 7808.462176] ? set_kthread_struct+0x70/0x70 [ 7808.463664] ret_from_fork+0x1f/0x30 [ 7812.576036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5895s! [ 7812.580797] Showing busy workqueues and worker pools: [ 7812.583673] workqueue events: flags=0x0 [ 7812.586011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7812.589680] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7812.599171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7812.605136] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7812.610069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.613706] pending: kfree_rcu_monitor [ 7812.616099] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7812.619102] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7812.627350] workqueue events_long: flags=0x0 [ 7812.629890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.636078] pending: lu_objects_flush [obdclass] [ 7812.644221] workqueue events_unbound: flags=0x2 [ 7812.650836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7812.657639] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7812.662455] workqueue events_power_efficient: flags=0x80 [ 7812.667800] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.674083] pending: check_lifetime [ 7812.677160] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.680875] pending: neigh_periodic_work [ 7812.688010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.696111] pending: neigh_periodic_work [ 7812.699365] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.706050] pending: do_cache_clean [ 7812.709922] workqueue mm_percpu_wq: flags=0x8 [ 7812.713686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.720676] pending: vmstat_update [ 7812.725992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.730958] pending: vmstat_update [ 7812.734194] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7812.739281] pending: vmstat_update [ 7812.742891] workqueue writeback: flags=0x4a [ 7812.745792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7812.749335] pending: wb_workfn, wb_workfn [ 7812.753233] workqueue xprtiod: flags=0xa [ 7812.756390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7812.761819] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7812.768034] workqueue ipv6_addrconf: flags=0x40008 [ 7812.772797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7812.778521] pending: addrconf_verify_work [ 7812.784633] workqueue knbd0-recv: flags=0x1a [ 7812.786184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7812.794619] in-flight: 651:recv_work [ 7812.799608] workqueue ptlrpc_pinger: flags=0x2 [ 7812.803560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7812.809763] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7812.811995] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5895s workers=4 idle: 37 31846 28897 [ 7812.823047] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7802s workers=2 idle: 653 [ 7816.058090] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7816.061364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7816.084262] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7816.088221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7816.090380] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7816.092486] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7816.099870] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7816.103122] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7816.106105] RDX: 00000000000029dd RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7816.109727] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7816.113495] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7816.116852] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7816.120812] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7816.124384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7816.126456] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7816.130052] Call Trace: [ 7816.133003] [ 7816.135782] ? show_regs.cold.9+0x22/0x2f [ 7816.139224] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7816.143544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7816.145998] ? watchdog+0x40/0x40 [ 7816.147074] ? hrtimer_interrupt+0x128/0x2c0 [ 7816.148255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7816.150441] ? apic_timer_interrupt+0xf/0x20 [ 7816.154563] [ 7816.155369] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7816.157872] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7816.161409] do_raw_spin_lock+0x177/0x190 [ 7816.162947] _raw_spin_lock+0x21/0x30 [ 7816.165220] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7816.171211] obd_mod_free+0x46/0x140 [ptlrpc] [ 7816.173499] mdc_commit_open+0x91/0xb0 [mdc] [ 7816.175503] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7816.180397] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7816.183657] after_reply+0xe2c/0x1780 [ptlrpc] [ 7816.189067] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7816.191426] ? __next_timer_interrupt+0x160/0x160 [ 7816.194335] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7816.197695] ? do_wait_intr+0xf0/0xf0 [ 7816.199981] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7816.202241] kthread+0x1d1/0x200 [ 7816.205656] ? set_kthread_struct+0x70/0x70 [ 7816.211929] ret_from_fork+0x1f/0x30 [ 7828.059018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7828.064544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7828.103125] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7828.114215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7828.119343] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7828.125470] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7828.139157] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7828.142974] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7828.146431] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7828.149230] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7828.151566] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7828.154540] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7828.158877] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7828.164326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7828.176405] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7828.185452] Call Trace: [ 7828.189145] [ 7828.190778] ? show_regs.cold.9+0x22/0x2f [ 7828.195422] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7828.196789] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7828.198924] ? watchdog+0x40/0x40 [ 7828.200726] ? hrtimer_interrupt+0x128/0x2c0 [ 7828.202296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7828.203981] ? apic_timer_interrupt+0xf/0x20 [ 7828.206105] [ 7828.206946] ? native_safe_halt+0x2e/0x50 [ 7828.208223] kvm_wait+0xcd/0xe0 [ 7828.209312] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7828.211170] do_raw_spin_lock+0x177/0x190 [ 7828.212450] _raw_spin_lock+0x21/0x30 [ 7828.213811] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7828.216063] ? do_raw_spin_unlock+0x75/0x190 [ 7828.217357] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7828.219831] ? do_wait_intr+0xf0/0xf0 [ 7828.230319] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7828.231909] mdc_close+0x53b/0xd70 [mdc] [ 7828.233807] lmv_close+0x18d/0x490 [lmv] [ 7828.235147] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7828.241485] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7828.245924] ll_md_close+0x274/0x800 [lustre] [ 7828.248885] ll_file_release+0x128/0xc50 [lustre] [ 7828.250274] __fput+0x10d/0x440 [ 7828.251091] ____fput+0x12/0x20 [ 7828.253746] task_work_run+0xc6/0x110 [ 7828.255981] exit_to_usermode_loop+0x1dd/0x1f0 [ 7828.257195] do_syscall_64+0x3d6/0x440 [ 7828.258711] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7828.262836] RIP: 0033:0x7fa749552cd8 [ 7828.263793] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7828.273485] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7828.276318] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7828.279415] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7828.282276] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7828.285144] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7828.287945] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7836.058171] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7836.068046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7836.133007] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7836.158072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7836.188769] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7836.191039] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7836.210752] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7836.244298] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7836.256712] RAX: 0000000000007f90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7836.260296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7836.266095] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7836.271661] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7836.275582] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7836.278820] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7836.283234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7836.284652] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7836.286914] Call Trace: [ 7836.288194] [ 7836.289095] ? show_regs.cold.9+0x22/0x2f [ 7836.292945] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7836.296231] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7836.298244] ? watchdog+0x40/0x40 [ 7836.299671] ? hrtimer_interrupt+0x128/0x2c0 [ 7836.301304] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7836.303609] ? apic_timer_interrupt+0xf/0x20 [ 7836.305745] [ 7836.306992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7836.309162] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7836.311805] do_raw_spin_lock+0x177/0x190 [ 7836.313405] _raw_spin_lock+0x21/0x30 [ 7836.315058] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7836.317766] process_one_work+0x2c8/0x700 [ 7836.320228] worker_thread+0x296/0x6e0 [ 7836.321933] ? rescuer_thread+0x570/0x570 [ 7836.323941] kthread+0x1d1/0x200 [ 7836.325929] ? set_kthread_struct+0x70/0x70 [ 7836.328201] ret_from_fork+0x1f/0x30 [ 7840.061378] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7840.066885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7840.109508] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7840.122969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7840.145137] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7840.152779] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7840.179809] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7840.186828] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7840.199573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7840.212228] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7840.214555] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7840.227266] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7840.234605] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7840.239249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7840.248191] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7840.254132] Call Trace: [ 7840.255125] [ 7840.255853] ? show_regs.cold.9+0x22/0x2f [ 7840.260589] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7840.266993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7840.269278] ? watchdog+0x40/0x40 [ 7840.272849] ? hrtimer_interrupt+0x128/0x2c0 [ 7840.275526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7840.278296] ? apic_timer_interrupt+0xf/0x20 [ 7840.280772] [ 7840.282430] ? native_safe_halt+0x2e/0x50 [ 7840.283690] kvm_wait+0xcd/0xe0 [ 7840.284775] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7840.286672] do_raw_spin_lock+0x177/0x190 [ 7840.288042] _raw_spin_lock+0x21/0x30 [ 7840.289538] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7840.291479] ? do_raw_spin_unlock+0x75/0x190 [ 7840.292781] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7840.295014] ? do_wait_intr+0xf0/0xf0 [ 7840.296259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7840.298473] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7840.300195] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7840.302286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.304253] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7840.306184] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7840.307787] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7840.309669] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.311275] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7840.313764] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7840.315756] lmv_intent_open+0x338/0x1150 [lmv] [ 7840.317738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.320233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.322056] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7840.323702] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7840.325189] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.326839] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7840.328317] ? do_raw_spin_unlock+0x75/0x190 [ 7840.329566] ? _raw_spin_unlock+0x12/0x30 [ 7840.331018] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7840.332529] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7840.334105] ll_atomic_open+0x36a/0xd70 [lustre] [ 7840.335585] lookup_open+0x356/0xb40 [ 7840.336536] do_last+0x313/0xfc0 [ 7840.337555] ? nd_jump_root+0xe5/0x160 [ 7840.338604] ? path_init+0x437/0x520 [ 7840.339704] path_openat+0xf7/0x500 [ 7840.340751] ? do_raw_spin_unlock+0x75/0x190 [ 7840.342039] do_filp_open+0x99/0x140 [ 7840.343110] ? getname_flags+0x6e/0x330 [ 7840.344246] ? __check_object_size+0xff/0x256 [ 7840.345518] ? do_raw_spin_unlock+0x75/0x190 [ 7840.346792] ? _raw_spin_unlock+0x12/0x30 [ 7840.348118] do_sys_openat2+0x2b4/0x410 [ 7840.349233] do_sys_open+0x73/0xa0 [ 7840.350290] __x64_sys_openat+0x24/0x30 [ 7840.351564] do_syscall_64+0xc1/0x440 [ 7840.352610] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7840.354202] RIP: 0033:0x7fa749552332 [ 7840.355259] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7840.360785] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7840.363121] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7840.365178] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7840.367286] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7840.369777] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7840.372000] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7843.296133] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5926s! [ 7843.301846] Showing busy workqueues and worker pools: [ 7843.303960] workqueue events: flags=0x0 [ 7843.305651] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7843.308469] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7843.314074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7843.316856] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7843.321500] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.324932] pending: kfree_rcu_monitor [ 7843.326969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7843.330425] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7843.335466] workqueue events_long: flags=0x0 [ 7843.337467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.340631] pending: lu_objects_flush [obdclass] [ 7843.343185] workqueue events_unbound: flags=0x2 [ 7843.345399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7843.349191] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7843.358704] workqueue events_power_efficient: flags=0x80 [ 7843.360972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.364507] pending: check_lifetime [ 7843.365949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.368305] pending: neigh_periodic_work [ 7843.370604] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.373094] pending: neigh_periodic_work [ 7843.374614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.378466] pending: do_cache_clean [ 7843.380147] workqueue mm_percpu_wq: flags=0x8 [ 7843.382626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.386390] pending: vmstat_update [ 7843.387628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.389571] pending: vmstat_update [ 7843.390700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7843.392446] pending: vmstat_update [ 7843.393609] workqueue writeback: flags=0x4a [ 7843.394676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7843.396421] pending: wb_workfn, wb_workfn [ 7843.397843] workqueue xprtiod: flags=0xa [ 7843.398837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7843.400675] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7843.402896] workqueue ipv6_addrconf: flags=0x40008 [ 7843.404156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7843.405934] pending: addrconf_verify_work [ 7843.407154] workqueue knbd0-recv: flags=0x1a [ 7843.408404] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7843.410376] in-flight: 651:recv_work [ 7843.411377] workqueue ptlrpc_pinger: flags=0x2 [ 7843.412469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7843.413969] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7843.415812] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5926s workers=4 idle: 37 31846 28897 [ 7843.417919] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7833s workers=2 idle: 653 [ 7844.058064] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 7844.061819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7844.083894] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7844.088544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7844.109802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7844.115803] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7844.137612] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7844.150449] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7844.152976] RDX: 0000000000002e49 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7844.163420] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7844.169592] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7844.176409] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7844.181846] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7844.188807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7844.195125] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7844.200124] Call Trace: [ 7844.202081] [ 7844.203587] ? show_regs.cold.9+0x22/0x2f [ 7844.204847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7844.207024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7844.208936] ? watchdog+0x40/0x40 [ 7844.214161] ? hrtimer_interrupt+0x128/0x2c0 [ 7844.216814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7844.219357] ? apic_timer_interrupt+0xf/0x20 [ 7844.220889] [ 7844.226575] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7844.230917] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7844.236545] do_raw_spin_lock+0x177/0x190 [ 7844.240023] _raw_spin_lock+0x21/0x30 [ 7844.241918] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7844.245646] obd_mod_free+0x46/0x140 [ptlrpc] [ 7844.251995] mdc_commit_open+0x91/0xb0 [mdc] [ 7844.255731] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7844.261133] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7844.270178] after_reply+0xe2c/0x1780 [ptlrpc] [ 7844.273995] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7844.279787] ? __next_timer_interrupt+0x160/0x160 [ 7844.285336] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7844.293588] ? do_wait_intr+0xf0/0xf0 [ 7844.295354] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7844.304824] kthread+0x1d1/0x200 [ 7844.307593] ? set_kthread_struct+0x70/0x70 [ 7844.313535] ret_from_fork+0x1f/0x30 [ 7856.060455] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7856.071761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7856.130730] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7856.138539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7856.149826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.154730] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7856.173671] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7856.177731] RAX: 0000000000004983 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7856.181532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7856.184839] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7856.188197] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7856.192318] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7856.195503] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7856.199416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7856.201769] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7856.204631] Call Trace: [ 7856.207552] [ 7856.209723] ? show_regs.cold.9+0x22/0x2f [ 7856.211868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7856.214961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7856.220720] ? watchdog+0x40/0x40 [ 7856.222655] ? hrtimer_interrupt+0x128/0x2c0 [ 7856.224954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7856.227651] ? apic_timer_interrupt+0xf/0x20 [ 7856.231906] [ 7856.232739] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.234576] do_raw_spin_lock+0x177/0x190 [ 7856.237704] _raw_spin_lock+0x21/0x30 [ 7856.241451] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7856.247277] ? do_raw_spin_unlock+0x75/0x190 [ 7856.248859] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7856.252642] ? do_wait_intr+0xf0/0xf0 [ 7856.254324] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7856.260244] mdc_close+0x53b/0xd70 [mdc] [ 7856.267269] lmv_close+0x18d/0x490 [lmv] [ 7856.276442] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7856.286651] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7856.289899] ll_md_close+0x274/0x800 [lustre] [ 7856.293022] ll_file_release+0x128/0xc50 [lustre] [ 7856.296527] __fput+0x10d/0x440 [ 7856.298667] ____fput+0x12/0x20 [ 7856.301017] task_work_run+0xc6/0x110 [ 7856.307732] exit_to_usermode_loop+0x1dd/0x1f0 [ 7856.321315] do_syscall_64+0x3d6/0x440 [ 7856.327203] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7856.334757] RIP: 0033:0x7fa749552cd8 [ 7856.342965] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7856.367971] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7856.375064] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7856.380901] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7856.391708] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7856.412980] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7856.419279] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7864.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7864.059280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7864.074475] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7864.078450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7864.081587] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7864.084323] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7864.085727] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7864.105302] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7864.109127] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7864.114618] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7864.121029] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7864.125056] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7864.130182] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7864.135297] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7864.140298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7864.143679] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7864.149101] Call Trace: [ 7864.150200] [ 7864.151291] ? show_regs.cold.9+0x22/0x2f [ 7864.154309] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7864.155872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7864.159383] ? watchdog+0x40/0x40 [ 7864.165536] ? hrtimer_interrupt+0x128/0x2c0 [ 7864.168313] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7864.171044] ? apic_timer_interrupt+0xf/0x20 [ 7864.173094] [ 7864.173918] ? native_safe_halt+0x2e/0x50 [ 7864.175505] kvm_wait+0xcd/0xe0 [ 7864.176838] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7864.180190] do_raw_spin_lock+0x177/0x190 [ 7864.181615] _raw_spin_lock+0x21/0x30 [ 7864.183474] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7864.185593] process_one_work+0x2c8/0x700 [ 7864.187228] worker_thread+0x296/0x6e0 [ 7864.188631] ? rescuer_thread+0x570/0x570 [ 7864.191911] kthread+0x1d1/0x200 [ 7864.194616] ? set_kthread_struct+0x70/0x70 [ 7864.197579] ret_from_fork+0x1f/0x30 [ 7868.054034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7868.061485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7868.120343] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7868.125618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7868.132657] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7868.134673] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7868.160298] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7868.172437] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7868.176762] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7868.181855] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7868.187561] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7868.195486] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7868.201975] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7868.207691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7868.213514] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7868.218755] Call Trace: [ 7868.219764] [ 7868.222998] ? show_regs.cold.9+0x22/0x2f [ 7868.225643] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7868.229113] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7868.234060] ? watchdog+0x40/0x40 [ 7868.238722] ? hrtimer_interrupt+0x128/0x2c0 [ 7868.240176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7868.241921] ? apic_timer_interrupt+0xf/0x20 [ 7868.249596] [ 7868.256047] ? native_safe_halt+0x2e/0x50 [ 7868.258116] kvm_wait+0xcd/0xe0 [ 7868.260609] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7868.266165] do_raw_spin_lock+0x177/0x190 [ 7868.268390] _raw_spin_lock+0x21/0x30 [ 7868.273124] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7868.279039] ? do_raw_spin_unlock+0x75/0x190 [ 7868.282735] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7868.288603] ? do_wait_intr+0xf0/0xf0 [ 7868.294384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7868.301799] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7868.308151] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7868.315153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.317170] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7868.331097] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7868.332646] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7868.341228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.348605] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7868.350917] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7868.356162] lmv_intent_open+0x338/0x1150 [lmv] [ 7868.359394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.362999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.373115] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7868.378345] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7868.384313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.388980] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7868.392650] ? do_raw_spin_unlock+0x75/0x190 [ 7868.394905] ? _raw_spin_unlock+0x12/0x30 [ 7868.396275] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7868.406117] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7868.408498] ll_atomic_open+0x36a/0xd70 [lustre] [ 7868.412659] lookup_open+0x356/0xb40 [ 7868.418034] do_last+0x313/0xfc0 [ 7868.419212] ? nd_jump_root+0xe5/0x160 [ 7868.420588] ? path_init+0x437/0x520 [ 7868.428134] path_openat+0xf7/0x500 [ 7868.438011] ? do_raw_spin_unlock+0x75/0x190 [ 7868.442599] do_filp_open+0x99/0x140 [ 7868.445429] ? getname_flags+0x6e/0x330 [ 7868.449405] ? __check_object_size+0xff/0x256 [ 7868.455532] ? do_raw_spin_unlock+0x75/0x190 [ 7868.459106] ? _raw_spin_unlock+0x12/0x30 [ 7868.466095] do_sys_openat2+0x2b4/0x410 [ 7868.467560] do_sys_open+0x73/0xa0 [ 7868.470208] __x64_sys_openat+0x24/0x30 [ 7868.474486] do_syscall_64+0xc1/0x440 [ 7868.478631] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7868.486358] RIP: 0033:0x7fa749552332 [ 7868.497993] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7868.516485] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7868.524021] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7868.536456] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7868.539102] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7868.558728] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7868.577028] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7872.058022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7872.063714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7872.117460] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7872.137506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7872.151577] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7872.153261] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7872.173999] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7872.178275] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7872.181910] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7872.186223] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7872.193479] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7872.200646] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7872.211645] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7872.221155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7872.226647] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7872.234490] Call Trace: [ 7872.235886] [ 7872.238241] ? show_regs.cold.9+0x22/0x2f [ 7872.241466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7872.245146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7872.249260] ? watchdog+0x40/0x40 [ 7872.251768] ? hrtimer_interrupt+0x128/0x2c0 [ 7872.256657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7872.263163] ? apic_timer_interrupt+0xf/0x20 [ 7872.265748] [ 7872.267268] ? native_safe_halt+0x2e/0x50 [ 7872.269369] kvm_wait+0xcd/0xe0 [ 7872.272699] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7872.276739] do_raw_spin_lock+0x177/0x190 [ 7872.282400] _raw_spin_lock+0x21/0x30 [ 7872.283801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7872.285689] obd_mod_free+0x46/0x140 [ptlrpc] [ 7872.293320] mdc_commit_open+0x91/0xb0 [mdc] [ 7872.301267] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7872.314397] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7872.318856] after_reply+0xe2c/0x1780 [ptlrpc] [ 7872.325334] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7872.332234] ? __next_timer_interrupt+0x160/0x160 [ 7872.338299] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7872.347530] ? do_wait_intr+0xf0/0xf0 [ 7872.351349] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7872.355355] kthread+0x1d1/0x200 [ 7872.357063] ? set_kthread_struct+0x70/0x70 [ 7872.359278] ret_from_fork+0x1f/0x30 [ 7873.507471] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5956s! [ 7873.517285] Showing busy workqueues and worker pools: [ 7873.523411] workqueue events: flags=0x0 [ 7873.525747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7873.531730] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7873.547188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7873.551557] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7873.557097] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.561292] pending: kfree_rcu_monitor [ 7873.563251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7873.566080] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7873.569468] workqueue events_long: flags=0x0 [ 7873.570700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.573379] pending: lu_objects_flush [obdclass] [ 7873.575390] workqueue events_unbound: flags=0x2 [ 7873.577397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7873.584572] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7873.590077] workqueue events_power_efficient: flags=0x80 [ 7873.593067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.600777] pending: check_lifetime [ 7873.603385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.607242] pending: neigh_periodic_work [ 7873.608996] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.615973] pending: neigh_periodic_work [ 7873.622544] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.631979] pending: do_cache_clean [ 7873.635171] workqueue mm_percpu_wq: flags=0x8 [ 7873.639567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.647438] pending: vmstat_update [ 7873.651931] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.653809] pending: vmstat_update [ 7873.660508] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7873.667001] pending: vmstat_update [ 7873.669477] workqueue writeback: flags=0x4a [ 7873.672341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7873.677060] pending: wb_workfn, wb_workfn [ 7873.683715] workqueue xprtiod: flags=0xa [ 7873.687272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7873.689289] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7873.699744] workqueue ipv6_addrconf: flags=0x40008 [ 7873.709675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7873.719345] pending: addrconf_verify_work [ 7873.724250] workqueue knbd0-recv: flags=0x1a [ 7873.726424] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7873.730100] in-flight: 651:recv_work [ 7873.732600] workqueue ptlrpc_pinger: flags=0x2 [ 7873.733784] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7873.741948] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7873.745295] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5956s workers=4 idle: 37 31846 28897 [ 7873.752080] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7863s workers=2 idle: 653 [ 7884.061508] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7884.074211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7884.115068] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7884.123157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7884.131037] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.136299] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7884.148692] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7884.159068] RAX: 000000000000648a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7884.162786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7884.166393] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7884.169937] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7884.178785] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7884.188567] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7884.192912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7884.200041] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7884.217453] Call Trace: [ 7884.222353] [ 7884.225567] ? show_regs.cold.9+0x22/0x2f [ 7884.234954] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7884.236622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7884.246997] ? watchdog+0x40/0x40 [ 7884.248355] ? hrtimer_interrupt+0x128/0x2c0 [ 7884.256683] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7884.260796] ? apic_timer_interrupt+0xf/0x20 [ 7884.264149] [ 7884.269237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.279627] do_raw_spin_lock+0x177/0x190 [ 7884.289402] _raw_spin_lock+0x21/0x30 [ 7884.292324] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7884.301051] ? do_raw_spin_unlock+0x75/0x190 [ 7884.303722] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7884.309629] ? do_wait_intr+0xf0/0xf0 [ 7884.319324] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7884.321178] mdc_close+0x53b/0xd70 [mdc] [ 7884.331972] lmv_close+0x18d/0x490 [lmv] [ 7884.335806] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7884.346804] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7884.355654] ll_md_close+0x274/0x800 [lustre] [ 7884.358079] ll_file_release+0x128/0xc50 [lustre] [ 7884.366025] __fput+0x10d/0x440 [ 7884.369738] ____fput+0x12/0x20 [ 7884.375481] task_work_run+0xc6/0x110 [ 7884.382078] exit_to_usermode_loop+0x1dd/0x1f0 [ 7884.394254] do_syscall_64+0x3d6/0x440 [ 7884.399887] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7884.420964] RIP: 0033:0x7fa749552cd8 [ 7884.422355] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7884.466504] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7884.475943] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7884.487899] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7884.490563] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7884.516733] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7884.528637] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7892.057052] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 7892.060548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7892.100056] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7892.107040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7892.114423] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7892.122077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7892.130784] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7892.148623] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7892.155292] RAX: 0000000000003412 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7892.160149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7892.164404] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7892.167021] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7892.173267] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7892.176978] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7892.186094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7892.190135] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7892.194245] Call Trace: [ 7892.196375] [ 7892.197184] ? show_regs.cold.9+0x22/0x2f [ 7892.198920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7892.202854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7892.205504] ? watchdog+0x40/0x40 [ 7892.209242] ? hrtimer_interrupt+0x128/0x2c0 [ 7892.212362] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7892.217105] ? apic_timer_interrupt+0xf/0x20 [ 7892.218512] [ 7892.219925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7892.223043] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7892.225521] do_raw_spin_lock+0x177/0x190 [ 7892.226946] _raw_spin_lock+0x21/0x30 [ 7892.233230] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7892.238342] process_one_work+0x2c8/0x700 [ 7892.242353] worker_thread+0x296/0x6e0 [ 7892.247364] ? rescuer_thread+0x570/0x570 [ 7892.254230] kthread+0x1d1/0x200 [ 7892.258310] ? set_kthread_struct+0x70/0x70 [ 7892.263672] ret_from_fork+0x1f/0x30 [ 7900.054189] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7900.058198] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7900.058216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 7900.062471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7900.062543] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7900.062550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7900.062552] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7900.062572] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7900.062576] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7900.062581] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7900.062583] RDX: 0000000000000c28 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7900.062585] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7900.062599] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7900.062601] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7900.062604] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7900.062606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7900.062609] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7900.062616] Call Trace: [ 7900.062622] [ 7900.062628] ? show_regs.cold.9+0x22/0x2f [ 7900.062637] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7900.062645] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7900.062648] ? watchdog+0x40/0x40 [ 7900.062652] ? hrtimer_interrupt+0x128/0x2c0 [ 7900.062656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7900.062663] ? apic_timer_interrupt+0xf/0x20 [ 7900.062668] [ 7900.062670] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7900.062674] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7900.062679] do_raw_spin_lock+0x177/0x190 [ 7900.062683] _raw_spin_lock+0x21/0x30 [ 7900.062688] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7900.063082] obd_mod_free+0x46/0x140 [ptlrpc] [ 7900.063487] mdc_commit_open+0x91/0xb0 [mdc] [ 7900.063525] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7900.063762] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7900.064151] after_reply+0xe2c/0x1780 [ptlrpc] [ 7900.064361] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7900.064646] ? __next_timer_interrupt+0x160/0x160 [ 7900.064657] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7900.064963] ? do_wait_intr+0xf0/0xf0 [ 7900.064973] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7900.071515] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7900.071581] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7900.071602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7900.071605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.071620] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7900.071628] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7900.071632] RAX: 00000000000078ed RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7900.071641] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7900.071644] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7900.071646] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7900.071648] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7900.071651] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7900.071654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7900.071656] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7900.071664] Call Trace: [ 7900.071669] [ 7900.071675] ? show_regs.cold.9+0x22/0x2f [ 7900.071685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7900.071691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7900.071695] ? watchdog+0x40/0x40 [ 7900.071699] ? hrtimer_interrupt+0x128/0x2c0 [ 7900.071703] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7900.071713] ? apic_timer_interrupt+0xf/0x20 [ 7900.071718] [ 7900.071721] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.071727] do_raw_spin_lock+0x177/0x190 [ 7900.071732] _raw_spin_lock+0x21/0x30 [ 7900.071737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7900.072194] ? do_raw_spin_unlock+0x75/0x190 [ 7900.072200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7900.072428] ? do_wait_intr+0xf0/0xf0 [ 7900.072436] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7900.072653] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7900.073144] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7900.097119] kthread+0x1d1/0x200 [ 7900.097140] ? set_kthread_struct+0x70/0x70 [ 7900.097143] ret_from_fork+0x1f/0x30 [ 7900.513136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.515885] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7900.525516] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7900.530498] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7900.532083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.539197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7900.543111] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7900.546309] lmv_intent_open+0x338/0x1150 [lmv] [ 7900.552670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.557240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.564039] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7900.568293] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7900.576293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.580686] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7900.586017] ? do_raw_spin_unlock+0x75/0x190 [ 7900.591187] ? _raw_spin_unlock+0x12/0x30 [ 7900.594120] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7900.597439] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7900.600143] ll_atomic_open+0x36a/0xd70 [lustre] [ 7900.602632] lookup_open+0x356/0xb40 [ 7900.612618] do_last+0x313/0xfc0 [ 7900.626459] ? nd_jump_root+0xe5/0x160 [ 7900.628651] ? path_init+0x437/0x520 [ 7900.629831] path_openat+0xf7/0x500 [ 7900.632572] ? do_raw_spin_unlock+0x75/0x190 [ 7900.635915] do_filp_open+0x99/0x140 [ 7900.638041] ? getname_flags+0x6e/0x330 [ 7900.640491] ? __check_object_size+0xff/0x256 [ 7900.643442] ? do_raw_spin_unlock+0x75/0x190 [ 7900.650290] ? _raw_spin_unlock+0x12/0x30 [ 7900.654135] do_sys_openat2+0x2b4/0x410 [ 7900.655610] do_sys_open+0x73/0xa0 [ 7900.663713] __x64_sys_openat+0x24/0x30 [ 7900.672816] do_syscall_64+0xc1/0x440 [ 7900.683437] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7900.686034] RIP: 0033:0x7fa749552332 [ 7900.687785] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7900.704916] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7900.718060] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7900.720018] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7900.737397] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7900.751847] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7900.759826] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7904.236963] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5987s! [ 7904.242528] Showing busy workqueues and worker pools: [ 7904.254214] workqueue events: flags=0x0 [ 7904.263149] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7904.272159] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7904.293710] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7904.307874] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7904.321932] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.332554] pending: kfree_rcu_monitor [ 7904.348797] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7904.366218] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7904.380882] workqueue events_long: flags=0x0 [ 7904.385336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.396566] pending: lu_objects_flush [obdclass] [ 7904.413285] workqueue events_unbound: flags=0x2 [ 7904.419333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7904.437072] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7904.446431] workqueue events_power_efficient: flags=0x80 [ 7904.452149] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.461161] pending: check_lifetime [ 7904.466186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.471359] pending: neigh_periodic_work [ 7904.477936] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.487785] pending: neigh_periodic_work [ 7904.491640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.499503] pending: do_cache_clean [ 7904.503186] workqueue mm_percpu_wq: flags=0x8 [ 7904.509360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.517957] pending: vmstat_update [ 7904.521351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.529151] pending: vmstat_update [ 7904.533859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7904.541473] pending: vmstat_update [ 7904.545046] workqueue writeback: flags=0x4a [ 7904.549321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7904.561739] pending: wb_workfn, wb_workfn [ 7904.570221] workqueue xprtiod: flags=0xa [ 7904.577128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7904.584221] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7904.588979] workqueue ipv6_addrconf: flags=0x40008 [ 7904.591668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7904.594983] pending: addrconf_verify_work [ 7904.597132] workqueue knbd0-recv: flags=0x1a [ 7904.598859] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7904.601719] in-flight: 651:recv_work [ 7904.603802] workqueue ptlrpc_pinger: flags=0x2 [ 7904.605370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7904.609594] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7904.612656] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5987s workers=4 idle: 37 31846 28897 [ 7904.616158] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7894s workers=2 idle: 653 [ 7912.059261] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7912.063385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7912.082853] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7912.087365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7912.091308] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.094001] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7912.111503] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7912.115563] RAX: 0000000000005406 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7912.117902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7912.122749] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7912.128770] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7912.134273] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7912.138343] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7912.141263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7912.142652] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7912.145779] Call Trace: [ 7912.147950] [ 7912.149564] ? show_regs.cold.9+0x22/0x2f [ 7912.151173] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7912.152308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7912.153242] ? watchdog+0x40/0x40 [ 7912.154014] ? hrtimer_interrupt+0x128/0x2c0 [ 7912.157557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7912.159672] ? apic_timer_interrupt+0xf/0x20 [ 7912.161585] [ 7912.162490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.165668] do_raw_spin_lock+0x177/0x190 [ 7912.167539] _raw_spin_lock+0x21/0x30 [ 7912.169228] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7912.171871] ? do_raw_spin_unlock+0x75/0x190 [ 7912.174251] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7912.176916] ? do_wait_intr+0xf0/0xf0 [ 7912.178558] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7912.181283] mdc_close+0x53b/0xd70 [mdc] [ 7912.182827] lmv_close+0x18d/0x490 [lmv] [ 7912.183976] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7912.185656] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7912.187088] ll_md_close+0x274/0x800 [lustre] [ 7912.188995] ll_file_release+0x128/0xc50 [lustre] [ 7912.191090] __fput+0x10d/0x440 [ 7912.192578] ____fput+0x12/0x20 [ 7912.193973] task_work_run+0xc6/0x110 [ 7912.195662] exit_to_usermode_loop+0x1dd/0x1f0 [ 7912.197914] do_syscall_64+0x3d6/0x440 [ 7912.199405] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7912.201552] RIP: 0033:0x7fa749552cd8 [ 7912.202669] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7912.209204] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7912.212215] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7912.215051] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7912.218167] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7912.221165] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7912.224832] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7913.822031] rcu: INFO: rcu_sched self-detected stall on CPU [ 7913.825427] rcu: 2-...!: (3338062 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 7913.833165] (t=6000133 jiffies g=258701 q=3185) [ 7913.837343] rcu: rcu_sched kthread starved for 5996928 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7913.843375] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7913.848159] rcu: RCU grace-period kthread stack dump: [ 7913.852492] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7913.862267] Call Trace: [ 7913.863412] __schedule+0x351/0xcb0 [ 7913.867535] schedule+0xc0/0x180 [ 7913.869763] schedule_timeout+0xb4/0x190 [ 7913.872035] ? do_raw_spin_unlock+0x75/0x190 [ 7913.873829] ? __next_timer_interrupt+0x160/0x160 [ 7913.876249] ? prepare_to_swait_event+0xf3/0x140 [ 7913.879271] rcu_gp_kthread+0x8dd/0x15b0 [ 7913.882967] ? force_qs_rnp+0x2c0/0x2c0 [ 7913.885400] kthread+0x1d1/0x200 [ 7913.890425] ? set_kthread_struct+0x70/0x70 [ 7913.892047] ret_from_fork+0x1f/0x30 [ 7913.893869] rcu: Stack dump where RCU GP kthread last ran: [ 7913.895932] Sending NMI from CPU 2 to CPUs 1: [ 7913.898556] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 7913.899621] NMI backtrace for cpu 2 [ 7913.912437] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7913.931150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7913.938513] Call Trace: [ 7913.943066] [ 7913.944172] ? dump_stack+0xbb/0x10e [ 7913.946724] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7913.950730] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7913.952956] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7913.956422] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7913.959473] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7913.962279] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7913.967925] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7913.970936] ? update_process_times+0x6d/0xc0 [ 7913.973417] ? tick_sched_handle.isra.18+0x27/0x90 [ 7913.975191] ? tick_sched_timer+0x53/0xb0 [ 7913.977322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7913.979446] ? tick_sched_handle.isra.18+0x90/0x90 [ 7913.982069] ? hrtimer_interrupt+0x128/0x2c0 [ 7913.985473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7913.991393] ? apic_timer_interrupt+0xf/0x20 [ 7913.997531] [ 7913.998826] ? native_safe_halt+0x2e/0x50 [ 7914.003258] ? kvm_wait+0xcd/0xe0 [ 7914.005727] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7914.011010] ? do_raw_spin_lock+0x177/0x190 [ 7914.013876] ? _raw_spin_lock+0x21/0x30 [ 7914.017838] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7914.022050] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7914.024450] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7914.025986] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7914.029573] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7914.033459] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7914.038610] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7914.046453] ? __next_timer_interrupt+0x160/0x160 [ 7914.049861] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7914.052778] ? do_wait_intr+0xf0/0xf0 [ 7914.054019] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7914.057453] ? kthread+0x1d1/0x200 [ 7914.060024] ? set_kthread_struct+0x70/0x70 [ 7914.064035] ? ret_from_fork+0x1f/0x30 [ 7932.059283] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7932.069639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7932.102397] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7932.107344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7932.111067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7932.113820] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7932.121544] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7932.124610] RAX: 0000000000007f89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7932.127187] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7932.130037] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7932.133567] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7932.136509] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7932.139200] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7932.142309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7932.144548] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7932.147518] Call Trace: [ 7932.148532] [ 7932.149428] ? show_regs.cold.9+0x22/0x2f [ 7932.151196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7932.152991] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7932.155207] ? watchdog+0x40/0x40 [ 7932.156474] ? hrtimer_interrupt+0x128/0x2c0 [ 7932.158091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7932.159963] ? apic_timer_interrupt+0xf/0x20 [ 7932.161908] [ 7932.162727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7932.164958] do_raw_spin_lock+0x177/0x190 [ 7932.167012] _raw_spin_lock+0x21/0x30 [ 7932.169307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7932.171288] ? do_raw_spin_unlock+0x75/0x190 [ 7932.173312] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7932.179501] ? do_wait_intr+0xf0/0xf0 [ 7932.181262] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7932.184066] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7932.186406] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7932.189259] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.192071] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7932.194330] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7932.197708] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7932.200918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.203811] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7932.206413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7932.208682] lmv_intent_open+0x338/0x1150 [lmv] [ 7932.218424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.227710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.231670] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7932.233985] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7932.238772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.242408] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7932.249544] ? do_raw_spin_unlock+0x75/0x190 [ 7932.251519] ? _raw_spin_unlock+0x12/0x30 [ 7932.253387] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7932.255590] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7932.257718] ll_atomic_open+0x36a/0xd70 [lustre] [ 7932.260066] lookup_open+0x356/0xb40 [ 7932.261513] do_last+0x313/0xfc0 [ 7932.263074] ? nd_jump_root+0xe5/0x160 [ 7932.265166] ? path_init+0x437/0x520 [ 7932.266643] path_openat+0xf7/0x500 [ 7932.268189] ? do_raw_spin_unlock+0x75/0x190 [ 7932.270055] do_filp_open+0x99/0x140 [ 7932.273174] ? getname_flags+0x6e/0x330 [ 7932.275162] ? __check_object_size+0xff/0x256 [ 7932.282247] ? do_raw_spin_unlock+0x75/0x190 [ 7932.283821] ? _raw_spin_unlock+0x12/0x30 [ 7932.288563] do_sys_openat2+0x2b4/0x410 [ 7932.292746] do_sys_open+0x73/0xa0 [ 7932.294429] __x64_sys_openat+0x24/0x30 [ 7932.297755] do_syscall_64+0xc1/0x440 [ 7932.300623] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7932.303350] RIP: 0033:0x7fa749552332 [ 7932.305283] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7932.318210] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7932.324939] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7932.338567] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7932.346037] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7932.349527] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7932.351761] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7934.947050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6017s! [ 7934.953051] Showing busy workqueues and worker pools: [ 7934.955241] workqueue events: flags=0x0 [ 7934.956598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7934.964097] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7934.969047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7934.975596] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7934.981502] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7934.985843] pending: kfree_rcu_monitor [ 7934.988507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7934.991510] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7935.001554] workqueue events_long: flags=0x0 [ 7935.004181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7935.009155] pending: lu_objects_flush [obdclass] [ 7935.017385] workqueue events_unbound: flags=0x2 [ 7935.023612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7935.031512] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7935.039763] workqueue events_power_efficient: flags=0x80 [ 7935.042439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7935.047422] pending: check_lifetime [ 7935.049755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7935.052917] pending: neigh_periodic_work [ 7935.056882] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7935.060373] pending: neigh_periodic_work [ 7935.062367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7935.065421] pending: do_cache_clean [ 7935.067172] workqueue mm_percpu_wq: flags=0x8 [ 7935.069129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7935.072316] pending: vmstat_update [ 7935.073738] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7935.076108] pending: vmstat_update [ 7935.077211] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7935.079425] pending: vmstat_update [ 7935.080967] workqueue writeback: flags=0x4a [ 7935.082837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7935.085545] pending: wb_workfn, wb_workfn [ 7935.090866] workqueue xprtiod: flags=0xa [ 7935.094537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7935.101515] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7935.108082] workqueue ipv6_addrconf: flags=0x40008 [ 7935.111137] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7935.120544] pending: addrconf_verify_work [ 7935.125603] workqueue knbd0-recv: flags=0x1a [ 7935.131517] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7935.135771] in-flight: 651:recv_work [ 7935.138525] workqueue ptlrpc_pinger: flags=0x2 [ 7935.140211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7935.144516] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7935.148058] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6017s workers=4 idle: 37 31846 28897 [ 7935.162297] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7925s workers=2 idle: 653 [ 7940.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7940.058089] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7940.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7940.059036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.059140] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.059147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.059149] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7940.059163] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7940.059167] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7940.059171] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7940.059174] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7940.059176] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7940.059178] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7940.059180] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7940.059183] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7940.059186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.059189] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7940.059197] Call Trace: [ 7940.059202] [ 7940.059207] ? show_regs.cold.9+0x22/0x2f [ 7940.059217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.059224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.059228] ? watchdog+0x40/0x40 [ 7940.059232] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.059237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.059243] ? apic_timer_interrupt+0xf/0x20 [ 7940.059248] [ 7940.059252] ? native_safe_halt+0x2e/0x50 [ 7940.059256] kvm_wait+0xcd/0xe0 [ 7940.059263] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7940.059272] do_raw_spin_lock+0x177/0x190 [ 7940.059277] _raw_spin_lock+0x21/0x30 [ 7940.059281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7940.059566] ? do_raw_spin_unlock+0x75/0x190 [ 7940.059776] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7940.060003] ? do_wait_intr+0xf0/0xf0 [ 7940.060012] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7940.060226] mdc_close+0x53b/0xd70 [mdc] [ 7940.060265] lmv_close+0x18d/0x490 [lmv] [ 7940.060302] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7940.060420] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7940.060503] ll_md_close+0x274/0x800 [lustre] [ 7940.060773] ll_file_release+0x128/0xc50 [lustre] [ 7940.060864] __fput+0x10d/0x440 [ 7940.060871] ____fput+0x12/0x20 [ 7940.060879] task_work_run+0xc6/0x110 [ 7940.060895] exit_to_usermode_loop+0x1dd/0x1f0 [ 7940.060903] do_syscall_64+0x3d6/0x440 [ 7940.060907] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7940.060915] RIP: 0033:0x7fa749552cd8 [ 7940.060921] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7940.060925] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7940.060931] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7940.060933] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7940.060935] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7940.060937] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7940.060940] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7940.063299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.063375] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.063382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.063389] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7940.068679] Modules linked in: [ 7940.072938] [ 7940.072947] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7940.072965] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7940.072969] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7940.072974] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7940.072977] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7940.072980] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7940.072983] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7940.072985] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7940.072988] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7940.072991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.072993] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7940.073001] Call Trace: [ 7940.073006] [ 7940.073012] ? show_regs.cold.9+0x22/0x2f [ 7940.073022] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.073029] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.073034] ? watchdog+0x40/0x40 [ 7940.073046] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.073050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.073057] ? apic_timer_interrupt+0xf/0x20 [ 7940.073063] [ 7940.073067] ? native_safe_halt+0x2e/0x50 [ 7940.127790] lustre(O) [ 7940.134912] kvm_wait+0xcd/0xe0 [ 7940.134934] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7940.134945] do_raw_spin_lock+0x177/0x190 [ 7940.134950] _raw_spin_lock+0x21/0x30 [ 7940.134957] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7940.138847] osp(O) [ 7940.140843] process_one_work+0x2c8/0x700 [ 7940.158175] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 7940.164943] worker_thread+0x296/0x6e0 [ 7940.168073] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.168119] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.168126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.168128] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7940.168142] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7940.168145] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7940.168151] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7940.168153] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7940.168155] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7940.168158] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7940.168161] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7940.168164] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7940.168167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.168169] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7940.168178] Call Trace: [ 7940.168183] [ 7940.168188] ? show_regs.cold.9+0x22/0x2f [ 7940.168197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.168204] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.168208] ? watchdog+0x40/0x40 [ 7940.168212] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.168216] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.168222] ? apic_timer_interrupt+0xf/0x20 [ 7940.168227] [ 7940.168230] ? native_safe_halt+0x2e/0x50 [ 7940.168233] kvm_wait+0xcd/0xe0 [ 7940.168240] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7940.168248] do_raw_spin_lock+0x177/0x190 [ 7940.168253] _raw_spin_lock+0x21/0x30 [ 7940.168259] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7940.168809] obd_mod_free+0x46/0x140 [ptlrpc] [ 7940.169014] mdc_commit_open+0x91/0xb0 [mdc] [ 7940.169047] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7940.169243] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7940.169572] after_reply+0xe2c/0x1780 [ptlrpc] [ 7940.169988] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7940.170186] ? __next_timer_interrupt+0x160/0x160 [ 7940.170198] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7940.175926] ? rescuer_thread+0x570/0x570 [ 7940.175947] kthread+0x1d1/0x200 [ 7940.175951] ? set_kthread_struct+0x70/0x70 [ 7940.175955] ret_from_fork+0x1f/0x30 [ 7940.562927] ? do_wait_intr+0xf0/0xf0 [ 7940.564731] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7940.567799] kthread+0x1d1/0x200 [ 7940.569492] ? set_kthread_struct+0x70/0x70 [ 7940.570738] ret_from_fork+0x1f/0x30 [ 7960.054021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 7960.056982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7960.106215] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7960.116505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7960.125186] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7960.130440] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7960.146053] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7960.149966] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7960.153132] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7960.156680] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7960.160207] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7960.164131] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7960.173401] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7960.179974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7960.183794] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7960.187462] Call Trace: [ 7960.188130] [ 7960.192187] ? show_regs.cold.9+0x22/0x2f [ 7960.197053] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7960.199126] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7960.201164] ? watchdog+0x40/0x40 [ 7960.202544] ? hrtimer_interrupt+0x128/0x2c0 [ 7960.205040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7960.208792] ? apic_timer_interrupt+0xf/0x20 [ 7960.211210] [ 7960.211978] ? native_safe_halt+0x2e/0x50 [ 7960.213611] kvm_wait+0xcd/0xe0 [ 7960.215185] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7960.217175] do_raw_spin_lock+0x177/0x190 [ 7960.218591] _raw_spin_lock+0x21/0x30 [ 7960.219845] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7960.221900] ? do_raw_spin_unlock+0x75/0x190 [ 7960.223221] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7960.224961] ? do_wait_intr+0xf0/0xf0 [ 7960.226484] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7960.228224] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7960.229998] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7960.232162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.234120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7960.236019] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7960.237567] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7960.238968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.240664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7960.242634] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7960.244166] lmv_intent_open+0x338/0x1150 [lmv] [ 7960.246318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.248215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.250444] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7960.251969] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7960.253874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.256422] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7960.258639] ? do_raw_spin_unlock+0x75/0x190 [ 7960.260300] ? _raw_spin_unlock+0x12/0x30 [ 7960.264117] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7960.266416] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7960.268391] ll_atomic_open+0x36a/0xd70 [lustre] [ 7960.270507] lookup_open+0x356/0xb40 [ 7960.272844] do_last+0x313/0xfc0 [ 7960.273782] ? nd_jump_root+0xe5/0x160 [ 7960.275346] ? path_init+0x437/0x520 [ 7960.277112] path_openat+0xf7/0x500 [ 7960.280846] ? do_raw_spin_unlock+0x75/0x190 [ 7960.284750] do_filp_open+0x99/0x140 [ 7960.288827] ? getname_flags+0x6e/0x330 [ 7960.294224] ? __check_object_size+0xff/0x256 [ 7960.297161] ? do_raw_spin_unlock+0x75/0x190 [ 7960.300131] ? _raw_spin_unlock+0x12/0x30 [ 7960.301785] do_sys_openat2+0x2b4/0x410 [ 7960.306966] do_sys_open+0x73/0xa0 [ 7960.308603] __x64_sys_openat+0x24/0x30 [ 7960.310472] do_syscall_64+0xc1/0x440 [ 7960.313857] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7960.316781] RIP: 0033:0x7fa749552332 [ 7960.318423] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7960.327704] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7960.332045] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7960.335929] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7960.340639] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7960.342666] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7960.350497] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7965.664225] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6048s! [ 7965.668400] Showing busy workqueues and worker pools: [ 7965.673412] workqueue events: flags=0x0 [ 7965.677346] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7965.680819] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7965.686205] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7965.689801] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7965.693617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.697180] pending: kfree_rcu_monitor [ 7965.700400] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7965.703685] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7965.712692] workqueue events_long: flags=0x0 [ 7965.714733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.718134] pending: lu_objects_flush [obdclass] [ 7965.723233] workqueue events_unbound: flags=0x2 [ 7965.725267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7965.731871] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7965.737022] workqueue events_power_efficient: flags=0x80 [ 7965.741858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.749295] pending: check_lifetime [ 7965.751124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.756806] pending: neigh_periodic_work [ 7965.759686] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.769911] pending: neigh_periodic_work [ 7965.777964] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.781006] pending: do_cache_clean [ 7965.782213] workqueue mm_percpu_wq: flags=0x8 [ 7965.784018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.786312] pending: vmstat_update [ 7965.787357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.789874] pending: vmstat_update [ 7965.791116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7965.794907] pending: vmstat_update [ 7965.796053] workqueue writeback: flags=0x4a [ 7965.797364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7965.799388] pending: wb_workfn, wb_workfn [ 7965.802270] workqueue xprtiod: flags=0xa [ 7965.803942] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7965.806008] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7965.810225] workqueue ipv6_addrconf: flags=0x40008 [ 7965.813344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7965.817964] pending: addrconf_verify_work [ 7965.819658] workqueue knbd0-recv: flags=0x1a [ 7965.821147] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7965.825860] in-flight: 651:recv_work [ 7965.827421] workqueue ptlrpc_pinger: flags=0x2 [ 7965.829017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7965.831905] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7965.834461] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6048s workers=4 idle: 37 31846 28897 [ 7965.837382] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7955s workers=2 idle: 653 [ 7968.060775] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7968.060866] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7968.062336] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7968.062346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.062425] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.062430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.062433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.062447] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7968.062451] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7968.062455] RAX: 00000000000037e8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7968.062458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7968.062460] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7968.062462] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7968.062465] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7968.062468] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7968.062471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.062473] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7968.062481] Call Trace: [ 7968.062487] [ 7968.062492] ? show_regs.cold.9+0x22/0x2f [ 7968.062501] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.062507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.062511] ? watchdog+0x40/0x40 [ 7968.062515] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.062519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.062527] ? apic_timer_interrupt+0xf/0x20 [ 7968.062531] [ 7968.062534] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.062542] do_raw_spin_lock+0x177/0x190 [ 7968.062547] _raw_spin_lock+0x21/0x30 [ 7968.062551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7968.062846] ? do_raw_spin_unlock+0x75/0x190 [ 7968.062852] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7968.063305] ? do_wait_intr+0xf0/0xf0 [ 7968.063313] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7968.063518] mdc_close+0x53b/0xd70 [mdc] [ 7968.063556] lmv_close+0x18d/0x490 [lmv] [ 7968.063583] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7968.063690] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7968.063771] ll_md_close+0x274/0x800 [lustre] [ 7968.063853] ll_file_release+0x128/0xc50 [lustre] [ 7968.063937] __fput+0x10d/0x440 [ 7968.063946] ____fput+0x12/0x20 [ 7968.067225] task_work_run+0xc6/0x110 [ 7968.067237] exit_to_usermode_loop+0x1dd/0x1f0 [ 7968.067244] do_syscall_64+0x3d6/0x440 [ 7968.067248] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7968.067256] RIP: 0033:0x7fa749552cd8 [ 7968.067265] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7968.067269] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7968.067274] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7968.067276] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7968.067278] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7968.067280] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7968.067282] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7968.073433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.073565] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.073573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.073580] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7968.074096] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.075636] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7968.075641] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7968.075647] RAX: 0000000000006d10 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7968.075649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7968.075652] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7968.075654] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7968.075657] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7968.075660] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7968.075664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.075666] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7968.075674] Call Trace: [ 7968.075680] [ 7968.075685] ? show_regs.cold.9+0x22/0x2f [ 7968.075695] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.075702] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.075705] ? watchdog+0x40/0x40 [ 7968.075710] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.075714] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.075723] ? apic_timer_interrupt+0xf/0x20 [ 7968.075727] [ 7968.075731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.075735] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7968.075743] do_raw_spin_lock+0x177/0x190 [ 7968.075748] _raw_spin_lock+0x21/0x30 [ 7968.075753] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7968.077616] process_one_work+0x2c8/0x700 [ 7968.077626] worker_thread+0x296/0x6e0 [ 7968.077632] ? rescuer_thread+0x570/0x570 [ 7968.077636] kthread+0x1d1/0x200 [ 7968.077640] ? set_kthread_struct+0x70/0x70 [ 7968.077644] ret_from_fork+0x1f/0x30 [ 7968.430882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.447664] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.451986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.456967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7968.459929] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7968.467750] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7968.470840] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7968.473795] RDX: 00000000000034aa RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7968.476419] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7968.478530] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7968.481478] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7968.484602] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7968.493377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.499469] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7968.507459] Call Trace: [ 7968.508353] [ 7968.509085] ? show_regs.cold.9+0x22/0x2f [ 7968.518161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.521867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.526190] ? watchdog+0x40/0x40 [ 7968.528624] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.530401] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.535777] ? apic_timer_interrupt+0xf/0x20 [ 7968.540723] [ 7968.543441] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7968.545324] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7968.548939] do_raw_spin_lock+0x177/0x190 [ 7968.550307] _raw_spin_lock+0x21/0x30 [ 7968.551850] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7968.554007] obd_mod_free+0x46/0x140 [ptlrpc] [ 7968.556299] mdc_commit_open+0x91/0xb0 [mdc] [ 7968.558472] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7968.560938] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7968.563227] after_reply+0xe2c/0x1780 [ptlrpc] [ 7968.565168] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7968.567010] ? __next_timer_interrupt+0x160/0x160 [ 7968.568364] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7968.569909] ? do_wait_intr+0xf0/0xf0 [ 7968.571238] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7968.572945] kthread+0x1d1/0x200 [ 7968.573910] ? set_kthread_struct+0x70/0x70 [ 7968.575235] ret_from_fork+0x1f/0x30 [ 7992.059516] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 7992.074096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7992.133839] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7992.140809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7992.151211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7992.160864] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7992.188514] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7992.198437] RAX: 00000000000045e1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7992.207098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 7992.213449] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 7992.228762] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 7992.241778] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 7992.255510] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 7992.265514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7992.269921] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 7992.273634] Call Trace: [ 7992.276641] [ 7992.279725] ? show_regs.cold.9+0x22/0x2f [ 7992.281145] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7992.287139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7992.291311] ? watchdog+0x40/0x40 [ 7992.294084] ? hrtimer_interrupt+0x128/0x2c0 [ 7992.297957] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7992.304266] ? apic_timer_interrupt+0xf/0x20 [ 7992.306576] [ 7992.309465] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7992.313295] do_raw_spin_lock+0x177/0x190 [ 7992.316477] _raw_spin_lock+0x21/0x30 [ 7992.319943] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7992.323405] ? do_raw_spin_unlock+0x75/0x190 [ 7992.325622] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7992.328218] ? do_wait_intr+0xf0/0xf0 [ 7992.330412] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7992.332659] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7992.335953] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7992.338535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7992.341029] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7992.343322] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7992.346015] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7992.347748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7992.350931] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7992.353589] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7992.356044] lmv_intent_open+0x338/0x1150 [lmv] [ 7992.358065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7992.360626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7992.363091] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7992.364872] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7992.366916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7992.369601] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7992.371729] ? do_raw_spin_unlock+0x75/0x190 [ 7992.373073] ? _raw_spin_unlock+0x12/0x30 [ 7992.374944] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7992.377291] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7992.379407] ll_atomic_open+0x36a/0xd70 [lustre] [ 7992.381413] lookup_open+0x356/0xb40 [ 7992.383002] do_last+0x313/0xfc0 [ 7992.384822] ? nd_jump_root+0xe5/0x160 [ 7992.388589] ? path_init+0x437/0x520 [ 7992.390576] path_openat+0xf7/0x500 [ 7992.392685] ? do_raw_spin_unlock+0x75/0x190 [ 7992.394453] do_filp_open+0x99/0x140 [ 7992.396243] ? getname_flags+0x6e/0x330 [ 7992.398077] ? __check_object_size+0xff/0x256 [ 7992.399568] ? do_raw_spin_unlock+0x75/0x190 [ 7992.401334] ? _raw_spin_unlock+0x12/0x30 [ 7992.403151] do_sys_openat2+0x2b4/0x410 [ 7992.404790] do_sys_open+0x73/0xa0 [ 7992.406067] __x64_sys_openat+0x24/0x30 [ 7992.407570] do_syscall_64+0xc1/0x440 [ 7992.409149] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7992.411080] RIP: 0033:0x7fa749552332 [ 7992.412379] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7992.420185] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7992.423110] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 7992.426042] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 7992.429021] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 7992.431612] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 7992.434691] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 7995.876047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6078s! [ 7995.885435] Showing busy workqueues and worker pools: [ 7995.887315] workqueue events: flags=0x0 [ 7995.895020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7995.901311] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 7995.915385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7995.922439] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7995.935421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7995.941470] pending: kfree_rcu_monitor [ 7995.948065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7995.960223] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 7995.973664] workqueue events_long: flags=0x0 [ 7995.983932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7995.995260] pending: lu_objects_flush [obdclass] [ 7995.999209] workqueue events_unbound: flags=0x2 [ 7996.004658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 7996.014999] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7996.026285] workqueue events_power_efficient: flags=0x80 [ 7996.036200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7996.049326] pending: check_lifetime [ 7996.056286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7996.057275] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 7996.059580] pending: neigh_periodic_work [ 7996.059642] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7996.059655] pending: neigh_periodic_work [ 7996.059661] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7996.059668] pending: do_cache_clean [ 7996.059681] workqueue mm_percpu_wq: flags=0x8 [ 7996.059683] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7996.059691] pending: vmstat_update [ 7996.059699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7996.059705] pending: vmstat_update [ 7996.059710] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7996.059715] pending: vmstat_update [ 7996.059721] workqueue writeback: flags=0x4a [ 7996.059720] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 7996.059724] pwq 8: [ 7996.059725] Modules linked in: [ 7996.059725] cpus=0-3 flags=0x4 nice=0 [ 7996.059728] lustre(O) [ 7996.059728] active=2/256 refcnt=4 [ 7996.059730] osp(O) [ 7996.059731] pending: [ 7996.059732] ofd(O) lod(O) [ 7996.059732] wb_workfn [ 7996.059736] mdt(O) [ 7996.059737] , wb_workfn [ 7996.059748] workqueue xprtiod: flags=0xa [ 7996.059751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7996.059756] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7996.059774] workqueue ipv6_addrconf: flags=0x40008 [ 7996.059778] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7996.059784] pending: addrconf_verify_work [ 7996.059964] workqueue knbd0-recv: flags=0x1a [ 7996.059968] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7996.059975] in-flight: 651:recv_work [ 7996.059985] workqueue ptlrpc_pinger: flags=0x2 [ 7996.059986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7996.059991] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7996.060308] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6078s workers=4 idle: 37 31846 28897 [ 7996.060321] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7985s workers=2 idle: 653 [ 7996.059737] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.060420] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.060426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.060429] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7996.060440] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7996.060444] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7996.060448] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 7996.060450] RDX: 00000000000008e3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 7996.060453] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 7996.060454] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 7996.060456] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7996.060459] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 7996.060463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.060465] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 7996.060472] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 7996.060475] Call Trace: [ 7996.060481] Modules linked in: lustre(O) [ 7996.060483] [ 7996.060490] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 7996.060492] ? show_regs.cold.9+0x22/0x2f [ 7996.060503] lfsck(O) [ 7996.060508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.060514] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.060518] ? watchdog+0x40/0x40 [ 7996.060522] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.060526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7996.060534] ? apic_timer_interrupt+0xf/0x20 [ 7996.060539] [ 7996.060541] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7996.060546] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7996.060550] do_raw_spin_lock+0x177/0x190 [ 7996.060554] _raw_spin_lock+0x21/0x30 [ 7996.060559] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7996.061409] obd_mod_free+0x46/0x140 [ptlrpc] [ 7996.062036] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.062161] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.062167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.062170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.062180] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7996.062184] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7996.062189] RAX: 000000000000220b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7996.062192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 7996.062194] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7996.062196] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 7996.062198] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 7996.062201] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 7996.062204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.062207] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 7996.062215] Call Trace: [ 7996.062218] [ 7996.062222] ? show_regs.cold.9+0x22/0x2f [ 7996.062233] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.062239] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.062243] ? watchdog+0x40/0x40 [ 7996.062248] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.062252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7996.062259] ? apic_timer_interrupt+0xf/0x20 [ 7996.062264] [ 7996.062267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.062273] do_raw_spin_lock+0x177/0x190 [ 7996.062278] _raw_spin_lock+0x21/0x30 [ 7996.062283] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7996.062505] ? do_raw_spin_unlock+0x75/0x190 [ 7996.062523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7996.062726] ? do_wait_intr+0xf0/0xf0 [ 7996.061623] mdc_commit_open+0x91/0xb0 [mdc] [ 7996.062732] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7996.062935] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7996.063165] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7996.063374] after_reply+0xe2c/0x1780 [ptlrpc] [ 7996.063719] mdc_close+0x53b/0xd70 [mdc] [ 7996.063752] lmv_close+0x18d/0x490 [lmv] [ 7996.063783] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7996.063889] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7996.064124] ll_md_close+0x274/0x800 [lustre] [ 7996.064211] ll_file_release+0x128/0xc50 [lustre] [ 7996.063625] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7996.067019] ? __next_timer_interrupt+0x160/0x160 [ 7996.067031] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7996.067230] ? do_wait_intr+0xf0/0xf0 [ 7996.067240] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7996.067446] kthread+0x1d1/0x200 [ 7996.067452] ? set_kthread_struct+0x70/0x70 [ 7996.067456] ret_from_fork+0x1f/0x30 [ 7996.068492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.068578] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.068585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.068591] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7996.068837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.068847] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7996.068851] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7996.068855] RAX: 00000000000060a7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 7996.068857] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 7996.068859] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7996.068862] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 7996.068864] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 7996.068867] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 7996.068870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.068872] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 7996.068881] Call Trace: [ 7996.068886] [ 7996.068890] ? show_regs.cold.9+0x22/0x2f [ 7996.068898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.069219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.069224] ? watchdog+0x40/0x40 [ 7996.069228] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.069232] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7996.069239] ? apic_timer_interrupt+0xf/0x20 [ 7996.069243] [ 7996.069246] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.069251] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7996.069256] do_raw_spin_lock+0x177/0x190 [ 7996.069261] _raw_spin_lock+0x21/0x30 [ 7996.069266] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7996.069494] process_one_work+0x2c8/0x700 [ 7996.069502] worker_thread+0x296/0x6e0 [ 7996.069507] ? rescuer_thread+0x570/0x570 [ 7996.069524] kthread+0x1d1/0x200 [ 7996.069529] ? set_kthread_struct+0x70/0x70 [ 7996.069533] ret_from_fork+0x1f/0x30 [ 7996.818832] __fput+0x10d/0x440 [ 7996.820036] ____fput+0x12/0x20 [ 7996.823940] task_work_run+0xc6/0x110 [ 7996.827393] exit_to_usermode_loop+0x1dd/0x1f0 [ 7996.830922] do_syscall_64+0x3d6/0x440 [ 7996.834080] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7996.843462] RIP: 0033:0x7fa749552cd8 [ 7996.850053] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7996.866736] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7996.875434] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 7996.880451] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 7996.888469] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 7996.896505] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 7996.911340] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8024.055178] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8024.058346] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8024.061226] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8024.061363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8024.061446] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8024.061452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8024.061455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.061469] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8024.061473] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8024.061479] RAX: 0000000000006a90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8024.061482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8024.061484] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8024.061486] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8024.061488] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8024.061492] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8024.061496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8024.061498] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8024.061507] Call Trace: [ 8024.061512] [ 8024.061517] ? show_regs.cold.9+0x22/0x2f [ 8024.061526] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8024.061533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8024.061538] ? watchdog+0x40/0x40 [ 8024.061543] ? hrtimer_interrupt+0x128/0x2c0 [ 8024.061547] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8024.061556] ? apic_timer_interrupt+0xf/0x20 [ 8024.061561] [ 8024.061564] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.061571] do_raw_spin_lock+0x177/0x190 [ 8024.061575] _raw_spin_lock+0x21/0x30 [ 8024.061580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8024.062713] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8024.062720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8024.062785] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8024.062791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8024.062878] ? do_raw_spin_unlock+0x75/0x190 [ 8024.062965] Modules linked in: lustre(O) [ 8024.062796] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8024.063262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.063269] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8024.063273] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8024.063278] RAX: 000000000000620f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8024.063281] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8024.063283] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8024.063285] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8024.063288] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8024.063291] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8024.063294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8024.063296] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8024.063303] Call Trace: [ 8024.063306] [ 8024.063308] ? show_regs.cold.9+0x22/0x2f [ 8024.063315] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8024.063320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8024.063323] ? watchdog+0x40/0x40 [ 8024.063327] ? hrtimer_interrupt+0x128/0x2c0 [ 8024.063331] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8024.063337] ? apic_timer_interrupt+0xf/0x20 [ 8024.063342] [ 8024.063345] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.063350] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8024.063355] do_raw_spin_lock+0x177/0x190 [ 8024.063359] _raw_spin_lock+0x21/0x30 [ 8024.063364] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8024.062885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8024.064514] ? do_wait_intr+0xf0/0xf0 [ 8024.064522] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8024.064688] process_one_work+0x2c8/0x700 [ 8024.064695] worker_thread+0x296/0x6e0 [ 8024.064700] ? rescuer_thread+0x570/0x570 [ 8024.064703] kthread+0x1d1/0x200 [ 8024.064709] ? set_kthread_struct+0x70/0x70 [ 8024.064712] ret_from_fork+0x1f/0x30 [ 8024.066710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8024.066781] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8024.066787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8024.066790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8024.066802] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8024.066806] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8024.066810] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8024.066813] RDX: 0000000000000deb RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8024.066815] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8024.066817] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8024.066820] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8024.066822] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8024.066826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8024.066828] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8024.066837] Call Trace: [ 8024.066841] [ 8024.066847] ? show_regs.cold.9+0x22/0x2f [ 8024.066857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8024.066864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8024.066936] ? watchdog+0x40/0x40 [ 8024.066941] ? hrtimer_interrupt+0x128/0x2c0 [ 8024.066945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8024.066952] ? apic_timer_interrupt+0xf/0x20 [ 8024.066957] [ 8024.066960] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8024.066965] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8024.066969] do_raw_spin_lock+0x177/0x190 [ 8024.066974] _raw_spin_lock+0x21/0x30 [ 8024.066979] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8024.067234] obd_mod_free+0x46/0x140 [ptlrpc] [ 8024.067449] mdc_commit_open+0x91/0xb0 [mdc] [ 8024.067552] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8024.067750] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8024.068152] mdc_close+0x53b/0xd70 [mdc] [ 8024.068257] lmv_close+0x18d/0x490 [lmv] [ 8024.068377] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8024.068698] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8024.068781] ll_md_close+0x274/0x800 [lustre] [ 8024.068865] ll_file_release+0x128/0xc50 [lustre] [ 8024.068946] __fput+0x10d/0x440 [ 8024.068955] ____fput+0x12/0x20 [ 8024.068959] task_work_run+0xc6/0x110 [ 8024.068966] exit_to_usermode_loop+0x1dd/0x1f0 [ 8024.068974] do_syscall_64+0x3d6/0x440 [ 8024.068978] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8024.068985] RIP: 0033:0x7fa749552cd8 [ 8024.068992] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8024.068996] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8024.069001] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8024.069004] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8024.069006] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8024.069009] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8024.069011] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8024.074941] osp(O) [ 8024.096546] after_reply+0xe2c/0x1780 [ptlrpc] [ 8024.096826] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8024.097085] ? __next_timer_interrupt+0x160/0x160 [ 8024.097096] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8024.097364] ? do_wait_intr+0xf0/0xf0 [ 8024.097375] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8024.097843] kthread+0x1d1/0x200 [ 8024.097849] ? set_kthread_struct+0x70/0x70 [ 8024.097853] ret_from_fork+0x1f/0x30 [ 8024.614416] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8024.633335] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8024.639295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8024.643809] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.645723] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8024.654333] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8024.658393] RAX: 00000000000056a9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8024.662061] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8024.665545] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8024.669331] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8024.672583] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8024.675134] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8024.679726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8024.680867] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8024.688336] Call Trace: [ 8024.690705] [ 8024.692337] ? show_regs.cold.9+0x22/0x2f [ 8024.694611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8024.698368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8024.700428] ? watchdog+0x40/0x40 [ 8024.703673] ? hrtimer_interrupt+0x128/0x2c0 [ 8024.707243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8024.709745] ? apic_timer_interrupt+0xf/0x20 [ 8024.711677] [ 8024.712768] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.716444] do_raw_spin_lock+0x177/0x190 [ 8024.718531] _raw_spin_lock+0x21/0x30 [ 8024.723557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8024.728261] ? do_raw_spin_unlock+0x75/0x190 [ 8024.731828] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8024.735980] ? do_wait_intr+0xf0/0xf0 [ 8024.737416] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8024.740076] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8024.742504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8024.744981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.747180] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8024.749225] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8024.751215] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8024.752726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.754456] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8024.756218] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8024.757933] lmv_intent_open+0x338/0x1150 [lmv] [ 8024.759082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.761054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.762622] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8024.763956] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8024.765295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.767589] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8024.769960] ? do_raw_spin_unlock+0x75/0x190 [ 8024.771660] ? _raw_spin_unlock+0x12/0x30 [ 8024.773384] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8024.775585] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8024.777827] ll_atomic_open+0x36a/0xd70 [lustre] [ 8024.780213] lookup_open+0x356/0xb40 [ 8024.781669] do_last+0x313/0xfc0 [ 8024.783186] ? nd_jump_root+0xe5/0x160 [ 8024.784768] ? path_init+0x437/0x520 [ 8024.786420] path_openat+0xf7/0x500 [ 8024.787553] ? do_raw_spin_unlock+0x75/0x190 [ 8024.788675] do_filp_open+0x99/0x140 [ 8024.789667] ? getname_flags+0x6e/0x330 [ 8024.794368] ? __check_object_size+0xff/0x256 [ 8024.796741] ? do_raw_spin_unlock+0x75/0x190 [ 8024.799768] ? _raw_spin_unlock+0x12/0x30 [ 8024.803388] do_sys_openat2+0x2b4/0x410 [ 8024.805303] do_sys_open+0x73/0xa0 [ 8024.806485] __x64_sys_openat+0x24/0x30 [ 8024.809338] do_syscall_64+0xc1/0x440 [ 8024.812067] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8024.814007] RIP: 0033:0x7fa749552332 [ 8024.815372] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8024.826381] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8024.829918] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8024.833795] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8024.837322] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8024.841168] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8024.844471] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8026.081041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6108s! [ 8026.085554] Showing busy workqueues and worker pools: [ 8026.087242] workqueue events: flags=0x0 [ 8026.090584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8026.096750] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8026.107487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8026.119289] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8026.129628] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.136689] pending: kfree_rcu_monitor [ 8026.142306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8026.145915] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8026.152556] workqueue events_long: flags=0x0 [ 8026.156566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.163417] pending: lu_objects_flush [obdclass] [ 8026.169453] workqueue events_unbound: flags=0x2 [ 8026.175770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8026.181527] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8026.191599] workqueue events_power_efficient: flags=0x80 [ 8026.193352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.200878] pending: check_lifetime [ 8026.206773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.211338] pending: neigh_periodic_work [ 8026.214269] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.222359] pending: neigh_periodic_work [ 8026.225885] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.229631] pending: do_cache_clean [ 8026.230943] workqueue mm_percpu_wq: flags=0x8 [ 8026.232355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.245500] pending: vmstat_update [ 8026.247719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.253858] pending: vmstat_update [ 8026.258633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8026.267472] pending: vmstat_update [ 8026.270865] workqueue writeback: flags=0x4a [ 8026.274274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8026.283548] pending: wb_workfn, wb_workfn [ 8026.287749] workqueue xprtiod: flags=0xa [ 8026.294303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8026.302704] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8026.315920] workqueue ipv6_addrconf: flags=0x40008 [ 8026.327845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8026.333907] pending: addrconf_verify_work [ 8026.335559] workqueue knbd0-recv: flags=0x1a [ 8026.337459] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8026.340402] in-flight: 651:recv_work [ 8026.342031] workqueue ptlrpc_pinger: flags=0x2 [ 8026.345454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8026.348995] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8026.356454] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6109s workers=4 idle: 37 31846 28897 [ 8026.361500] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8016s workers=2 idle: 653 [ 8052.057122] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8052.057940] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8052.059096] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8052.059112] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8052.059189] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8052.059194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8052.059196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8052.059210] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8052.059214] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8052.059218] RAX: 0000000000004b25 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8052.059221] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8052.059223] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8052.059225] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8052.059228] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8052.059231] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8052.059437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.059439] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8052.059449] Call Trace: [ 8052.059455] [ 8052.059462] ? show_regs.cold.9+0x22/0x2f [ 8052.059474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8052.059481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8052.059486] ? watchdog+0x40/0x40 [ 8052.059490] ? hrtimer_interrupt+0x128/0x2c0 [ 8052.059493] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8052.059501] ? apic_timer_interrupt+0xf/0x20 [ 8052.059506] [ 8052.059509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8052.059515] do_raw_spin_lock+0x177/0x190 [ 8052.059520] _raw_spin_lock+0x21/0x30 [ 8052.059524] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8052.059891] ? do_raw_spin_unlock+0x75/0x190 [ 8052.059897] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8052.060095] ? do_wait_intr+0xf0/0xf0 [ 8052.060102] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8052.060589] mdc_close+0x53b/0xd70 [mdc] [ 8052.060766] lmv_close+0x18d/0x490 [lmv] [ 8052.060797] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8052.060940] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8052.061031] ll_md_close+0x274/0x800 [lustre] [ 8052.061124] ll_file_release+0x128/0xc50 [lustre] [ 8052.061438] __fput+0x10d/0x440 [ 8052.061453] ____fput+0x12/0x20 [ 8052.061458] task_work_run+0xc6/0x110 [ 8052.061466] exit_to_usermode_loop+0x1dd/0x1f0 [ 8052.061472] do_syscall_64+0x3d6/0x440 [ 8052.061477] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8052.061485] RIP: 0033:0x7fa749552cd8 [ 8052.061493] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8052.061497] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8052.061502] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8052.061504] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8052.061507] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8052.061509] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8052.061511] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8052.062254] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8052.062264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8052.062329] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8052.062334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8052.062338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8052.062348] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8052.062352] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8052.062356] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8052.062359] RDX: 00000000000068d6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8052.062361] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8052.062363] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8052.062366] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8052.062369] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8052.062372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.062374] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8052.062385] Call Trace: [ 8052.062388] [ 8052.062392] ? show_regs.cold.9+0x22/0x2f [ 8052.062400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8052.062405] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8052.062409] ? watchdog+0x40/0x40 [ 8052.062413] ? hrtimer_interrupt+0x128/0x2c0 [ 8052.062416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8052.062422] ? apic_timer_interrupt+0xf/0x20 [ 8052.062426] [ 8052.062429] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8052.062433] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8052.062438] do_raw_spin_lock+0x177/0x190 [ 8052.062442] _raw_spin_lock+0x21/0x30 [ 8052.062447] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8052.062988] obd_mod_free+0x46/0x140 [ptlrpc] [ 8052.063356] mdc_commit_open+0x91/0xb0 [mdc] [ 8052.063393] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8052.063871] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8052.064465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8052.064935] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8052.064942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8052.064944] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8052.064956] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8052.064960] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8052.064965] RAX: 0000000000007583 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8052.064967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8052.064969] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8052.064972] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8052.064974] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8052.064977] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8052.064980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.064983] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8052.064992] Call Trace: [ 8052.064996] [ 8052.064999] ? show_regs.cold.9+0x22/0x2f [ 8052.065007] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8052.065013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8052.065017] ? watchdog+0x40/0x40 [ 8052.065021] ? hrtimer_interrupt+0x128/0x2c0 [ 8052.065026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8052.065033] ? apic_timer_interrupt+0xf/0x20 [ 8052.065038] [ 8052.065041] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8052.065047] do_raw_spin_lock+0x177/0x190 [ 8052.065052] _raw_spin_lock+0x21/0x30 [ 8052.065057] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8052.065966] ? do_raw_spin_unlock+0x75/0x190 [ 8052.065973] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8052.066404] ? do_wait_intr+0xf0/0xf0 [ 8052.066414] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8052.066704] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8052.064076] after_reply+0xe2c/0x1780 [ptlrpc] [ 8052.067217] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8052.067063] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8052.069129] ? __next_timer_interrupt+0x160/0x160 [ 8052.069480] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8052.070428] ? do_wait_intr+0xf0/0xf0 [ 8052.070438] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8052.070680] kthread+0x1d1/0x200 [ 8052.070687] ? set_kthread_struct+0x70/0x70 [ 8052.070691] ret_from_fork+0x1f/0x30 [ 8052.075852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8052.075945] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8052.075952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8052.075959] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8052.076247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8052.076257] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8052.076261] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8052.076265] RAX: 0000000000007d76 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8052.076268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8052.076270] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8052.076272] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8052.076274] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8052.076277] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8052.076280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.076282] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8052.076293] Call Trace: [ 8052.076299] [ 8052.076305] ? show_regs.cold.9+0x22/0x2f [ 8052.076315] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8052.076322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8052.076327] ? watchdog+0x40/0x40 [ 8052.076331] ? hrtimer_interrupt+0x128/0x2c0 [ 8052.076335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8052.076345] ? apic_timer_interrupt+0xf/0x20 [ 8052.076349] [ 8052.076352] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8052.076357] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8052.076362] do_raw_spin_lock+0x177/0x190 [ 8052.076367] _raw_spin_lock+0x21/0x30 [ 8052.076371] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8052.076575] process_one_work+0x2c8/0x700 [ 8052.076584] worker_thread+0x296/0x6e0 [ 8052.076588] ? rescuer_thread+0x570/0x570 [ 8052.076592] kthread+0x1d1/0x200 [ 8052.076597] ? set_kthread_struct+0x70/0x70 [ 8052.076600] ret_from_fork+0x1f/0x30 [ 8052.932732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.938147] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8052.941855] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8052.944606] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8052.946740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.950049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8052.953138] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8052.956002] lmv_intent_open+0x338/0x1150 [lmv] [ 8052.960980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.966194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.971314] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8052.974221] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8052.978332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.984949] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8052.990296] ? do_raw_spin_unlock+0x75/0x190 [ 8052.992445] ? _raw_spin_unlock+0x12/0x30 [ 8052.995764] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8053.000208] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8053.007959] ll_atomic_open+0x36a/0xd70 [lustre] [ 8053.010811] lookup_open+0x356/0xb40 [ 8053.016111] do_last+0x313/0xfc0 [ 8053.017377] ? nd_jump_root+0xe5/0x160 [ 8053.019298] ? path_init+0x437/0x520 [ 8053.023082] path_openat+0xf7/0x500 [ 8053.026955] ? do_raw_spin_unlock+0x75/0x190 [ 8053.029236] do_filp_open+0x99/0x140 [ 8053.030991] ? getname_flags+0x6e/0x330 [ 8053.035096] ? __check_object_size+0xff/0x256 [ 8053.039240] ? do_raw_spin_unlock+0x75/0x190 [ 8053.044847] ? _raw_spin_unlock+0x12/0x30 [ 8053.048437] do_sys_openat2+0x2b4/0x410 [ 8053.052045] do_sys_open+0x73/0xa0 [ 8053.054346] __x64_sys_openat+0x24/0x30 [ 8053.058127] do_syscall_64+0xc1/0x440 [ 8053.060012] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8053.064943] RIP: 0033:0x7fa749552332 [ 8053.070770] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8053.083895] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8053.097185] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8053.107202] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8053.116873] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8053.124371] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8053.135643] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8056.801082] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6139s! [ 8056.805644] Showing busy workqueues and worker pools: [ 8056.808036] workqueue events: flags=0x0 [ 8056.810617] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8056.814118] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8056.820205] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8056.823156] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8056.827266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.829980] pending: kfree_rcu_monitor [ 8056.832048] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8056.835270] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8056.839246] workqueue events_long: flags=0x0 [ 8056.841361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.844983] pending: lu_objects_flush [obdclass] [ 8056.847152] workqueue events_unbound: flags=0x2 [ 8056.849164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8056.851742] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8056.855739] workqueue events_power_efficient: flags=0x80 [ 8056.858146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.860569] pending: check_lifetime [ 8056.861968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.867087] pending: neigh_periodic_work [ 8056.869186] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.872258] pending: neigh_periodic_work [ 8056.874523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.876990] pending: do_cache_clean [ 8056.878487] workqueue mm_percpu_wq: flags=0x8 [ 8056.881122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.884100] pending: vmstat_update [ 8056.886058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.890068] pending: vmstat_update [ 8056.892542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8056.896527] pending: vmstat_update [ 8056.904148] workqueue writeback: flags=0x4a [ 8056.907153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8056.912118] pending: wb_workfn, wb_workfn [ 8056.914759] workqueue xprtiod: flags=0xa [ 8056.917435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8056.921128] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8056.925066] workqueue ipv6_addrconf: flags=0x40008 [ 8056.927013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8056.930992] pending: addrconf_verify_work [ 8056.933105] workqueue knbd0-recv: flags=0x1a [ 8056.935168] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8056.938143] in-flight: 651:recv_work [ 8056.940153] workqueue ptlrpc_pinger: flags=0x2 [ 8056.943065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8056.947513] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8056.950974] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6139s workers=4 idle: 37 31846 28897 [ 8056.955438] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8046s workers=2 idle: 653 [ 8080.057143] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8080.058199] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8080.059624] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8080.059635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8080.059705] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8080.059710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8080.059713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8080.059726] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8080.059730] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8080.059735] RAX: 0000000000001483 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8080.059737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8080.059739] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8080.059741] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8080.059744] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8080.059747] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8080.059750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.059752] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8080.059760] Call Trace: [ 8080.059765] [ 8080.059771] ? show_regs.cold.9+0x22/0x2f [ 8080.059780] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8080.059785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8080.059790] ? watchdog+0x40/0x40 [ 8080.059794] ? hrtimer_interrupt+0x128/0x2c0 [ 8080.059798] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8080.059805] ? apic_timer_interrupt+0xf/0x20 [ 8080.059810] [ 8080.059812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8080.059818] do_raw_spin_lock+0x177/0x190 [ 8080.059823] _raw_spin_lock+0x21/0x30 [ 8080.059827] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8080.060083] ? do_raw_spin_unlock+0x75/0x190 [ 8080.060088] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8080.060496] ? do_wait_intr+0xf0/0xf0 [ 8080.060505] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8080.060714] mdc_close+0x53b/0xd70 [mdc] [ 8080.060749] lmv_close+0x18d/0x490 [lmv] [ 8080.060779] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8080.060891] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8080.060976] ll_md_close+0x274/0x800 [lustre] [ 8080.061146] ll_file_release+0x128/0xc50 [lustre] [ 8080.061357] __fput+0x10d/0x440 [ 8080.061368] ____fput+0x12/0x20 [ 8080.061373] task_work_run+0xc6/0x110 [ 8080.061380] exit_to_usermode_loop+0x1dd/0x1f0 [ 8080.061387] do_syscall_64+0x3d6/0x440 [ 8080.061392] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8080.061399] RIP: 0033:0x7fa749552cd8 [ 8080.061406] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8080.061410] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8080.061414] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8080.061416] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8080.061418] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8080.061420] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8080.061422] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8080.061967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8080.062034] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8080.062040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8080.062045] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8080.062306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8080.062315] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8080.062319] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8080.062323] RAX: 0000000000007715 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8080.062326] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8080.062328] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8080.062331] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8080.062333] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8080.062337] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8080.062340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.062342] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8080.062352] Call Trace: [ 8080.062356] [ 8080.062360] ? show_regs.cold.9+0x22/0x2f [ 8080.062368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8080.062375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8080.062379] ? watchdog+0x40/0x40 [ 8080.062384] ? hrtimer_interrupt+0x128/0x2c0 [ 8080.062389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8080.062395] ? apic_timer_interrupt+0xf/0x20 [ 8080.062399] [ 8080.062403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8080.062407] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8080.062413] do_raw_spin_lock+0x177/0x190 [ 8080.062418] _raw_spin_lock+0x21/0x30 [ 8080.062423] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8080.062655] process_one_work+0x2c8/0x700 [ 8080.062661] worker_thread+0x296/0x6e0 [ 8080.062666] ? rescuer_thread+0x570/0x570 [ 8080.062669] kthread+0x1d1/0x200 [ 8080.062674] ? set_kthread_struct+0x70/0x70 [ 8080.062678] ret_from_fork+0x1f/0x30 [ 8080.369136] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8080.389887] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8080.394872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8080.398502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8080.401076] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8080.408123] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8080.410352] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8080.412850] RDX: 0000000000007aac RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8080.415834] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8080.418235] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8080.420529] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8080.423097] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8080.425794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.427750] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8080.430080] Call Trace: [ 8080.430782] [ 8080.431340] ? show_regs.cold.9+0x22/0x2f [ 8080.432835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8080.434600] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8080.436464] ? watchdog+0x40/0x40 [ 8080.437484] ? hrtimer_interrupt+0x128/0x2c0 [ 8080.439015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8080.440853] ? apic_timer_interrupt+0xf/0x20 [ 8080.442633] [ 8080.443517] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8080.445661] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8080.448581] do_raw_spin_lock+0x177/0x190 [ 8080.450271] _raw_spin_lock+0x21/0x30 [ 8080.451826] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8080.456194] obd_mod_free+0x46/0x140 [ptlrpc] [ 8080.459255] mdc_commit_open+0x91/0xb0 [mdc] [ 8080.461212] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8080.463873] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8080.466373] after_reply+0xe2c/0x1780 [ptlrpc] [ 8080.468385] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8080.470674] ? __next_timer_interrupt+0x160/0x160 [ 8080.472700] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8080.474773] ? do_wait_intr+0xf0/0xf0 [ 8080.476344] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8080.478849] kthread+0x1d1/0x200 [ 8080.480328] ? set_kthread_struct+0x70/0x70 [ 8080.481920] ret_from_fork+0x1f/0x30 [ 8084.054856] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8084.061747] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8084.098414] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8084.106811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8084.111545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8084.114407] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8084.124702] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8084.129074] RAX: 0000000000007f85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8084.132957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8084.136903] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8084.140631] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8084.148378] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8084.153587] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8084.158477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.161829] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8084.165190] Call Trace: [ 8084.166255] [ 8084.167251] ? show_regs.cold.9+0x22/0x2f [ 8084.169019] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8084.171686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8084.173767] ? watchdog+0x40/0x40 [ 8084.175230] ? hrtimer_interrupt+0x128/0x2c0 [ 8084.177346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8084.179757] ? apic_timer_interrupt+0xf/0x20 [ 8084.181793] [ 8084.182731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8084.185263] do_raw_spin_lock+0x177/0x190 [ 8084.187161] _raw_spin_lock+0x21/0x30 [ 8084.188901] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8084.191412] ? do_raw_spin_unlock+0x75/0x190 [ 8084.193308] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8084.195950] ? do_wait_intr+0xf0/0xf0 [ 8084.197409] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8084.199816] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8084.202236] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8084.205291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8084.207793] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8084.210296] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8084.212398] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8084.214441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8084.217017] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8084.219821] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8084.224594] lmv_intent_open+0x338/0x1150 [lmv] [ 8084.226917] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8084.229831] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8084.232710] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8084.235333] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8084.238160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8084.240821] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8084.242979] ? do_raw_spin_unlock+0x75/0x190 [ 8084.245040] ? _raw_spin_unlock+0x12/0x30 [ 8084.246856] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8084.248913] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8084.251430] ll_atomic_open+0x36a/0xd70 [lustre] [ 8084.254645] lookup_open+0x356/0xb40 [ 8084.256398] do_last+0x313/0xfc0 [ 8084.257969] ? nd_jump_root+0xe5/0x160 [ 8084.259722] ? path_init+0x437/0x520 [ 8084.261340] path_openat+0xf7/0x500 [ 8084.262940] ? do_raw_spin_unlock+0x75/0x190 [ 8084.264798] do_filp_open+0x99/0x140 [ 8084.266370] ? getname_flags+0x6e/0x330 [ 8084.268198] ? __check_object_size+0xff/0x256 [ 8084.270174] ? do_raw_spin_unlock+0x75/0x190 [ 8084.272369] ? _raw_spin_unlock+0x12/0x30 [ 8084.274293] do_sys_openat2+0x2b4/0x410 [ 8084.276169] do_sys_open+0x73/0xa0 [ 8084.277710] __x64_sys_openat+0x24/0x30 [ 8084.279392] do_syscall_64+0xc1/0x440 [ 8084.281228] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8084.283972] RIP: 0033:0x7fa749552332 [ 8084.285941] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8084.302680] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8084.306474] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8084.309863] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8084.314290] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8084.317218] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8084.321186] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8087.008059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6169s! [ 8087.011720] Showing busy workqueues and worker pools: [ 8087.014426] workqueue events: flags=0x0 [ 8087.016371] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8087.019457] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8087.032255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8087.036668] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8087.042956] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.046703] pending: kfree_rcu_monitor [ 8087.049471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8087.056509] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8087.068529] workqueue events_long: flags=0x0 [ 8087.074785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.083034] pending: lu_objects_flush [obdclass] [ 8087.085026] workqueue events_unbound: flags=0x2 [ 8087.093581] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8087.099947] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8087.105216] workqueue events_power_efficient: flags=0x80 [ 8087.110923] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.117698] pending: check_lifetime [ 8087.124764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.128874] pending: neigh_periodic_work [ 8087.134188] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.139611] pending: neigh_periodic_work [ 8087.142523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.148225] pending: do_cache_clean [ 8087.150652] workqueue mm_percpu_wq: flags=0x8 [ 8087.159053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.169549] pending: vmstat_update [ 8087.173768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.182960] pending: vmstat_update [ 8087.185038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8087.189718] pending: vmstat_update [ 8087.194512] workqueue writeback: flags=0x4a [ 8087.198496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8087.206928] pending: wb_workfn, wb_workfn [ 8087.213703] workqueue xprtiod: flags=0xa [ 8087.216734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8087.226964] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8087.238559] workqueue ipv6_addrconf: flags=0x40008 [ 8087.247444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8087.249888] pending: addrconf_verify_work [ 8087.251716] workqueue knbd0-recv: flags=0x1a [ 8087.254712] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8087.258870] in-flight: 651:recv_work [ 8087.262832] workqueue ptlrpc_pinger: flags=0x2 [ 8087.268500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8087.274934] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8087.279467] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6170s workers=4 idle: 37 31846 28897 [ 8087.284887] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8077s workers=2 idle: 653 [ 8093.825022] rcu: INFO: rcu_sched self-detected stall on CPU [ 8093.827497] rcu: 2-...!: (3444343 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 8093.833301] (t=6180136 jiffies g=258701 q=3259) [ 8093.838411] rcu: rcu_sched kthread starved for 6176931 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8093.849500] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8093.862402] rcu: RCU grace-period kthread stack dump: [ 8093.866339] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8093.877079] Call Trace: [ 8093.877898] __schedule+0x351/0xcb0 [ 8093.883478] schedule+0xc0/0x180 [ 8093.888476] schedule_timeout+0xb4/0x190 [ 8093.891664] ? do_raw_spin_unlock+0x75/0x190 [ 8093.903612] ? __next_timer_interrupt+0x160/0x160 [ 8093.913324] ? prepare_to_swait_event+0xf3/0x140 [ 8093.924686] rcu_gp_kthread+0x8dd/0x15b0 [ 8093.932327] ? force_qs_rnp+0x2c0/0x2c0 [ 8093.938771] kthread+0x1d1/0x200 [ 8093.941625] ? set_kthread_struct+0x70/0x70 [ 8093.949529] ret_from_fork+0x1f/0x30 [ 8093.952728] rcu: Stack dump where RCU GP kthread last ran: [ 8093.967387] NMI backtrace for cpu 2 [ 8093.973781] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8093.989557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8094.005328] Call Trace: [ 8094.009435] [ 8094.011667] ? dump_stack+0xbb/0x10e [ 8094.016501] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8094.028297] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8094.033414] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8094.042630] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8094.050635] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8094.056365] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8094.060564] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 8094.063498] ? update_process_times+0x6d/0xc0 [ 8094.069399] ? tick_sched_handle.isra.18+0x27/0x90 [ 8094.073654] ? tick_sched_timer+0x53/0xb0 [ 8094.078615] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8094.080101] ? tick_sched_handle.isra.18+0x90/0x90 [ 8094.085010] ? hrtimer_interrupt+0x128/0x2c0 [ 8094.086414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8094.089578] ? apic_timer_interrupt+0xf/0x20 [ 8094.093273] [ 8094.094173] ? native_safe_halt+0x2e/0x50 [ 8094.095267] ? kvm_wait+0xcd/0xe0 [ 8094.096286] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8094.097772] ? do_raw_spin_lock+0x177/0x190 [ 8094.099091] ? _raw_spin_lock+0x21/0x30 [ 8094.100450] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8094.102372] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8094.104352] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8094.106187] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8094.108233] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8094.110477] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8094.112331] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8094.114352] ? __next_timer_interrupt+0x160/0x160 [ 8094.115946] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8094.117474] ? do_wait_intr+0xf0/0xf0 [ 8094.118673] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8094.120562] ? kthread+0x1d1/0x200 [ 8094.121649] ? set_kthread_struct+0x70/0x70 [ 8094.123053] ? ret_from_fork+0x1f/0x30 [ 8094.124301] NMI backtrace for cpu 2 [ 8094.125374] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8094.128789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8094.131328] Call Trace: [ 8094.132041] [ 8094.132586] ? dump_stack+0xbb/0x10e [ 8094.133631] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8094.135226] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8094.136735] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8094.138359] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8094.140105] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8094.141728] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8094.143341] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8094.144891] ? update_process_times+0x6d/0xc0 [ 8094.146377] ? tick_sched_handle.isra.18+0x27/0x90 [ 8094.148156] ? tick_sched_timer+0x53/0xb0 [ 8094.149459] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8094.150981] ? tick_sched_handle.isra.18+0x90/0x90 [ 8094.152457] ? hrtimer_interrupt+0x128/0x2c0 [ 8094.153871] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8094.155399] ? apic_timer_interrupt+0xf/0x20 [ 8094.156806] [ 8094.157628] ? native_safe_halt+0x2e/0x50 [ 8094.159238] ? kvm_wait+0xcd/0xe0 [ 8094.160477] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8094.162401] ? do_raw_spin_lock+0x177/0x190 [ 8094.164049] ? _raw_spin_lock+0x21/0x30 [ 8094.165268] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8094.167291] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8094.169346] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8094.171036] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8094.173161] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8094.175487] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8094.177495] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8094.179671] ? __next_timer_interrupt+0x160/0x160 [ 8094.181439] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8094.183295] ? do_wait_intr+0xf0/0xf0 [ 8094.184669] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8094.186740] ? kthread+0x1d1/0x200 [ 8094.187822] ? set_kthread_struct+0x70/0x70 [ 8094.189298] ? ret_from_fork+0x1f/0x30 [ 8108.062964] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8108.063333] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8108.065298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8108.065373] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8108.065380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8108.065386] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8108.065662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8108.065673] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8108.065677] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8108.065682] RAX: 0000000000007e04 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8108.065684] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8108.065686] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8108.065688] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8108.065691] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8108.065694] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8108.065696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8108.065698] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8108.065707] Call Trace: [ 8108.065713] [ 8108.065718] ? show_regs.cold.9+0x22/0x2f [ 8108.065727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8108.065733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8108.065738] ? watchdog+0x40/0x40 [ 8108.065742] ? hrtimer_interrupt+0x128/0x2c0 [ 8108.065746] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8108.065755] ? apic_timer_interrupt+0xf/0x20 [ 8108.065759] [ 8108.065762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8108.065767] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8108.065772] do_raw_spin_lock+0x177/0x190 [ 8108.065778] _raw_spin_lock+0x21/0x30 [ 8108.065783] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8108.073724] process_one_work+0x2c8/0x700 [ 8108.073741] worker_thread+0x296/0x6e0 [ 8108.073747] ? rescuer_thread+0x570/0x570 [ 8108.073751] kthread+0x1d1/0x200 [ 8108.073755] ? set_kthread_struct+0x70/0x70 [ 8108.073759] ret_from_fork+0x1f/0x30 [ 8108.371713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8108.424519] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8108.445017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8108.465468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8108.472360] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8108.481968] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8108.490042] RAX: 0000000000007e12 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8108.496192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8108.502885] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8108.510827] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8108.517989] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8108.523208] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8108.528025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8108.534588] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8108.542133] Call Trace: [ 8108.543504] [ 8108.544529] ? show_regs.cold.9+0x22/0x2f [ 8108.547769] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8108.550837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8108.561104] ? watchdog+0x40/0x40 [ 8108.562287] ? hrtimer_interrupt+0x128/0x2c0 [ 8108.563786] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8108.570607] ? apic_timer_interrupt+0xf/0x20 [ 8108.572527] [ 8108.573441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8108.575766] do_raw_spin_lock+0x177/0x190 [ 8108.582670] _raw_spin_lock+0x21/0x30 [ 8108.583816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8108.585594] ? do_raw_spin_unlock+0x75/0x190 [ 8108.588633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8108.594481] ? do_wait_intr+0xf0/0xf0 [ 8108.596691] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8108.600175] mdc_close+0x53b/0xd70 [mdc] [ 8108.602175] lmv_close+0x18d/0x490 [lmv] [ 8108.606472] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8108.618130] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8108.622451] ll_md_close+0x274/0x800 [lustre] [ 8108.626237] ll_file_release+0x128/0xc50 [lustre] [ 8108.629600] __fput+0x10d/0x440 [ 8108.634541] ____fput+0x12/0x20 [ 8108.635567] task_work_run+0xc6/0x110 [ 8108.640086] exit_to_usermode_loop+0x1dd/0x1f0 [ 8108.643985] do_syscall_64+0x3d6/0x440 [ 8108.649269] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8108.656278] RIP: 0033:0x7fa749552cd8 [ 8108.659745] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8108.673635] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8108.681619] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8108.685412] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8108.689087] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8108.692707] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8108.696527] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8112.056001] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8112.059465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8112.115219] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8112.122376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8112.131297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8112.138089] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8112.153118] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8112.159140] RAX: 0000000000007418 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8112.167166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8112.170938] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8112.175302] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8112.186052] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8112.188903] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8112.192818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8112.195534] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8112.201653] Call Trace: [ 8112.207095] [ 8112.208294] ? show_regs.cold.9+0x22/0x2f [ 8112.210613] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8112.218389] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8112.220158] ? watchdog+0x40/0x40 [ 8112.230261] ? hrtimer_interrupt+0x128/0x2c0 [ 8112.236637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8112.239481] ? apic_timer_interrupt+0xf/0x20 [ 8112.242788] [ 8112.247495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8112.259282] do_raw_spin_lock+0x177/0x190 [ 8112.264896] _raw_spin_lock+0x21/0x30 [ 8112.266660] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8112.272460] ? do_raw_spin_unlock+0x75/0x190 [ 8112.276191] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8112.282644] ? do_wait_intr+0xf0/0xf0 [ 8112.284710] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8112.288523] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8112.292664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8112.296001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8112.299222] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8112.302034] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8112.304877] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8112.307525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8112.311891] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8112.315982] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8112.319907] lmv_intent_open+0x338/0x1150 [lmv] [ 8112.322684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8112.325884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8112.328665] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8112.330566] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8112.332730] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8112.335712] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8112.338517] ? do_raw_spin_unlock+0x75/0x190 [ 8112.340897] ? _raw_spin_unlock+0x12/0x30 [ 8112.342936] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8112.345341] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8112.348820] ll_atomic_open+0x36a/0xd70 [lustre] [ 8112.351206] lookup_open+0x356/0xb40 [ 8112.353163] do_last+0x313/0xfc0 [ 8112.354367] ? nd_jump_root+0xe5/0x160 [ 8112.356107] ? path_init+0x437/0x520 [ 8112.358152] path_openat+0xf7/0x500 [ 8112.359999] ? do_raw_spin_unlock+0x75/0x190 [ 8112.362239] do_filp_open+0x99/0x140 [ 8112.367599] ? getname_flags+0x6e/0x330 [ 8112.369368] ? __check_object_size+0xff/0x256 [ 8112.372200] ? do_raw_spin_unlock+0x75/0x190 [ 8112.374022] ? _raw_spin_unlock+0x12/0x30 [ 8112.376147] do_sys_openat2+0x2b4/0x410 [ 8112.377916] do_sys_open+0x73/0xa0 [ 8112.379589] __x64_sys_openat+0x24/0x30 [ 8112.381279] do_syscall_64+0xc1/0x440 [ 8112.383218] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8112.386156] RIP: 0033:0x7fa749552332 [ 8112.387683] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8112.396726] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8112.400420] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8112.403907] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8112.407179] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8112.410670] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8112.413893] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8117.730247] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6200s! [ 8117.735916] Showing busy workqueues and worker pools: [ 8117.738870] workqueue events: flags=0x0 [ 8117.741798] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8117.745607] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8117.752842] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8117.756907] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8117.762706] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.767127] pending: kfree_rcu_monitor [ 8117.769687] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8117.772445] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8117.779201] workqueue events_long: flags=0x0 [ 8117.782148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.785738] pending: lu_objects_flush [obdclass] [ 8117.788859] workqueue events_unbound: flags=0x2 [ 8117.791654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8117.795262] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8117.800045] workqueue events_power_efficient: flags=0x80 [ 8117.803643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.807265] pending: check_lifetime [ 8117.809611] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.813464] pending: neigh_periodic_work [ 8117.816010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.820118] pending: neigh_periodic_work [ 8117.822932] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.826708] pending: do_cache_clean [ 8117.828810] workqueue mm_percpu_wq: flags=0x8 [ 8117.832107] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.835995] pending: vmstat_update [ 8117.838595] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.841872] pending: vmstat_update [ 8117.844741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8117.848903] pending: vmstat_update [ 8117.851434] workqueue writeback: flags=0x4a [ 8117.854732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8117.858548] pending: wb_workfn, wb_workfn [ 8117.861431] workqueue xprtiod: flags=0xa [ 8117.863449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8117.867545] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8117.872864] workqueue ipv6_addrconf: flags=0x40008 [ 8117.875534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8117.879151] pending: addrconf_verify_work [ 8117.881196] workqueue knbd0-recv: flags=0x1a [ 8117.883899] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8117.887118] in-flight: 651:recv_work [ 8117.889073] workqueue ptlrpc_pinger: flags=0x2 [ 8117.891645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8117.894748] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8117.898102] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6200s workers=4 idle: 37 31846 28897 [ 8117.902634] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8107s workers=2 idle: 653 [ 8120.058108] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8120.075519] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8120.114012] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8120.130684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8120.142359] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8120.150348] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8120.170466] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8120.174248] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8120.177659] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8120.180553] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8120.189335] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8120.196005] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8120.199340] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8120.205618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8120.212331] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8120.219383] Call Trace: [ 8120.220016] [ 8120.220456] ? show_regs.cold.9+0x22/0x2f [ 8120.221635] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8120.230236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8120.238057] ? watchdog+0x40/0x40 [ 8120.239632] ? hrtimer_interrupt+0x128/0x2c0 [ 8120.244310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8120.247194] ? apic_timer_interrupt+0xf/0x20 [ 8120.249799] [ 8120.250696] ? native_safe_halt+0x2e/0x50 [ 8120.254776] kvm_wait+0xcd/0xe0 [ 8120.260539] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8120.262232] do_raw_spin_lock+0x177/0x190 [ 8120.264234] _raw_spin_lock+0x21/0x30 [ 8120.265561] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8120.271403] obd_mod_free+0x46/0x140 [ptlrpc] [ 8120.273277] mdc_commit_open+0x91/0xb0 [mdc] [ 8120.281426] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8120.286376] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8120.290423] after_reply+0xe2c/0x1780 [ptlrpc] [ 8120.293120] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8120.298072] ? __next_timer_interrupt+0x160/0x160 [ 8120.302918] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8120.304438] ? do_wait_intr+0xf0/0xf0 [ 8120.310055] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8120.312613] kthread+0x1d1/0x200 [ 8120.314636] ? set_kthread_struct+0x70/0x70 [ 8120.317124] ret_from_fork+0x1f/0x30 [ 8136.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8136.059025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8136.063997] Modules linked in: lustre(O) [ 8136.067216] Modules linked in: [ 8136.067218] osp(O) ofd(O) [ 8136.068427] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 8136.070359] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8136.070658] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8136.070666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8136.070672] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8136.071073] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8136.071084] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8136.071089] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8136.071093] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8136.071095] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8136.071098] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8136.071100] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8136.071102] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8136.071106] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8136.071109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8136.071111] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8136.071187] Call Trace: [ 8136.071193] [ 8136.071198] ? show_regs.cold.9+0x22/0x2f [ 8136.071210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8136.071218] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8136.071222] ? watchdog+0x40/0x40 [ 8136.071226] ? hrtimer_interrupt+0x128/0x2c0 [ 8136.071230] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8136.071237] ? apic_timer_interrupt+0xf/0x20 [ 8136.071241] [ 8136.071243] ? native_safe_halt+0x2e/0x50 [ 8136.071247] kvm_wait+0xcd/0xe0 [ 8136.071254] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8136.071262] do_raw_spin_lock+0x177/0x190 [ 8136.071266] _raw_spin_lock+0x21/0x30 [ 8136.071271] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8136.071726] process_one_work+0x2c8/0x700 [ 8136.071734] worker_thread+0x296/0x6e0 [ 8136.071738] ? rescuer_thread+0x570/0x570 [ 8136.071742] kthread+0x1d1/0x200 [ 8136.071747] ? set_kthread_struct+0x70/0x70 [ 8136.071750] ret_from_fork+0x1f/0x30 [ 8136.287892] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8136.305472] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8136.317621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8136.325333] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8136.332533] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8136.349187] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8136.353414] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8136.358059] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8136.364046] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8136.371825] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8136.378054] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8136.383498] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8136.388783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8136.393989] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8136.398879] Call Trace: [ 8136.400577] [ 8136.403818] ? show_regs.cold.9+0x22/0x2f [ 8136.409511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8136.414603] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8136.421197] ? watchdog+0x40/0x40 [ 8136.422890] ? hrtimer_interrupt+0x128/0x2c0 [ 8136.426334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8136.428028] ? apic_timer_interrupt+0xf/0x20 [ 8136.434458] [ 8136.436217] ? native_safe_halt+0x2e/0x50 [ 8136.440741] kvm_wait+0xcd/0xe0 [ 8136.442490] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8136.447047] do_raw_spin_lock+0x177/0x190 [ 8136.449820] _raw_spin_lock+0x21/0x30 [ 8136.454761] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8136.461209] ? do_raw_spin_unlock+0x75/0x190 [ 8136.462686] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8136.473731] ? do_wait_intr+0xf0/0xf0 [ 8136.475074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8136.480959] mdc_close+0x53b/0xd70 [mdc] [ 8136.483969] lmv_close+0x18d/0x490 [lmv] [ 8136.489001] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8136.495762] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8136.499904] ll_md_close+0x274/0x800 [lustre] [ 8136.501689] ll_file_release+0x128/0xc50 [lustre] [ 8136.504114] __fput+0x10d/0x440 [ 8136.506140] ____fput+0x12/0x20 [ 8136.509913] task_work_run+0xc6/0x110 [ 8136.512910] exit_to_usermode_loop+0x1dd/0x1f0 [ 8136.517505] do_syscall_64+0x3d6/0x440 [ 8136.520815] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8136.525449] RIP: 0033:0x7fa749552cd8 [ 8136.527905] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8136.550294] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8136.554903] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8136.559568] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8136.566775] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8136.579601] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8136.590376] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8144.057432] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8144.068262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8144.106409] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8144.115425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8144.121345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8144.122946] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8144.141688] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8144.147546] RAX: 000000000000158b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8144.151797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8144.164850] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8144.180868] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8144.189648] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8144.203290] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8144.217518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8144.227294] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8144.238284] Call Trace: [ 8144.241367] [ 8144.253527] ? show_regs.cold.9+0x22/0x2f [ 8144.258018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8144.268547] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8144.279865] ? watchdog+0x40/0x40 [ 8144.282956] ? hrtimer_interrupt+0x128/0x2c0 [ 8144.285832] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8144.288527] ? apic_timer_interrupt+0xf/0x20 [ 8144.290486] [ 8144.292778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8144.296717] do_raw_spin_lock+0x177/0x190 [ 8144.299933] _raw_spin_lock+0x21/0x30 [ 8144.301850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8144.307629] ? do_raw_spin_unlock+0x75/0x190 [ 8144.311389] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8144.316338] ? do_wait_intr+0xf0/0xf0 [ 8144.317687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8144.320115] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8144.322135] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8144.324895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8144.327938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8144.333409] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8144.336339] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8144.340819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8144.344706] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8144.348751] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8144.351458] lmv_intent_open+0x338/0x1150 [lmv] [ 8144.358559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8144.364149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8144.367654] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8144.373865] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8144.377515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8144.382753] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8144.387042] ? do_raw_spin_unlock+0x75/0x190 [ 8144.393449] ? _raw_spin_unlock+0x12/0x30 [ 8144.396499] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8144.400840] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8144.403779] ll_atomic_open+0x36a/0xd70 [lustre] [ 8144.407902] lookup_open+0x356/0xb40 [ 8144.409567] do_last+0x313/0xfc0 [ 8144.411253] ? nd_jump_root+0xe5/0x160 [ 8144.412875] ? path_init+0x437/0x520 [ 8144.414492] path_openat+0xf7/0x500 [ 8144.416831] ? do_raw_spin_unlock+0x75/0x190 [ 8144.419401] do_filp_open+0x99/0x140 [ 8144.421372] ? getname_flags+0x6e/0x330 [ 8144.423363] ? __check_object_size+0xff/0x256 [ 8144.425456] ? do_raw_spin_unlock+0x75/0x190 [ 8144.427481] ? _raw_spin_unlock+0x12/0x30 [ 8144.429374] do_sys_openat2+0x2b4/0x410 [ 8144.431293] do_sys_open+0x73/0xa0 [ 8144.432774] __x64_sys_openat+0x24/0x30 [ 8144.434633] do_syscall_64+0xc1/0x440 [ 8144.436284] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8144.438383] RIP: 0033:0x7fa749552332 [ 8144.439957] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8144.448543] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8144.451877] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8144.454672] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8144.459175] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8144.462310] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8144.466206] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8147.937021] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6230s! [ 8147.940917] Showing busy workqueues and worker pools: [ 8147.943676] workqueue events: flags=0x0 [ 8147.945678] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8147.948772] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8147.959213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8147.963330] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8147.977857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8147.988341] pending: kfree_rcu_monitor [ 8147.992685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8148.002813] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8148.013477] workqueue events_long: flags=0x0 [ 8148.015064] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8148.024087] pending: lu_objects_flush [obdclass] [ 8148.027539] workqueue events_unbound: flags=0x2 [ 8148.029559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8148.033943] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8148.039756] workqueue events_power_efficient: flags=0x80 [ 8148.042412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8148.045574] pending: check_lifetime [ 8148.046943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8148.049676] pending: neigh_periodic_work [ 8148.052830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8148.057332] pending: neigh_periodic_work [ 8148.058040] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8148.059908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8148.059928] pending: do_cache_clean [ 8148.059943] workqueue mm_percpu_wq: flags=0x8 [ 8148.059945] pwq 6: [ 8148.064907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8148.064994] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8148.065003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8148.065006] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8148.065020] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8148.065025] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8148.065030] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8148.065033] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8148.065035] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8148.065038] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8148.065041] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8148.065044] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8148.065047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8148.065049] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8148.065058] Call Trace: [ 8148.065064] [ 8148.065070] ? show_regs.cold.9+0x22/0x2f [ 8148.065079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8148.065086] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8148.065090] ? watchdog+0x40/0x40 [ 8148.065094] ? hrtimer_interrupt+0x128/0x2c0 [ 8148.065098] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8148.065106] ? apic_timer_interrupt+0xf/0x20 [ 8148.065118] [ 8148.065121] ? native_safe_halt+0x2e/0x50 [ 8148.065126] kvm_wait+0xcd/0xe0 [ 8148.065133] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8148.065141] do_raw_spin_lock+0x177/0x190 [ 8148.065145] _raw_spin_lock+0x21/0x30 [ 8148.065151] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8148.065500] obd_mod_free+0x46/0x140 [ptlrpc] [ 8148.065705] mdc_commit_open+0x91/0xb0 [mdc] [ 8148.065755] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8148.065970] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8148.066177] after_reply+0xe2c/0x1780 [ptlrpc] [ 8148.066402] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8148.066704] ? __next_timer_interrupt+0x160/0x160 [ 8148.066716] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8148.067002] ? do_wait_intr+0xf0/0xf0 [ 8148.072518] cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8148.072537] pending: vmstat_update [ 8148.072571] pwq 4: [ 8148.074228] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8148.074515] kthread+0x1d1/0x200 [ 8148.074524] ? set_kthread_struct+0x70/0x70 [ 8148.074529] ret_from_fork+0x1f/0x30 [ 8148.246475] cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8148.253006] pending: vmstat_update [ 8148.254842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8148.263174] pending: vmstat_update [ 8148.266671] workqueue writeback: flags=0x4a [ 8148.270642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8148.274530] pending: wb_workfn, wb_workfn [ 8148.278238] workqueue xprtiod: flags=0xa [ 8148.279531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8148.287933] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8148.291459] workqueue ipv6_addrconf: flags=0x40008 [ 8148.295652] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8148.299797] pending: addrconf_verify_work [ 8148.302667] workqueue knbd0-recv: flags=0x1a [ 8148.304916] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8148.307830] in-flight: 651:recv_work [ 8148.313422] workqueue ptlrpc_pinger: flags=0x2 [ 8148.315335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8148.322191] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8148.324962] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6231s workers=4 idle: 37 31846 28897 [ 8148.329618] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8138s workers=2 idle: 653 [ 8164.057059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8164.060538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8164.065160] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8164.081839] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8164.081852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8164.081858] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8164.082280] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8164.082335] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8164.082340] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8164.082345] RAX: 0000000000000c8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8164.082347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8164.082349] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8164.082351] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8164.082354] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8164.082357] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8164.082360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8164.082362] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8164.082370] Call Trace: [ 8164.082386] [ 8164.082392] ? show_regs.cold.9+0x22/0x2f [ 8164.082402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8164.082409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8164.082414] ? watchdog+0x40/0x40 [ 8164.082419] ? hrtimer_interrupt+0x128/0x2c0 [ 8164.082423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8164.082431] ? apic_timer_interrupt+0xf/0x20 [ 8164.082436] [ 8164.082439] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8164.082444] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8164.082449] do_raw_spin_lock+0x177/0x190 [ 8164.082455] _raw_spin_lock+0x21/0x30 [ 8164.082460] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8164.082846] process_one_work+0x2c8/0x700 [ 8164.082854] worker_thread+0x296/0x6e0 [ 8164.082859] ? rescuer_thread+0x570/0x570 [ 8164.082863] kthread+0x1d1/0x200 [ 8164.082868] ? set_kthread_struct+0x70/0x70 [ 8164.082872] ret_from_fork+0x1f/0x30 [ 8164.176485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8164.195563] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8164.200085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8164.204284] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8164.206551] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8164.212772] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8164.215135] RAX: 0000000000002026 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8164.217487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8164.220176] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8164.223539] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8164.226615] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8164.229199] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8164.231771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8164.234158] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8164.237083] Call Trace: [ 8164.238239] [ 8164.239119] ? show_regs.cold.9+0x22/0x2f [ 8164.240568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8164.242907] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8164.244930] ? watchdog+0x40/0x40 [ 8164.246362] ? hrtimer_interrupt+0x128/0x2c0 [ 8164.248224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8164.250476] ? apic_timer_interrupt+0xf/0x20 [ 8164.252205] [ 8164.253181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8164.255668] do_raw_spin_lock+0x177/0x190 [ 8164.257423] _raw_spin_lock+0x21/0x30 [ 8164.258941] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8164.261277] ? do_raw_spin_unlock+0x75/0x190 [ 8164.263040] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8164.265903] ? do_wait_intr+0xf0/0xf0 [ 8164.268342] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8164.271055] mdc_close+0x53b/0xd70 [mdc] [ 8164.272631] lmv_close+0x18d/0x490 [lmv] [ 8164.274875] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8164.277216] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8164.279645] ll_md_close+0x274/0x800 [lustre] [ 8164.282300] ll_file_release+0x128/0xc50 [lustre] [ 8164.285268] __fput+0x10d/0x440 [ 8164.286374] ____fput+0x12/0x20 [ 8164.287469] task_work_run+0xc6/0x110 [ 8164.288470] exit_to_usermode_loop+0x1dd/0x1f0 [ 8164.290137] do_syscall_64+0x3d6/0x440 [ 8164.292120] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8164.294417] RIP: 0033:0x7fa749552cd8 [ 8164.295987] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8164.310379] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8164.315343] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8164.318525] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8164.321783] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8164.325066] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8164.328167] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8176.054032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8176.056645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8176.058012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8176.096666] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8176.096681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8176.096684] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8176.096699] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8176.096703] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8176.096709] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8176.096711] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8176.096714] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8176.096715] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8176.096718] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8176.096721] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8176.096724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8176.096726] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8176.096735] Call Trace: [ 8176.096749] [ 8176.096755] ? show_regs.cold.9+0x22/0x2f [ 8176.096764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8176.096771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8176.096776] ? watchdog+0x40/0x40 [ 8176.096780] ? hrtimer_interrupt+0x128/0x2c0 [ 8176.096785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8176.096792] ? apic_timer_interrupt+0xf/0x20 [ 8176.096797] [ 8176.096799] ? native_safe_halt+0x2e/0x50 [ 8176.096803] kvm_wait+0xcd/0xe0 [ 8176.096811] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8176.096821] do_raw_spin_lock+0x177/0x190 [ 8176.096825] _raw_spin_lock+0x21/0x30 [ 8176.096830] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8176.097104] ? do_raw_spin_unlock+0x75/0x190 [ 8176.097109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8176.097296] ? do_wait_intr+0xf0/0xf0 [ 8176.097305] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8176.097514] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8176.097726] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8176.098053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.098161] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8176.098199] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8176.098225] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8176.098250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.098331] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8176.098627] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8176.098657] lmv_intent_open+0x338/0x1150 [lmv] [ 8176.098685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.098780] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.098863] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8176.098886] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8176.098970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.099099] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8176.099176] ? do_raw_spin_unlock+0x75/0x190 [ 8176.099182] ? _raw_spin_unlock+0x12/0x30 [ 8176.099187] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8176.099215] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8176.099235] ll_atomic_open+0x36a/0xd70 [lustre] [ 8176.099319] lookup_open+0x356/0xb40 [ 8176.099330] do_last+0x313/0xfc0 [ 8176.099335] ? nd_jump_root+0xe5/0x160 [ 8176.099340] ? path_init+0x437/0x520 [ 8176.099345] path_openat+0xf7/0x500 [ 8176.099351] ? do_raw_spin_unlock+0x75/0x190 [ 8176.099355] do_filp_open+0x99/0x140 [ 8176.099362] ? getname_flags+0x6e/0x330 [ 8176.099366] ? __check_object_size+0xff/0x256 [ 8176.099373] ? do_raw_spin_unlock+0x75/0x190 [ 8176.099376] ? _raw_spin_unlock+0x12/0x30 [ 8176.099380] do_sys_openat2+0x2b4/0x410 [ 8176.099384] do_sys_open+0x73/0xa0 [ 8176.099388] __x64_sys_openat+0x24/0x30 [ 8176.099393] do_syscall_64+0xc1/0x440 [ 8176.099399] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8176.099406] RIP: 0033:0x7fa749552332 [ 8176.099446] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8176.099450] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8176.102677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8176.102763] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8176.102770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8176.102772] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8176.102784] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8176.102788] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8176.102793] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8176.102795] RDX: 0000000000003f94 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8176.102797] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8176.102799] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8176.102802] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8176.102805] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8176.102808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8176.102810] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8176.102818] Call Trace: [ 8176.102823] [ 8176.102829] ? show_regs.cold.9+0x22/0x2f [ 8176.102838] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8176.102845] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8176.102849] ? watchdog+0x40/0x40 [ 8176.102853] ? hrtimer_interrupt+0x128/0x2c0 [ 8176.102856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8176.102864] ? apic_timer_interrupt+0xf/0x20 [ 8176.102868] [ 8176.102871] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8176.102875] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8176.102879] do_raw_spin_lock+0x177/0x190 [ 8176.102883] _raw_spin_lock+0x21/0x30 [ 8176.102889] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8176.103312] obd_mod_free+0x46/0x140 [ptlrpc] [ 8176.107326] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8176.107334] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8176.107336] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8176.107344] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8176.107347] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8176.357449] mdc_commit_open+0x91/0xb0 [mdc] [ 8176.359201] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8176.361408] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8176.367036] after_reply+0xe2c/0x1780 [ptlrpc] [ 8176.369115] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8176.372220] ? __next_timer_interrupt+0x160/0x160 [ 8176.374411] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8176.376309] ? do_wait_intr+0xf0/0xf0 [ 8176.377809] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8176.380126] kthread+0x1d1/0x200 [ 8176.382036] ? set_kthread_struct+0x70/0x70 [ 8176.383362] ret_from_fork+0x1f/0x30 [ 8178.664307] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6261s! [ 8178.672693] Showing busy workqueues and worker pools: [ 8178.674479] workqueue events: flags=0x0 [ 8178.683165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8178.686774] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8178.693740] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8178.697765] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8178.701959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.705094] pending: kfree_rcu_monitor [ 8178.708240] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8178.710737] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8178.717336] workqueue events_long: flags=0x0 [ 8178.720904] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.726283] pending: lu_objects_flush [obdclass] [ 8178.728809] workqueue events_unbound: flags=0x2 [ 8178.731309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8178.733943] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8178.737730] workqueue events_power_efficient: flags=0x80 [ 8178.740357] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.744064] pending: check_lifetime [ 8178.746198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.750567] pending: neigh_periodic_work [ 8178.752931] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.756500] pending: neigh_periodic_work [ 8178.758851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.761692] pending: do_cache_clean [ 8178.764095] workqueue mm_percpu_wq: flags=0x8 [ 8178.765841] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.768793] pending: vmstat_update [ 8178.769800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.771793] pending: vmstat_update [ 8178.773255] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8178.775691] pending: vmstat_update [ 8178.778372] workqueue writeback: flags=0x4a [ 8178.779957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8178.787618] pending: wb_workfn, wb_workfn [ 8178.790125] workqueue xprtiod: flags=0xa [ 8178.794400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8178.797888] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8178.802464] workqueue ipv6_addrconf: flags=0x40008 [ 8178.805292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8178.808893] pending: addrconf_verify_work [ 8178.810286] workqueue knbd0-recv: flags=0x1a [ 8178.811430] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8178.813635] in-flight: 651:recv_work [ 8178.816952] workqueue ptlrpc_pinger: flags=0x2 [ 8178.818978] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8178.824295] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8178.827353] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6261s workers=4 idle: 37 31846 28897 [ 8178.830942] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8168s workers=2 idle: 653 [ 8192.057032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8192.060620] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8192.061995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8192.062067] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8192.062074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8192.062080] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8192.062354] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8192.062364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8192.062368] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8192.062372] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8192.062375] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8192.062377] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8192.062379] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8192.062381] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8192.062385] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8192.062387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8192.062390] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8192.062398] Call Trace: [ 8192.062404] [ 8192.062410] ? show_regs.cold.9+0x22/0x2f [ 8192.062420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8192.062426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8192.062432] ? watchdog+0x40/0x40 [ 8192.062437] ? hrtimer_interrupt+0x128/0x2c0 [ 8192.062441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8192.062448] ? apic_timer_interrupt+0xf/0x20 [ 8192.062452] [ 8192.062455] ? native_safe_halt+0x2e/0x50 [ 8192.062459] kvm_wait+0xcd/0xe0 [ 8192.062482] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8192.062490] do_raw_spin_lock+0x177/0x190 [ 8192.062495] _raw_spin_lock+0x21/0x30 [ 8192.062499] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8192.062697] process_one_work+0x2c8/0x700 [ 8192.062704] worker_thread+0x296/0x6e0 [ 8192.062708] ? rescuer_thread+0x570/0x570 [ 8192.062711] kthread+0x1d1/0x200 [ 8192.062716] ? set_kthread_struct+0x70/0x70 [ 8192.062720] ret_from_fork+0x1f/0x30 [ 8192.234834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8192.272793] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8192.289115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8192.300949] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8192.305439] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8192.317954] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8192.322802] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8192.331273] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8192.341529] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8192.345739] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8192.352444] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8192.358828] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8192.369474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8192.377365] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8192.381366] Call Trace: [ 8192.382673] [ 8192.385613] ? show_regs.cold.9+0x22/0x2f [ 8192.390312] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8192.398262] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8192.406752] ? watchdog+0x40/0x40 [ 8192.408116] ? hrtimer_interrupt+0x128/0x2c0 [ 8192.413154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8192.415594] ? apic_timer_interrupt+0xf/0x20 [ 8192.418312] [ 8192.419224] ? native_safe_halt+0x2e/0x50 [ 8192.420942] kvm_wait+0xcd/0xe0 [ 8192.422166] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8192.424379] do_raw_spin_lock+0x177/0x190 [ 8192.425954] _raw_spin_lock+0x21/0x30 [ 8192.427553] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8192.432040] ? do_raw_spin_unlock+0x75/0x190 [ 8192.435113] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8192.438773] ? do_wait_intr+0xf0/0xf0 [ 8192.442087] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8192.447334] mdc_close+0x53b/0xd70 [mdc] [ 8192.450423] lmv_close+0x18d/0x490 [lmv] [ 8192.454084] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8192.456476] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8192.461526] ll_md_close+0x274/0x800 [lustre] [ 8192.466323] ll_file_release+0x128/0xc50 [lustre] [ 8192.469930] __fput+0x10d/0x440 [ 8192.472041] ____fput+0x12/0x20 [ 8192.473062] task_work_run+0xc6/0x110 [ 8192.474826] exit_to_usermode_loop+0x1dd/0x1f0 [ 8192.476311] do_syscall_64+0x3d6/0x440 [ 8192.477933] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8192.479601] RIP: 0033:0x7fa749552cd8 [ 8192.488919] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8192.502470] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8192.506191] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8192.508417] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8192.511913] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8192.514628] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8192.519425] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8204.054434] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8204.057808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8204.061108] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8204.078555] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8204.078569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8204.078572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8204.078586] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8204.078590] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8204.078596] RAX: 0000000000005314 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8204.078599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8204.078601] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8204.078603] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8204.078605] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8204.078608] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8204.078611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8204.078613] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8204.078622] Call Trace: [ 8204.078627] [ 8204.078633] ? show_regs.cold.9+0x22/0x2f [ 8204.078642] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8204.078649] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8204.078654] ? watchdog+0x40/0x40 [ 8204.078658] ? hrtimer_interrupt+0x128/0x2c0 [ 8204.078662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8204.078669] ? apic_timer_interrupt+0xf/0x20 [ 8204.078674] [ 8204.078677] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8204.078684] do_raw_spin_lock+0x177/0x190 [ 8204.078689] _raw_spin_lock+0x21/0x30 [ 8204.078694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8204.079019] ? do_raw_spin_unlock+0x75/0x190 [ 8204.079026] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8204.079248] ? do_wait_intr+0xf0/0xf0 [ 8204.079268] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8204.079549] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8204.079783] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8204.080088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.080246] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8204.080298] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8204.091565] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 8204.096526] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8204.096585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.096694] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8204.104889] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8204.105291] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8204.108194] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8204.108247] lmv_intent_open+0x338/0x1150 [lmv] [ 8204.108293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.108412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.108502] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8204.108527] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8204.108614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.108697] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8204.108777] ? do_raw_spin_unlock+0x75/0x190 [ 8204.108785] ? _raw_spin_unlock+0x12/0x30 [ 8204.108790] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8204.108817] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8204.108837] ll_atomic_open+0x36a/0xd70 [lustre] [ 8204.108919] lookup_open+0x356/0xb40 [ 8204.108928] do_last+0x313/0xfc0 [ 8204.108932] ? nd_jump_root+0xe5/0x160 [ 8204.108937] ? path_init+0x437/0x520 [ 8204.108941] path_openat+0xf7/0x500 [ 8204.108947] ? do_raw_spin_unlock+0x75/0x190 [ 8204.108952] do_filp_open+0x99/0x140 [ 8204.108958] ? getname_flags+0x6e/0x330 [ 8204.109320] ? __check_object_size+0xff/0x256 [ 8204.109328] ? do_raw_spin_unlock+0x75/0x190 [ 8204.109332] ? _raw_spin_unlock+0x12/0x30 [ 8204.109338] do_sys_openat2+0x2b4/0x410 [ 8204.109343] do_sys_open+0x73/0xa0 [ 8204.109348] __x64_sys_openat+0x24/0x30 [ 8204.109352] do_syscall_64+0xc1/0x440 [ 8204.109358] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8204.109365] RIP: 0033:0x7fa749552332 [ 8204.109373] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8204.109376] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8204.109382] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8204.109385] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8204.109387] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8204.109389] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8204.109392] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8204.562087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8204.566957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8204.571439] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8204.586039] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8204.590731] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8204.596919] RDX: 0000000000007c82 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8204.599403] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8204.610931] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8204.616456] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8204.622512] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8204.634735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8204.645647] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8204.650159] Call Trace: [ 8204.651571] [ 8204.653634] ? show_regs.cold.9+0x22/0x2f [ 8204.658573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8204.662095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8204.666205] ? watchdog+0x40/0x40 [ 8204.668752] ? hrtimer_interrupt+0x128/0x2c0 [ 8204.671747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8204.672874] ? apic_timer_interrupt+0xf/0x20 [ 8204.673921] [ 8204.679307] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8204.684459] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8204.690193] do_raw_spin_lock+0x177/0x190 [ 8204.693693] _raw_spin_lock+0x21/0x30 [ 8204.694872] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8204.697775] obd_mod_free+0x46/0x140 [ptlrpc] [ 8204.700435] mdc_commit_open+0x91/0xb0 [mdc] [ 8204.704317] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8204.708530] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8204.714291] after_reply+0xe2c/0x1780 [ptlrpc] [ 8204.717654] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8204.724271] ? __next_timer_interrupt+0x160/0x160 [ 8204.729669] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8204.732044] ? do_wait_intr+0xf0/0xf0 [ 8204.733227] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8204.736922] kthread+0x1d1/0x200 [ 8204.738538] ? set_kthread_struct+0x70/0x70 [ 8204.745200] ret_from_fork+0x1f/0x30 [ 8208.866708] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6291s! [ 8208.875821] Showing busy workqueues and worker pools: [ 8208.880304] workqueue events: flags=0x0 [ 8208.886339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8208.888811] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8208.907834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8208.916955] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8208.923850] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8208.929446] pending: kfree_rcu_monitor [ 8208.930931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8208.934682] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8208.940607] workqueue events_long: flags=0x0 [ 8208.942343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8208.945379] pending: lu_objects_flush [obdclass] [ 8208.948188] workqueue events_unbound: flags=0x2 [ 8208.950275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8208.956201] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8208.961049] workqueue events_power_efficient: flags=0x80 [ 8208.965281] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8208.968554] pending: check_lifetime [ 8208.970576] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8208.973478] pending: neigh_periodic_work [ 8208.975800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8208.981292] pending: neigh_periodic_work [ 8208.983392] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8208.987752] pending: do_cache_clean [ 8208.988686] workqueue mm_percpu_wq: flags=0x8 [ 8208.992531] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8208.995359] pending: vmstat_update [ 8208.996696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8209.005924] pending: vmstat_update [ 8209.008787] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8209.014631] pending: vmstat_update [ 8209.015984] workqueue writeback: flags=0x4a [ 8209.022763] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8209.027027] pending: wb_workfn, wb_workfn [ 8209.028098] workqueue xprtiod: flags=0xa [ 8209.036123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8209.040555] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8209.047136] workqueue ipv6_addrconf: flags=0x40008 [ 8209.050302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8209.058442] pending: addrconf_verify_work [ 8209.061085] workqueue knbd0-recv: flags=0x1a [ 8209.063931] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8209.066132] in-flight: 651:recv_work [ 8209.070458] workqueue ptlrpc_pinger: flags=0x2 [ 8209.072059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8209.075569] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8209.079313] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6291s workers=4 idle: 37 31846 28897 [ 8209.083011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8198s workers=2 idle: 653 [ 8220.057059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8220.059818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8220.063714] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8220.087248] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8220.087262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8220.087269] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8220.092141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8220.092222] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8220.092230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8220.092233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8220.092247] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8220.092252] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8220.092257] RAX: 0000000000000418 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8220.092259] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8220.092262] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8220.092265] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8220.092268] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8220.092272] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8220.092275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8220.092278] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8220.092287] Call Trace: [ 8220.092292] [ 8220.092298] ? show_regs.cold.9+0x22/0x2f [ 8220.092308] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8220.092314] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8220.092318] ? watchdog+0x40/0x40 [ 8220.092323] ? hrtimer_interrupt+0x128/0x2c0 [ 8220.092328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8220.092336] ? apic_timer_interrupt+0xf/0x20 [ 8220.092341] [ 8220.092344] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8220.092352] do_raw_spin_lock+0x177/0x190 [ 8220.092357] _raw_spin_lock+0x21/0x30 [ 8220.092362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8220.093282] ? do_raw_spin_unlock+0x75/0x190 [ 8220.093290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8220.095299] ? do_wait_intr+0xf0/0xf0 [ 8220.095312] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8220.109861] [ 8220.114272] mdc_close+0x53b/0xd70 [mdc] [ 8220.114321] lmv_close+0x18d/0x490 [lmv] [ 8220.114354] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8220.114481] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8220.117541] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8220.117562] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8220.117566] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8220.117572] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8220.117575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8220.117577] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8220.117580] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8220.117582] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8220.117586] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8220.117589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8220.117591] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8220.117599] Call Trace: [ 8220.117604] [ 8220.117609] ? show_regs.cold.9+0x22/0x2f [ 8220.117618] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8220.117625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8220.117629] ? watchdog+0x40/0x40 [ 8220.117634] ? hrtimer_interrupt+0x128/0x2c0 [ 8220.117638] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8220.117645] ? apic_timer_interrupt+0xf/0x20 [ 8220.140503] ll_md_close+0x274/0x800 [lustre] [ 8220.147837] [ 8220.147856] ? native_safe_halt+0x2e/0x50 [ 8220.147870] kvm_wait+0xcd/0xe0 [ 8220.147879] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8220.147890] do_raw_spin_lock+0x177/0x190 [ 8220.147895] _raw_spin_lock+0x21/0x30 [ 8220.147899] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8220.148291] process_one_work+0x2c8/0x700 [ 8220.148299] worker_thread+0x296/0x6e0 [ 8220.148304] ? rescuer_thread+0x570/0x570 [ 8220.148308] kthread+0x1d1/0x200 [ 8220.148313] ? set_kthread_struct+0x70/0x70 [ 8220.148317] ret_from_fork+0x1f/0x30 [ 8220.313603] ll_file_release+0x128/0xc50 [lustre] [ 8220.315956] __fput+0x10d/0x440 [ 8220.317612] ____fput+0x12/0x20 [ 8220.319144] task_work_run+0xc6/0x110 [ 8220.321412] exit_to_usermode_loop+0x1dd/0x1f0 [ 8220.323326] do_syscall_64+0x3d6/0x440 [ 8220.324995] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8220.327639] RIP: 0033:0x7fa749552cd8 [ 8220.328935] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8220.344236] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8220.351261] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8220.354306] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8220.357375] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8220.360291] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8220.363057] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8232.058012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8232.061372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8232.089215] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8232.099080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8232.105402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8232.114306] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8232.133695] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8232.141006] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8232.146160] RDX: 000000000000167d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8232.148517] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8232.151538] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8232.155661] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8232.158668] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8232.161817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8232.168337] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8232.170942] Call Trace: [ 8232.171772] [ 8232.172407] ? show_regs.cold.9+0x22/0x2f [ 8232.175336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8232.178021] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8232.179977] ? watchdog+0x40/0x40 [ 8232.181689] ? hrtimer_interrupt+0x128/0x2c0 [ 8232.185459] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8232.188425] ? apic_timer_interrupt+0xf/0x20 [ 8232.190949] [ 8232.191846] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8232.194622] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8232.197751] do_raw_spin_lock+0x177/0x190 [ 8232.199842] _raw_spin_lock+0x21/0x30 [ 8232.201883] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8232.205016] obd_mod_free+0x46/0x140 [ptlrpc] [ 8232.207705] mdc_commit_open+0x91/0xb0 [mdc] [ 8232.209989] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8232.212720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8232.216061] after_reply+0xe2c/0x1780 [ptlrpc] [ 8232.218891] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8232.222834] ? __next_timer_interrupt+0x160/0x160 [ 8232.225172] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8232.228287] ? do_wait_intr+0xf0/0xf0 [ 8232.230851] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8232.233844] kthread+0x1d1/0x200 [ 8232.235973] ? set_kthread_struct+0x70/0x70 [ 8232.237748] ret_from_fork+0x1f/0x30 [ 8236.054023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8236.059669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8236.112967] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8236.132170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8236.143835] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8236.146639] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8236.158784] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8236.162079] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8236.165275] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8236.173156] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8236.178054] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8236.181814] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8236.192019] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8236.202046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8236.209466] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8236.213528] Call Trace: [ 8236.215294] [ 8236.215955] ? show_regs.cold.9+0x22/0x2f [ 8236.217744] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8236.221329] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8236.224242] ? watchdog+0x40/0x40 [ 8236.227374] ? hrtimer_interrupt+0x128/0x2c0 [ 8236.229775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8236.232270] ? apic_timer_interrupt+0xf/0x20 [ 8236.234926] [ 8236.236350] ? native_safe_halt+0x2e/0x50 [ 8236.239785] kvm_wait+0xcd/0xe0 [ 8236.243795] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8236.252227] do_raw_spin_lock+0x177/0x190 [ 8236.256942] _raw_spin_lock+0x21/0x30 [ 8236.258576] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8236.266348] ? do_raw_spin_unlock+0x75/0x190 [ 8236.269746] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8236.273935] ? do_wait_intr+0xf0/0xf0 [ 8236.278903] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8236.283095] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8236.291026] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8236.293898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8236.298046] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8236.302283] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8236.311395] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8236.315581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8236.320812] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8236.327611] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8236.331309] lmv_intent_open+0x338/0x1150 [lmv] [ 8236.335770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8236.339251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8236.342625] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8236.345296] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8236.350156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8236.356077] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8236.361107] ? do_raw_spin_unlock+0x75/0x190 [ 8236.365995] ? _raw_spin_unlock+0x12/0x30 [ 8236.368322] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8236.376063] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8236.382194] ll_atomic_open+0x36a/0xd70 [lustre] [ 8236.387378] lookup_open+0x356/0xb40 [ 8236.394244] do_last+0x313/0xfc0 [ 8236.395881] ? nd_jump_root+0xe5/0x160 [ 8236.402408] ? path_init+0x437/0x520 [ 8236.405274] path_openat+0xf7/0x500 [ 8236.409295] ? do_raw_spin_unlock+0x75/0x190 [ 8236.413300] do_filp_open+0x99/0x140 [ 8236.417000] ? getname_flags+0x6e/0x330 [ 8236.429994] ? __check_object_size+0xff/0x256 [ 8236.433335] ? do_raw_spin_unlock+0x75/0x190 [ 8236.437675] ? _raw_spin_unlock+0x12/0x30 [ 8236.439114] do_sys_openat2+0x2b4/0x410 [ 8236.440565] do_sys_open+0x73/0xa0 [ 8236.449362] __x64_sys_openat+0x24/0x30 [ 8236.459489] do_syscall_64+0xc1/0x440 [ 8236.469082] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8236.472839] RIP: 0033:0x7fa749552332 [ 8236.477983] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8236.499204] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8236.508980] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8236.516807] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8236.530079] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8236.537176] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8236.542954] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8239.586218] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6322s! [ 8239.600943] Showing busy workqueues and worker pools: [ 8239.609276] workqueue events: flags=0x0 [ 8239.617871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8239.624721] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8239.645941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8239.657719] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8239.665360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.673022] pending: kfree_rcu_monitor [ 8239.676685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8239.682035] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8239.693268] workqueue events_long: flags=0x0 [ 8239.695783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.699220] pending: lu_objects_flush [obdclass] [ 8239.703477] workqueue events_unbound: flags=0x2 [ 8239.704893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8239.708676] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8239.714724] workqueue events_power_efficient: flags=0x80 [ 8239.716903] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.719870] pending: check_lifetime [ 8239.723692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.727680] pending: neigh_periodic_work [ 8239.729472] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.732678] pending: neigh_periodic_work [ 8239.734047] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.737426] pending: do_cache_clean [ 8239.739222] workqueue mm_percpu_wq: flags=0x8 [ 8239.741247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.744018] pending: vmstat_update [ 8239.746160] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.749801] pending: vmstat_update [ 8239.751064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8239.754218] pending: vmstat_update [ 8239.755930] workqueue writeback: flags=0x4a [ 8239.758182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8239.761811] pending: wb_workfn, wb_workfn [ 8239.764056] workqueue xprtiod: flags=0xa [ 8239.766682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8239.772131] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8239.779802] workqueue ipv6_addrconf: flags=0x40008 [ 8239.783903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8239.791276] pending: addrconf_verify_work [ 8239.798893] workqueue knbd0-recv: flags=0x1a [ 8239.806715] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8239.815774] in-flight: 651:recv_work [ 8239.820916] workqueue ptlrpc_pinger: flags=0x2 [ 8239.825584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8239.833274] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8239.841800] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6322s workers=4 idle: 37 31846 28897 [ 8239.850994] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8229s workers=2 idle: 653 [ 8248.057021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8248.059508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8248.062442] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8248.096082] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8248.096096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8248.096102] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8248.100123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata [ 8248.113758] [ 8248.113769] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8248.113787] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8248.113792] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8248.113797] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8248.113800] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8248.113802] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8248.113805] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8248.113807] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8248.113810] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8248.113813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8248.113816] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8248.113824] Call Trace: [ 8248.113829] [ 8248.113836] ? show_regs.cold.9+0x22/0x2f [ 8248.113846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8248.113852] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8248.113857] ? watchdog+0x40/0x40 [ 8248.113862] ? hrtimer_interrupt+0x128/0x2c0 [ 8248.113866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8248.113874] ? apic_timer_interrupt+0xf/0x20 [ 8248.113878] [ 8248.113882] ? native_safe_halt+0x2e/0x50 [ 8248.113886] kvm_wait+0xcd/0xe0 [ 8248.113892] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8248.113903] do_raw_spin_lock+0x177/0x190 [ 8248.113907] _raw_spin_lock+0x21/0x30 [ 8248.113912] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8248.114703] process_one_work+0x2c8/0x700 [ 8248.114711] worker_thread+0x296/0x6e0 [ 8248.114716] ? rescuer_thread+0x570/0x570 [ 8248.114719] kthread+0x1d1/0x200 [ 8248.114723] ? set_kthread_struct+0x70/0x70 [ 8248.114727] ret_from_fork+0x1f/0x30 [ 8248.283786] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8248.291498] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8248.296716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8248.307555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8248.313938] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8248.332377] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8248.342470] RAX: 0000000000007f8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8248.350052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8248.356829] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8248.370554] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8248.382972] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8248.385222] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8248.403463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8248.410591] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8248.418997] Call Trace: [ 8248.423798] [ 8248.424557] ? show_regs.cold.9+0x22/0x2f [ 8248.426983] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8248.428884] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8248.435201] ? watchdog+0x40/0x40 [ 8248.438920] ? hrtimer_interrupt+0x128/0x2c0 [ 8248.448017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8248.454701] ? apic_timer_interrupt+0xf/0x20 [ 8248.456402] [ 8248.465511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8248.472487] do_raw_spin_lock+0x177/0x190 [ 8248.477006] _raw_spin_lock+0x21/0x30 [ 8248.479112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8248.486402] ? do_raw_spin_unlock+0x75/0x190 [ 8248.489493] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8248.497859] ? do_wait_intr+0xf0/0xf0 [ 8248.503335] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8248.506907] mdc_close+0x53b/0xd70 [mdc] [ 8248.508447] lmv_close+0x18d/0x490 [lmv] [ 8248.511644] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8248.515516] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8248.521718] ll_md_close+0x274/0x800 [lustre] [ 8248.527538] ll_file_release+0x128/0xc50 [lustre] [ 8248.531909] __fput+0x10d/0x440 [ 8248.533158] ____fput+0x12/0x20 [ 8248.534326] task_work_run+0xc6/0x110 [ 8248.535388] exit_to_usermode_loop+0x1dd/0x1f0 [ 8248.538804] do_syscall_64+0x3d6/0x440 [ 8248.540907] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8248.545975] RIP: 0033:0x7fa749552cd8 [ 8248.549873] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8248.564856] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8248.568802] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8248.574087] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8248.578334] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8248.580367] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8248.584833] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8260.058201] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8260.070804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8260.105000] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8260.119371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8260.123472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8260.133407] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8260.156760] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8260.167255] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8260.178547] RDX: 0000000000002aa0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8260.186622] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8260.194209] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8260.196565] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8260.201744] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8260.215880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8260.221491] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8260.232161] Call Trace: [ 8260.233767] [ 8260.235626] ? show_regs.cold.9+0x22/0x2f [ 8260.242166] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8260.250410] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8260.252055] ? watchdog+0x40/0x40 [ 8260.259436] ? hrtimer_interrupt+0x128/0x2c0 [ 8260.263539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8260.269701] ? apic_timer_interrupt+0xf/0x20 [ 8260.274907] [ 8260.277115] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8260.283299] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8260.288222] do_raw_spin_lock+0x177/0x190 [ 8260.292198] _raw_spin_lock+0x21/0x30 [ 8260.294784] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8260.298438] obd_mod_free+0x46/0x140 [ptlrpc] [ 8260.304454] mdc_commit_open+0x91/0xb0 [mdc] [ 8260.306404] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8260.313402] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8260.320544] after_reply+0xe2c/0x1780 [ptlrpc] [ 8260.324344] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8260.334191] ? __next_timer_interrupt+0x160/0x160 [ 8260.343613] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8260.346820] ? do_wait_intr+0xf0/0xf0 [ 8260.348422] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8260.350575] kthread+0x1d1/0x200 [ 8260.352461] ? set_kthread_struct+0x70/0x70 [ 8260.355731] ret_from_fork+0x1f/0x30 [ 8264.054128] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8264.057405] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8264.102543] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8264.109063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8264.116110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8264.122656] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8264.143134] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8264.153141] RAX: 0000000000004e81 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8264.161159] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8264.167210] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8264.176325] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8264.183396] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8264.187401] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8264.194410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8264.198361] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8264.201468] Call Trace: [ 8264.205727] [ 8264.206727] ? show_regs.cold.9+0x22/0x2f [ 8264.209353] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8264.211976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8264.214495] ? watchdog+0x40/0x40 [ 8264.218613] ? hrtimer_interrupt+0x128/0x2c0 [ 8264.221479] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8264.229430] ? apic_timer_interrupt+0xf/0x20 [ 8264.240292] [ 8264.243433] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8264.250074] do_raw_spin_lock+0x177/0x190 [ 8264.253672] _raw_spin_lock+0x21/0x30 [ 8264.258323] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8264.261515] ? do_raw_spin_unlock+0x75/0x190 [ 8264.264203] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8264.267266] ? do_wait_intr+0xf0/0xf0 [ 8264.269805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8264.272590] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8264.277253] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8264.285735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8264.295370] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8264.301669] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8264.309518] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8264.311564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8264.316715] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8264.318916] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8264.322413] lmv_intent_open+0x338/0x1150 [lmv] [ 8264.324974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8264.328577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8264.334101] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8264.335631] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8264.346215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8264.355278] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8264.359461] ? do_raw_spin_unlock+0x75/0x190 [ 8264.369517] ? _raw_spin_unlock+0x12/0x30 [ 8264.374326] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8264.380618] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8264.385947] ll_atomic_open+0x36a/0xd70 [lustre] [ 8264.387673] lookup_open+0x356/0xb40 [ 8264.392279] do_last+0x313/0xfc0 [ 8264.394494] ? nd_jump_root+0xe5/0x160 [ 8264.398142] ? path_init+0x437/0x520 [ 8264.400959] path_openat+0xf7/0x500 [ 8264.403691] ? do_raw_spin_unlock+0x75/0x190 [ 8264.405418] do_filp_open+0x99/0x140 [ 8264.407417] ? getname_flags+0x6e/0x330 [ 8264.413121] ? __check_object_size+0xff/0x256 [ 8264.415685] ? do_raw_spin_unlock+0x75/0x190 [ 8264.419617] ? _raw_spin_unlock+0x12/0x30 [ 8264.421741] do_sys_openat2+0x2b4/0x410 [ 8264.423258] do_sys_open+0x73/0xa0 [ 8264.425740] __x64_sys_openat+0x24/0x30 [ 8264.427318] do_syscall_64+0xc1/0x440 [ 8264.429116] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8264.432493] RIP: 0033:0x7fa749552332 [ 8264.441883] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8264.464317] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8264.467099] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8264.472704] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8264.476229] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8264.482287] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8264.485620] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8270.305128] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6353s! [ 8270.311526] Showing busy workqueues and worker pools: [ 8270.317288] workqueue events: flags=0x0 [ 8270.318655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8270.324349] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8270.334714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8270.337173] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8270.341333] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.344052] pending: kfree_rcu_monitor [ 8270.346000] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8270.351222] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8270.360339] workqueue events_long: flags=0x0 [ 8270.362544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.368267] pending: lu_objects_flush [obdclass] [ 8270.375104] workqueue events_unbound: flags=0x2 [ 8270.381762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8270.387446] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8270.401202] workqueue events_power_efficient: flags=0x80 [ 8270.405147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.410707] pending: check_lifetime [ 8270.414785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.419202] pending: neigh_periodic_work [ 8270.422767] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.426970] pending: neigh_periodic_work [ 8270.429455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.436712] pending: do_cache_clean [ 8270.441678] workqueue mm_percpu_wq: flags=0x8 [ 8270.445846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.448806] pending: vmstat_update [ 8270.449863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.452641] pending: vmstat_update [ 8270.454941] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8270.457557] pending: vmstat_update [ 8270.459040] workqueue writeback: flags=0x4a [ 8270.461334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8270.464315] pending: wb_workfn, wb_workfn [ 8270.466522] workqueue xprtiod: flags=0xa [ 8270.470606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8270.474434] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8270.478239] workqueue ipv6_addrconf: flags=0x40008 [ 8270.480868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8270.484266] pending: addrconf_verify_work [ 8270.486485] workqueue knbd0-recv: flags=0x1a [ 8270.488520] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8270.492352] in-flight: 651:recv_work [ 8270.494032] workqueue ptlrpc_pinger: flags=0x2 [ 8270.495636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8270.503624] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8270.509924] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6353s workers=4 idle: 37 31846 28897 [ 8270.514062] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8260s workers=2 idle: 653 [ 8273.828070] rcu: INFO: rcu_sched self-detected stall on CPU [ 8273.831066] rcu: 2-...!: (3560123 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 8273.836680] (t=6360139 jiffies g=258701 q=3276) [ 8273.839140] rcu: rcu_sched kthread starved for 6356934 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8273.847640] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8273.850644] rcu: RCU grace-period kthread stack dump: [ 8273.859420] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8273.868411] Call Trace: [ 8273.870450] __schedule+0x351/0xcb0 [ 8273.872211] schedule+0xc0/0x180 [ 8273.873367] schedule_timeout+0xb4/0x190 [ 8273.874653] ? do_raw_spin_unlock+0x75/0x190 [ 8273.876185] ? __next_timer_interrupt+0x160/0x160 [ 8273.877799] ? prepare_to_swait_event+0xf3/0x140 [ 8273.879348] rcu_gp_kthread+0x8dd/0x15b0 [ 8273.880799] ? force_qs_rnp+0x2c0/0x2c0 [ 8273.882256] kthread+0x1d1/0x200 [ 8273.883623] ? set_kthread_struct+0x70/0x70 [ 8273.885071] ret_from_fork+0x1f/0x30 [ 8273.886237] rcu: Stack dump where RCU GP kthread last ran: [ 8273.888128] Sending NMI from CPU 2 to CPUs 0: [ 8273.889602] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 8273.892637] NMI backtrace for cpu 2 [ 8273.894277] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8273.898873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8273.902473] Call Trace: [ 8273.903475] [ 8273.904305] ? dump_stack+0xbb/0x10e [ 8273.905715] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8273.907848] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8273.909794] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8273.911874] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8273.914181] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8273.916273] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8273.918148] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8273.920160] ? update_process_times+0x6d/0xc0 [ 8273.921921] ? tick_sched_handle.isra.18+0x27/0x90 [ 8273.923893] ? tick_sched_timer+0x53/0xb0 [ 8273.925545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8273.927443] ? tick_sched_handle.isra.18+0x90/0x90 [ 8273.929640] ? hrtimer_interrupt+0x128/0x2c0 [ 8273.931435] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8273.933609] ? apic_timer_interrupt+0xf/0x20 [ 8273.935424] [ 8273.936247] ? native_safe_halt+0x2e/0x50 [ 8273.937888] ? kvm_wait+0xcd/0xe0 [ 8273.939244] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8273.941412] ? do_raw_spin_lock+0x177/0x190 [ 8273.943057] ? _raw_spin_lock+0x21/0x30 [ 8273.944604] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8273.947058] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8273.949472] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8273.951391] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8273.954558] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8273.957089] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8273.959240] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8273.961889] ? __next_timer_interrupt+0x160/0x160 [ 8273.963822] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8273.965894] ? do_wait_intr+0xf0/0xf0 [ 8273.967407] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8273.969879] ? kthread+0x1d1/0x200 [ 8273.971261] ? set_kthread_struct+0x70/0x70 [ 8273.972906] ? ret_from_fork+0x1f/0x30 [ 8276.057091] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8276.059065] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8276.061404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8276.061476] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8276.061482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8276.061488] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8276.062131] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8276.062142] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8276.062145] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8276.062150] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8276.062152] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8276.062155] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8276.062157] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8276.062159] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8276.062162] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8276.062165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8276.062167] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8276.062175] Call Trace: [ 8276.062181] [ 8276.062186] ? show_regs.cold.9+0x22/0x2f [ 8276.062196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8276.062330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8276.062335] ? watchdog+0x40/0x40 [ 8276.062340] ? hrtimer_interrupt+0x128/0x2c0 [ 8276.062344] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8276.062350] ? apic_timer_interrupt+0xf/0x20 [ 8276.062355] [ 8276.062358] ? native_safe_halt+0x2e/0x50 [ 8276.062362] kvm_wait+0xcd/0xe0 [ 8276.062370] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8276.062377] do_raw_spin_lock+0x177/0x190 [ 8276.062382] _raw_spin_lock+0x21/0x30 [ 8276.062444] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8276.062657] process_one_work+0x2c8/0x700 [ 8276.062664] worker_thread+0x296/0x6e0 [ 8276.062668] ? rescuer_thread+0x570/0x570 [ 8276.062671] kthread+0x1d1/0x200 [ 8276.062675] ? set_kthread_struct+0x70/0x70 [ 8276.062679] ret_from_fork+0x1f/0x30 [ 8276.210620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8276.237980] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8276.258268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8276.268319] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8276.275117] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8276.295035] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8276.300047] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8276.303489] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8276.307511] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8276.312014] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8276.316443] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8276.320721] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8276.325952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8276.332387] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8276.343859] Call Trace: [ 8276.344426] [ 8276.344977] ? show_regs.cold.9+0x22/0x2f [ 8276.345889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8276.346905] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8276.354034] ? watchdog+0x40/0x40 [ 8276.356335] ? hrtimer_interrupt+0x128/0x2c0 [ 8276.358681] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8276.361379] ? apic_timer_interrupt+0xf/0x20 [ 8276.363576] [ 8276.365008] ? native_safe_halt+0x2e/0x50 [ 8276.367448] kvm_wait+0xcd/0xe0 [ 8276.369376] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8276.372376] do_raw_spin_lock+0x177/0x190 [ 8276.374475] _raw_spin_lock+0x21/0x30 [ 8276.376319] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8276.379178] ? do_raw_spin_unlock+0x75/0x190 [ 8276.381328] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8276.384378] ? do_wait_intr+0xf0/0xf0 [ 8276.387136] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8276.397413] mdc_close+0x53b/0xd70 [mdc] [ 8276.400012] lmv_close+0x18d/0x490 [lmv] [ 8276.403255] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8276.407415] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8276.410678] ll_md_close+0x274/0x800 [lustre] [ 8276.413303] ll_file_release+0x128/0xc50 [lustre] [ 8276.416085] __fput+0x10d/0x440 [ 8276.417625] ____fput+0x12/0x20 [ 8276.419609] task_work_run+0xc6/0x110 [ 8276.421495] exit_to_usermode_loop+0x1dd/0x1f0 [ 8276.423823] do_syscall_64+0x3d6/0x440 [ 8276.426162] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8276.428665] RIP: 0033:0x7fa749552cd8 [ 8276.430660] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8276.440413] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8276.443417] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8276.446989] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8276.450116] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8276.452559] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8276.456204] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8300.054065] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8300.059546] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.061139] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8300.097700] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.097716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.097719] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8300.097733] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8300.097738] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8300.097744] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8300.097747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8300.097750] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8300.097752] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8300.097755] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8300.097758] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8300.097761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.097764] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8300.097773] Call Trace: [ 8300.097779] [ 8300.097785] ? show_regs.cold.9+0x22/0x2f [ 8300.097795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.097803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.097807] ? watchdog+0x40/0x40 [ 8300.097812] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.097816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.097823] ? apic_timer_interrupt+0xf/0x20 [ 8300.097828] [ 8300.097831] ? native_safe_halt+0x2e/0x50 [ 8300.097835] kvm_wait+0xcd/0xe0 [ 8300.097844] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8300.097854] do_raw_spin_lock+0x177/0x190 [ 8300.097859] _raw_spin_lock+0x21/0x30 [ 8300.097864] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8300.106767] Modules linked in: [ 8300.112057] ? do_raw_spin_unlock+0x75/0x190 [ 8300.112077] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8300.116967] lustre(O) [ 8300.119870] ? do_wait_intr+0xf0/0xf0 [ 8300.119893] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8300.137830] osp(O) [ 8300.142084] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8300.152446] ofd(O) [ 8300.156597] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8300.156889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.157079] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8300.157235] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8300.157270] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8300.157301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.157456] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8300.157770] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8300.157804] lmv_intent_open+0x338/0x1150 [lmv] [ 8300.157838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.157929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.158086] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8300.158228] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8300.158321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.158546] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8300.158648] ? do_raw_spin_unlock+0x75/0x190 [ 8300.158657] ? _raw_spin_unlock+0x12/0x30 [ 8300.158663] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8300.158691] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8300.158713] ll_atomic_open+0x36a/0xd70 [lustre] [ 8300.166503] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [ 8300.170870] lookup_open+0x356/0xb40 [ 8300.180897] i2c_piix4 [ 8300.185820] do_last+0x313/0xfc0 [ 8300.185841] ? nd_jump_root+0xe5/0x160 [ 8300.185847] ? path_init+0x437/0x520 [ 8300.185852] path_openat+0xf7/0x500 [ 8300.185858] ? do_raw_spin_unlock+0x75/0x190 [ 8300.185866] do_filp_open+0x99/0x140 [ 8300.185873] ? getname_flags+0x6e/0x330 [ 8300.185878] ? __check_object_size+0xff/0x256 [ 8300.185885] ? do_raw_spin_unlock+0x75/0x190 [ 8300.185889] ? _raw_spin_unlock+0x12/0x30 [ 8300.185897] do_sys_openat2+0x2b4/0x410 [ 8300.185902] do_sys_open+0x73/0xa0 [ 8300.185907] __x64_sys_openat+0x24/0x30 [ 8300.185911] do_syscall_64+0xc1/0x440 [ 8300.185918] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8300.185925] RIP: 0033:0x7fa749552332 [ 8300.185934] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8300.185940] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8300.185946] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8300.185948] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8300.185951] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8300.185954] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8300.185956] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8300.640099] pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.673094] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.692887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.719195] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8300.728965] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8300.766137] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8300.783692] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8300.786267] RDX: 0000000000001af7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8300.802057] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8300.816718] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8300.832329] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8300.848008] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8300.872579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.881971] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8300.892449] Call Trace: [ 8300.894527] [ 8300.895867] ? show_regs.cold.9+0x22/0x2f [ 8300.901373] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.906857] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.912891] ? watchdog+0x40/0x40 [ 8300.919472] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.924932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.938816] ? apic_timer_interrupt+0xf/0x20 [ 8300.944660] [ 8300.948438] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8300.955611] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8300.965726] do_raw_spin_lock+0x177/0x190 [ 8300.975094] _raw_spin_lock+0x21/0x30 [ 8300.981698] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8300.995574] obd_mod_free+0x46/0x140 [ptlrpc] [ 8301.003977] mdc_commit_open+0x91/0xb0 [mdc] [ 8301.005712] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8301.014099] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8301.023081] after_reply+0xe2c/0x1780 [ptlrpc] [ 8301.030447] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8301.035975] ? __next_timer_interrupt+0x160/0x160 [ 8301.040237] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8301.041971] ? do_wait_intr+0xf0/0xf0 [ 8301.049851] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8301.064225] kthread+0x1d1/0x200 [ 8301.069564] ? set_kthread_struct+0x70/0x70 [ 8301.079849] ret_from_fork+0x1f/0x30 [ 8301.095452] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6383s! [ 8301.116447] Showing busy workqueues and worker pools: [ 8301.120217] workqueue events: flags=0x0 [ 8301.135840] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8301.147789] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8301.158137] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8301.164248] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8301.173398] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.178713] pending: kfree_rcu_monitor [ 8301.182055] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8301.189102] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8301.203134] workqueue events_long: flags=0x0 [ 8301.209202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.214000] pending: lu_objects_flush [obdclass] [ 8301.227104] workqueue events_unbound: flags=0x2 [ 8301.232223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8301.242049] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8301.258948] workqueue events_power_efficient: flags=0x80 [ 8301.266944] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.273012] pending: check_lifetime [ 8301.279675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.287153] pending: neigh_periodic_work [ 8301.290878] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.303659] pending: neigh_periodic_work [ 8301.313197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.319183] pending: do_cache_clean [ 8301.323618] workqueue mm_percpu_wq: flags=0x8 [ 8301.329458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.338008] pending: vmstat_update [ 8301.343079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.350321] pending: vmstat_update [ 8301.356455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8301.367090] pending: vmstat_update [ 8301.370432] workqueue writeback: flags=0x4a [ 8301.373865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8301.378173] pending: wb_workfn, wb_workfn [ 8301.382049] workqueue xprtiod: flags=0xa [ 8301.384855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8301.387467] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8301.390762] workqueue ipv6_addrconf: flags=0x40008 [ 8301.395143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8301.404898] pending: addrconf_verify_work [ 8301.414167] workqueue knbd0-recv: flags=0x1a [ 8301.419430] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8301.424466] in-flight: 651:recv_work [ 8301.426292] workqueue ptlrpc_pinger: flags=0x2 [ 8301.433187] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8301.442022] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8301.444252] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6384s workers=4 idle: 37 31846 28897 [ 8301.462957] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8291s workers=2 idle: 653 [ 8304.057186] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8304.062227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8304.062540] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8304.125668] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8304.125682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8304.125688] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8304.126007] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8304.126899] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8304.126904] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8304.126909] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8304.126911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8304.126914] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8304.126916] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8304.126919] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8304.126922] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8304.126925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8304.126928] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8304.126936] Call Trace: [ 8304.127315] [ 8304.127323] ? show_regs.cold.9+0x22/0x2f [ 8304.127333] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8304.127339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8304.127344] ? watchdog+0x40/0x40 [ 8304.127348] ? hrtimer_interrupt+0x128/0x2c0 [ 8304.127353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8304.127359] ? apic_timer_interrupt+0xf/0x20 [ 8304.127364] [ 8304.127367] ? native_safe_halt+0x2e/0x50 [ 8304.127371] kvm_wait+0xcd/0xe0 [ 8304.127379] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8304.127387] do_raw_spin_lock+0x177/0x190 [ 8304.127391] _raw_spin_lock+0x21/0x30 [ 8304.127396] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8304.127687] process_one_work+0x2c8/0x700 [ 8304.127695] worker_thread+0x296/0x6e0 [ 8304.127700] ? rescuer_thread+0x570/0x570 [ 8304.127703] kthread+0x1d1/0x200 [ 8304.127708] ? set_kthread_struct+0x70/0x70 [ 8304.127713] ret_from_fork+0x1f/0x30 [ 8304.366647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8304.406078] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8304.413427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8304.417752] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8304.420485] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8304.432271] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8304.439575] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8304.442467] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8304.447685] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8304.452025] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8304.456066] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8304.464656] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8304.471717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8304.474707] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8304.480638] Call Trace: [ 8304.486573] [ 8304.488774] ? show_regs.cold.9+0x22/0x2f [ 8304.494310] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8304.499296] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8304.501511] ? watchdog+0x40/0x40 [ 8304.504153] ? hrtimer_interrupt+0x128/0x2c0 [ 8304.505717] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8304.507447] ? apic_timer_interrupt+0xf/0x20 [ 8304.518289] [ 8304.519392] ? native_safe_halt+0x2e/0x50 [ 8304.524406] kvm_wait+0xcd/0xe0 [ 8304.525955] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8304.530368] do_raw_spin_lock+0x177/0x190 [ 8304.536470] _raw_spin_lock+0x21/0x30 [ 8304.541196] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8304.544594] ? do_raw_spin_unlock+0x75/0x190 [ 8304.550835] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8304.558560] ? do_wait_intr+0xf0/0xf0 [ 8304.560416] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8304.563918] mdc_close+0x53b/0xd70 [mdc] [ 8304.568128] lmv_close+0x18d/0x490 [lmv] [ 8304.574635] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8304.582808] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8304.584649] ll_md_close+0x274/0x800 [lustre] [ 8304.587788] ll_file_release+0x128/0xc50 [lustre] [ 8304.590926] __fput+0x10d/0x440 [ 8304.592552] ____fput+0x12/0x20 [ 8304.593655] task_work_run+0xc6/0x110 [ 8304.598880] exit_to_usermode_loop+0x1dd/0x1f0 [ 8304.605737] do_syscall_64+0x3d6/0x440 [ 8304.610380] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8304.615361] RIP: 0033:0x7fa749552cd8 [ 8304.620066] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8304.630471] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8304.633400] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8304.639451] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8304.642333] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8304.648557] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8304.661911] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8328.055306] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8328.060391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.060704] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8328.098713] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.098728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.098731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.098745] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8328.098750] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8328.098757] RAX: 0000000000006114 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8328.098759] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8328.098762] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8328.098764] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8328.098766] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8328.098769] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8328.098773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.098775] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8328.098784] Call Trace: [ 8328.099387] [ 8328.099392] ? show_regs.cold.9+0x22/0x2f [ 8328.099402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.099409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.099414] ? watchdog+0x40/0x40 [ 8328.099418] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.099422] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.099430] ? apic_timer_interrupt+0xf/0x20 [ 8328.099435] [ 8328.099438] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.099446] do_raw_spin_lock+0x177/0x190 [ 8328.099451] _raw_spin_lock+0x21/0x30 [ 8328.099455] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8328.099741] ? do_raw_spin_unlock+0x75/0x190 [ 8328.099747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8328.111661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.111744] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.111752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.111755] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8328.111770] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8328.111774] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8328.111778] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8328.111780] RDX: 0000000000001f32 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8328.111783] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8328.111796] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8328.111799] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8328.111802] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8328.111804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.111806] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8328.111814] Call Trace: [ 8328.111819] [ 8328.111826] ? show_regs.cold.9+0x22/0x2f [ 8328.111837] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.111843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.111847] ? watchdog+0x40/0x40 [ 8328.111852] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.111856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.111864] ? apic_timer_interrupt+0xf/0x20 [ 8328.111869] [ 8328.111872] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8328.111877] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8328.111882] do_raw_spin_lock+0x177/0x190 [ 8328.111886] _raw_spin_lock+0x21/0x30 [ 8328.111892] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8328.112331] obd_mod_free+0x46/0x140 [ptlrpc] [ 8328.112549] mdc_commit_open+0x91/0xb0 [mdc] [ 8328.112589] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8328.114384] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8328.114650] after_reply+0xe2c/0x1780 [ptlrpc] [ 8328.124515] ? do_wait_intr+0xf0/0xf0 [ 8328.124537] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8328.124880] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8328.125134] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8328.125619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.125740] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8328.127341] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8328.137892] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8328.143564] ? __next_timer_interrupt+0x160/0x160 [ 8328.143575] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8328.145418] ? do_wait_intr+0xf0/0xf0 [ 8328.145431] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8328.145702] kthread+0x1d1/0x200 [ 8328.145708] ? set_kthread_struct+0x70/0x70 [ 8328.145712] ret_from_fork+0x1f/0x30 [ 8328.606088] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8328.610883] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.614924] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8328.621765] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8328.627848] lmv_intent_open+0x338/0x1150 [lmv] [ 8328.634413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.643011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.646344] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8328.650964] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8328.652717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.659632] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8328.662503] ? do_raw_spin_unlock+0x75/0x190 [ 8328.665771] ? _raw_spin_unlock+0x12/0x30 [ 8328.670667] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8328.674718] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8328.678134] ll_atomic_open+0x36a/0xd70 [lustre] [ 8328.684594] lookup_open+0x356/0xb40 [ 8328.685957] do_last+0x313/0xfc0 [ 8328.688749] ? nd_jump_root+0xe5/0x160 [ 8328.692575] ? path_init+0x437/0x520 [ 8328.695007] path_openat+0xf7/0x500 [ 8328.699605] ? do_raw_spin_unlock+0x75/0x190 [ 8328.703029] do_filp_open+0x99/0x140 [ 8328.706899] ? getname_flags+0x6e/0x330 [ 8328.708689] ? __check_object_size+0xff/0x256 [ 8328.713974] ? do_raw_spin_unlock+0x75/0x190 [ 8328.716625] ? _raw_spin_unlock+0x12/0x30 [ 8328.720893] do_sys_openat2+0x2b4/0x410 [ 8328.722724] do_sys_open+0x73/0xa0 [ 8328.725787] __x64_sys_openat+0x24/0x30 [ 8328.728141] do_syscall_64+0xc1/0x440 [ 8328.729170] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8328.730504] RIP: 0033:0x7fa749552332 [ 8328.735992] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8328.741173] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8328.745709] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8328.750965] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8328.754416] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8328.760130] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8328.761885] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8331.747571] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6414s! [ 8331.755919] Showing busy workqueues and worker pools: [ 8331.759911] workqueue events: flags=0x0 [ 8331.763788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8331.773979] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8331.794418] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8331.800250] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8331.811556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.813914] pending: kfree_rcu_monitor [ 8331.826172] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8331.836429] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8331.844844] workqueue events_long: flags=0x0 [ 8331.849063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.853324] pending: lu_objects_flush [obdclass] [ 8331.862447] workqueue events_unbound: flags=0x2 [ 8331.866362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8331.873568] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8331.886238] workqueue events_power_efficient: flags=0x80 [ 8331.893862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.899622] pending: check_lifetime [ 8331.901797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.904997] pending: neigh_periodic_work [ 8331.908336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.914228] pending: neigh_periodic_work [ 8331.920081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.924425] pending: do_cache_clean [ 8331.927235] workqueue mm_percpu_wq: flags=0x8 [ 8331.929576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.937955] pending: vmstat_update [ 8331.945197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.952444] pending: vmstat_update [ 8331.966227] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8331.977271] pending: vmstat_update [ 8331.980689] workqueue writeback: flags=0x4a [ 8331.985421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8331.992119] pending: wb_workfn, wb_workfn [ 8331.997711] workqueue xprtiod: flags=0xa [ 8332.005035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8332.007454] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8332.019371] workqueue ipv6_addrconf: flags=0x40008 [ 8332.026028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8332.031508] pending: addrconf_verify_work [ 8332.036048] workqueue knbd0-recv: flags=0x1a [ 8332.039039] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8332.045031] in-flight: 651:recv_work [ 8332.047509] workqueue ptlrpc_pinger: flags=0x2 [ 8332.049273] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8332.058225] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8332.059121] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8332.063860] pool 8: [ 8332.067625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8332.067694] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8332.067701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8332.067704] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8332.067782] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8332.067788] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8332.067793] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8332.067796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8332.067798] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8332.067800] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8332.067802] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8332.067805] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8332.067808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8332.067810] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8332.067818] Call Trace: [ 8332.067824] [ 8332.067829] ? show_regs.cold.9+0x22/0x2f [ 8332.067839] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8332.067845] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8332.067849] ? watchdog+0x40/0x40 [ 8332.067853] ? hrtimer_interrupt+0x128/0x2c0 [ 8332.067857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8332.067863] ? apic_timer_interrupt+0xf/0x20 [ 8332.067867] [ 8332.067870] ? native_safe_halt+0x2e/0x50 [ 8332.067874] kvm_wait+0xcd/0xe0 [ 8332.067881] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8332.067891] do_raw_spin_lock+0x177/0x190 [ 8332.067894] _raw_spin_lock+0x21/0x30 [ 8332.067899] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8332.068189] ? do_raw_spin_unlock+0x75/0x190 [ 8332.068196] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8332.068461] ? do_wait_intr+0xf0/0xf0 [ 8332.068469] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8332.068663] mdc_close+0x53b/0xd70 [mdc] [ 8332.068700] lmv_close+0x18d/0x490 [lmv] [ 8332.068804] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8332.068917] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8332.068997] ll_md_close+0x274/0x800 [lustre] [ 8332.069090] ll_file_release+0x128/0xc50 [lustre] [ 8332.069169] __fput+0x10d/0x440 [ 8332.069178] ____fput+0x12/0x20 [ 8332.069182] task_work_run+0xc6/0x110 [ 8332.069189] exit_to_usermode_loop+0x1dd/0x1f0 [ 8332.069195] do_syscall_64+0x3d6/0x440 [ 8332.069199] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8332.069205] RIP: 0033:0x7fa749552cd8 [ 8332.069212] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8332.069216] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8332.069287] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8332.069290] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8332.069292] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8332.069293] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8332.069296] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8332.069805] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8332.069811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8332.069855] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8332.069858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8332.069862] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8332.070329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8332.070336] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8332.070339] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8332.070342] RAX: 0000000000007f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8332.070344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8332.070345] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8332.070347] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8332.070348] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8332.070350] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8332.070352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8332.070354] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8332.070360] Call Trace: [ 8332.070364] [ 8332.070367] ? show_regs.cold.9+0x22/0x2f [ 8332.070375] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8332.070379] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8332.070382] ? watchdog+0x40/0x40 [ 8332.070385] ? hrtimer_interrupt+0x128/0x2c0 [ 8332.070388] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8332.070394] ? apic_timer_interrupt+0xf/0x20 [ 8332.070397] [ 8332.070399] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8332.070402] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8332.070405] do_raw_spin_lock+0x177/0x190 [ 8332.070408] _raw_spin_lock+0x21/0x30 [ 8332.070411] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8332.073996] cpus=0-3 flags=0x4 nice=0 hung=6414s workers=4 idle: 37 31846 28897 [ 8332.074062] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8321s workers=2 idle: 653 [ 8332.368453] process_one_work+0x2c8/0x700 [ 8332.369914] worker_thread+0x296/0x6e0 [ 8332.371472] ? rescuer_thread+0x570/0x570 [ 8332.373166] kthread+0x1d1/0x200 [ 8332.374573] ? set_kthread_struct+0x70/0x70 [ 8332.376391] ret_from_fork+0x1f/0x30 [ 8356.058256] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8356.060516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8356.075777] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8356.079452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8356.083285] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8356.085932] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8356.092905] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8356.095926] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8356.098487] RDX: 00000000000077c3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8356.101157] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8356.104758] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8356.107789] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8356.110685] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8356.113373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8356.115559] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8356.118387] Call Trace: [ 8356.119076] [ 8356.120016] ? show_regs.cold.9+0x22/0x2f [ 8356.121527] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8356.123289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8356.125335] ? watchdog+0x40/0x40 [ 8356.126208] ? hrtimer_interrupt+0x128/0x2c0 [ 8356.127519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8356.130076] ? apic_timer_interrupt+0xf/0x20 [ 8356.132434] [ 8356.133232] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8356.135062] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8356.136719] do_raw_spin_lock+0x177/0x190 [ 8356.138008] _raw_spin_lock+0x21/0x30 [ 8356.139172] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8356.142102] obd_mod_free+0x46/0x140 [ptlrpc] [ 8356.144033] mdc_commit_open+0x91/0xb0 [mdc] [ 8356.145803] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8356.148618] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8356.151747] after_reply+0xe2c/0x1780 [ptlrpc] [ 8356.154377] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8356.156078] ? __next_timer_interrupt+0x160/0x160 [ 8356.157593] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8356.159357] ? do_wait_intr+0xf0/0xf0 [ 8356.160470] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8356.163139] kthread+0x1d1/0x200 [ 8356.164217] ? set_kthread_struct+0x70/0x70 [ 8356.165924] ret_from_fork+0x1f/0x30 [ 8360.055469] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8360.058439] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8360.068314] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8360.068328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8360.068399] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8360.068405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8360.068408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8360.068423] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8360.068427] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8360.068432] RAX: 0000000000003283 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8360.068435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8360.068437] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8360.068439] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8360.068441] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8360.068445] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8360.068448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8360.068450] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8360.068459] Call Trace: [ 8360.068465] [ 8360.068472] ? show_regs.cold.9+0x22/0x2f [ 8360.068481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8360.068488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8360.068494] ? watchdog+0x40/0x40 [ 8360.068498] ? hrtimer_interrupt+0x128/0x2c0 [ 8360.068502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8360.068521] ? apic_timer_interrupt+0xf/0x20 [ 8360.068527] [ 8360.068530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8360.068537] do_raw_spin_lock+0x177/0x190 [ 8360.068542] _raw_spin_lock+0x21/0x30 [ 8360.068703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 8360.068547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8360.068864] ? do_raw_spin_unlock+0x75/0x190 [ 8360.068871] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8360.070106] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8360.070153] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8360.070160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8360.070162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8360.070172] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8360.070176] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8360.070179] RAX: 0000000000003586 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8360.070182] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8360.070183] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8360.070185] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8360.070188] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8360.070190] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8360.070192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8360.070194] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8360.070210] Call Trace: [ 8360.070215] [ 8360.070219] ? show_regs.cold.9+0x22/0x2f [ 8360.070316] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8360.070323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8360.070328] ? watchdog+0x40/0x40 [ 8360.070332] ? hrtimer_interrupt+0x128/0x2c0 [ 8360.070337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8360.070344] ? apic_timer_interrupt+0xf/0x20 [ 8360.070348] [ 8360.070351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8360.070358] do_raw_spin_lock+0x177/0x190 [ 8360.070362] _raw_spin_lock+0x21/0x30 [ 8360.070368] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8360.070696] ? do_raw_spin_unlock+0x75/0x190 [ 8360.071263] ? do_wait_intr+0xf0/0xf0 [ 8360.071272] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8360.071485] mdc_close+0x53b/0xd70 [mdc] [ 8360.071767] lmv_close+0x18d/0x490 [lmv] [ 8360.071802] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8360.071943] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8360.072022] ll_md_close+0x274/0x800 [lustre] [ 8360.072124] ll_file_release+0x128/0xc50 [lustre] [ 8360.072211] __fput+0x10d/0x440 [ 8360.072221] ____fput+0x12/0x20 [ 8360.072226] task_work_run+0xc6/0x110 [ 8360.072233] exit_to_usermode_loop+0x1dd/0x1f0 [ 8360.072241] do_syscall_64+0x3d6/0x440 [ 8360.072245] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8360.072254] RIP: 0033:0x7fa749552cd8 [ 8360.072261] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8360.072265] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8360.072271] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8360.072273] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8360.072276] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8360.072278] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8360.072281] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8360.070702] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8360.082569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8360.082651] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8360.082670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8360.082678] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8360.083175] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8360.083187] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8360.083191] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8360.083196] RAX: 0000000000004587 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8360.083198] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8360.083200] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8360.083203] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8360.083206] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8360.083209] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8360.083212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8360.083214] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8360.083222] Call Trace: [ 8360.083226] [ 8360.083232] ? show_regs.cold.9+0x22/0x2f [ 8360.083239] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8360.083245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8360.083249] ? watchdog+0x40/0x40 [ 8360.083253] ? hrtimer_interrupt+0x128/0x2c0 [ 8360.083257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8360.083267] ? apic_timer_interrupt+0xf/0x20 [ 8360.083271] [ 8360.083275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8360.083279] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8360.083334] do_raw_spin_lock+0x177/0x190 [ 8360.083340] _raw_spin_lock+0x21/0x30 [ 8360.083346] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8360.083590] process_one_work+0x2c8/0x700 [ 8360.083603] worker_thread+0x296/0x6e0 [ 8360.083608] ? rescuer_thread+0x570/0x570 [ 8360.083611] kthread+0x1d1/0x200 [ 8360.083615] ? set_kthread_struct+0x70/0x70 [ 8360.083619] ret_from_fork+0x1f/0x30 [ 8360.944154] ? do_wait_intr+0xf0/0xf0 [ 8360.947788] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8360.955147] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8360.960088] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8360.971335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8360.974585] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8360.979776] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8360.984150] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8360.991735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8360.995665] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8361.004906] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8361.007961] lmv_intent_open+0x338/0x1150 [lmv] [ 8361.009664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8361.018402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8361.020450] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8361.040229] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8361.046599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8361.052952] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8361.055274] ? do_raw_spin_unlock+0x75/0x190 [ 8361.058705] ? _raw_spin_unlock+0x12/0x30 [ 8361.060959] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8361.071528] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8361.086864] ll_atomic_open+0x36a/0xd70 [lustre] [ 8361.090633] lookup_open+0x356/0xb40 [ 8361.095017] do_last+0x313/0xfc0 [ 8361.096900] ? nd_jump_root+0xe5/0x160 [ 8361.100608] ? path_init+0x437/0x520 [ 8361.106084] path_openat+0xf7/0x500 [ 8361.110078] ? do_raw_spin_unlock+0x75/0x190 [ 8361.121136] do_filp_open+0x99/0x140 [ 8361.123168] ? getname_flags+0x6e/0x330 [ 8361.125641] ? __check_object_size+0xff/0x256 [ 8361.127807] ? do_raw_spin_unlock+0x75/0x190 [ 8361.129667] ? _raw_spin_unlock+0x12/0x30 [ 8361.133985] do_sys_openat2+0x2b4/0x410 [ 8361.135840] do_sys_open+0x73/0xa0 [ 8361.140783] __x64_sys_openat+0x24/0x30 [ 8361.144776] do_syscall_64+0xc1/0x440 [ 8361.148765] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8361.153766] RIP: 0033:0x7fa749552332 [ 8361.155880] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8361.175900] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8361.181098] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8361.186084] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8361.190432] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8361.208433] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8361.222183] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8362.474691] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6445s! [ 8362.491989] Showing busy workqueues and worker pools: [ 8362.496827] workqueue events: flags=0x0 [ 8362.501396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8362.511009] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8362.534872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8362.550291] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8362.557961] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.577353] pending: kfree_rcu_monitor [ 8362.589423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8362.603024] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8362.618789] workqueue events_long: flags=0x0 [ 8362.623839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.629126] pending: lu_objects_flush [obdclass] [ 8362.642310] workqueue events_unbound: flags=0x2 [ 8362.654848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8362.657152] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8362.683806] workqueue events_power_efficient: flags=0x80 [ 8362.694843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.708055] pending: check_lifetime [ 8362.711822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.714260] pending: neigh_periodic_work [ 8362.727995] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.737933] pending: neigh_periodic_work [ 8362.742900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.758256] pending: do_cache_clean [ 8362.763578] workqueue mm_percpu_wq: flags=0x8 [ 8362.767160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.779405] pending: vmstat_update [ 8362.785101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.801715] pending: vmstat_update [ 8362.803127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8362.813134] pending: vmstat_update [ 8362.816392] workqueue writeback: flags=0x4a [ 8362.818392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8362.826637] pending: wb_workfn, wb_workfn [ 8362.830572] workqueue xprtiod: flags=0xa [ 8362.832356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8362.835975] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8362.846070] workqueue ipv6_addrconf: flags=0x40008 [ 8362.853394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8362.859376] pending: addrconf_verify_work [ 8362.864041] workqueue knbd0-recv: flags=0x1a [ 8362.867685] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8362.874062] in-flight: 651:recv_work [ 8362.886180] workqueue ptlrpc_pinger: flags=0x2 [ 8362.890927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8362.896598] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8362.902700] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6445s workers=4 idle: 37 31846 28897 [ 8362.913744] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8352s workers=2 idle: 653 [ 8384.068761] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8384.083603] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8384.133888] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8384.145449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8384.152714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8384.155963] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8384.167376] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8384.173306] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8384.177701] RDX: 00000000000006fe RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8384.182776] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8384.190949] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8384.197120] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8384.200792] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8384.206674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8384.215440] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8384.220614] Call Trace: [ 8384.222700] [ 8384.224037] ? show_regs.cold.9+0x22/0x2f [ 8384.226726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8384.229152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8384.232138] ? watchdog+0x40/0x40 [ 8384.233614] ? hrtimer_interrupt+0x128/0x2c0 [ 8384.237708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8384.240894] ? apic_timer_interrupt+0xf/0x20 [ 8384.244657] [ 8384.246233] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8384.249163] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8384.251710] do_raw_spin_lock+0x177/0x190 [ 8384.253674] _raw_spin_lock+0x21/0x30 [ 8384.256043] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8384.258482] obd_mod_free+0x46/0x140 [ptlrpc] [ 8384.261578] mdc_commit_open+0x91/0xb0 [mdc] [ 8384.264282] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8384.268214] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8384.270864] after_reply+0xe2c/0x1780 [ptlrpc] [ 8384.276785] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8384.280103] ? __next_timer_interrupt+0x160/0x160 [ 8384.281871] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8384.286911] ? do_wait_intr+0xf0/0xf0 [ 8384.289235] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8384.294339] kthread+0x1d1/0x200 [ 8384.295532] ? set_kthread_struct+0x70/0x70 [ 8384.300106] ret_from_fork+0x1f/0x30 [ 8388.055860] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8388.059646] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8388.061547] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8388.061559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8388.061628] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8388.061634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8388.061640] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8388.061928] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8388.062312] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8388.062317] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8388.062321] RAX: 0000000000005a9b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8388.062323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8388.062326] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8388.062327] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8388.062330] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8388.062332] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8388.062335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8388.062337] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8388.062358] Call Trace: [ 8388.062364] [ 8388.062370] ? show_regs.cold.9+0x22/0x2f [ 8388.062380] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8388.062387] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8388.062391] ? watchdog+0x40/0x40 [ 8388.062396] ? hrtimer_interrupt+0x128/0x2c0 [ 8388.062400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8388.062408] ? apic_timer_interrupt+0xf/0x20 [ 8388.062412] [ 8388.062415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8388.062420] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8388.062425] do_raw_spin_lock+0x177/0x190 [ 8388.062430] _raw_spin_lock+0x21/0x30 [ 8388.062435] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8388.062651] process_one_work+0x2c8/0x700 [ 8388.062658] worker_thread+0x296/0x6e0 [ 8388.062663] ? rescuer_thread+0x570/0x570 [ 8388.062667] kthread+0x1d1/0x200 [ 8388.062672] ? set_kthread_struct+0x70/0x70 [ 8388.062675] ret_from_fork+0x1f/0x30 [ 8388.064174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8388.064253] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8388.064260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8388.064263] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8388.064276] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8388.064280] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8388.064286] RAX: 0000000000007e91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8388.064288] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8388.064290] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8388.064293] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8388.064295] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8388.064306] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8388.064310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8388.064313] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8388.064319] Call Trace: [ 8388.064324] [ 8388.064329] ? show_regs.cold.9+0x22/0x2f [ 8388.064338] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8388.064344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8388.064348] ? watchdog+0x40/0x40 [ 8388.064352] ? hrtimer_interrupt+0x128/0x2c0 [ 8388.064356] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8388.064362] ? apic_timer_interrupt+0xf/0x20 [ 8388.064367] [ 8388.064371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8388.064377] do_raw_spin_lock+0x177/0x190 [ 8388.064382] _raw_spin_lock+0x21/0x30 [ 8388.064387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8388.065174] ? do_raw_spin_unlock+0x75/0x190 [ 8388.065392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8388.066925] ? do_wait_intr+0xf0/0xf0 [ 8388.067042] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8388.067275] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8388.069176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8388.069391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8388.069525] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8388.069565] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8388.069595] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8388.069621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8388.069708] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8388.071334] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8388.071376] lmv_intent_open+0x338/0x1150 [lmv] [ 8388.071409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8388.071532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8388.082262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8388.082338] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8388.082346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8388.082349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8388.082362] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8388.082366] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8388.082371] RAX: 0000000000007c88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8388.082373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8388.082375] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8388.082377] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8388.082379] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8388.082382] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8388.082384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8388.082386] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8388.082396] Call Trace: [ 8388.082403] [ 8388.082411] ? show_regs.cold.9+0x22/0x2f [ 8388.082422] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8388.082428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8388.082432] ? watchdog+0x40/0x40 [ 8388.082437] ? hrtimer_interrupt+0x128/0x2c0 [ 8388.082441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8388.082450] ? apic_timer_interrupt+0xf/0x20 [ 8388.082454] [ 8388.082458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8388.082465] do_raw_spin_lock+0x177/0x190 [ 8388.082470] _raw_spin_lock+0x21/0x30 [ 8388.082475] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8388.082784] ? do_raw_spin_unlock+0x75/0x190 [ 8388.082790] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8388.083827] ? do_wait_intr+0xf0/0xf0 [ 8388.083838] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8388.084324] mdc_close+0x53b/0xd70 [mdc] [ 8388.084361] lmv_close+0x18d/0x490 [lmv] [ 8388.084388] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8388.084497] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8388.084580] ll_md_close+0x274/0x800 [lustre] [ 8388.084675] ll_file_release+0x128/0xc50 [lustre] [ 8388.084785] __fput+0x10d/0x440 [ 8388.084796] ____fput+0x12/0x20 [ 8388.084803] task_work_run+0xc6/0x110 [ 8388.084812] exit_to_usermode_loop+0x1dd/0x1f0 [ 8388.084820] do_syscall_64+0x3d6/0x440 [ 8388.084824] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8388.084832] RIP: 0033:0x7fa749552cd8 [ 8388.084840] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8388.084844] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8388.084848] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8388.084851] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8388.084853] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8388.084855] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8388.084857] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8388.941492] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8388.945308] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8388.953433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8388.960343] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8388.963988] ? do_raw_spin_unlock+0x75/0x190 [ 8388.970092] ? _raw_spin_unlock+0x12/0x30 [ 8388.971602] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8388.974566] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8388.980147] ll_atomic_open+0x36a/0xd70 [lustre] [ 8388.989941] lookup_open+0x356/0xb40 [ 8388.991295] do_last+0x313/0xfc0 [ 8388.992496] ? nd_jump_root+0xe5/0x160 [ 8389.001074] ? path_init+0x437/0x520 [ 8389.004618] path_openat+0xf7/0x500 [ 8389.009249] ? do_raw_spin_unlock+0x75/0x190 [ 8389.013654] do_filp_open+0x99/0x140 [ 8389.018633] ? getname_flags+0x6e/0x330 [ 8389.020447] ? __check_object_size+0xff/0x256 [ 8389.023812] ? do_raw_spin_unlock+0x75/0x190 [ 8389.027541] ? _raw_spin_unlock+0x12/0x30 [ 8389.029687] do_sys_openat2+0x2b4/0x410 [ 8389.033718] do_sys_open+0x73/0xa0 [ 8389.035849] __x64_sys_openat+0x24/0x30 [ 8389.037743] do_syscall_64+0xc1/0x440 [ 8389.039714] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8389.045252] RIP: 0033:0x7fa749552332 [ 8389.049129] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8389.059942] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8389.067486] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8389.076282] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8389.084458] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8389.091520] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8389.106394] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8393.190570] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6476s! [ 8393.199853] Showing busy workqueues and worker pools: [ 8393.206278] workqueue events: flags=0x0 [ 8393.211926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8393.215877] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8393.221301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8393.225222] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8393.229363] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.233540] pending: kfree_rcu_monitor [ 8393.235719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8393.238809] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8393.242274] workqueue events_long: flags=0x0 [ 8393.243876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.246301] pending: lu_objects_flush [obdclass] [ 8393.248909] workqueue events_unbound: flags=0x2 [ 8393.251025] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8393.253727] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8393.257967] workqueue events_power_efficient: flags=0x80 [ 8393.260248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.262942] pending: check_lifetime [ 8393.265174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.269034] pending: neigh_periodic_work [ 8393.274819] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.280332] pending: neigh_periodic_work [ 8393.285235] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.289002] pending: do_cache_clean [ 8393.290840] workqueue mm_percpu_wq: flags=0x8 [ 8393.293020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.298191] pending: vmstat_update [ 8393.299950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.305039] pending: vmstat_update [ 8393.307982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8393.316395] pending: vmstat_update [ 8393.320010] workqueue writeback: flags=0x4a [ 8393.322885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8393.326617] pending: wb_workfn, wb_workfn [ 8393.328791] workqueue xprtiod: flags=0xa [ 8393.332017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8393.335358] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8393.339594] workqueue ipv6_addrconf: flags=0x40008 [ 8393.341954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8393.345969] pending: addrconf_verify_work [ 8393.348971] workqueue knbd0-recv: flags=0x1a [ 8393.351161] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8393.354261] in-flight: 651:recv_work [ 8393.356040] workqueue ptlrpc_pinger: flags=0x2 [ 8393.358046] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8393.362004] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8393.370568] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6476s workers=4 idle: 37 31846 28897 [ 8393.377329] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8383s workers=2 idle: 653 [ 8412.067339] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8412.079186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8412.145980] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8412.157281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8412.173067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8412.182392] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8412.208264] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8412.223182] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8412.230184] RDX: 00000000000000e2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8412.233529] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8412.241691] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8412.257400] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8412.272209] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8412.295179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8412.314426] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8412.317208] Call Trace: [ 8412.318376] [ 8412.319212] ? show_regs.cold.9+0x22/0x2f [ 8412.321077] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8412.322962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8412.324891] ? watchdog+0x40/0x40 [ 8412.326844] ? hrtimer_interrupt+0x128/0x2c0 [ 8412.328621] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8412.330670] ? apic_timer_interrupt+0xf/0x20 [ 8412.332535] [ 8412.333522] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8412.335724] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8412.337834] do_raw_spin_lock+0x177/0x190 [ 8412.339063] _raw_spin_lock+0x21/0x30 [ 8412.340629] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8412.343057] obd_mod_free+0x46/0x140 [ptlrpc] [ 8412.345066] mdc_commit_open+0x91/0xb0 [mdc] [ 8412.346970] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8412.349183] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8412.351717] after_reply+0xe2c/0x1780 [ptlrpc] [ 8412.353576] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8412.355894] ? __next_timer_interrupt+0x160/0x160 [ 8412.357914] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8412.359996] ? do_wait_intr+0xf0/0xf0 [ 8412.361677] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8412.364033] kthread+0x1d1/0x200 [ 8412.365224] ? set_kthread_struct+0x70/0x70 [ 8412.367208] ret_from_fork+0x1f/0x30 [ 8416.057010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8416.059234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8416.061038] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8416.074003] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8416.074014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8416.074020] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8416.074230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8416.074238] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 8416.074289] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8416.074293] RAX: 0000000000000f4a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8416.074294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8416.074296] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8416.074297] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8416.074299] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8416.074301] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8416.074303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8416.074305] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8416.074311] Call Trace: [ 8416.074315] [ 8416.074319] ? show_regs.cold.9+0x22/0x2f [ 8416.074326] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8416.074331] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8416.074335] ? watchdog+0x40/0x40 [ 8416.074337] ? hrtimer_interrupt+0x128/0x2c0 [ 8416.074340] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8416.074346] ? apic_timer_interrupt+0xf/0x20 [ 8416.074349] [ 8416.074352] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8416.074355] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8416.074358] do_raw_spin_lock+0x177/0x190 [ 8416.074361] _raw_spin_lock+0x21/0x30 [ 8416.074365] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8416.074518] process_one_work+0x2c8/0x700 [ 8416.074523] worker_thread+0x296/0x6e0 [ 8416.074526] ? rescuer_thread+0x570/0x570 [ 8416.074543] kthread+0x1d1/0x200 [ 8416.074606] ? set_kthread_struct+0x70/0x70 [ 8416.074611] ret_from_fork+0x1f/0x30 [ 8416.224197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8416.254932] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8416.261394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8416.266787] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8416.270994] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8416.282853] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8416.285295] RAX: 000000000000535a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8416.291410] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8416.293772] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8416.299688] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8416.307492] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8416.312455] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8416.317162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8416.319338] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8416.322349] Call Trace: [ 8416.324956] [ 8416.325782] ? show_regs.cold.9+0x22/0x2f [ 8416.328682] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8416.330393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8416.332688] ? watchdog+0x40/0x40 [ 8416.334024] ? hrtimer_interrupt+0x128/0x2c0 [ 8416.336374] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8416.338536] ? apic_timer_interrupt+0xf/0x20 [ 8416.340764] [ 8416.341860] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8416.343653] do_raw_spin_lock+0x177/0x190 [ 8416.346206] _raw_spin_lock+0x21/0x30 [ 8416.351476] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8416.355710] ? do_raw_spin_unlock+0x75/0x190 [ 8416.358713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8416.360803] ? do_wait_intr+0xf0/0xf0 [ 8416.361905] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8416.363971] mdc_close+0x53b/0xd70 [mdc] [ 8416.365472] lmv_close+0x18d/0x490 [lmv] [ 8416.366552] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8416.368147] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8416.370399] ll_md_close+0x274/0x800 [lustre] [ 8416.371677] ll_file_release+0x128/0xc50 [lustre] [ 8416.378899] __fput+0x10d/0x440 [ 8416.379965] ____fput+0x12/0x20 [ 8416.381464] task_work_run+0xc6/0x110 [ 8416.382923] exit_to_usermode_loop+0x1dd/0x1f0 [ 8416.386441] do_syscall_64+0x3d6/0x440 [ 8416.389217] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8416.391622] RIP: 0033:0x7fa749552cd8 [ 8416.393473] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8416.402173] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8416.409427] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8416.414063] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8416.428741] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8416.437439] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8416.450208] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8420.054369] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8420.058238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8420.098449] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8420.112776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8420.120394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8420.125568] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8420.144516] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8420.151330] RAX: 0000000000003004 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8420.158006] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8420.161406] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8420.167983] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8420.173839] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8420.180692] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8420.185763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8420.190022] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8420.195831] Call Trace: [ 8420.198835] [ 8420.201994] ? show_regs.cold.9+0x22/0x2f [ 8420.211058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8420.213810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8420.217271] ? watchdog+0x40/0x40 [ 8420.219056] ? hrtimer_interrupt+0x128/0x2c0 [ 8420.221007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8420.224729] ? apic_timer_interrupt+0xf/0x20 [ 8420.226150] [ 8420.226826] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8420.233050] do_raw_spin_lock+0x177/0x190 [ 8420.236979] _raw_spin_lock+0x21/0x30 [ 8420.241673] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8420.247980] ? do_raw_spin_unlock+0x75/0x190 [ 8420.251407] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8420.256680] ? do_wait_intr+0xf0/0xf0 [ 8420.259556] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8420.262093] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8420.266503] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8420.269228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8420.274701] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8420.276640] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8420.283327] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8420.288048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8420.297184] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8420.303574] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8420.309728] lmv_intent_open+0x338/0x1150 [lmv] [ 8420.313548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8420.318105] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8420.328255] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8420.330828] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8420.337034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8420.346715] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8420.348876] ? do_raw_spin_unlock+0x75/0x190 [ 8420.359489] ? _raw_spin_unlock+0x12/0x30 [ 8420.361023] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8420.371917] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8420.378826] ll_atomic_open+0x36a/0xd70 [lustre] [ 8420.382243] lookup_open+0x356/0xb40 [ 8420.387648] do_last+0x313/0xfc0 [ 8420.392861] ? nd_jump_root+0xe5/0x160 [ 8420.397262] ? path_init+0x437/0x520 [ 8420.399963] path_openat+0xf7/0x500 [ 8420.402767] ? do_raw_spin_unlock+0x75/0x190 [ 8420.407331] do_filp_open+0x99/0x140 [ 8420.408706] ? getname_flags+0x6e/0x330 [ 8420.410865] ? __check_object_size+0xff/0x256 [ 8420.413549] ? do_raw_spin_unlock+0x75/0x190 [ 8420.417755] ? _raw_spin_unlock+0x12/0x30 [ 8420.419320] do_sys_openat2+0x2b4/0x410 [ 8420.423510] do_sys_open+0x73/0xa0 [ 8420.426507] __x64_sys_openat+0x24/0x30 [ 8420.432200] do_syscall_64+0xc1/0x440 [ 8420.436538] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8420.439515] RIP: 0033:0x7fa749552332 [ 8420.442062] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8420.461308] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8420.469684] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8420.476957] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8420.482012] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8420.487657] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8420.492049] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8423.401283] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6506s! [ 8423.425316] Showing busy workqueues and worker pools: [ 8423.439670] workqueue events: flags=0x0 [ 8423.448381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8423.454290] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8423.466697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8423.485505] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8423.495841] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.502049] pending: kfree_rcu_monitor [ 8423.512988] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8423.522105] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8423.540210] workqueue events_long: flags=0x0 [ 8423.550404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.552701] pending: lu_objects_flush [obdclass] [ 8423.564636] workqueue events_unbound: flags=0x2 [ 8423.573217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8423.581669] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8423.590296] workqueue events_power_efficient: flags=0x80 [ 8423.599668] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.608460] pending: check_lifetime [ 8423.618123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.627624] pending: neigh_periodic_work [ 8423.630072] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.633198] pending: neigh_periodic_work [ 8423.635378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.639358] pending: do_cache_clean [ 8423.642303] workqueue mm_percpu_wq: flags=0x8 [ 8423.646537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.654166] pending: vmstat_update [ 8423.655850] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.661801] pending: vmstat_update [ 8423.665555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8423.674750] pending: vmstat_update [ 8423.677299] workqueue writeback: flags=0x4a [ 8423.682283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8423.685808] pending: wb_workfn, wb_workfn [ 8423.691537] workqueue xprtiod: flags=0xa [ 8423.694238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8423.698439] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8423.705077] workqueue ipv6_addrconf: flags=0x40008 [ 8423.711059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8423.713262] pending: addrconf_verify_work [ 8423.717930] workqueue knbd0-recv: flags=0x1a [ 8423.719535] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8423.723460] in-flight: 651:recv_work [ 8423.726330] workqueue ptlrpc_pinger: flags=0x2 [ 8423.730023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8423.733213] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8423.736655] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6506s workers=4 idle: 37 31846 28897 [ 8423.742455] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8413s workers=2 idle: 653 [ 8440.058022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8440.060372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8440.081409] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8440.086026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8440.093429] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8440.096136] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8440.104473] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8440.107782] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8440.110774] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8440.113526] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8440.119015] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8440.122403] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8440.125436] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8440.128996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8440.131103] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8440.134014] Call Trace: [ 8440.135035] [ 8440.135957] ? show_regs.cold.9+0x22/0x2f [ 8440.141570] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8440.144310] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8440.146195] ? watchdog+0x40/0x40 [ 8440.147975] ? hrtimer_interrupt+0x128/0x2c0 [ 8440.149270] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8440.153962] ? apic_timer_interrupt+0xf/0x20 [ 8440.156077] [ 8440.157036] ? native_safe_halt+0x2e/0x50 [ 8440.158920] kvm_wait+0xcd/0xe0 [ 8440.161237] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8440.163594] do_raw_spin_lock+0x177/0x190 [ 8440.165584] _raw_spin_lock+0x21/0x30 [ 8440.167686] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8440.173510] obd_mod_free+0x46/0x140 [ptlrpc] [ 8440.180220] mdc_commit_open+0x91/0xb0 [mdc] [ 8440.183588] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8440.188998] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8440.192956] after_reply+0xe2c/0x1780 [ptlrpc] [ 8440.195380] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8440.201499] ? __next_timer_interrupt+0x160/0x160 [ 8440.204570] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8440.207803] ? do_wait_intr+0xf0/0xf0 [ 8440.208721] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8440.210366] kthread+0x1d1/0x200 [ 8440.211285] ? set_kthread_struct+0x70/0x70 [ 8440.212519] ret_from_fork+0x1f/0x30 [ 8444.057761] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8444.059199] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8444.060690] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 8444.063394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8444.063470] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8444.063478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8444.063480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8444.063494] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8444.063498] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8444.063503] RAX: 0000000000001c9c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8444.063505] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8444.063507] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8444.063510] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8444.063513] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8444.063516] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8444.063519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8444.063522] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8444.063530] Call Trace: [ 8444.063535] [ 8444.063540] ? show_regs.cold.9+0x22/0x2f [ 8444.063550] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8444.063556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8444.063560] ? watchdog+0x40/0x40 [ 8444.063565] ? hrtimer_interrupt+0x128/0x2c0 [ 8444.063569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8444.063577] ? apic_timer_interrupt+0xf/0x20 [ 8444.063581] [ 8444.063584] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8444.063591] do_raw_spin_lock+0x177/0x190 [ 8444.063596] _raw_spin_lock+0x21/0x30 [ 8444.063601] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8444.063876] ? do_raw_spin_unlock+0x75/0x190 [ 8444.063882] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8444.064171] ? do_wait_intr+0xf0/0xf0 [ 8444.064180] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8444.080623] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8444.080677] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8444.080684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8444.080691] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8444.106901] mdc_close+0x53b/0xd70 [mdc] [ 8444.117978] [ 8444.117986] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8444.118003] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8444.118008] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8444.118013] RAX: 0000000000007e85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8444.118016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8444.118018] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8444.118020] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8444.118023] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8444.118026] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8444.118028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8444.118031] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8444.118048] Call Trace: [ 8444.118054] [ 8444.118060] ? show_regs.cold.9+0x22/0x2f [ 8444.118070] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8444.118077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8444.118083] ? watchdog+0x40/0x40 [ 8444.118086] ? hrtimer_interrupt+0x128/0x2c0 [ 8444.118090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8444.118646] ? apic_timer_interrupt+0xf/0x20 [ 8444.118652] [ 8444.118655] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8444.118661] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8444.118668] do_raw_spin_lock+0x177/0x190 [ 8444.118673] _raw_spin_lock+0x21/0x30 [ 8444.118677] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8444.119493] process_one_work+0x2c8/0x700 [ 8444.119502] worker_thread+0x296/0x6e0 [ 8444.119507] ? rescuer_thread+0x570/0x570 [ 8444.119510] kthread+0x1d1/0x200 [ 8444.119514] ? set_kthread_struct+0x70/0x70 [ 8444.119517] ret_from_fork+0x1f/0x30 [ 8444.347954] lmv_close+0x18d/0x490 [lmv] [ 8444.349634] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8444.357129] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8444.361638] ll_md_close+0x274/0x800 [lustre] [ 8444.364695] ll_file_release+0x128/0xc50 [lustre] [ 8444.369387] __fput+0x10d/0x440 [ 8444.370724] ____fput+0x12/0x20 [ 8444.375618] task_work_run+0xc6/0x110 [ 8444.377740] exit_to_usermode_loop+0x1dd/0x1f0 [ 8444.380515] do_syscall_64+0x3d6/0x440 [ 8444.383531] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8444.389344] RIP: 0033:0x7fa749552cd8 [ 8444.394604] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8444.403838] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8444.412257] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8444.418236] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8444.424182] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8444.430200] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8444.436357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8448.057424] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8448.063340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8448.109958] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8448.116507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8448.120401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8448.123073] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8448.131176] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8448.134336] RAX: 0000000000001085 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8448.137544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8448.140505] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8448.143530] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8448.146991] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8448.150786] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8448.156461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8448.159627] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8448.170217] Call Trace: [ 8448.171752] [ 8448.173944] ? show_regs.cold.9+0x22/0x2f [ 8448.178248] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8448.187368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8448.193036] ? watchdog+0x40/0x40 [ 8448.195349] ? hrtimer_interrupt+0x128/0x2c0 [ 8448.197727] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8448.202282] ? apic_timer_interrupt+0xf/0x20 [ 8448.206445] [ 8448.207531] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8448.214425] do_raw_spin_lock+0x177/0x190 [ 8448.219626] _raw_spin_lock+0x21/0x30 [ 8448.224093] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8448.231218] ? do_raw_spin_unlock+0x75/0x190 [ 8448.237989] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8448.245406] ? do_wait_intr+0xf0/0xf0 [ 8448.248638] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8448.251986] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8448.256039] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8448.263781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8448.269006] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8448.276057] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8448.284168] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8448.288123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8448.293841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8448.301516] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8448.308475] lmv_intent_open+0x338/0x1150 [lmv] [ 8448.316199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8448.322995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8448.329799] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8448.334483] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8448.336650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8448.339251] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8448.341201] ? do_raw_spin_unlock+0x75/0x190 [ 8448.343843] ? _raw_spin_unlock+0x12/0x30 [ 8448.349453] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8448.351440] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8448.353903] ll_atomic_open+0x36a/0xd70 [lustre] [ 8448.357209] lookup_open+0x356/0xb40 [ 8448.359380] do_last+0x313/0xfc0 [ 8448.364826] ? nd_jump_root+0xe5/0x160 [ 8448.368853] ? path_init+0x437/0x520 [ 8448.370918] path_openat+0xf7/0x500 [ 8448.373168] ? do_raw_spin_unlock+0x75/0x190 [ 8448.375323] do_filp_open+0x99/0x140 [ 8448.378631] ? getname_flags+0x6e/0x330 [ 8448.382624] ? __check_object_size+0xff/0x256 [ 8448.384614] ? do_raw_spin_unlock+0x75/0x190 [ 8448.386579] ? _raw_spin_unlock+0x12/0x30 [ 8448.389092] do_sys_openat2+0x2b4/0x410 [ 8448.391170] do_sys_open+0x73/0xa0 [ 8448.392609] __x64_sys_openat+0x24/0x30 [ 8448.394509] do_syscall_64+0xc1/0x440 [ 8448.401271] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8448.408653] RIP: 0033:0x7fa749552332 [ 8448.414451] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8448.434752] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8448.437650] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8448.440558] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8448.447141] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8448.457034] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8448.460660] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8453.831119] rcu: INFO: rcu_sched self-detected stall on CPU [ 8453.833986] rcu: 2-...!: (3662126 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 8453.838271] (t=6540142 jiffies g=258701 q=3378) [ 8453.842125] rcu: rcu_sched kthread starved for 6536937 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8453.851503] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8453.856234] rcu: RCU grace-period kthread stack dump: [ 8453.861003] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8453.874614] Call Trace: [ 8453.876849] __schedule+0x351/0xcb0 [ 8453.880244] schedule+0xc0/0x180 [ 8453.886038] schedule_timeout+0xb4/0x190 [ 8453.890543] ? do_raw_spin_unlock+0x75/0x190 [ 8453.894541] ? __next_timer_interrupt+0x160/0x160 [ 8453.902397] ? prepare_to_swait_event+0xf3/0x140 [ 8453.909157] rcu_gp_kthread+0x8dd/0x15b0 [ 8453.916369] ? force_qs_rnp+0x2c0/0x2c0 [ 8453.917715] kthread+0x1d1/0x200 [ 8453.926290] ? set_kthread_struct+0x70/0x70 [ 8453.933245] ret_from_fork+0x1f/0x30 [ 8453.939757] rcu: Stack dump where RCU GP kthread last ran: [ 8453.944077] NMI backtrace for cpu 2 [ 8453.948052] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8453.962350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8453.970769] Call Trace: [ 8453.973356] [ 8453.974666] ? dump_stack+0xbb/0x10e [ 8453.979140] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8453.983105] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8453.987293] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8453.990197] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8453.993851] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8454.000659] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8454.004943] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 8454.007307] ? update_process_times+0x6d/0xc0 [ 8454.008812] ? tick_sched_handle.isra.18+0x27/0x90 [ 8454.013503] ? tick_sched_timer+0x53/0xb0 [ 8454.015370] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8454.020723] ? tick_sched_handle.isra.18+0x90/0x90 [ 8454.023165] ? hrtimer_interrupt+0x128/0x2c0 [ 8454.024660] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8454.031131] ? apic_timer_interrupt+0xf/0x20 [ 8454.033209] [ 8454.038282] ? native_safe_halt+0x2e/0x50 [ 8454.040487] ? kvm_wait+0xcd/0xe0 [ 8454.044304] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8454.050849] ? do_raw_spin_lock+0x177/0x190 [ 8454.052719] ? _raw_spin_lock+0x21/0x30 [ 8454.056492] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8454.067842] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8454.076638] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8454.078157] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8454.086761] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8454.092088] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8454.098381] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8454.104444] ? __next_timer_interrupt+0x160/0x160 [ 8454.113076] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8454.117717] ? do_wait_intr+0xf0/0xf0 [ 8454.120893] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8454.129129] ? kthread+0x1d1/0x200 [ 8454.132255] ? set_kthread_struct+0x70/0x70 [ 8454.138316] ? ret_from_fork+0x1f/0x30 [ 8454.140952] NMI backtrace for cpu 2 [ 8454.147835] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8454.156813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8454.167303] Call Trace: [ 8454.168255] [ 8454.169037] ? dump_stack+0xbb/0x10e [ 8454.172270] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8454.177702] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8454.182521] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8454.189448] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8454.194371] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8454.196175] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8454.198924] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8454.201949] ? update_process_times+0x6d/0xc0 [ 8454.204485] ? tick_sched_handle.isra.18+0x27/0x90 [ 8454.212487] ? tick_sched_timer+0x53/0xb0 [ 8454.218173] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8454.223957] ? tick_sched_handle.isra.18+0x90/0x90 [ 8454.229245] ? hrtimer_interrupt+0x128/0x2c0 [ 8454.232659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8454.235100] ? apic_timer_interrupt+0xf/0x20 [ 8454.237731] [ 8454.238828] ? native_safe_halt+0x2e/0x50 [ 8454.240193] ? kvm_wait+0xcd/0xe0 [ 8454.241894] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8454.244776] ? do_raw_spin_lock+0x177/0x190 [ 8454.247311] ? _raw_spin_lock+0x21/0x30 [ 8454.248586] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8454.250966] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8454.253332] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8454.255262] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8454.257552] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8454.261918] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8454.265921] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8454.269306] ? __next_timer_interrupt+0x160/0x160 [ 8454.271754] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8454.276256] ? do_wait_intr+0xf0/0xf0 [ 8454.278047] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8454.281090] ? kthread+0x1d1/0x200 [ 8454.283111] ? set_kthread_struct+0x70/0x70 [ 8454.284576] ? ret_from_fork+0x1f/0x30 [ 8454.287430] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6537s! [ 8454.289444] Showing busy workqueues and worker pools: [ 8454.291335] workqueue events: flags=0x0 [ 8454.292575] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8454.295706] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8454.300666] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8454.303330] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8454.311231] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.316403] pending: kfree_rcu_monitor [ 8454.317820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8454.320231] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8454.325816] workqueue events_long: flags=0x0 [ 8454.327747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.331320] pending: lu_objects_flush [obdclass] [ 8454.336885] workqueue events_unbound: flags=0x2 [ 8454.339771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8454.343054] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8454.349068] workqueue events_power_efficient: flags=0x80 [ 8454.353219] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.358294] pending: check_lifetime [ 8454.363515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.375055] pending: neigh_periodic_work [ 8454.380165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.386814] pending: neigh_periodic_work [ 8454.391853] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.399111] pending: do_cache_clean [ 8454.402066] workqueue mm_percpu_wq: flags=0x8 [ 8454.404547] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.409830] pending: vmstat_update [ 8454.416439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.424549] pending: vmstat_update [ 8454.427226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8454.431070] pending: vmstat_update [ 8454.435697] workqueue writeback: flags=0x4a [ 8454.438167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8454.450169] pending: wb_workfn, wb_workfn [ 8454.453127] workqueue xprtiod: flags=0xa [ 8454.456333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8454.463240] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8454.467482] workqueue ipv6_addrconf: flags=0x40008 [ 8454.469838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8454.472557] pending: addrconf_verify_work [ 8454.475556] workqueue knbd0-recv: flags=0x1a [ 8454.480538] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8454.486536] in-flight: 651:recv_work [ 8454.495447] workqueue ptlrpc_pinger: flags=0x2 [ 8454.505389] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8454.514098] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8454.517749] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6537s workers=4 idle: 37 31846 28897 [ 8454.521861] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8444s workers=2 idle: 653 [ 8472.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8472.059029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8472.059902] Modules linked in: [ 8472.064847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8472.064925] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8472.064932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8472.064935] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8472.064949] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8472.064953] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8472.064958] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8472.064960] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8472.064963] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8472.064965] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8472.064967] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8472.064970] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8472.064974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8472.064976] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8472.064985] Call Trace: [ 8472.064991] [ 8472.064996] ? show_regs.cold.9+0x22/0x2f [ 8472.065006] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8472.065012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8472.065017] ? watchdog+0x40/0x40 [ 8472.065021] ? hrtimer_interrupt+0x128/0x2c0 [ 8472.065025] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8472.065032] ? apic_timer_interrupt+0xf/0x20 [ 8472.065036] [ 8472.065039] ? native_safe_halt+0x2e/0x50 [ 8472.065042] kvm_wait+0xcd/0xe0 [ 8472.065050] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8472.065058] do_raw_spin_lock+0x177/0x190 [ 8472.065063] _raw_spin_lock+0x21/0x30 [ 8472.065067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8472.065392] ? do_raw_spin_unlock+0x75/0x190 [ 8472.065398] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8472.065920] ? do_wait_intr+0xf0/0xf0 [ 8472.065933] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8472.068096] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8472.068182] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8472.068187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8472.068193] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8472.068585] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8472.068594] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8472.068601] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8472.068605] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8472.068608] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8472.068610] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8472.068612] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8472.068618] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8472.068621] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8472.068624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8472.068626] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8472.068635] Call Trace: [ 8472.068741] [ 8472.068748] ? show_regs.cold.9+0x22/0x2f [ 8472.068757] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8472.068766] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8472.068771] ? watchdog+0x40/0x40 [ 8472.068840] ? hrtimer_interrupt+0x128/0x2c0 [ 8472.068844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8472.068851] ? apic_timer_interrupt+0xf/0x20 [ 8472.068855] [ 8472.068858] ? native_safe_halt+0x2e/0x50 [ 8472.068862] kvm_wait+0xcd/0xe0 [ 8472.068869] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8472.068877] do_raw_spin_lock+0x177/0x190 [ 8472.068881] _raw_spin_lock+0x21/0x30 [ 8472.068885] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8472.069193] process_one_work+0x2c8/0x700 [ 8472.069199] worker_thread+0x296/0x6e0 [ 8472.069203] ? rescuer_thread+0x570/0x570 [ 8472.069207] kthread+0x1d1/0x200 [ 8472.069211] ? set_kthread_struct+0x70/0x70 [ 8472.069215] ret_from_fork+0x1f/0x30 [ 8472.303205] mdc_close+0x53b/0xd70 [mdc] [ 8472.304453] lmv_close+0x18d/0x490 [lmv] [ 8472.305674] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8472.307559] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8472.308994] ll_md_close+0x274/0x800 [lustre] [ 8472.310381] ll_file_release+0x128/0xc50 [lustre] [ 8472.314500] __fput+0x10d/0x440 [ 8472.315921] ____fput+0x12/0x20 [ 8472.325758] task_work_run+0xc6/0x110 [ 8472.331458] exit_to_usermode_loop+0x1dd/0x1f0 [ 8472.336661] do_syscall_64+0x3d6/0x440 [ 8472.339815] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8472.342137] RIP: 0033:0x7fa749552cd8 [ 8472.346880] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8472.363846] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8472.367993] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8472.371170] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8472.374225] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8472.377951] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8472.381310] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8480.057330] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8480.058085] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8480.065093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix [ 8480.067757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.067832] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.067839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.067841] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8480.067913] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8480.067916] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8480.067921] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8480.067923] RDX: 00000000000006dc RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8480.067925] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8480.067927] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8480.067930] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8480.067932] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8480.067935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.067937] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8480.067945] Call Trace: [ 8480.067951] [ 8480.067956] ? show_regs.cold.9+0x22/0x2f [ 8480.067966] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.067971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.067974] ? watchdog+0x40/0x40 [ 8480.067978] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.067981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.067988] ? apic_timer_interrupt+0xf/0x20 [ 8480.067992] [ 8480.067994] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8480.067999] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8480.068003] do_raw_spin_lock+0x177/0x190 [ 8480.068008] _raw_spin_lock+0x21/0x30 [ 8480.068013] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8480.068667] obd_mod_free+0x46/0x140 [ptlrpc] [ 8480.069197] mdc_commit_open+0x91/0xb0 [mdc] [ 8480.069234] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8480.069466] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8480.070491] after_reply+0xe2c/0x1780 [ptlrpc] [ 8480.071153] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8480.071393] ? __next_timer_interrupt+0x160/0x160 [ 8480.071402] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8480.071628] ? do_wait_intr+0xf0/0xf0 [ 8480.071638] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8480.071962] kthread+0x1d1/0x200 [ 8480.071968] ? set_kthread_struct+0x70/0x70 [ 8480.071971] ret_from_fork+0x1f/0x30 [ 8480.383411] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.387125] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.394304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.398791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.403781] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8480.412031] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8480.418733] RAX: 0000000000007e89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8480.422699] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8480.425736] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8480.428991] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8480.432059] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8480.435338] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8480.438919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.441384] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8480.445383] Call Trace: [ 8480.446290] [ 8480.453980] ? show_regs.cold.9+0x22/0x2f [ 8480.456472] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.465025] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.470934] ? watchdog+0x40/0x40 [ 8480.473514] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.475031] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.481250] ? apic_timer_interrupt+0xf/0x20 [ 8480.482771] [ 8480.484416] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.487992] do_raw_spin_lock+0x177/0x190 [ 8480.490936] _raw_spin_lock+0x21/0x30 [ 8480.495348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8480.501669] ? do_raw_spin_unlock+0x75/0x190 [ 8480.506019] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8480.509581] ? do_wait_intr+0xf0/0xf0 [ 8480.515816] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8480.519558] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8480.526708] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8480.530681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.535924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8480.537775] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8480.541059] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8480.547056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.549718] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8480.555561] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8480.557762] lmv_intent_open+0x338/0x1150 [lmv] [ 8480.559858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.562270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.564904] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8480.566476] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8480.568083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.570903] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8480.572327] ? do_raw_spin_unlock+0x75/0x190 [ 8480.575915] ? _raw_spin_unlock+0x12/0x30 [ 8480.577817] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8480.579802] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8480.583101] ll_atomic_open+0x36a/0xd70 [lustre] [ 8480.586532] lookup_open+0x356/0xb40 [ 8480.587892] do_last+0x313/0xfc0 [ 8480.590832] ? nd_jump_root+0xe5/0x160 [ 8480.597719] ? path_init+0x437/0x520 [ 8480.600519] path_openat+0xf7/0x500 [ 8480.602222] ? do_raw_spin_unlock+0x75/0x190 [ 8480.606065] do_filp_open+0x99/0x140 [ 8480.607283] ? getname_flags+0x6e/0x330 [ 8480.608890] ? __check_object_size+0xff/0x256 [ 8480.610797] ? do_raw_spin_unlock+0x75/0x190 [ 8480.613162] ? _raw_spin_unlock+0x12/0x30 [ 8480.616091] do_sys_openat2+0x2b4/0x410 [ 8480.620767] do_sys_open+0x73/0xa0 [ 8480.622745] __x64_sys_openat+0x24/0x30 [ 8480.625171] do_syscall_64+0xc1/0x440 [ 8480.627285] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8480.628864] RIP: 0033:0x7fa749552332 [ 8480.634418] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8480.649190] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8480.656016] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8480.663863] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8480.667556] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8480.670470] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8480.674802] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8484.833333] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6567s! [ 8484.844384] Showing busy workqueues and worker pools: [ 8484.846797] workqueue events: flags=0x0 [ 8484.848977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8484.852860] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8484.860647] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8484.865618] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8484.871947] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.877386] pending: kfree_rcu_monitor [ 8484.879365] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8484.883457] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8484.888622] workqueue events_long: flags=0x0 [ 8484.891585] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.896391] pending: lu_objects_flush [obdclass] [ 8484.899768] workqueue events_unbound: flags=0x2 [ 8484.902229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8484.905833] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8484.911990] workqueue events_power_efficient: flags=0x80 [ 8484.914677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.918104] pending: check_lifetime [ 8484.920473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.924528] pending: neigh_periodic_work [ 8484.926486] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.930983] pending: neigh_periodic_work [ 8484.933244] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.936763] pending: do_cache_clean [ 8484.938713] workqueue mm_percpu_wq: flags=0x8 [ 8484.940333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.943146] pending: vmstat_update [ 8484.944658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.949945] pending: vmstat_update [ 8484.952558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8484.956160] pending: vmstat_update [ 8484.959421] workqueue writeback: flags=0x4a [ 8484.962497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8484.965441] pending: wb_workfn, wb_workfn [ 8484.969784] workqueue xprtiod: flags=0xa [ 8484.972435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8484.976846] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8484.985143] workqueue ipv6_addrconf: flags=0x40008 [ 8484.987429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8484.991871] pending: addrconf_verify_work [ 8484.994575] workqueue knbd0-recv: flags=0x1a [ 8484.996704] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8485.000022] in-flight: 651:recv_work [ 8485.001748] workqueue ptlrpc_pinger: flags=0x2 [ 8485.005025] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8485.007851] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8485.011950] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6567s workers=4 idle: 37 31846 28897 [ 8485.016027] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8474s workers=2 idle: 653 [ 8500.058248] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8500.059100] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8500.072141] Modules linked in: lustre(O) [ 8500.075168] Modules linked in: [ 8500.075171] osp(O) ofd(O) [ 8500.076758] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8500.076823] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8500.076830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8500.076833] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8500.076846] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8500.076851] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8500.076855] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8500.076858] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8500.076860] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8500.076862] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8500.076865] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8500.076868] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8500.076870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8500.076873] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8500.076882] Call Trace: [ 8500.076888] [ 8500.076895] ? show_regs.cold.9+0x22/0x2f [ 8500.076906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8500.076912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8500.076917] ? watchdog+0x40/0x40 [ 8500.076921] ? hrtimer_interrupt+0x128/0x2c0 [ 8500.076925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8500.076932] ? apic_timer_interrupt+0xf/0x20 [ 8500.076937] [ 8500.076940] ? native_safe_halt+0x2e/0x50 [ 8500.076944] kvm_wait+0xcd/0xe0 [ 8500.076952] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8500.076962] do_raw_spin_lock+0x177/0x190 [ 8500.076967] _raw_spin_lock+0x21/0x30 [ 8500.076972] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8500.077326] ? do_raw_spin_unlock+0x75/0x190 [ 8500.077333] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8500.077525] ? do_wait_intr+0xf0/0xf0 [ 8500.077534] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8500.077727] mdc_close+0x53b/0xd70 [mdc] [ 8500.077761] lmv_close+0x18d/0x490 [lmv] [ 8500.077788] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8500.077896] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8500.077976] ll_md_close+0x274/0x800 [lustre] [ 8500.078130] ll_file_release+0x128/0xc50 [lustre] [ 8500.078213] __fput+0x10d/0x440 [ 8500.078223] ____fput+0x12/0x20 [ 8500.078227] task_work_run+0xc6/0x110 [ 8500.078234] exit_to_usermode_loop+0x1dd/0x1f0 [ 8500.078241] do_syscall_64+0x3d6/0x440 [ 8500.078246] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8500.078251] RIP: 0033:0x7fa749552cd8 [ 8500.078258] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8500.078262] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8500.078266] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8500.078268] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8500.078270] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8500.078273] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8500.078275] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8500.606668] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8500.653702] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8500.671645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8500.674805] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8500.676786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8500.678861] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8500.685916] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8500.688731] RAX: 0000000000007f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8500.691534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8500.694520] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8500.697422] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8500.702281] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8500.708741] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8500.714922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8500.717426] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8500.720857] Call Trace: [ 8500.721965] [ 8500.722815] ? show_regs.cold.9+0x22/0x2f [ 8500.724259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8500.726272] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8500.728054] ? watchdog+0x40/0x40 [ 8500.729569] ? hrtimer_interrupt+0x128/0x2c0 [ 8500.731237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8500.733138] ? apic_timer_interrupt+0xf/0x20 [ 8500.735023] [ 8500.735959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8500.738104] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8500.740551] do_raw_spin_lock+0x177/0x190 [ 8500.742219] _raw_spin_lock+0x21/0x30 [ 8500.744131] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8500.746585] process_one_work+0x2c8/0x700 [ 8500.748188] worker_thread+0x296/0x6e0 [ 8500.749647] ? rescuer_thread+0x570/0x570 [ 8500.751248] kthread+0x1d1/0x200 [ 8500.752581] ? set_kthread_struct+0x70/0x70 [ 8500.756623] ret_from_fork+0x1f/0x30 [ 8508.068017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8508.077743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.121770] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.129143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.134945] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8508.138489] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8508.148448] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8508.152373] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8508.157977] RDX: 0000000000004d61 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8508.162720] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8508.167036] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8508.179428] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8508.185012] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8508.196622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.202085] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8508.209445] Call Trace: [ 8508.210902] [ 8508.214146] ? show_regs.cold.9+0x22/0x2f [ 8508.216406] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.220985] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.228366] ? watchdog+0x40/0x40 [ 8508.233059] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.237549] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.245705] ? apic_timer_interrupt+0xf/0x20 [ 8508.250676] [ 8508.257436] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8508.262093] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8508.273238] do_raw_spin_lock+0x177/0x190 [ 8508.279506] _raw_spin_lock+0x21/0x30 [ 8508.281935] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8508.291089] obd_mod_free+0x46/0x140 [ptlrpc] [ 8508.296944] mdc_commit_open+0x91/0xb0 [mdc] [ 8508.301930] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8508.308684] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8508.313965] after_reply+0xe2c/0x1780 [ptlrpc] [ 8508.315797] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8508.322572] ? __next_timer_interrupt+0x160/0x160 [ 8508.326304] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8508.328863] ? do_wait_intr+0xf0/0xf0 [ 8508.332668] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8508.337429] kthread+0x1d1/0x200 [ 8508.340304] ? set_kthread_struct+0x70/0x70 [ 8508.343125] ret_from_fork+0x1f/0x30 [ 8512.062022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8512.068782] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8512.111837] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8512.126227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8512.133317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8512.140280] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8512.162444] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8512.168143] RAX: 0000000000000210 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8512.174810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8512.182228] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8512.187355] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8512.196921] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8512.200484] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8512.208105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8512.216019] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8512.223401] Call Trace: [ 8512.224420] [ 8512.228300] ? show_regs.cold.9+0x22/0x2f [ 8512.234159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8512.240451] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8512.250386] ? watchdog+0x40/0x40 [ 8512.256063] ? hrtimer_interrupt+0x128/0x2c0 [ 8512.265522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8512.272699] ? apic_timer_interrupt+0xf/0x20 [ 8512.278212] [ 8512.279029] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8512.281534] do_raw_spin_lock+0x177/0x190 [ 8512.283304] _raw_spin_lock+0x21/0x30 [ 8512.285146] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8512.291000] ? do_raw_spin_unlock+0x75/0x190 [ 8512.296350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8512.304337] ? do_wait_intr+0xf0/0xf0 [ 8512.305742] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8512.308565] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8512.311489] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8512.314750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8512.316821] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8512.319279] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8512.321191] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8512.323290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8512.325464] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8512.328790] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8512.331084] lmv_intent_open+0x338/0x1150 [lmv] [ 8512.334192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8512.337016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8512.339541] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8512.342168] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8512.343801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8512.348579] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8512.351508] ? do_raw_spin_unlock+0x75/0x190 [ 8512.355067] ? _raw_spin_unlock+0x12/0x30 [ 8512.359932] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8512.363559] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8512.370141] ll_atomic_open+0x36a/0xd70 [lustre] [ 8512.373733] lookup_open+0x356/0xb40 [ 8512.375162] do_last+0x313/0xfc0 [ 8512.376259] ? nd_jump_root+0xe5/0x160 [ 8512.377237] ? path_init+0x437/0x520 [ 8512.378500] path_openat+0xf7/0x500 [ 8512.379792] ? do_raw_spin_unlock+0x75/0x190 [ 8512.380945] do_filp_open+0x99/0x140 [ 8512.382018] ? getname_flags+0x6e/0x330 [ 8512.383453] ? __check_object_size+0xff/0x256 [ 8512.385064] ? do_raw_spin_unlock+0x75/0x190 [ 8512.386812] ? _raw_spin_unlock+0x12/0x30 [ 8512.387912] do_sys_openat2+0x2b4/0x410 [ 8512.389018] do_sys_open+0x73/0xa0 [ 8512.389957] __x64_sys_openat+0x24/0x30 [ 8512.391315] do_syscall_64+0xc1/0x440 [ 8512.392982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8512.395810] RIP: 0033:0x7fa749552332 [ 8512.397149] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8512.406127] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8512.410970] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8512.418971] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8512.425217] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8512.430241] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8512.433449] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8515.041049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6597s! [ 8515.046453] Showing busy workqueues and worker pools: [ 8515.049462] workqueue events: flags=0x0 [ 8515.051459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8515.054536] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8515.066542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8515.071138] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8515.081500] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.087416] pending: kfree_rcu_monitor [ 8515.091155] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8515.101538] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8515.113411] workqueue events_long: flags=0x0 [ 8515.116624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.121547] pending: lu_objects_flush [obdclass] [ 8515.126679] workqueue events_unbound: flags=0x2 [ 8515.132128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8515.138390] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8515.148519] workqueue events_power_efficient: flags=0x80 [ 8515.156445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.160134] pending: check_lifetime [ 8515.163378] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.171209] pending: neigh_periodic_work [ 8515.176056] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.178252] pending: neigh_periodic_work [ 8515.182013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.188171] pending: do_cache_clean [ 8515.195333] workqueue mm_percpu_wq: flags=0x8 [ 8515.199214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.204904] pending: vmstat_update [ 8515.207164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.212868] pending: vmstat_update [ 8515.217165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8515.220232] pending: vmstat_update [ 8515.221761] workqueue writeback: flags=0x4a [ 8515.223326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8515.225360] pending: wb_workfn, wb_workfn [ 8515.227695] workqueue xprtiod: flags=0xa [ 8515.229030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8515.231584] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8515.234842] workqueue ipv6_addrconf: flags=0x40008 [ 8515.236182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8515.238165] pending: addrconf_verify_work [ 8515.239481] workqueue knbd0-recv: flags=0x1a [ 8515.241185] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8515.243702] in-flight: 651:recv_work [ 8515.245065] workqueue ptlrpc_pinger: flags=0x2 [ 8515.246486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8515.248854] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8515.250811] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6598s workers=4 idle: 37 31846 28897 [ 8515.253989] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8505s workers=2 idle: 653 [ 8528.060330] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8528.062738] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8528.070822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8528.071090] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8528.071097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8528.071103] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8528.071533] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8528.071543] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8528.071548] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8528.071552] RAX: 0000000000005816 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8528.071555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8528.071557] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8528.071560] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8528.071573] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8528.071577] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8528.071580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8528.071582] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8528.071589] Call Trace: [ 8528.071594] [ 8528.071599] ? show_regs.cold.9+0x22/0x2f [ 8528.071609] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8528.071615] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8528.071620] ? watchdog+0x40/0x40 [ 8528.071624] ? hrtimer_interrupt+0x128/0x2c0 [ 8528.071628] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8528.072336] ? apic_timer_interrupt+0xf/0x20 [ 8528.072343] [ 8528.072346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8528.072351] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8528.072356] do_raw_spin_lock+0x177/0x190 [ 8528.072361] _raw_spin_lock+0x21/0x30 [ 8528.072366] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8528.073784] process_one_work+0x2c8/0x700 [ 8528.073799] worker_thread+0x296/0x6e0 [ 8528.073804] ? rescuer_thread+0x570/0x570 [ 8528.073808] kthread+0x1d1/0x200 [ 8528.073813] ? set_kthread_struct+0x70/0x70 [ 8528.073817] ret_from_fork+0x1f/0x30 [ 8528.217734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8528.274357] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8528.289132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8528.293657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8528.296414] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8528.306228] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8528.315106] RAX: 00000000000039a0 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8528.319961] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8528.331051] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8528.340815] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8528.345497] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8528.349427] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8528.354144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8528.356701] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8528.360469] Call Trace: [ 8528.362295] [ 8528.363498] ? show_regs.cold.9+0x22/0x2f [ 8528.366163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8528.368860] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8528.377546] ? watchdog+0x40/0x40 [ 8528.379533] ? hrtimer_interrupt+0x128/0x2c0 [ 8528.385415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8528.390063] ? apic_timer_interrupt+0xf/0x20 [ 8528.395553] [ 8528.397154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8528.399589] do_raw_spin_lock+0x177/0x190 [ 8528.405521] _raw_spin_lock+0x21/0x30 [ 8528.407735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8528.410910] ? do_raw_spin_unlock+0x75/0x190 [ 8528.412673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8528.414600] ? do_wait_intr+0xf0/0xf0 [ 8528.417434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8528.420146] mdc_close+0x53b/0xd70 [mdc] [ 8528.423285] lmv_close+0x18d/0x490 [lmv] [ 8528.424736] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8528.427316] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8528.430083] ll_md_close+0x274/0x800 [lustre] [ 8528.432526] ll_file_release+0x128/0xc50 [lustre] [ 8528.435000] __fput+0x10d/0x440 [ 8528.436628] ____fput+0x12/0x20 [ 8528.438235] task_work_run+0xc6/0x110 [ 8528.439699] exit_to_usermode_loop+0x1dd/0x1f0 [ 8528.442154] do_syscall_64+0x3d6/0x440 [ 8528.444254] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8528.446469] RIP: 0033:0x7fa749552cd8 [ 8528.448065] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8528.453417] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8528.455189] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8528.457065] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8528.458796] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8528.460576] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8528.462565] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8536.067723] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8536.081594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8536.125605] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8536.135397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8536.140556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8536.146098] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8536.165437] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8536.174843] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8536.179504] RDX: 00000000000047c0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8536.186728] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8536.191521] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8536.197352] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8536.208620] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8536.213924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8536.221597] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8536.228954] Call Trace: [ 8536.230430] [ 8536.231556] ? show_regs.cold.9+0x22/0x2f [ 8536.234339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8536.237518] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8536.241171] ? watchdog+0x40/0x40 [ 8536.242276] ? hrtimer_interrupt+0x128/0x2c0 [ 8536.246596] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8536.254198] ? apic_timer_interrupt+0xf/0x20 [ 8536.259513] [ 8536.262903] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8536.267053] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8536.276602] do_raw_spin_lock+0x177/0x190 [ 8536.281855] _raw_spin_lock+0x21/0x30 [ 8536.286055] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8536.295973] obd_mod_free+0x46/0x140 [ptlrpc] [ 8536.303127] mdc_commit_open+0x91/0xb0 [mdc] [ 8536.309000] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8536.318512] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8536.325822] after_reply+0xe2c/0x1780 [ptlrpc] [ 8536.335458] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8536.344758] ? __next_timer_interrupt+0x160/0x160 [ 8536.355518] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8536.357205] ? do_wait_intr+0xf0/0xf0 [ 8536.365401] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8536.367406] kthread+0x1d1/0x200 [ 8536.371414] ? set_kthread_struct+0x70/0x70 [ 8536.379186] ret_from_fork+0x1f/0x30 [ 8540.054166] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8540.057013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8540.093901] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8540.114971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8540.123295] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8540.125488] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8540.133668] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8540.137452] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8540.140453] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8540.144566] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8540.149739] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8540.154804] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8540.159113] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8540.164304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8540.166071] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8540.169778] Call Trace: [ 8540.170812] [ 8540.173073] ? show_regs.cold.9+0x22/0x2f [ 8540.174562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8540.176866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8540.179283] ? watchdog+0x40/0x40 [ 8540.180560] ? hrtimer_interrupt+0x128/0x2c0 [ 8540.182817] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8540.184811] ? apic_timer_interrupt+0xf/0x20 [ 8540.187973] [ 8540.188879] ? native_safe_halt+0x2e/0x50 [ 8540.194832] kvm_wait+0xcd/0xe0 [ 8540.196067] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8540.199412] do_raw_spin_lock+0x177/0x190 [ 8540.200991] _raw_spin_lock+0x21/0x30 [ 8540.203410] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8540.206830] ? do_raw_spin_unlock+0x75/0x190 [ 8540.208880] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8540.211235] ? do_wait_intr+0xf0/0xf0 [ 8540.212859] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8540.215438] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8540.218023] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8540.223146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8540.226970] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8540.230853] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8540.233120] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8540.235264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8540.238153] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8540.241080] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8540.244911] lmv_intent_open+0x338/0x1150 [lmv] [ 8540.248637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8540.253274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8540.257477] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8540.259980] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8540.263816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8540.266832] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8540.268530] ? do_raw_spin_unlock+0x75/0x190 [ 8540.270453] ? _raw_spin_unlock+0x12/0x30 [ 8540.271714] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8540.274238] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8540.276020] ll_atomic_open+0x36a/0xd70 [lustre] [ 8540.278266] lookup_open+0x356/0xb40 [ 8540.279394] do_last+0x313/0xfc0 [ 8540.281368] ? nd_jump_root+0xe5/0x160 [ 8540.282591] ? path_init+0x437/0x520 [ 8540.284285] path_openat+0xf7/0x500 [ 8540.285501] ? do_raw_spin_unlock+0x75/0x190 [ 8540.289246] do_filp_open+0x99/0x140 [ 8540.290427] ? getname_flags+0x6e/0x330 [ 8540.292102] ? __check_object_size+0xff/0x256 [ 8540.293976] ? do_raw_spin_unlock+0x75/0x190 [ 8540.296297] ? _raw_spin_unlock+0x12/0x30 [ 8540.298265] do_sys_openat2+0x2b4/0x410 [ 8540.299916] do_sys_open+0x73/0xa0 [ 8540.301248] __x64_sys_openat+0x24/0x30 [ 8540.302473] do_syscall_64+0xc1/0x440 [ 8540.303665] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8540.305566] RIP: 0033:0x7fa749552332 [ 8540.306750] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8540.313049] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8540.315752] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8540.317937] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8540.320219] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8540.322366] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8540.324825] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8545.761095] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6628s! [ 8545.764113] Showing busy workqueues and worker pools: [ 8545.765935] workqueue events: flags=0x0 [ 8545.773137] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8545.784299] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8545.798725] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8545.801290] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8545.811125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.820581] pending: kfree_rcu_monitor [ 8545.824553] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8545.835554] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8545.843587] workqueue events_long: flags=0x0 [ 8545.852479] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.856940] pending: lu_objects_flush [obdclass] [ 8545.858984] workqueue events_unbound: flags=0x2 [ 8545.866683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8545.877263] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8545.889085] workqueue events_power_efficient: flags=0x80 [ 8545.895367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.902504] pending: check_lifetime [ 8545.904878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.912342] pending: neigh_periodic_work [ 8545.913927] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.918291] pending: neigh_periodic_work [ 8545.923572] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.926042] pending: do_cache_clean [ 8545.928279] workqueue mm_percpu_wq: flags=0x8 [ 8545.929909] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.935533] pending: vmstat_update [ 8545.939473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.946551] pending: vmstat_update [ 8545.950115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8545.961883] pending: vmstat_update [ 8545.964701] workqueue writeback: flags=0x4a [ 8545.973665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8545.983368] pending: wb_workfn, wb_workfn [ 8545.984490] workqueue xprtiod: flags=0xa [ 8545.997804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8546.003758] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8546.024369] workqueue ipv6_addrconf: flags=0x40008 [ 8546.030734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8546.037199] pending: addrconf_verify_work [ 8546.045887] workqueue knbd0-recv: flags=0x1a [ 8546.047522] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8546.054750] in-flight: 651:recv_work [ 8546.063916] workqueue ptlrpc_pinger: flags=0x2 [ 8546.071505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8546.083636] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8546.086563] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6628s workers=4 idle: 37 31846 28897 [ 8546.090973] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8535s workers=2 idle: 653 [ 8556.057403] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8556.061078] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8556.062797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8556.062870] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8556.063599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8556.063606] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8556.063942] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8556.064120] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8556.064125] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8556.064130] RAX: 0000000000007983 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8556.064133] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8556.064135] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8556.064137] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8556.064140] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8556.064143] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8556.064146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8556.064148] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8556.064157] Call Trace: [ 8556.064163] [ 8556.069682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8556.069762] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8556.069769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8556.069773] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8556.069786] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8556.069791] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8556.069796] RAX: 000000000000773f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8556.069798] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8556.069800] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8556.069803] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8556.069805] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8556.069809] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8556.069812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8556.069815] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8556.069823] Call Trace: [ 8556.069830] [ 8556.069837] ? show_regs.cold.9+0x22/0x2f [ 8556.069847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8556.069854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8556.069858] ? watchdog+0x40/0x40 [ 8556.069863] ? hrtimer_interrupt+0x128/0x2c0 [ 8556.069868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8556.069876] ? apic_timer_interrupt+0xf/0x20 [ 8556.069880] [ 8556.069883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8556.069891] do_raw_spin_lock+0x177/0x190 [ 8556.069896] _raw_spin_lock+0x21/0x30 [ 8556.069900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8556.071298] ? do_raw_spin_unlock+0x75/0x190 [ 8556.071305] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8556.071505] ? do_wait_intr+0xf0/0xf0 [ 8556.071513] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8556.071714] mdc_close+0x53b/0xd70 [mdc] [ 8556.071754] lmv_close+0x18d/0x490 [lmv] [ 8556.071785] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8556.071895] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8556.073062] ll_md_close+0x274/0x800 [lustre] [ 8556.073175] ll_file_release+0x128/0xc50 [lustre] [ 8556.073255] __fput+0x10d/0x440 [ 8556.073266] ____fput+0x12/0x20 [ 8556.073271] task_work_run+0xc6/0x110 [ 8556.073279] exit_to_usermode_loop+0x1dd/0x1f0 [ 8556.073287] do_syscall_64+0x3d6/0x440 [ 8556.073292] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8556.073301] RIP: 0033:0x7fa749552cd8 [ 8556.073310] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8556.073315] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8556.073320] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8556.073323] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8556.073326] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8556.073328] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8556.073331] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8556.539478] ? show_regs.cold.9+0x22/0x2f [ 8556.548130] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8556.550971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8556.558011] ? watchdog+0x40/0x40 [ 8556.560127] ? hrtimer_interrupt+0x128/0x2c0 [ 8556.562275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8556.565171] ? apic_timer_interrupt+0xf/0x20 [ 8556.569150] [ 8556.575894] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8556.577770] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8556.591114] do_raw_spin_lock+0x177/0x190 [ 8556.595322] _raw_spin_lock+0x21/0x30 [ 8556.600956] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8556.612916] process_one_work+0x2c8/0x700 [ 8556.615169] worker_thread+0x296/0x6e0 [ 8556.619971] ? rescuer_thread+0x570/0x570 [ 8556.625127] kthread+0x1d1/0x200 [ 8556.626202] ? set_kthread_struct+0x70/0x70 [ 8556.631306] ret_from_fork+0x1f/0x30 [ 8564.065834] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8564.069066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8564.086874] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8564.091770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8564.095745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8564.097999] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8564.112404] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8564.119669] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8564.122478] RDX: 0000000000000cb2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8564.125463] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8564.128438] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8564.131551] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8564.134409] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8564.138526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8564.140863] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8564.144671] Call Trace: [ 8564.145758] [ 8564.146644] ? show_regs.cold.9+0x22/0x2f [ 8564.148350] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8564.150437] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8564.152505] ? watchdog+0x40/0x40 [ 8564.153776] ? hrtimer_interrupt+0x128/0x2c0 [ 8564.156154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8564.158488] ? apic_timer_interrupt+0xf/0x20 [ 8564.160503] [ 8564.161437] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8564.163896] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8564.166189] do_raw_spin_lock+0x177/0x190 [ 8564.167781] _raw_spin_lock+0x21/0x30 [ 8564.169393] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8564.172027] obd_mod_free+0x46/0x140 [ptlrpc] [ 8564.174459] mdc_commit_open+0x91/0xb0 [mdc] [ 8564.176463] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8564.179083] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8564.181901] after_reply+0xe2c/0x1780 [ptlrpc] [ 8564.184149] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8564.186495] ? __next_timer_interrupt+0x160/0x160 [ 8564.188240] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8564.190311] ? do_wait_intr+0xf0/0xf0 [ 8564.191869] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8564.194687] kthread+0x1d1/0x200 [ 8564.196277] ? set_kthread_struct+0x70/0x70 [ 8564.198277] ret_from_fork+0x1f/0x30 [ 8572.054067] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8572.056760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8572.093473] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8572.098326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8572.103175] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8572.106345] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8572.122301] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8572.130311] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8572.136599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8572.140271] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8572.150177] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8572.158350] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8572.166402] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8572.168460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8572.177678] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8572.187287] Call Trace: [ 8572.188560] [ 8572.189420] ? show_regs.cold.9+0x22/0x2f [ 8572.192143] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8572.193971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8572.199372] ? watchdog+0x40/0x40 [ 8572.203302] ? hrtimer_interrupt+0x128/0x2c0 [ 8572.207931] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8572.217903] ? apic_timer_interrupt+0xf/0x20 [ 8572.221705] [ 8572.223671] ? native_safe_halt+0x2e/0x50 [ 8572.227675] kvm_wait+0xcd/0xe0 [ 8572.228616] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8572.231546] do_raw_spin_lock+0x177/0x190 [ 8572.234297] _raw_spin_lock+0x21/0x30 [ 8572.236859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8572.240686] ? do_raw_spin_unlock+0x75/0x190 [ 8572.244155] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8572.249039] ? do_wait_intr+0xf0/0xf0 [ 8572.252954] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8572.256156] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8572.259827] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8572.264140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.267166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8572.270815] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8572.273672] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8572.277266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.281452] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8572.285977] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8572.289977] lmv_intent_open+0x338/0x1150 [lmv] [ 8572.293234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.298133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.300761] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8572.303608] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8572.304960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.306488] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8572.309041] ? do_raw_spin_unlock+0x75/0x190 [ 8572.311760] ? _raw_spin_unlock+0x12/0x30 [ 8572.313985] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8572.317927] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8572.319964] ll_atomic_open+0x36a/0xd70 [lustre] [ 8572.327022] lookup_open+0x356/0xb40 [ 8572.328305] do_last+0x313/0xfc0 [ 8572.330650] ? nd_jump_root+0xe5/0x160 [ 8572.332695] ? path_init+0x437/0x520 [ 8572.337460] path_openat+0xf7/0x500 [ 8572.339328] ? do_raw_spin_unlock+0x75/0x190 [ 8572.341023] do_filp_open+0x99/0x140 [ 8572.345704] ? getname_flags+0x6e/0x330 [ 8572.348106] ? __check_object_size+0xff/0x256 [ 8572.351217] ? do_raw_spin_unlock+0x75/0x190 [ 8572.353580] ? _raw_spin_unlock+0x12/0x30 [ 8572.356392] do_sys_openat2+0x2b4/0x410 [ 8572.359024] do_sys_open+0x73/0xa0 [ 8572.361779] __x64_sys_openat+0x24/0x30 [ 8572.364356] do_syscall_64+0xc1/0x440 [ 8572.365656] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8572.368359] RIP: 0033:0x7fa749552332 [ 8572.371853] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8572.385826] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8572.388328] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8572.392217] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8572.395403] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8572.400561] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8572.404228] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8576.489393] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6659s! [ 8576.495152] Showing busy workqueues and worker pools: [ 8576.498250] workqueue events: flags=0x0 [ 8576.503014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8576.513531] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8576.521343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8576.526047] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8576.531037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.533065] pending: kfree_rcu_monitor [ 8576.535096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8576.538345] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8576.542325] workqueue events_long: flags=0x0 [ 8576.545729] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.548748] pending: lu_objects_flush [obdclass] [ 8576.551179] workqueue events_unbound: flags=0x2 [ 8576.560874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8576.564284] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8576.571153] workqueue events_power_efficient: flags=0x80 [ 8576.574097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.577766] pending: check_lifetime [ 8576.580447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.587757] pending: neigh_periodic_work [ 8576.590248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.594180] pending: neigh_periodic_work [ 8576.596071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.598917] pending: do_cache_clean [ 8576.600126] workqueue mm_percpu_wq: flags=0x8 [ 8576.602056] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.605067] pending: vmstat_update [ 8576.606292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.608616] pending: vmstat_update [ 8576.610073] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8576.612616] pending: vmstat_update [ 8576.614095] workqueue writeback: flags=0x4a [ 8576.615939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8576.618256] pending: wb_workfn, wb_workfn [ 8576.620323] workqueue xprtiod: flags=0xa [ 8576.622288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8576.624420] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8576.628115] workqueue ipv6_addrconf: flags=0x40008 [ 8576.630197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8576.633012] pending: addrconf_verify_work [ 8576.634923] workqueue knbd0-recv: flags=0x1a [ 8576.636388] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8576.639039] in-flight: 651:recv_work [ 8576.640828] workqueue ptlrpc_pinger: flags=0x2 [ 8576.643195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8576.647059] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8576.649665] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6659s workers=4 idle: 37 31846 28897 [ 8576.654502] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8566s workers=2 idle: 653 [ 8584.061457] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8584.061543] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8584.068209] Modules linked in: lustre(O) [ 8584.073627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8584.073703] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8584.073711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8584.073719] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8584.075119] osp(O) [ 8584.101485] [ 8584.101495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8584.101515] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8584.101519] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8584.101524] RAX: 00000000000014c5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8584.101527] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8584.101529] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8584.101531] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8584.101534] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8584.101537] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8584.101540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8584.101542] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8584.101551] Call Trace: [ 8584.101556] [ 8584.101562] ? show_regs.cold.9+0x22/0x2f [ 8584.101572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8584.101593] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8584.101598] ? watchdog+0x40/0x40 [ 8584.101603] ? hrtimer_interrupt+0x128/0x2c0 [ 8584.101608] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8584.101617] ? apic_timer_interrupt+0xf/0x20 [ 8584.101622] [ 8584.101626] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8584.101631] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8584.101639] do_raw_spin_lock+0x177/0x190 [ 8584.101644] _raw_spin_lock+0x21/0x30 [ 8584.101650] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8584.101940] process_one_work+0x2c8/0x700 [ 8584.101948] worker_thread+0x296/0x6e0 [ 8584.101952] ? rescuer_thread+0x570/0x570 [ 8584.101957] kthread+0x1d1/0x200 [ 8584.101962] ? set_kthread_struct+0x70/0x70 [ 8584.101966] ret_from_fork+0x1f/0x30 [ 8584.233142] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8584.260071] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8584.266935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8584.273618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8584.278869] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8584.290229] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8584.297532] RAX: 000000000000711e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8584.303675] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8584.309558] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8584.315031] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8584.321845] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8584.325477] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8584.330622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8584.333814] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8584.339760] Call Trace: [ 8584.342679] [ 8584.343409] ? show_regs.cold.9+0x22/0x2f [ 8584.344816] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8584.350752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8584.352430] ? watchdog+0x40/0x40 [ 8584.357675] ? hrtimer_interrupt+0x128/0x2c0 [ 8584.365538] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8584.367304] ? apic_timer_interrupt+0xf/0x20 [ 8584.374149] [ 8584.374980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8584.376872] do_raw_spin_lock+0x177/0x190 [ 8584.379891] _raw_spin_lock+0x21/0x30 [ 8584.381864] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8584.386129] ? do_raw_spin_unlock+0x75/0x190 [ 8584.387866] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8584.390060] ? do_wait_intr+0xf0/0xf0 [ 8584.395061] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8584.402262] mdc_close+0x53b/0xd70 [mdc] [ 8584.406102] lmv_close+0x18d/0x490 [lmv] [ 8584.407963] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8584.412091] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8584.414066] ll_md_close+0x274/0x800 [lustre] [ 8584.415998] ll_file_release+0x128/0xc50 [lustre] [ 8584.418738] __fput+0x10d/0x440 [ 8584.419935] ____fput+0x12/0x20 [ 8584.422956] task_work_run+0xc6/0x110 [ 8584.424107] exit_to_usermode_loop+0x1dd/0x1f0 [ 8584.426676] do_syscall_64+0x3d6/0x440 [ 8584.431594] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8584.433129] RIP: 0033:0x7fa749552cd8 [ 8584.435648] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8584.456448] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8584.459261] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8584.464067] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8584.468148] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8584.471171] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8584.477838] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8592.058108] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8592.063894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8592.088009] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8592.093990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8592.099122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8592.102671] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8592.111799] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8592.115359] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8592.119589] RDX: 0000000000003517 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8592.122735] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8592.126373] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8592.129929] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8592.133355] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8592.137137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8592.139684] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8592.142604] Call Trace: [ 8592.143584] [ 8592.144946] ? show_regs.cold.9+0x22/0x2f [ 8592.146933] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8592.149487] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8592.151591] ? watchdog+0x40/0x40 [ 8592.153640] ? hrtimer_interrupt+0x128/0x2c0 [ 8592.155493] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8592.157852] ? apic_timer_interrupt+0xf/0x20 [ 8592.159914] [ 8592.160897] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8592.163468] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8592.165638] do_raw_spin_lock+0x177/0x190 [ 8592.167755] _raw_spin_lock+0x21/0x30 [ 8592.169121] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8592.172014] obd_mod_free+0x46/0x140 [ptlrpc] [ 8592.174598] mdc_commit_open+0x91/0xb0 [mdc] [ 8592.177073] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8592.179818] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8592.183290] after_reply+0xe2c/0x1780 [ptlrpc] [ 8592.185630] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8592.188326] ? __next_timer_interrupt+0x160/0x160 [ 8592.190493] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8592.192768] ? do_wait_intr+0xf0/0xf0 [ 8592.194663] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8592.197837] kthread+0x1d1/0x200 [ 8592.200025] ? set_kthread_struct+0x70/0x70 [ 8592.202038] ret_from_fork+0x1f/0x30 [ 8604.054045] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8604.059643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8604.101409] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8604.106823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8604.111073] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8604.114835] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8604.125592] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8604.129737] RAX: 0000000000000fd5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8604.133389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8604.137045] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8604.140627] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8604.144604] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8604.148138] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8604.152275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8604.155012] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8604.158712] Call Trace: [ 8604.159833] [ 8604.160723] ? show_regs.cold.9+0x22/0x2f [ 8604.162877] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8604.165379] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8604.167757] ? watchdog+0x40/0x40 [ 8604.169896] ? hrtimer_interrupt+0x128/0x2c0 [ 8604.171971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8604.174406] ? apic_timer_interrupt+0xf/0x20 [ 8604.176447] [ 8604.177422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8604.179690] do_raw_spin_lock+0x177/0x190 [ 8604.181149] _raw_spin_lock+0x21/0x30 [ 8604.182621] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8604.185154] ? do_raw_spin_unlock+0x75/0x190 [ 8604.187549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8604.189935] ? do_wait_intr+0xf0/0xf0 [ 8604.192609] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8604.194996] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8604.198192] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8604.201389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8604.203840] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8604.206595] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8604.209063] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8604.211654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8604.214312] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8604.217389] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8604.220380] lmv_intent_open+0x338/0x1150 [lmv] [ 8604.222217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8604.224892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8604.227606] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8604.229596] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8604.231599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8604.236396] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8604.238981] ? do_raw_spin_unlock+0x75/0x190 [ 8604.240935] ? _raw_spin_unlock+0x12/0x30 [ 8604.243731] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8604.246664] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8604.248900] ll_atomic_open+0x36a/0xd70 [lustre] [ 8604.251392] lookup_open+0x356/0xb40 [ 8604.253361] do_last+0x313/0xfc0 [ 8604.254830] ? nd_jump_root+0xe5/0x160 [ 8604.256567] ? path_init+0x437/0x520 [ 8604.258017] path_openat+0xf7/0x500 [ 8604.259255] ? do_raw_spin_unlock+0x75/0x190 [ 8604.261176] do_filp_open+0x99/0x140 [ 8604.262875] ? getname_flags+0x6e/0x330 [ 8604.264667] ? __check_object_size+0xff/0x256 [ 8604.266639] ? do_raw_spin_unlock+0x75/0x190 [ 8604.268587] ? _raw_spin_unlock+0x12/0x30 [ 8604.270687] do_sys_openat2+0x2b4/0x410 [ 8604.272651] do_sys_open+0x73/0xa0 [ 8604.274168] __x64_sys_openat+0x24/0x30 [ 8604.276023] do_syscall_64+0xc1/0x440 [ 8604.277738] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8604.280127] RIP: 0033:0x7fa749552332 [ 8604.282285] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8604.291365] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8604.295143] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8604.298854] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8604.302861] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8604.306850] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8604.310361] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8606.693127] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6689s! [ 8606.697415] Showing busy workqueues and worker pools: [ 8606.699684] workqueue events: flags=0x0 [ 8606.701534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8606.704790] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8606.709643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8606.712626] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8606.717532] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.723378] pending: kfree_rcu_monitor [ 8606.725759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8606.732509] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8606.742825] workqueue events_long: flags=0x0 [ 8606.746908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.750896] pending: lu_objects_flush [obdclass] [ 8606.756103] workqueue events_unbound: flags=0x2 [ 8606.758367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8606.761782] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8606.767426] workqueue events_power_efficient: flags=0x80 [ 8606.770177] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.773768] pending: check_lifetime [ 8606.776011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.779731] pending: neigh_periodic_work [ 8606.781765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.786482] pending: neigh_periodic_work [ 8606.789450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.792884] pending: do_cache_clean [ 8606.794413] workqueue mm_percpu_wq: flags=0x8 [ 8606.796925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.799302] pending: vmstat_update [ 8606.800914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.803959] pending: vmstat_update [ 8606.805209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8606.807157] pending: vmstat_update [ 8606.808323] workqueue writeback: flags=0x4a [ 8606.810074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8606.812137] pending: wb_workfn, wb_workfn [ 8606.814041] workqueue xprtiod: flags=0xa [ 8606.815541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8606.818759] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8606.821845] workqueue ipv6_addrconf: flags=0x40008 [ 8606.823605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8606.825746] pending: addrconf_verify_work [ 8606.827568] workqueue knbd0-recv: flags=0x1a [ 8606.830369] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8606.834711] in-flight: 651:recv_work [ 8606.836731] workqueue ptlrpc_pinger: flags=0x2 [ 8606.840767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8606.843961] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8606.846866] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6689s workers=4 idle: 37 31846 28897 [ 8606.850700] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8596s workers=2 idle: 653 [ 8612.059229] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8612.060101] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8612.076121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8612.076201] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8612.076207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8612.076213] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8612.077040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8612.077053] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8612.077057] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8612.077062] RAX: 0000000000006af2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8612.077064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8612.077066] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8612.077068] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8612.077070] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8612.077073] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8612.077076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8612.077078] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8612.077099] Call Trace: [ 8612.077104] [ 8612.077110] ? show_regs.cold.9+0x22/0x2f [ 8612.077119] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8612.077126] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8612.077131] ? watchdog+0x40/0x40 [ 8612.077136] ? hrtimer_interrupt+0x128/0x2c0 [ 8612.077141] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8612.077151] ? apic_timer_interrupt+0xf/0x20 [ 8612.077157] [ 8612.077160] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8612.077165] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8612.077171] do_raw_spin_lock+0x177/0x190 [ 8612.077176] _raw_spin_lock+0x21/0x30 [ 8612.077181] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8612.077426] process_one_work+0x2c8/0x700 [ 8612.077434] worker_thread+0x296/0x6e0 [ 8612.077439] ? rescuer_thread+0x570/0x570 [ 8612.077443] kthread+0x1d1/0x200 [ 8612.078628] ? set_kthread_struct+0x70/0x70 [ 8612.078633] ret_from_fork+0x1f/0x30 [ 8612.256222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8612.293033] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8612.299289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8612.304633] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8612.307144] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8612.315349] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8612.318731] RAX: 0000000000007f17 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8612.323046] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8612.327504] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8612.334271] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8612.338243] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8612.340865] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8612.344155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8612.346773] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8612.351329] Call Trace: [ 8612.352649] [ 8612.355304] ? show_regs.cold.9+0x22/0x2f [ 8612.358991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8612.361860] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8612.367185] ? watchdog+0x40/0x40 [ 8612.368442] ? hrtimer_interrupt+0x128/0x2c0 [ 8612.370969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8612.372752] ? apic_timer_interrupt+0xf/0x20 [ 8612.374585] [ 8612.375784] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8612.379202] do_raw_spin_lock+0x177/0x190 [ 8612.384304] _raw_spin_lock+0x21/0x30 [ 8612.386453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8612.392370] ? do_raw_spin_unlock+0x75/0x190 [ 8612.395539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8612.401360] ? do_wait_intr+0xf0/0xf0 [ 8612.403292] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8612.410375] mdc_close+0x53b/0xd70 [mdc] [ 8612.412517] lmv_close+0x18d/0x490 [lmv] [ 8612.416328] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8612.418335] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8612.420173] ll_md_close+0x274/0x800 [lustre] [ 8612.421994] ll_file_release+0x128/0xc50 [lustre] [ 8612.427489] __fput+0x10d/0x440 [ 8612.428861] ____fput+0x12/0x20 [ 8612.431181] task_work_run+0xc6/0x110 [ 8612.433166] exit_to_usermode_loop+0x1dd/0x1f0 [ 8612.436247] do_syscall_64+0x3d6/0x440 [ 8612.440849] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8612.442269] RIP: 0033:0x7fa749552cd8 [ 8612.443251] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8612.454165] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8612.458486] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8612.461001] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8612.467363] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8612.471021] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8612.477605] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8620.061135] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8620.068380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8620.105771] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8620.112633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8620.116503] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8620.119273] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8620.128786] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8620.132654] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8620.135892] RDX: 0000000000005ed4 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8620.139115] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8620.142378] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8620.144597] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8620.147475] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8620.151001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8620.154281] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8620.158595] Call Trace: [ 8620.162014] [ 8620.165371] ? show_regs.cold.9+0x22/0x2f [ 8620.171354] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8620.179326] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8620.184908] ? watchdog+0x40/0x40 [ 8620.186582] ? hrtimer_interrupt+0x128/0x2c0 [ 8620.191010] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8620.194275] ? apic_timer_interrupt+0xf/0x20 [ 8620.197055] [ 8620.198481] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8620.201085] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8620.204312] do_raw_spin_lock+0x177/0x190 [ 8620.206266] _raw_spin_lock+0x21/0x30 [ 8620.208566] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8620.211006] obd_mod_free+0x46/0x140 [ptlrpc] [ 8620.213119] mdc_commit_open+0x91/0xb0 [mdc] [ 8620.217604] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8620.219746] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8620.225514] after_reply+0xe2c/0x1780 [ptlrpc] [ 8620.231896] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8620.234995] ? __next_timer_interrupt+0x160/0x160 [ 8620.236810] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8620.238915] ? do_wait_intr+0xf0/0xf0 [ 8620.240979] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8620.243800] kthread+0x1d1/0x200 [ 8620.245479] ? set_kthread_struct+0x70/0x70 [ 8620.247499] ret_from_fork+0x1f/0x30 [ 8632.056565] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8632.067796] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8632.088153] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8632.092931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8632.096076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8632.100970] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8632.111256] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8632.116440] RAX: 0000000000007e0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8632.120065] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8632.123937] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8632.127186] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8632.130717] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8632.134003] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8632.137315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8632.139700] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8632.143076] Call Trace: [ 8632.144084] [ 8632.144921] ? show_regs.cold.9+0x22/0x2f [ 8632.146303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8632.148336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8632.150264] ? watchdog+0x40/0x40 [ 8632.151712] ? hrtimer_interrupt+0x128/0x2c0 [ 8632.153793] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8632.156558] ? apic_timer_interrupt+0xf/0x20 [ 8632.158411] [ 8632.159155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8632.161037] do_raw_spin_lock+0x177/0x190 [ 8632.162567] _raw_spin_lock+0x21/0x30 [ 8632.164098] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8632.166939] ? do_raw_spin_unlock+0x75/0x190 [ 8632.168341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8632.172243] ? do_wait_intr+0xf0/0xf0 [ 8632.175547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8632.178136] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8632.180788] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8632.183598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8632.186126] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8632.188207] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8632.190345] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8632.192089] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8632.194559] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8632.197807] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8632.200046] lmv_intent_open+0x338/0x1150 [lmv] [ 8632.201659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8632.204137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8632.206375] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8632.208577] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8632.210709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8632.212769] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8632.214902] ? do_raw_spin_unlock+0x75/0x190 [ 8632.216151] ? _raw_spin_unlock+0x12/0x30 [ 8632.217395] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8632.219254] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8632.221810] ll_atomic_open+0x36a/0xd70 [lustre] [ 8632.224306] lookup_open+0x356/0xb40 [ 8632.226208] do_last+0x313/0xfc0 [ 8632.227550] ? nd_jump_root+0xe5/0x160 [ 8632.229288] ? path_init+0x437/0x520 [ 8632.230965] path_openat+0xf7/0x500 [ 8632.232715] ? do_raw_spin_unlock+0x75/0x190 [ 8632.234643] do_filp_open+0x99/0x140 [ 8632.236183] ? getname_flags+0x6e/0x330 [ 8632.238005] ? __check_object_size+0xff/0x256 [ 8632.239632] ? do_raw_spin_unlock+0x75/0x190 [ 8632.241637] ? _raw_spin_unlock+0x12/0x30 [ 8632.243243] do_sys_openat2+0x2b4/0x410 [ 8632.244662] do_sys_open+0x73/0xa0 [ 8632.246279] __x64_sys_openat+0x24/0x30 [ 8632.248136] do_syscall_64+0xc1/0x440 [ 8632.250008] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8632.251984] RIP: 0033:0x7fa749552332 [ 8632.253258] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8632.261054] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8632.264328] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8632.267647] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8632.271731] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8632.275240] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8632.278166] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8633.835433] rcu: INFO: rcu_sched self-detected stall on CPU [ 8633.838839] rcu: 2-...!: (3770104 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 8633.851931] (t=6720146 jiffies g=258701 q=3434) [ 8633.863457] rcu: rcu_sched kthread starved for 6716941 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 8633.877762] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8633.890411] rcu: RCU grace-period kthread stack dump: [ 8633.896884] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8633.909061] Call Trace: [ 8633.911799] __schedule+0x351/0xcb0 [ 8633.916259] schedule+0xc0/0x180 [ 8633.918988] schedule_timeout+0xb4/0x190 [ 8633.922210] ? do_raw_spin_unlock+0x75/0x190 [ 8633.923682] ? __next_timer_interrupt+0x160/0x160 [ 8633.926936] ? prepare_to_swait_event+0xf3/0x140 [ 8633.930361] rcu_gp_kthread+0x8dd/0x15b0 [ 8633.932387] ? force_qs_rnp+0x2c0/0x2c0 [ 8633.933938] kthread+0x1d1/0x200 [ 8633.937775] ? set_kthread_struct+0x70/0x70 [ 8633.941072] ret_from_fork+0x1f/0x30 [ 8633.944467] rcu: Stack dump where RCU GP kthread last ran: [ 8633.948302] Sending NMI from CPU 2 to CPUs 1: [ 8633.952833] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 8633.955427] NMI backtrace for cpu 2 [ 8633.956570] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8633.969328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8633.980682] Call Trace: [ 8633.982071] [ 8633.983943] ? dump_stack+0xbb/0x10e [ 8633.988378] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8633.999235] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8634.000645] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8634.002455] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8634.013143] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8634.024444] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8634.030771] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8634.036302] ? update_process_times+0x6d/0xc0 [ 8634.041731] ? tick_sched_handle.isra.18+0x27/0x90 [ 8634.049510] ? tick_sched_timer+0x53/0xb0 [ 8634.050835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8634.056794] ? tick_sched_handle.isra.18+0x90/0x90 [ 8634.069841] ? hrtimer_interrupt+0x128/0x2c0 [ 8634.073181] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8634.077892] ? apic_timer_interrupt+0xf/0x20 [ 8634.083762] [ 8634.085923] ? native_safe_halt+0x2e/0x50 [ 8634.087543] ? kvm_wait+0xcd/0xe0 [ 8634.091228] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8634.100157] ? do_raw_spin_lock+0x177/0x190 [ 8634.108245] ? _raw_spin_lock+0x21/0x30 [ 8634.111986] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8634.119484] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8634.126803] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8634.133982] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8634.142142] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8634.147780] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8634.153075] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8634.159048] ? __next_timer_interrupt+0x160/0x160 [ 8634.165782] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8634.177334] ? do_wait_intr+0xf0/0xf0 [ 8634.183996] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8634.194403] ? kthread+0x1d1/0x200 [ 8634.202979] ? set_kthread_struct+0x70/0x70 [ 8634.208044] ? ret_from_fork+0x1f/0x30 [ 8636.896052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6719s! [ 8636.901079] Showing busy workqueues and worker pools: [ 8636.905350] workqueue events: flags=0x0 [ 8636.912993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8636.923101] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8636.931549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8636.937617] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8636.946140] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8636.952233] pending: kfree_rcu_monitor [ 8636.954583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8636.962197] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8636.972077] workqueue events_long: flags=0x0 [ 8636.974256] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8636.980417] pending: lu_objects_flush [obdclass] [ 8636.983343] workqueue events_unbound: flags=0x2 [ 8636.988381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8636.995847] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8637.001086] workqueue events_power_efficient: flags=0x80 [ 8637.003341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8637.008442] pending: check_lifetime [ 8637.012347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8637.019963] pending: neigh_periodic_work [ 8637.025338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8637.029974] pending: neigh_periodic_work [ 8637.035931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8637.042190] pending: do_cache_clean [ 8637.046693] workqueue mm_percpu_wq: flags=0x8 [ 8637.050186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8637.055538] pending: vmstat_update [ 8637.059130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8637.063514] pending: vmstat_update [ 8637.071465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8637.080873] pending: vmstat_update [ 8637.082212] workqueue writeback: flags=0x4a [ 8637.090944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8637.105949] pending: wb_workfn, wb_workfn [ 8637.110537] workqueue xprtiod: flags=0xa [ 8637.114695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8637.126438] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8637.131911] workqueue ipv6_addrconf: flags=0x40008 [ 8637.138456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8637.143877] pending: addrconf_verify_work [ 8637.146896] workqueue knbd0-recv: flags=0x1a [ 8637.149143] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8637.153477] in-flight: 651:recv_work [ 8637.156144] workqueue ptlrpc_pinger: flags=0x2 [ 8637.159219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8637.166311] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8637.172157] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6720s workers=4 idle: 37 31846 28897 [ 8637.178909] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8627s workers=2 idle: 653 [ 8640.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8640.062806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8640.117204] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8640.132063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8640.143753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8640.149455] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8640.169538] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8640.176239] RAX: 0000000000003530 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8640.182552] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8640.190303] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8640.197990] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8640.204264] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8640.211015] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8640.214447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8640.221155] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8640.229169] Call Trace: [ 8640.230435] [ 8640.233435] ? show_regs.cold.9+0x22/0x2f [ 8640.241182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8640.243457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8640.247408] ? watchdog+0x40/0x40 [ 8640.248942] ? hrtimer_interrupt+0x128/0x2c0 [ 8640.250148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8640.251836] ? apic_timer_interrupt+0xf/0x20 [ 8640.259112] [ 8640.261272] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8640.267284] do_raw_spin_lock+0x177/0x190 [ 8640.271793] _raw_spin_lock+0x21/0x30 [ 8640.273929] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8640.277805] ? do_raw_spin_unlock+0x75/0x190 [ 8640.278908] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8640.284899] ? do_wait_intr+0xf0/0xf0 [ 8640.288147] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8640.294916] mdc_close+0x53b/0xd70 [mdc] [ 8640.299106] lmv_close+0x18d/0x490 [lmv] [ 8640.303363] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8640.308203] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8640.310047] ll_md_close+0x274/0x800 [lustre] [ 8640.316403] ll_file_release+0x128/0xc50 [lustre] [ 8640.319015] __fput+0x10d/0x440 [ 8640.324560] ____fput+0x12/0x20 [ 8640.327674] task_work_run+0xc6/0x110 [ 8640.330158] exit_to_usermode_loop+0x1dd/0x1f0 [ 8640.331585] do_syscall_64+0x3d6/0x440 [ 8640.334104] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8640.336112] RIP: 0033:0x7fa749552cd8 [ 8640.338438] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8640.348988] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8640.352717] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8640.355985] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8640.363444] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8640.369959] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8640.373092] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8660.060262] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8660.061830] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8660.066405] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8660.066502] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8660.066509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8660.066519] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8660.072445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8660.072529] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8660.072537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8660.108542] [ 8660.108552] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8660.108571] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8660.108576] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8660.108581] RAX: 0000000000006f0f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8660.108583] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8660.108585] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8660.108588] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8660.108590] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8660.108593] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8660.108596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8660.108599] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8660.108608] Call Trace: [ 8660.108613] [ 8660.108618] ? show_regs.cold.9+0x22/0x2f [ 8660.108628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8660.108633] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8660.108637] ? watchdog+0x40/0x40 [ 8660.108641] ? hrtimer_interrupt+0x128/0x2c0 [ 8660.108645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8660.108653] ? apic_timer_interrupt+0xf/0x20 [ 8660.108656] [ 8660.108659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8660.108664] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8660.108671] do_raw_spin_lock+0x177/0x190 [ 8660.108675] _raw_spin_lock+0x21/0x30 [ 8660.108680] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8660.117742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8660.117766] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8660.117771] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8660.117777] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8660.117780] RDX: 00000000000034f9 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8660.117782] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8660.117784] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8660.117786] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8660.117790] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8660.117792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8660.117795] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8660.117812] Call Trace: [ 8660.117818] [ 8660.117824] ? show_regs.cold.9+0x22/0x2f [ 8660.117833] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8660.117839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8660.117843] ? watchdog+0x40/0x40 [ 8660.117848] ? hrtimer_interrupt+0x128/0x2c0 [ 8660.117851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8660.117859] ? apic_timer_interrupt+0xf/0x20 [ 8660.117864] [ 8660.117867] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8660.117872] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8660.117877] do_raw_spin_lock+0x177/0x190 [ 8660.117881] _raw_spin_lock+0x21/0x30 [ 8660.117889] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8660.123291] process_one_work+0x2c8/0x700 [ 8660.123319] worker_thread+0x296/0x6e0 [ 8660.123325] ? rescuer_thread+0x570/0x570 [ 8660.123329] kthread+0x1d1/0x200 [ 8660.123333] ? set_kthread_struct+0x70/0x70 [ 8660.123343] ret_from_fork+0x1f/0x30 [ 8660.335551] obd_mod_free+0x46/0x140 [ptlrpc] [ 8660.337827] mdc_commit_open+0x91/0xb0 [mdc] [ 8660.339852] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8660.342572] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8660.345427] after_reply+0xe2c/0x1780 [ptlrpc] [ 8660.347505] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8660.350010] ? __next_timer_interrupt+0x160/0x160 [ 8660.352080] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8660.354060] ? do_wait_intr+0xf0/0xf0 [ 8660.355537] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8660.357838] kthread+0x1d1/0x200 [ 8660.359038] ? set_kthread_struct+0x70/0x70 [ 8660.360459] ret_from_fork+0x1f/0x30 [ 8664.055915] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8664.063218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8664.087724] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8664.094597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8664.097701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8664.100608] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8664.109105] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8664.112690] RAX: 0000000000007d0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8664.116127] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8664.120004] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8664.123182] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8664.126242] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8664.129411] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8664.132713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8664.135278] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8664.138338] Call Trace: [ 8664.139579] [ 8664.140454] ? show_regs.cold.9+0x22/0x2f [ 8664.142373] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8664.144865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8664.146483] ? watchdog+0x40/0x40 [ 8664.148033] ? hrtimer_interrupt+0x128/0x2c0 [ 8664.149214] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8664.150722] ? apic_timer_interrupt+0xf/0x20 [ 8664.152251] [ 8664.153013] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8664.154774] do_raw_spin_lock+0x177/0x190 [ 8664.156429] _raw_spin_lock+0x21/0x30 [ 8664.158220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8664.160996] ? do_raw_spin_unlock+0x75/0x190 [ 8664.162856] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8664.165368] ? do_wait_intr+0xf0/0xf0 [ 8664.167239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8664.169528] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8664.172238] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8664.174658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.177263] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8664.179821] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8664.182058] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8664.184215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.186231] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8664.188452] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8664.191088] lmv_intent_open+0x338/0x1150 [lmv] [ 8664.193445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.196020] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.199193] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8664.201569] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8664.204130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.206525] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8664.208950] ? do_raw_spin_unlock+0x75/0x190 [ 8664.211022] ? _raw_spin_unlock+0x12/0x30 [ 8664.212726] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8664.215073] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8664.217614] ll_atomic_open+0x36a/0xd70 [lustre] [ 8664.219575] lookup_open+0x356/0xb40 [ 8664.220961] do_last+0x313/0xfc0 [ 8664.222321] ? nd_jump_root+0xe5/0x160 [ 8664.223700] ? path_init+0x437/0x520 [ 8664.225002] path_openat+0xf7/0x500 [ 8664.226606] ? do_raw_spin_unlock+0x75/0x190 [ 8664.228521] do_filp_open+0x99/0x140 [ 8664.229999] ? getname_flags+0x6e/0x330 [ 8664.231011] ? __check_object_size+0xff/0x256 [ 8664.231919] ? do_raw_spin_unlock+0x75/0x190 [ 8664.233406] ? _raw_spin_unlock+0x12/0x30 [ 8664.235342] do_sys_openat2+0x2b4/0x410 [ 8664.237025] do_sys_open+0x73/0xa0 [ 8664.238492] __x64_sys_openat+0x24/0x30 [ 8664.240285] do_syscall_64+0xc1/0x440 [ 8664.242031] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8664.244119] RIP: 0033:0x7fa749552332 [ 8664.245695] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8664.255310] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8664.259075] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8664.262085] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8664.265401] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8664.268499] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8664.271764] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8667.618120] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6750s! [ 8667.621372] Showing busy workqueues and worker pools: [ 8667.624252] workqueue events: flags=0x0 [ 8667.626255] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8667.636901] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8667.657204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8667.669911] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8667.686399] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.696123] pending: kfree_rcu_monitor [ 8667.701001] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8667.708286] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8667.716306] workqueue events_long: flags=0x0 [ 8667.721382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.725979] pending: lu_objects_flush [obdclass] [ 8667.729705] workqueue events_unbound: flags=0x2 [ 8667.732887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8667.737030] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8667.744789] workqueue events_power_efficient: flags=0x80 [ 8667.748756] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.752068] pending: check_lifetime [ 8667.754169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.758544] pending: neigh_periodic_work [ 8667.760795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.764617] pending: neigh_periodic_work [ 8667.771058] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.775767] pending: do_cache_clean [ 8667.778090] workqueue mm_percpu_wq: flags=0x8 [ 8667.781144] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.785928] pending: vmstat_update [ 8667.789294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.792751] pending: vmstat_update [ 8667.794613] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8667.803380] pending: vmstat_update [ 8667.807287] workqueue writeback: flags=0x4a [ 8667.815087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8667.818967] pending: wb_workfn, wb_workfn [ 8667.822261] workqueue xprtiod: flags=0xa [ 8667.825455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8667.827615] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8667.832872] workqueue ipv6_addrconf: flags=0x40008 [ 8667.837208] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8667.843256] pending: addrconf_verify_work [ 8667.847332] workqueue knbd0-recv: flags=0x1a [ 8667.849882] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8667.853791] in-flight: 651:recv_work [ 8667.856795] workqueue ptlrpc_pinger: flags=0x2 [ 8667.861456] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8667.864356] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8667.872726] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6750s workers=4 idle: 37 31846 28897 [ 8667.880908] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8657s workers=2 idle: 653 [ 8668.060279] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8668.078200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8668.107285] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8668.111959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8668.115102] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8668.117475] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8668.125244] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8668.128031] RAX: 0000000000004f0e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8668.130604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8668.132596] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8668.134561] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8668.136695] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8668.138964] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8668.141089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8668.142572] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8668.144397] Call Trace: [ 8668.145010] [ 8668.145555] ? show_regs.cold.9+0x22/0x2f [ 8668.146707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8668.148031] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8668.149709] ? watchdog+0x40/0x40 [ 8668.151087] ? hrtimer_interrupt+0x128/0x2c0 [ 8668.152773] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8668.155064] ? apic_timer_interrupt+0xf/0x20 [ 8668.156678] [ 8668.157581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8668.159628] do_raw_spin_lock+0x177/0x190 [ 8668.161345] _raw_spin_lock+0x21/0x30 [ 8668.162675] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8668.165155] ? do_raw_spin_unlock+0x75/0x190 [ 8668.166792] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8668.168707] ? do_wait_intr+0xf0/0xf0 [ 8668.171515] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8668.177359] mdc_close+0x53b/0xd70 [mdc] [ 8668.181431] lmv_close+0x18d/0x490 [lmv] [ 8668.182578] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8668.185183] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8668.188619] ll_md_close+0x274/0x800 [lustre] [ 8668.190468] ll_file_release+0x128/0xc50 [lustre] [ 8668.192386] __fput+0x10d/0x440 [ 8668.193524] ____fput+0x12/0x20 [ 8668.194778] task_work_run+0xc6/0x110 [ 8668.196242] exit_to_usermode_loop+0x1dd/0x1f0 [ 8668.197833] do_syscall_64+0x3d6/0x440 [ 8668.199345] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8668.201253] RIP: 0033:0x7fa749552cd8 [ 8668.202560] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8668.209429] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8668.211445] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8668.213441] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8668.215877] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8668.218715] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8668.221742] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8688.058336] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8688.060687] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8688.066262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8688.066342] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8688.066350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8688.066353] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8688.066367] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8688.066371] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8688.066376] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8688.066378] RDX: 00000000000077f7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8688.066381] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8688.066383] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8688.066385] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8688.066388] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8688.066391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8688.066394] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8688.066402] Call Trace: [ 8688.066409] [ 8688.066415] ? show_regs.cold.9+0x22/0x2f [ 8688.066424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8688.066431] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8688.066435] ? watchdog+0x40/0x40 [ 8688.066440] ? hrtimer_interrupt+0x128/0x2c0 [ 8688.066444] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8688.066451] ? apic_timer_interrupt+0xf/0x20 [ 8688.066456] [ 8688.066459] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8688.066464] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8688.066468] do_raw_spin_lock+0x177/0x190 [ 8688.066473] _raw_spin_lock+0x21/0x30 [ 8688.066479] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8688.066764] obd_mod_free+0x46/0x140 [ptlrpc] [ 8688.067146] mdc_commit_open+0x91/0xb0 [mdc] [ 8688.067182] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8688.067395] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8688.076990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8688.077072] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8688.077078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8688.077084] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8688.077388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8688.077398] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8688.077402] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8688.077406] RAX: 000000000000678b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8688.077409] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8688.077411] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8688.077413] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8688.077415] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8688.077419] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8688.077421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8688.077424] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8688.077432] Call Trace: [ 8688.077436] [ 8688.077441] ? show_regs.cold.9+0x22/0x2f [ 8688.077454] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8688.077460] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8688.077464] ? watchdog+0x40/0x40 [ 8688.077530] ? hrtimer_interrupt+0x128/0x2c0 [ 8688.077534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8688.077544] ? apic_timer_interrupt+0xf/0x20 [ 8688.077549] [ 8688.077552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8688.077557] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8688.077563] do_raw_spin_lock+0x177/0x190 [ 8688.077567] _raw_spin_lock+0x21/0x30 [ 8688.077572] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8688.077865] process_one_work+0x2c8/0x700 [ 8688.077873] worker_thread+0x296/0x6e0 [ 8688.077877] ? rescuer_thread+0x570/0x570 [ 8688.077881] kthread+0x1d1/0x200 [ 8688.077885] ? set_kthread_struct+0x70/0x70 [ 8688.077893] ret_from_fork+0x1f/0x30 [ 8688.364701] after_reply+0xe2c/0x1780 [ptlrpc] [ 8688.366883] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8688.369241] ? __next_timer_interrupt+0x160/0x160 [ 8688.371160] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8688.373137] ? do_wait_intr+0xf0/0xf0 [ 8688.374649] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8688.377102] kthread+0x1d1/0x200 [ 8688.380995] ? set_kthread_struct+0x70/0x70 [ 8688.383337] ret_from_fork+0x1f/0x30 [ 8692.054124] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8692.061006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8692.106308] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8692.113304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8692.120246] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.122321] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8692.138127] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8692.142816] RAX: 0000000000005bad RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8692.146665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8692.150239] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8692.155877] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8692.160702] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8692.163183] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8692.172849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8692.174905] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8692.179347] Call Trace: [ 8692.182382] [ 8692.188280] ? show_regs.cold.9+0x22/0x2f [ 8692.192635] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8692.197801] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8692.202955] ? watchdog+0x40/0x40 [ 8692.208050] ? hrtimer_interrupt+0x128/0x2c0 [ 8692.210282] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8692.212871] ? apic_timer_interrupt+0xf/0x20 [ 8692.215357] [ 8692.216363] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.220107] do_raw_spin_lock+0x177/0x190 [ 8692.222330] _raw_spin_lock+0x21/0x30 [ 8692.224057] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8692.226957] ? do_raw_spin_unlock+0x75/0x190 [ 8692.228975] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8692.231361] ? do_wait_intr+0xf0/0xf0 [ 8692.232931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8692.236082] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8692.238179] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8692.244151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.249354] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8692.251383] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8692.253112] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8692.254968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.257262] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8692.259306] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8692.261265] lmv_intent_open+0x338/0x1150 [lmv] [ 8692.263885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.266625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.269770] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8692.272491] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8692.275256] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.278730] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8692.281292] ? do_raw_spin_unlock+0x75/0x190 [ 8692.283971] ? _raw_spin_unlock+0x12/0x30 [ 8692.285370] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8692.287689] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8692.289318] ll_atomic_open+0x36a/0xd70 [lustre] [ 8692.293260] lookup_open+0x356/0xb40 [ 8692.298958] do_last+0x313/0xfc0 [ 8692.301344] ? nd_jump_root+0xe5/0x160 [ 8692.303282] ? path_init+0x437/0x520 [ 8692.308395] path_openat+0xf7/0x500 [ 8692.312109] ? do_raw_spin_unlock+0x75/0x190 [ 8692.316108] do_filp_open+0x99/0x140 [ 8692.319934] ? getname_flags+0x6e/0x330 [ 8692.324707] ? __check_object_size+0xff/0x256 [ 8692.326834] ? do_raw_spin_unlock+0x75/0x190 [ 8692.332851] ? _raw_spin_unlock+0x12/0x30 [ 8692.338979] do_sys_openat2+0x2b4/0x410 [ 8692.344791] do_sys_open+0x73/0xa0 [ 8692.348636] __x64_sys_openat+0x24/0x30 [ 8692.355052] do_syscall_64+0xc1/0x440 [ 8692.360102] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8692.366931] RIP: 0033:0x7fa749552332 [ 8692.369614] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8692.384136] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8692.388861] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8692.392243] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8692.397373] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8692.404080] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8692.412363] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8696.062728] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8696.071842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8696.099666] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8696.107098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8696.110829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8696.113114] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8696.125972] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8696.128858] RAX: 000000000000428d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8696.130502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8696.141624] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8696.149688] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8696.159108] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8696.163444] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8696.168223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8696.170690] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8696.180136] Call Trace: [ 8696.180986] [ 8696.182048] ? show_regs.cold.9+0x22/0x2f [ 8696.184185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8696.190028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8696.193311] ? watchdog+0x40/0x40 [ 8696.196988] ? hrtimer_interrupt+0x128/0x2c0 [ 8696.200037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8696.202085] ? apic_timer_interrupt+0xf/0x20 [ 8696.203708] [ 8696.204683] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8696.206926] do_raw_spin_lock+0x177/0x190 [ 8696.208783] _raw_spin_lock+0x21/0x30 [ 8696.210197] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8696.213931] ? do_raw_spin_unlock+0x75/0x190 [ 8696.215587] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8696.218158] ? do_wait_intr+0xf0/0xf0 [ 8696.219639] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8696.221842] mdc_close+0x53b/0xd70 [mdc] [ 8696.223193] lmv_close+0x18d/0x490 [lmv] [ 8696.224683] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8696.227062] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8696.229021] ll_md_close+0x274/0x800 [lustre] [ 8696.231008] ll_file_release+0x128/0xc50 [lustre] [ 8696.233168] __fput+0x10d/0x440 [ 8696.234669] ____fput+0x12/0x20 [ 8696.236121] task_work_run+0xc6/0x110 [ 8696.237718] exit_to_usermode_loop+0x1dd/0x1f0 [ 8696.239667] do_syscall_64+0x3d6/0x440 [ 8696.241066] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8696.242988] RIP: 0033:0x7fa749552cd8 [ 8696.246655] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8696.258090] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8696.261159] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8696.267508] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8696.275211] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8696.278174] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8696.281303] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8698.338291] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6781s! [ 8698.350008] Showing busy workqueues and worker pools: [ 8698.356928] workqueue events: flags=0x0 [ 8698.362078] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8698.371108] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8698.383143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8698.390051] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8698.397975] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.402095] pending: kfree_rcu_monitor [ 8698.404105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8698.408034] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8698.413519] workqueue events_long: flags=0x0 [ 8698.415882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.418955] pending: lu_objects_flush [obdclass] [ 8698.421795] workqueue events_unbound: flags=0x2 [ 8698.424158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8698.428163] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8698.433150] workqueue events_power_efficient: flags=0x80 [ 8698.435865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.443945] pending: check_lifetime [ 8698.449057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.455913] pending: neigh_periodic_work [ 8698.458889] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.463228] pending: neigh_periodic_work [ 8698.466229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.470876] pending: do_cache_clean [ 8698.473193] workqueue mm_percpu_wq: flags=0x8 [ 8698.474800] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.479862] pending: vmstat_update [ 8698.482142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.486725] pending: vmstat_update [ 8698.488852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8698.492994] pending: vmstat_update [ 8698.495644] workqueue writeback: flags=0x4a [ 8698.497190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8698.507714] pending: wb_workfn, wb_workfn [ 8698.510755] workqueue xprtiod: flags=0xa [ 8698.513081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8698.522100] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8698.528932] workqueue ipv6_addrconf: flags=0x40008 [ 8698.532022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8698.536495] pending: addrconf_verify_work [ 8698.538895] workqueue knbd0-recv: flags=0x1a [ 8698.541672] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8698.545092] in-flight: 651:recv_work [ 8698.547316] workqueue ptlrpc_pinger: flags=0x2 [ 8698.551070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8698.554923] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8698.558838] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6781s workers=4 idle: 37 31846 28897 [ 8698.563374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8688s workers=2 idle: 653 [ 8716.057059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8716.058484] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8716.059126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8716.059195] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8716.059201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8716.059207] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8716.063167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8716.063245] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8716.063252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8716.063255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8716.063296] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8716.063300] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8716.063305] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8716.063307] RDX: 0000000000004cf7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8716.063310] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8716.063312] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8716.063314] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8716.063317] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8716.063320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8716.063322] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8716.063331] Call Trace: [ 8716.063338] [ 8716.063345] ? show_regs.cold.9+0x22/0x2f [ 8716.063353] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8716.063359] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8716.063363] ? watchdog+0x40/0x40 [ 8716.063367] ? hrtimer_interrupt+0x128/0x2c0 [ 8716.063371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8716.063379] ? apic_timer_interrupt+0xf/0x20 [ 8716.063384] [ 8716.063387] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8716.063392] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8716.063397] do_raw_spin_lock+0x177/0x190 [ 8716.063401] _raw_spin_lock+0x21/0x30 [ 8716.063407] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8716.063736] obd_mod_free+0x46/0x140 [ptlrpc] [ 8716.064053] mdc_commit_open+0x91/0xb0 [mdc] [ 8716.064095] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8716.064319] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8716.064581] after_reply+0xe2c/0x1780 [ptlrpc] [ 8716.064830] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8716.065065] ? __next_timer_interrupt+0x160/0x160 [ 8716.065076] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8716.065295] ? do_wait_intr+0xf0/0xf0 [ 8716.065304] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8716.065508] kthread+0x1d1/0x200 [ 8716.065518] ? set_kthread_struct+0x70/0x70 [ 8716.065523] ret_from_fork+0x1f/0x30 [ 8716.243473] [ 8716.243935] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8716.247963] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8716.263666] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8716.267652] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8716.270124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8716.276592] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8716.279171] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8716.282898] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8716.291804] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8716.296684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8716.300548] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8716.305240] Call Trace: [ 8716.305869] [ 8716.307389] ? show_regs.cold.9+0x22/0x2f [ 8716.308491] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8716.309841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8716.318820] ? watchdog+0x40/0x40 [ 8716.326819] ? hrtimer_interrupt+0x128/0x2c0 [ 8716.332191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8716.333745] ? apic_timer_interrupt+0xf/0x20 [ 8716.335036] [ 8716.341410] ? native_safe_halt+0x2e/0x50 [ 8716.342537] kvm_wait+0xcd/0xe0 [ 8716.343573] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8716.354428] do_raw_spin_lock+0x177/0x190 [ 8716.357352] _raw_spin_lock+0x21/0x30 [ 8716.359426] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8716.362219] process_one_work+0x2c8/0x700 [ 8716.364227] worker_thread+0x296/0x6e0 [ 8716.365366] ? rescuer_thread+0x570/0x570 [ 8716.371421] kthread+0x1d1/0x200 [ 8716.375101] ? set_kthread_struct+0x70/0x70 [ 8716.378841] ret_from_fork+0x1f/0x30 [ 8724.057297] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8724.060033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8724.060252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8724.060323] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8724.060328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8724.060331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8724.060345] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8724.060349] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8724.060353] RAX: 0000000000003e83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8724.060356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8724.060357] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8724.060359] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8724.060362] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8724.060365] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8724.060368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8724.060371] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8724.060379] Call Trace: [ 8724.060384] [ 8724.060389] ? show_regs.cold.9+0x22/0x2f [ 8724.060399] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8724.060405] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8724.060409] ? watchdog+0x40/0x40 [ 8724.060487] ? hrtimer_interrupt+0x128/0x2c0 [ 8724.060498] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8724.060507] ? apic_timer_interrupt+0xf/0x20 [ 8724.060512] [ 8724.060514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8724.060521] do_raw_spin_lock+0x177/0x190 [ 8724.060526] _raw_spin_lock+0x21/0x30 [ 8724.060531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8724.061096] ? do_raw_spin_unlock+0x75/0x190 [ 8724.061103] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8724.061384] ? do_wait_intr+0xf0/0xf0 [ 8724.061392] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8724.068428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel [ 8724.091078] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8724.091471] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8724.091998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8724.092149] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8724.092218] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8724.092328] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8724.092360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8724.092462] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8724.092889] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8724.092921] lmv_intent_open+0x338/0x1150 [lmv] [ 8724.092952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8724.093042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8724.093577] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8724.093606] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8724.093775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8724.093970] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8724.094055] ? do_raw_spin_unlock+0x75/0x190 [ 8724.094064] ? _raw_spin_unlock+0x12/0x30 [ 8724.094070] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8724.104556] ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8724.104585] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8724.104594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8724.104597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8724.104614] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8724.104619] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8724.104624] RAX: 0000000000006f07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8724.109671] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8724.109727] ll_atomic_open+0x36a/0xd70 [lustre] [ 8724.109920] lookup_open+0x356/0xb40 [ 8724.109932] do_last+0x313/0xfc0 [ 8724.109937] ? nd_jump_root+0xe5/0x160 [ 8724.109942] ? path_init+0x437/0x520 [ 8724.109946] path_openat+0xf7/0x500 [ 8724.109951] ? do_raw_spin_unlock+0x75/0x190 [ 8724.109959] do_filp_open+0x99/0x140 [ 8724.109966] ? getname_flags+0x6e/0x330 [ 8724.109970] ? __check_object_size+0xff/0x256 [ 8724.109978] ? do_raw_spin_unlock+0x75/0x190 [ 8724.109982] ? _raw_spin_unlock+0x12/0x30 [ 8724.109990] do_sys_openat2+0x2b4/0x410 [ 8724.109995] do_sys_open+0x73/0xa0 [ 8724.110000] __x64_sys_openat+0x24/0x30 [ 8724.110004] do_syscall_64+0xc1/0x440 [ 8724.110012] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8724.110018] RIP: 0033:0x7fa749552332 [ 8724.110025] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8724.110030] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8724.110035] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8724.110037] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8724.110039] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8724.110041] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8724.110044] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8724.474654] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8724.479648] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8724.490721] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8724.494795] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8724.500221] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8724.507506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8724.509602] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8724.513328] Call Trace: [ 8724.514540] [ 8724.515465] ? show_regs.cold.9+0x22/0x2f [ 8724.517480] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8724.519663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8724.521752] ? watchdog+0x40/0x40 [ 8724.523474] ? hrtimer_interrupt+0x128/0x2c0 [ 8724.525566] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8724.527772] ? apic_timer_interrupt+0xf/0x20 [ 8724.529938] [ 8724.531025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8724.533791] do_raw_spin_lock+0x177/0x190 [ 8724.535763] _raw_spin_lock+0x21/0x30 [ 8724.537925] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8724.541350] ? do_raw_spin_unlock+0x75/0x190 [ 8724.543051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8724.547737] ? do_wait_intr+0xf0/0xf0 [ 8724.550649] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8724.557476] mdc_close+0x53b/0xd70 [mdc] [ 8724.562510] lmv_close+0x18d/0x490 [lmv] [ 8724.565345] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8724.569251] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8724.572286] ll_md_close+0x274/0x800 [lustre] [ 8724.574609] ll_file_release+0x128/0xc50 [lustre] [ 8724.578103] __fput+0x10d/0x440 [ 8724.579218] ____fput+0x12/0x20 [ 8724.580301] task_work_run+0xc6/0x110 [ 8724.584313] exit_to_usermode_loop+0x1dd/0x1f0 [ 8724.586894] do_syscall_64+0x3d6/0x440 [ 8724.590014] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8724.608845] RIP: 0033:0x7fa749552cd8 [ 8724.610555] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8724.623509] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8724.629624] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8724.636703] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8724.640777] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8724.644299] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8724.648511] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8729.056090] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6811s! [ 8729.059321] Showing busy workqueues and worker pools: [ 8729.061833] workqueue events: flags=0x0 [ 8729.063422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8729.065700] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8729.070712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8729.072996] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8729.076360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.079282] pending: kfree_rcu_monitor [ 8729.080573] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8729.085973] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8729.090840] workqueue events_long: flags=0x0 [ 8729.092395] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.094598] pending: lu_objects_flush [obdclass] [ 8729.096901] workqueue events_unbound: flags=0x2 [ 8729.099130] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8729.101194] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8729.105963] workqueue events_power_efficient: flags=0x80 [ 8729.108252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.111818] pending: check_lifetime [ 8729.113417] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.115923] pending: neigh_periodic_work [ 8729.117692] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.120125] pending: neigh_periodic_work [ 8729.122320] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.125280] pending: do_cache_clean [ 8729.126733] workqueue mm_percpu_wq: flags=0x8 [ 8729.128562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.131943] pending: vmstat_update [ 8729.133331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.135517] pending: vmstat_update [ 8729.137348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8729.139668] pending: vmstat_update [ 8729.141257] workqueue writeback: flags=0x4a [ 8729.142728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8729.145410] pending: wb_workfn, wb_workfn [ 8729.147273] workqueue xprtiod: flags=0xa [ 8729.148693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8729.151563] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8729.154700] workqueue ipv6_addrconf: flags=0x40008 [ 8729.156797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8729.159433] pending: addrconf_verify_work [ 8729.161533] workqueue knbd0-recv: flags=0x1a [ 8729.163222] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8729.165680] in-flight: 651:recv_work [ 8729.168388] workqueue ptlrpc_pinger: flags=0x2 [ 8729.172645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8729.174866] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8729.178866] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6812s workers=4 idle: 37 31846 28897 [ 8729.185958] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8719s workers=2 idle: 653 [ 8744.057113] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8744.059783] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8744.060918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8744.060992] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8744.060998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8744.061005] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8744.061535] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8744.061546] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8744.061550] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8744.061555] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8744.061557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8744.061559] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8744.061561] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8744.061564] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8744.061567] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8744.061570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8744.061572] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8744.061580] Call Trace: [ 8744.061586] [ 8744.061591] ? show_regs.cold.9+0x22/0x2f [ 8744.061600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8744.061606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8744.061610] ? watchdog+0x40/0x40 [ 8744.061614] ? hrtimer_interrupt+0x128/0x2c0 [ 8744.061618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8744.061625] ? apic_timer_interrupt+0xf/0x20 [ 8744.061629] [ 8744.061632] ? native_safe_halt+0x2e/0x50 [ 8744.061635] kvm_wait+0xcd/0xe0 [ 8744.061644] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8744.061652] do_raw_spin_lock+0x177/0x190 [ 8744.061657] _raw_spin_lock+0x21/0x30 [ 8744.061661] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8744.061951] process_one_work+0x2c8/0x700 [ 8744.061958] worker_thread+0x296/0x6e0 [ 8744.061963] ? rescuer_thread+0x570/0x570 [ 8744.061967] kthread+0x1d1/0x200 [ 8744.061971] ? set_kthread_struct+0x70/0x70 [ 8744.061975] ret_from_fork+0x1f/0x30 [ 8744.204547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8744.235331] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8744.241196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8744.245069] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8744.247098] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8744.253018] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8744.259460] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8744.263260] RDX: 0000000000007f80 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8744.266778] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8744.274385] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8744.277644] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8744.281192] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8744.287909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8744.290342] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8744.293417] Call Trace: [ 8744.294476] [ 8744.296064] ? show_regs.cold.9+0x22/0x2f [ 8744.297682] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8744.300224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8744.303044] ? watchdog+0x40/0x40 [ 8744.304398] ? hrtimer_interrupt+0x128/0x2c0 [ 8744.311372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8744.317296] ? apic_timer_interrupt+0xf/0x20 [ 8744.320089] [ 8744.321048] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8744.323831] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8744.326303] do_raw_spin_lock+0x177/0x190 [ 8744.330435] _raw_spin_lock+0x21/0x30 [ 8744.333712] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8744.336214] obd_mod_free+0x46/0x140 [ptlrpc] [ 8744.339193] mdc_commit_open+0x91/0xb0 [mdc] [ 8744.343468] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8744.351715] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8744.358059] after_reply+0xe2c/0x1780 [ptlrpc] [ 8744.361922] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8744.371806] ? __next_timer_interrupt+0x160/0x160 [ 8744.374745] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8744.379166] ? do_wait_intr+0xf0/0xf0 [ 8744.381459] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8744.384857] kthread+0x1d1/0x200 [ 8744.387234] ? set_kthread_struct+0x70/0x70 [ 8744.389827] ret_from_fork+0x1f/0x30 [ 8752.059666] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8752.068321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8752.114005] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8752.122673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8752.129842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8752.135712] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8752.152119] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8752.157058] RAX: 0000000000001d85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8752.160938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8752.170055] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8752.177540] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8752.184498] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8752.188498] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8752.193613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8752.196237] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8752.199914] Call Trace: [ 8752.201025] [ 8752.202132] ? show_regs.cold.9+0x22/0x2f [ 8752.204004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8752.206941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8752.209502] ? watchdog+0x40/0x40 [ 8752.211022] ? hrtimer_interrupt+0x128/0x2c0 [ 8752.212563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8752.214661] ? apic_timer_interrupt+0xf/0x20 [ 8752.215965] [ 8752.216605] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8752.218562] do_raw_spin_lock+0x177/0x190 [ 8752.219978] _raw_spin_lock+0x21/0x30 [ 8752.221210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8752.223100] ? do_raw_spin_unlock+0x75/0x190 [ 8752.224495] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8752.226268] ? do_wait_intr+0xf0/0xf0 [ 8752.227483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8752.229552] mdc_close+0x53b/0xd70 [mdc] [ 8752.230955] lmv_close+0x18d/0x490 [lmv] [ 8752.232035] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8752.233797] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8752.235138] ll_md_close+0x274/0x800 [lustre] [ 8752.236457] ll_file_release+0x128/0xc50 [lustre] [ 8752.237981] __fput+0x10d/0x440 [ 8752.238814] ____fput+0x12/0x20 [ 8752.239590] task_work_run+0xc6/0x110 [ 8752.240618] exit_to_usermode_loop+0x1dd/0x1f0 [ 8752.241792] do_syscall_64+0x3d6/0x440 [ 8752.242602] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8752.244063] RIP: 0033:0x7fa749552cd8 [ 8752.244993] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8752.249579] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8752.252137] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8752.255263] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8752.258922] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8752.262202] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8752.265494] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8756.054033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8756.055881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8756.082427] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8756.090039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8756.096621] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8756.099527] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8756.111586] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8756.117456] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8756.123411] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8756.126004] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8756.132160] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8756.137540] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8756.142865] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8756.148923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8756.151598] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8756.157233] Call Trace: [ 8756.157997] [ 8756.158534] ? show_regs.cold.9+0x22/0x2f [ 8756.159995] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8756.161667] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8756.165911] ? watchdog+0x40/0x40 [ 8756.168048] ? hrtimer_interrupt+0x128/0x2c0 [ 8756.170809] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8756.174878] ? apic_timer_interrupt+0xf/0x20 [ 8756.176012] [ 8756.178850] ? native_safe_halt+0x2e/0x50 [ 8756.181428] kvm_wait+0xcd/0xe0 [ 8756.182885] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8756.187924] do_raw_spin_lock+0x177/0x190 [ 8756.191065] _raw_spin_lock+0x21/0x30 [ 8756.192292] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8756.195692] ? do_raw_spin_unlock+0x75/0x190 [ 8756.199180] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8756.204074] ? do_wait_intr+0xf0/0xf0 [ 8756.205245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8756.207116] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8756.209220] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8756.215663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8756.218664] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8756.223099] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8756.228158] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8756.230096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8756.234475] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8756.238746] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8756.242908] lmv_intent_open+0x338/0x1150 [lmv] [ 8756.245197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8756.247446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8756.253109] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8756.256953] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8756.260202] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8756.263992] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8756.266205] ? do_raw_spin_unlock+0x75/0x190 [ 8756.268252] ? _raw_spin_unlock+0x12/0x30 [ 8756.272245] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8756.274160] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8756.278512] ll_atomic_open+0x36a/0xd70 [lustre] [ 8756.280959] lookup_open+0x356/0xb40 [ 8756.283244] do_last+0x313/0xfc0 [ 8756.286486] ? nd_jump_root+0xe5/0x160 [ 8756.288098] ? path_init+0x437/0x520 [ 8756.289328] path_openat+0xf7/0x500 [ 8756.292124] ? do_raw_spin_unlock+0x75/0x190 [ 8756.294579] do_filp_open+0x99/0x140 [ 8756.297234] ? getname_flags+0x6e/0x330 [ 8756.299126] ? __check_object_size+0xff/0x256 [ 8756.300699] ? do_raw_spin_unlock+0x75/0x190 [ 8756.303528] ? _raw_spin_unlock+0x12/0x30 [ 8756.307363] do_sys_openat2+0x2b4/0x410 [ 8756.308563] do_sys_open+0x73/0xa0 [ 8756.309593] __x64_sys_openat+0x24/0x30 [ 8756.312460] do_syscall_64+0xc1/0x440 [ 8756.315795] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8756.319569] RIP: 0033:0x7fa749552332 [ 8756.322026] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8756.332693] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8756.339985] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8756.345224] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8756.351346] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8756.357258] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8756.359685] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8759.265077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6842s! [ 8759.268871] Showing busy workqueues and worker pools: [ 8759.271294] workqueue events: flags=0x0 [ 8759.273131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8759.276545] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8759.281093] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8759.284703] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8759.289154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.291372] pending: kfree_rcu_monitor [ 8759.294165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8759.297482] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8759.301774] workqueue events_long: flags=0x0 [ 8759.305013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.310536] pending: lu_objects_flush [obdclass] [ 8759.320471] workqueue events_unbound: flags=0x2 [ 8759.324548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8759.331117] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8759.348631] workqueue events_power_efficient: flags=0x80 [ 8759.357123] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.366419] pending: check_lifetime [ 8759.369071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.375180] pending: neigh_periodic_work [ 8759.378146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.381248] pending: neigh_periodic_work [ 8759.384201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.388301] pending: do_cache_clean [ 8759.390705] workqueue mm_percpu_wq: flags=0x8 [ 8759.393612] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.405167] pending: vmstat_update [ 8759.409109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.416093] pending: vmstat_update [ 8759.420712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8759.426903] pending: vmstat_update [ 8759.428918] workqueue writeback: flags=0x4a [ 8759.434522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8759.440959] pending: wb_workfn, wb_workfn [ 8759.442781] workqueue xprtiod: flags=0xa [ 8759.446801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8759.454424] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8759.458171] workqueue ipv6_addrconf: flags=0x40008 [ 8759.460805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8759.464114] pending: addrconf_verify_work [ 8759.466155] workqueue knbd0-recv: flags=0x1a [ 8759.469044] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8759.472679] in-flight: 651:recv_work [ 8759.474398] workqueue ptlrpc_pinger: flags=0x2 [ 8759.476597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8759.480007] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8759.490112] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6842s workers=4 idle: 37 31846 28897 [ 8759.497863] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8749s workers=2 idle: 653 [ 8772.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8772.058114] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8772.060637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8772.060787] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8772.060794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8772.060801] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8772.061258] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8772.061268] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8772.061272] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8772.061276] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8772.061290] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8772.061292] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8772.061295] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8772.061297] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8772.061301] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8772.061304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8772.061306] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8772.061314] Call Trace: [ 8772.061320] [ 8772.061325] ? show_regs.cold.9+0x22/0x2f [ 8772.061334] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8772.061341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8772.061345] ? watchdog+0x40/0x40 [ 8772.061350] ? hrtimer_interrupt+0x128/0x2c0 [ 8772.061355] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8772.061361] ? apic_timer_interrupt+0xf/0x20 [ 8772.061365] [ 8772.061368] ? native_safe_halt+0x2e/0x50 [ 8772.061373] kvm_wait+0xcd/0xe0 [ 8772.061378] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8772.061386] do_raw_spin_lock+0x177/0x190 [ 8772.061390] _raw_spin_lock+0x21/0x30 [ 8772.061394] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8772.061765] process_one_work+0x2c8/0x700 [ 8772.061773] worker_thread+0x296/0x6e0 [ 8772.061777] ? rescuer_thread+0x570/0x570 [ 8772.061781] kthread+0x1d1/0x200 [ 8772.061786] ? set_kthread_struct+0x70/0x70 [ 8772.061790] ret_from_fork+0x1f/0x30 [ 8772.278938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8772.309725] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8772.318924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8772.324895] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8772.329204] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8772.343796] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8772.349979] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8772.356368] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8772.362489] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8772.365204] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8772.370873] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8772.373541] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8772.379026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8772.381157] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8772.383995] Call Trace: [ 8772.384764] [ 8772.386375] ? show_regs.cold.9+0x22/0x2f [ 8772.388828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8772.392791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8772.394023] ? watchdog+0x40/0x40 [ 8772.395170] ? hrtimer_interrupt+0x128/0x2c0 [ 8772.397696] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8772.402238] ? apic_timer_interrupt+0xf/0x20 [ 8772.404100] [ 8772.405181] ? native_safe_halt+0x2e/0x50 [ 8772.409739] kvm_wait+0xcd/0xe0 [ 8772.411711] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8772.415764] do_raw_spin_lock+0x177/0x190 [ 8772.417163] _raw_spin_lock+0x21/0x30 [ 8772.420254] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8772.425325] obd_mod_free+0x46/0x140 [ptlrpc] [ 8772.428747] mdc_commit_open+0x91/0xb0 [mdc] [ 8772.431830] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8772.436412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8772.440618] after_reply+0xe2c/0x1780 [ptlrpc] [ 8772.443738] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8772.446961] ? __next_timer_interrupt+0x160/0x160 [ 8772.452557] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8772.454852] ? do_wait_intr+0xf0/0xf0 [ 8772.456708] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8772.458435] kthread+0x1d1/0x200 [ 8772.460398] ? set_kthread_struct+0x70/0x70 [ 8772.462369] ret_from_fork+0x1f/0x30 [ 8780.059375] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8780.064627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8780.084360] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8780.088998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8780.092444] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8780.094369] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8780.103021] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8780.106366] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8780.110518] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8780.113530] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8780.116354] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8780.120396] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8780.124546] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8780.131444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8780.137827] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8780.143555] Call Trace: [ 8780.146633] [ 8780.147922] ? show_regs.cold.9+0x22/0x2f [ 8780.150548] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8780.153060] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8780.155487] ? watchdog+0x40/0x40 [ 8780.157247] ? hrtimer_interrupt+0x128/0x2c0 [ 8780.159502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8780.161539] ? apic_timer_interrupt+0xf/0x20 [ 8780.163583] [ 8780.164527] ? native_safe_halt+0x2e/0x50 [ 8780.166381] kvm_wait+0xcd/0xe0 [ 8780.167640] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8780.170395] do_raw_spin_lock+0x177/0x190 [ 8780.171929] _raw_spin_lock+0x21/0x30 [ 8780.173340] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8780.175471] ? do_raw_spin_unlock+0x75/0x190 [ 8780.177123] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8780.181106] ? do_wait_intr+0xf0/0xf0 [ 8780.184789] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8780.191785] mdc_close+0x53b/0xd70 [mdc] [ 8780.193251] lmv_close+0x18d/0x490 [lmv] [ 8780.196423] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8780.199643] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8780.204610] ll_md_close+0x274/0x800 [lustre] [ 8780.207451] ll_file_release+0x128/0xc50 [lustre] [ 8780.210793] __fput+0x10d/0x440 [ 8780.212622] ____fput+0x12/0x20 [ 8780.214486] task_work_run+0xc6/0x110 [ 8780.217090] exit_to_usermode_loop+0x1dd/0x1f0 [ 8780.219190] do_syscall_64+0x3d6/0x440 [ 8780.221503] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8780.226374] RIP: 0033:0x7fa749552cd8 [ 8780.229345] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8780.239931] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8780.246697] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8780.249008] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8780.256987] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8780.269716] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8780.281237] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8784.058223] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8784.069581] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8784.118618] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8784.124890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8784.138340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.142700] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8784.166019] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8784.174779] RAX: 0000000000004b0d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8784.180569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8784.183771] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8784.189730] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8784.195296] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8784.201433] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8784.211129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8784.215288] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8784.219390] Call Trace: [ 8784.221283] [ 8784.222343] ? show_regs.cold.9+0x22/0x2f [ 8784.226373] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8784.229170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8784.232600] ? watchdog+0x40/0x40 [ 8784.233877] ? hrtimer_interrupt+0x128/0x2c0 [ 8784.237753] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8784.241379] ? apic_timer_interrupt+0xf/0x20 [ 8784.245860] [ 8784.250250] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.257248] do_raw_spin_lock+0x177/0x190 [ 8784.261243] _raw_spin_lock+0x21/0x30 [ 8784.267145] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8784.268949] ? do_raw_spin_unlock+0x75/0x190 [ 8784.272353] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8784.276015] ? do_wait_intr+0xf0/0xf0 [ 8784.278239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8784.282343] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8784.286986] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8784.290317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8784.295214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8784.302347] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8784.304052] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8784.307722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8784.317421] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8784.324734] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8784.331041] lmv_intent_open+0x338/0x1150 [lmv] [ 8784.337401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8784.341057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8784.354148] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8784.357120] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8784.358852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8784.367024] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8784.371952] ? do_raw_spin_unlock+0x75/0x190 [ 8784.377877] ? _raw_spin_unlock+0x12/0x30 [ 8784.381460] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8784.385009] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8784.388228] ll_atomic_open+0x36a/0xd70 [lustre] [ 8784.392273] lookup_open+0x356/0xb40 [ 8784.393640] do_last+0x313/0xfc0 [ 8784.397144] ? nd_jump_root+0xe5/0x160 [ 8784.401262] ? path_init+0x437/0x520 [ 8784.404315] path_openat+0xf7/0x500 [ 8784.406417] ? do_raw_spin_unlock+0x75/0x190 [ 8784.412111] do_filp_open+0x99/0x140 [ 8784.416160] ? getname_flags+0x6e/0x330 [ 8784.420495] ? __check_object_size+0xff/0x256 [ 8784.421677] ? do_raw_spin_unlock+0x75/0x190 [ 8784.422784] ? _raw_spin_unlock+0x12/0x30 [ 8784.429078] do_sys_openat2+0x2b4/0x410 [ 8784.432930] do_sys_open+0x73/0xa0 [ 8784.435689] __x64_sys_openat+0x24/0x30 [ 8784.439677] do_syscall_64+0xc1/0x440 [ 8784.441367] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8784.449922] RIP: 0033:0x7fa749552332 [ 8784.451977] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8784.475097] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8784.484419] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8784.496861] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8784.510354] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8784.512784] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8784.529014] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8789.986102] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6872s! [ 8789.996115] Showing busy workqueues and worker pools: [ 8790.001605] workqueue events: flags=0x0 [ 8790.003664] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8790.010774] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8790.021126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8790.024908] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8790.029233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.032464] pending: kfree_rcu_monitor [ 8790.034041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8790.036612] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8790.041201] workqueue events_long: flags=0x0 [ 8790.042587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.045097] pending: lu_objects_flush [obdclass] [ 8790.046840] workqueue events_unbound: flags=0x2 [ 8790.048428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8790.050594] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8790.054200] workqueue events_power_efficient: flags=0x80 [ 8790.056159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.058078] pending: check_lifetime [ 8790.059546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.062259] pending: neigh_periodic_work [ 8790.063668] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.066366] pending: neigh_periodic_work [ 8790.067862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.070281] pending: do_cache_clean [ 8790.072750] workqueue mm_percpu_wq: flags=0x8 [ 8790.078344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.081918] pending: vmstat_update [ 8790.086698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.091898] pending: vmstat_update [ 8790.094062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8790.096972] pending: vmstat_update [ 8790.098609] workqueue writeback: flags=0x4a [ 8790.100450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8790.104384] pending: wb_workfn, wb_workfn [ 8790.106346] workqueue xprtiod: flags=0xa [ 8790.107798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8790.109946] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8790.113158] workqueue ipv6_addrconf: flags=0x40008 [ 8790.114936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8790.117312] pending: addrconf_verify_work [ 8790.118887] workqueue knbd0-recv: flags=0x1a [ 8790.120407] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8790.122876] in-flight: 651:recv_work [ 8790.124834] workqueue ptlrpc_pinger: flags=0x2 [ 8790.133450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8790.139243] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8790.145309] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6872s workers=4 idle: 37 31846 28897 [ 8790.151587] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8780s workers=2 idle: 653 [ 8800.058822] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8800.063754] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8800.070138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.070224] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.070238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.070245] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8800.070663] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8800.070675] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8800.070679] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8800.070684] RAX: 0000000000005209 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8800.070687] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8800.070689] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8800.070691] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8800.070693] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8800.070697] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8800.070699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.070702] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8800.070710] Call Trace: [ 8800.070715] [ 8800.070721] ? show_regs.cold.9+0x22/0x2f [ 8800.070731] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.070737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.070742] ? watchdog+0x40/0x40 [ 8800.070746] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.070758] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.070765] ? apic_timer_interrupt+0xf/0x20 [ 8800.070769] [ 8800.070772] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8800.070776] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8800.070782] do_raw_spin_lock+0x177/0x190 [ 8800.070803] _raw_spin_lock+0x21/0x30 [ 8800.070810] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8800.071119] process_one_work+0x2c8/0x700 [ 8800.071126] worker_thread+0x296/0x6e0 [ 8800.071130] ? rescuer_thread+0x570/0x570 [ 8800.071134] kthread+0x1d1/0x200 [ 8800.071138] ? set_kthread_struct+0x70/0x70 [ 8800.071141] ret_from_fork+0x1f/0x30 [ 8800.349004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.394114] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.420155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.431171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8800.438315] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8800.461404] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8800.470914] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8800.481117] RDX: 00000000000003da RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8800.485365] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8800.495356] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8800.499856] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8800.513377] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8800.523456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.527312] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8800.531300] Call Trace: [ 8800.532205] [ 8800.536225] ? show_regs.cold.9+0x22/0x2f [ 8800.538223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.543809] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.549845] ? watchdog+0x40/0x40 [ 8800.553900] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.562439] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.570232] ? apic_timer_interrupt+0xf/0x20 [ 8800.579225] [ 8800.582877] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8800.591296] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8800.601041] do_raw_spin_lock+0x177/0x190 [ 8800.606423] _raw_spin_lock+0x21/0x30 [ 8800.607707] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8800.620381] obd_mod_free+0x46/0x140 [ptlrpc] [ 8800.624461] mdc_commit_open+0x91/0xb0 [mdc] [ 8800.631859] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8800.636362] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8800.639346] after_reply+0xe2c/0x1780 [ptlrpc] [ 8800.642465] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8800.649827] ? __next_timer_interrupt+0x160/0x160 [ 8800.652274] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8800.654708] ? do_wait_intr+0xf0/0xf0 [ 8800.656154] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8800.658093] kthread+0x1d1/0x200 [ 8800.659803] ? set_kthread_struct+0x70/0x70 [ 8800.663265] ret_from_fork+0x1f/0x30 [ 8808.059057] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8808.065121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8808.101661] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8808.112948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8808.118460] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8808.122286] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8808.148068] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8808.155097] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8808.161017] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8808.167072] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8808.174612] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8808.187826] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8808.194485] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8808.199763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8808.203310] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8808.210894] Call Trace: [ 8808.212585] [ 8808.213295] ? show_regs.cold.9+0x22/0x2f [ 8808.215701] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8808.218772] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8808.222340] ? watchdog+0x40/0x40 [ 8808.223931] ? hrtimer_interrupt+0x128/0x2c0 [ 8808.227073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8808.230206] ? apic_timer_interrupt+0xf/0x20 [ 8808.234795] [ 8808.235770] ? native_safe_halt+0x2e/0x50 [ 8808.238504] kvm_wait+0xcd/0xe0 [ 8808.239377] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8808.242158] do_raw_spin_lock+0x177/0x190 [ 8808.243340] _raw_spin_lock+0x21/0x30 [ 8808.245210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8808.247072] ? do_raw_spin_unlock+0x75/0x190 [ 8808.248612] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8808.250528] ? do_wait_intr+0xf0/0xf0 [ 8808.251597] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8808.253442] mdc_close+0x53b/0xd70 [mdc] [ 8808.254757] lmv_close+0x18d/0x490 [lmv] [ 8808.255962] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8808.258153] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8808.260291] ll_md_close+0x274/0x800 [lustre] [ 8808.261904] ll_file_release+0x128/0xc50 [lustre] [ 8808.263528] __fput+0x10d/0x440 [ 8808.264817] ____fput+0x12/0x20 [ 8808.266295] task_work_run+0xc6/0x110 [ 8808.267482] exit_to_usermode_loop+0x1dd/0x1f0 [ 8808.268831] do_syscall_64+0x3d6/0x440 [ 8808.269892] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8808.271471] RIP: 0033:0x7fa749552cd8 [ 8808.272516] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8808.278809] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8808.285787] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8808.288595] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8808.290653] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8808.293078] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8808.295654] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8813.838031] rcu: INFO: rcu_sched self-detected stall on CPU [ 8813.840334] rcu: 2-...!: (3880737 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 8813.844955] (t=6900149 jiffies g=258701 q=3464) [ 8813.846955] rcu: rcu_sched kthread starved for 6896944 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8813.851619] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8813.854927] rcu: RCU grace-period kthread stack dump: [ 8813.857331] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8813.861229] Call Trace: [ 8813.862472] __schedule+0x351/0xcb0 [ 8813.864277] schedule+0xc0/0x180 [ 8813.865632] schedule_timeout+0xb4/0x190 [ 8813.867205] ? do_raw_spin_unlock+0x75/0x190 [ 8813.869170] ? __next_timer_interrupt+0x160/0x160 [ 8813.871646] ? prepare_to_swait_event+0xf3/0x140 [ 8813.873963] rcu_gp_kthread+0x8dd/0x15b0 [ 8813.876118] ? force_qs_rnp+0x2c0/0x2c0 [ 8813.878481] kthread+0x1d1/0x200 [ 8813.879829] ? set_kthread_struct+0x70/0x70 [ 8813.881603] ret_from_fork+0x1f/0x30 [ 8813.882878] rcu: Stack dump where RCU GP kthread last ran: [ 8813.885529] NMI backtrace for cpu 2 [ 8813.887025] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8813.891835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8813.895593] Call Trace: [ 8813.896720] [ 8813.898044] ? dump_stack+0xbb/0x10e [ 8813.899325] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8813.901474] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8813.903197] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8813.904812] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8813.906775] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8813.908776] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8813.911148] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 8813.913369] ? update_process_times+0x6d/0xc0 [ 8813.915419] ? tick_sched_handle.isra.18+0x27/0x90 [ 8813.917246] ? tick_sched_timer+0x53/0xb0 [ 8813.918826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8813.920752] ? tick_sched_handle.isra.18+0x90/0x90 [ 8813.922741] ? hrtimer_interrupt+0x128/0x2c0 [ 8813.924809] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8813.926821] ? apic_timer_interrupt+0xf/0x20 [ 8813.928495] [ 8813.929362] ? native_safe_halt+0x2e/0x50 [ 8813.930978] ? kvm_wait+0xcd/0xe0 [ 8813.932361] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8813.934651] ? do_raw_spin_lock+0x177/0x190 [ 8813.936242] ? _raw_spin_lock+0x21/0x30 [ 8813.937887] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8813.939916] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8813.942141] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8813.944175] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8813.946529] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8813.948850] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8813.950861] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8813.952611] ? __next_timer_interrupt+0x160/0x160 [ 8813.953975] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8813.955655] ? do_wait_intr+0xf0/0xf0 [ 8813.956768] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8813.958714] ? kthread+0x1d1/0x200 [ 8813.959655] ? set_kthread_struct+0x70/0x70 [ 8813.961219] ? ret_from_fork+0x1f/0x30 [ 8813.962367] NMI backtrace for cpu 2 [ 8813.963365] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8813.966652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8813.969210] Call Trace: [ 8813.970102] [ 8813.970895] ? dump_stack+0xbb/0x10e [ 8813.971920] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8813.973933] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8813.975140] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8813.976836] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8813.978449] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8813.980223] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8813.981660] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8813.983228] ? update_process_times+0x6d/0xc0 [ 8813.984986] ? tick_sched_handle.isra.18+0x27/0x90 [ 8813.986359] ? tick_sched_timer+0x53/0xb0 [ 8813.987566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8813.988908] ? tick_sched_handle.isra.18+0x90/0x90 [ 8813.990360] ? hrtimer_interrupt+0x128/0x2c0 [ 8813.991752] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8813.993160] ? apic_timer_interrupt+0xf/0x20 [ 8813.994515] [ 8813.995082] ? native_safe_halt+0x2e/0x50 [ 8813.996228] ? kvm_wait+0xcd/0xe0 [ 8813.997231] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8813.999186] ? do_raw_spin_lock+0x177/0x190 [ 8814.000718] ? _raw_spin_lock+0x21/0x30 [ 8814.002057] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8814.003963] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8814.005746] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8814.007184] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8814.009101] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8814.011332] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8814.013116] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8814.015202] ? __next_timer_interrupt+0x160/0x160 [ 8814.016781] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8814.018252] ? do_wait_intr+0xf0/0xf0 [ 8814.019231] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8814.020886] ? kthread+0x1d1/0x200 [ 8814.021812] ? set_kthread_struct+0x70/0x70 [ 8814.023152] ? ret_from_fork+0x1f/0x30 [ 8816.054037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8816.058153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8816.084798] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8816.090748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8816.095727] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8816.098076] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8816.108840] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8816.116039] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8816.119057] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8816.122459] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8816.126354] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8816.131629] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8816.136565] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8816.142759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8816.146503] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8816.150158] Call Trace: [ 8816.151204] [ 8816.152073] ? show_regs.cold.9+0x22/0x2f [ 8816.153749] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8816.155561] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8816.157531] ? watchdog+0x40/0x40 [ 8816.159615] ? hrtimer_interrupt+0x128/0x2c0 [ 8816.162084] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8816.165979] ? apic_timer_interrupt+0xf/0x20 [ 8816.169536] [ 8816.170906] ? native_safe_halt+0x2e/0x50 [ 8816.173385] kvm_wait+0xcd/0xe0 [ 8816.175203] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8816.177853] do_raw_spin_lock+0x177/0x190 [ 8816.180760] _raw_spin_lock+0x21/0x30 [ 8816.183427] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8816.187374] ? do_raw_spin_unlock+0x75/0x190 [ 8816.189498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8816.192552] ? do_wait_intr+0xf0/0xf0 [ 8816.194150] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8816.197050] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8816.199654] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8816.202610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8816.204980] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8816.209869] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8816.217915] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8816.227571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8816.233962] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8816.238609] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8816.240681] lmv_intent_open+0x338/0x1150 [lmv] [ 8816.242569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8816.244512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8816.247786] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8816.250673] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8816.252383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8816.254415] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8816.256062] ? do_raw_spin_unlock+0x75/0x190 [ 8816.258612] ? _raw_spin_unlock+0x12/0x30 [ 8816.259884] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8816.265638] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8816.267481] ll_atomic_open+0x36a/0xd70 [lustre] [ 8816.270867] lookup_open+0x356/0xb40 [ 8816.273654] do_last+0x313/0xfc0 [ 8816.275213] ? nd_jump_root+0xe5/0x160 [ 8816.277630] ? path_init+0x437/0x520 [ 8816.279139] path_openat+0xf7/0x500 [ 8816.280979] ? do_raw_spin_unlock+0x75/0x190 [ 8816.283374] do_filp_open+0x99/0x140 [ 8816.286220] ? getname_flags+0x6e/0x330 [ 8816.289911] ? __check_object_size+0xff/0x256 [ 8816.294839] ? do_raw_spin_unlock+0x75/0x190 [ 8816.299077] ? _raw_spin_unlock+0x12/0x30 [ 8816.304122] do_sys_openat2+0x2b4/0x410 [ 8816.307153] do_sys_open+0x73/0xa0 [ 8816.311647] __x64_sys_openat+0x24/0x30 [ 8816.313188] do_syscall_64+0xc1/0x440 [ 8816.316679] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8816.318459] RIP: 0033:0x7fa749552332 [ 8816.322599] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8816.334959] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8816.339377] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8816.344456] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8816.351689] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8816.356442] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8816.359682] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8820.193189] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6903s! [ 8820.196293] Showing busy workqueues and worker pools: [ 8820.198277] workqueue events: flags=0x0 [ 8820.199816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8820.202669] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8820.208197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8820.210700] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8820.214561] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.217425] pending: kfree_rcu_monitor [ 8820.219036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8820.228614] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8820.237411] workqueue events_long: flags=0x0 [ 8820.241718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.248351] pending: lu_objects_flush [obdclass] [ 8820.256095] workqueue events_unbound: flags=0x2 [ 8820.260073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8820.270628] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8820.280243] workqueue events_power_efficient: flags=0x80 [ 8820.284112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.290493] pending: check_lifetime [ 8820.294346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.301375] pending: neigh_periodic_work [ 8820.304493] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.309883] pending: neigh_periodic_work [ 8820.311855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.315090] pending: do_cache_clean [ 8820.317594] workqueue mm_percpu_wq: flags=0x8 [ 8820.319410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.326838] pending: vmstat_update [ 8820.328898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.336337] pending: vmstat_update [ 8820.339216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8820.348408] pending: vmstat_update [ 8820.350825] workqueue writeback: flags=0x4a [ 8820.356172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8820.360895] pending: wb_workfn, wb_workfn [ 8820.362092] workqueue xprtiod: flags=0xa [ 8820.367915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8820.375867] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8820.383051] workqueue ipv6_addrconf: flags=0x40008 [ 8820.385300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8820.392041] pending: addrconf_verify_work [ 8820.395934] workqueue knbd0-recv: flags=0x1a [ 8820.399104] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8820.402797] in-flight: 651:recv_work [ 8820.404803] workqueue ptlrpc_pinger: flags=0x2 [ 8820.406362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8820.408390] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8820.410609] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6903s workers=4 idle: 37 31846 28897 [ 8820.413716] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8810s workers=2 idle: 653 [ 8828.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8828.062934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8828.089957] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8828.097646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8828.104128] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8828.108852] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8828.113896] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8828.128974] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8828.132986] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8828.136733] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8828.140452] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8828.144487] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8828.148368] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8828.151976] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8828.156134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8828.158825] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8828.164324] Call Trace: [ 8828.165994] [ 8828.167990] ? show_regs.cold.9+0x22/0x2f [ 8828.170309] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8828.175203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8828.176987] ? watchdog+0x40/0x40 [ 8828.179151] ? hrtimer_interrupt+0x128/0x2c0 [ 8828.181986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8828.185144] ? apic_timer_interrupt+0xf/0x20 [ 8828.188124] [ 8828.189431] ? native_safe_halt+0x2e/0x50 [ 8828.193488] kvm_wait+0xcd/0xe0 [ 8828.196779] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8828.200557] do_raw_spin_lock+0x177/0x190 [ 8828.203482] _raw_spin_lock+0x21/0x30 [ 8828.206356] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8828.212861] process_one_work+0x2c8/0x700 [ 8828.217171] worker_thread+0x296/0x6e0 [ 8828.225025] ? rescuer_thread+0x570/0x570 [ 8828.230131] kthread+0x1d1/0x200 [ 8828.233247] ? set_kthread_struct+0x70/0x70 [ 8828.234727] ret_from_fork+0x1f/0x30 [ 8836.065581] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8836.074998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8836.133509] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8836.147581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8836.157165] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8836.164120] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8836.177974] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8836.185929] RAX: 0000000000005d92 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8836.192265] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8836.197355] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8836.202419] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8836.207520] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8836.212469] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8836.224333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8836.227234] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8836.231001] Call Trace: [ 8836.232210] [ 8836.233161] ? show_regs.cold.9+0x22/0x2f [ 8836.234920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8836.236571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8836.239187] ? watchdog+0x40/0x40 [ 8836.241394] ? hrtimer_interrupt+0x128/0x2c0 [ 8836.244136] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8836.246569] ? apic_timer_interrupt+0xf/0x20 [ 8836.248444] [ 8836.252240] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8836.255124] do_raw_spin_lock+0x177/0x190 [ 8836.263502] _raw_spin_lock+0x21/0x30 [ 8836.267195] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8836.271189] ? do_raw_spin_unlock+0x75/0x190 [ 8836.272859] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8836.275612] ? do_wait_intr+0xf0/0xf0 [ 8836.281124] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8836.285914] mdc_close+0x53b/0xd70 [mdc] [ 8836.289030] lmv_close+0x18d/0x490 [lmv] [ 8836.294040] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8836.296246] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8836.301173] ll_md_close+0x274/0x800 [lustre] [ 8836.304507] ll_file_release+0x128/0xc50 [lustre] [ 8836.309246] __fput+0x10d/0x440 [ 8836.313092] ____fput+0x12/0x20 [ 8836.314646] task_work_run+0xc6/0x110 [ 8836.316348] exit_to_usermode_loop+0x1dd/0x1f0 [ 8836.319208] do_syscall_64+0x3d6/0x440 [ 8836.321281] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8836.323821] RIP: 0033:0x7fa749552cd8 [ 8836.325823] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8836.338983] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8836.343013] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8836.346332] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8836.353428] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8836.355634] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8836.358225] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8840.058083] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 8840.061247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8840.085166] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8840.090511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8840.093761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8840.096753] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8840.105661] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8840.109512] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8840.112126] RDX: 0000000000004c97 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8840.115905] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8840.118615] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8840.121234] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8840.123657] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8840.126539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8840.128342] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8840.131584] Call Trace: [ 8840.132620] [ 8840.133447] ? show_regs.cold.9+0x22/0x2f [ 8840.134965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8840.136595] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8840.138456] ? watchdog+0x40/0x40 [ 8840.139650] ? hrtimer_interrupt+0x128/0x2c0 [ 8840.141262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8840.142952] ? apic_timer_interrupt+0xf/0x20 [ 8840.144501] [ 8840.145299] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8840.146685] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8840.148318] do_raw_spin_lock+0x177/0x190 [ 8840.149995] _raw_spin_lock+0x21/0x30 [ 8840.151479] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8840.153502] obd_mod_free+0x46/0x140 [ptlrpc] [ 8840.155360] mdc_commit_open+0x91/0xb0 [mdc] [ 8840.156554] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8840.159146] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8840.161730] after_reply+0xe2c/0x1780 [ptlrpc] [ 8840.164266] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8840.167173] ? __next_timer_interrupt+0x160/0x160 [ 8840.169674] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8840.171854] ? do_wait_intr+0xf0/0xf0 [ 8840.173951] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8840.176223] kthread+0x1d1/0x200 [ 8840.177625] ? set_kthread_struct+0x70/0x70 [ 8840.178727] ret_from_fork+0x1f/0x30 [ 8848.054809] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8848.063029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8848.095384] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8848.101291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8848.106132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8848.109282] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 8848.119304] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8848.122953] RAX: 000000000000608e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8848.125298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8848.128527] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8848.133003] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8848.137043] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8848.140728] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8848.145511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8848.149476] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8848.154435] Call Trace: [ 8848.157070] [ 8848.160401] ? show_regs.cold.9+0x22/0x2f [ 8848.161785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8848.164786] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8848.168487] ? watchdog+0x40/0x40 [ 8848.179517] ? hrtimer_interrupt+0x128/0x2c0 [ 8848.185132] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8848.189743] ? apic_timer_interrupt+0xf/0x20 [ 8848.201482] [ 8848.203275] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8848.211391] do_raw_spin_lock+0x177/0x190 [ 8848.215517] _raw_spin_lock+0x21/0x30 [ 8848.222483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8848.227299] ? do_raw_spin_unlock+0x75/0x190 [ 8848.231378] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8848.236202] ? do_wait_intr+0xf0/0xf0 [ 8848.243952] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8848.252544] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8848.257162] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8848.274297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8848.279464] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8848.281370] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8848.287126] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8848.298248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8848.302282] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8848.307413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8848.310507] lmv_intent_open+0x338/0x1150 [lmv] [ 8848.313029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8848.316804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8848.319517] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8848.321391] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8848.323324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8848.326042] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8848.328232] ? do_raw_spin_unlock+0x75/0x190 [ 8848.330276] ? _raw_spin_unlock+0x12/0x30 [ 8848.331913] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8848.333951] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8848.336430] ll_atomic_open+0x36a/0xd70 [lustre] [ 8848.338774] lookup_open+0x356/0xb40 [ 8848.340472] do_last+0x313/0xfc0 [ 8848.341927] ? nd_jump_root+0xe5/0x160 [ 8848.343549] ? path_init+0x437/0x520 [ 8848.345469] path_openat+0xf7/0x500 [ 8848.346783] ? do_raw_spin_unlock+0x75/0x190 [ 8848.348888] do_filp_open+0x99/0x140 [ 8848.352920] ? getname_flags+0x6e/0x330 [ 8848.354184] ? __check_object_size+0xff/0x256 [ 8848.355607] ? do_raw_spin_unlock+0x75/0x190 [ 8848.358128] ? _raw_spin_unlock+0x12/0x30 [ 8848.359848] do_sys_openat2+0x2b4/0x410 [ 8848.361290] do_sys_open+0x73/0xa0 [ 8848.362547] __x64_sys_openat+0x24/0x30 [ 8848.364314] do_syscall_64+0xc1/0x440 [ 8848.366563] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8848.368793] RIP: 0033:0x7fa749552332 [ 8848.370438] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8848.378189] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8848.381513] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8848.384556] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8848.387918] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8848.394295] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8848.398055] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8850.912211] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6933s! [ 8850.917561] Showing busy workqueues and worker pools: [ 8850.921274] workqueue events: flags=0x0 [ 8850.922719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8850.928097] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8850.938865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8850.941829] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8850.947237] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8850.953293] pending: kfree_rcu_monitor [ 8850.958169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8850.963358] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8850.968885] workqueue events_long: flags=0x0 [ 8850.970978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8850.973285] pending: lu_objects_flush [obdclass] [ 8850.984449] workqueue events_unbound: flags=0x2 [ 8850.994017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8851.001577] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8851.014535] workqueue events_power_efficient: flags=0x80 [ 8851.016252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8851.025443] pending: check_lifetime [ 8851.026863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8851.036464] pending: neigh_periodic_work [ 8851.043327] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8851.048438] pending: neigh_periodic_work [ 8851.058365] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8851.068366] pending: do_cache_clean [ 8851.077248] workqueue mm_percpu_wq: flags=0x8 [ 8851.080337] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8851.088524] pending: vmstat_update [ 8851.091187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8851.095744] pending: vmstat_update [ 8851.097554] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8851.103531] pending: vmstat_update [ 8851.106909] workqueue writeback: flags=0x4a [ 8851.108308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8851.110240] pending: wb_workfn, wb_workfn [ 8851.111937] workqueue xprtiod: flags=0xa [ 8851.113408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8851.115265] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8851.121447] workqueue ipv6_addrconf: flags=0x40008 [ 8851.123675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8851.129022] pending: addrconf_verify_work [ 8851.133270] workqueue knbd0-recv: flags=0x1a [ 8851.137208] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8851.141585] in-flight: 651:recv_work [ 8851.143508] workqueue ptlrpc_pinger: flags=0x2 [ 8851.145087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8851.152988] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8851.160518] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6934s workers=4 idle: 37 31846 28897 [ 8851.168009] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8841s workers=2 idle: 653 [ 8856.057038] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8856.060221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8856.082994] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8856.087913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8856.092453] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8856.097020] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8856.099635] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8856.108211] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8856.112286] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8856.115406] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8856.119208] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8856.122943] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8856.126269] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8856.130094] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8856.133746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8856.136198] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8856.139506] Call Trace: [ 8856.140889] [ 8856.141818] ? show_regs.cold.9+0x22/0x2f [ 8856.143696] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8856.145940] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8856.148345] ? watchdog+0x40/0x40 [ 8856.149796] ? hrtimer_interrupt+0x128/0x2c0 [ 8856.152120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8856.154678] ? apic_timer_interrupt+0xf/0x20 [ 8856.156524] [ 8856.157651] ? native_safe_halt+0x2e/0x50 [ 8856.159556] kvm_wait+0xcd/0xe0 [ 8856.161106] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8856.163926] do_raw_spin_lock+0x177/0x190 [ 8856.166132] _raw_spin_lock+0x21/0x30 [ 8856.167862] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8856.170621] process_one_work+0x2c8/0x700 [ 8856.171832] worker_thread+0x296/0x6e0 [ 8856.173222] ? rescuer_thread+0x570/0x570 [ 8856.175418] kthread+0x1d1/0x200 [ 8856.176626] ? set_kthread_struct+0x70/0x70 [ 8856.178977] ret_from_fork+0x1f/0x30 [ 8864.061054] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 8864.071682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8864.126527] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8864.145395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8864.156722] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8864.164260] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8864.191549] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8864.200841] RAX: 000000000000018e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8864.210459] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8864.220788] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8864.231808] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8864.242815] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8864.253381] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8864.264444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8864.270915] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8864.285316] Call Trace: [ 8864.286487] [ 8864.286992] ? show_regs.cold.9+0x22/0x2f [ 8864.293164] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8864.295338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8864.301874] ? watchdog+0x40/0x40 [ 8864.305209] ? hrtimer_interrupt+0x128/0x2c0 [ 8864.309197] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8864.314456] ? apic_timer_interrupt+0xf/0x20 [ 8864.317575] [ 8864.322958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8864.327021] do_raw_spin_lock+0x177/0x190 [ 8864.331283] _raw_spin_lock+0x21/0x30 [ 8864.336259] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8864.340797] ? do_raw_spin_unlock+0x75/0x190 [ 8864.347305] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8864.353357] ? do_wait_intr+0xf0/0xf0 [ 8864.358587] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8864.364393] mdc_close+0x53b/0xd70 [mdc] [ 8864.365808] lmv_close+0x18d/0x490 [lmv] [ 8864.371902] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8864.376794] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8864.378862] ll_md_close+0x274/0x800 [lustre] [ 8864.382138] ll_file_release+0x128/0xc50 [lustre] [ 8864.383532] __fput+0x10d/0x440 [ 8864.384364] ____fput+0x12/0x20 [ 8864.385155] task_work_run+0xc6/0x110 [ 8864.387087] exit_to_usermode_loop+0x1dd/0x1f0 [ 8864.388937] do_syscall_64+0x3d6/0x440 [ 8864.391920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8864.394569] RIP: 0033:0x7fa749552cd8 [ 8864.396064] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8864.405680] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8864.410601] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8864.423508] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8864.433460] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8864.438143] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8864.444103] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8868.060033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8868.063974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8868.088527] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8868.093553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8868.097385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8868.100128] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8868.108523] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8868.111951] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8868.115076] RDX: 0000000000002061 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8868.117932] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8868.120821] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8868.123897] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8868.127052] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8868.130611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8868.133031] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8868.135994] Call Trace: [ 8868.137189] [ 8868.138086] ? show_regs.cold.9+0x22/0x2f [ 8868.140490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8868.142630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8868.144603] ? watchdog+0x40/0x40 [ 8868.146014] ? hrtimer_interrupt+0x128/0x2c0 [ 8868.147908] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8868.149892] ? apic_timer_interrupt+0xf/0x20 [ 8868.151510] [ 8868.152114] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8868.153997] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8868.156004] do_raw_spin_lock+0x177/0x190 [ 8868.157665] _raw_spin_lock+0x21/0x30 [ 8868.159137] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8868.161435] obd_mod_free+0x46/0x140 [ptlrpc] [ 8868.163151] mdc_commit_open+0x91/0xb0 [mdc] [ 8868.164630] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8868.166653] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8868.168868] after_reply+0xe2c/0x1780 [ptlrpc] [ 8868.170893] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8868.172680] ? __next_timer_interrupt+0x160/0x160 [ 8868.174012] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8868.175640] ? do_wait_intr+0xf0/0xf0 [ 8868.176888] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8868.179177] kthread+0x1d1/0x200 [ 8868.180459] ? set_kthread_struct+0x70/0x70 [ 8868.181799] ret_from_fork+0x1f/0x30 [ 8876.054222] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 8876.059592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8876.092955] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8876.101372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8876.108309] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8876.116341] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8876.137611] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8876.145503] RAX: 0000000000007e0e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8876.153311] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8876.167590] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8876.176279] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8876.188138] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8876.197406] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8876.210341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8876.216087] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8876.218564] Call Trace: [ 8876.219390] [ 8876.232121] ? show_regs.cold.9+0x22/0x2f [ 8876.236826] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8876.240717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8876.247602] ? watchdog+0x40/0x40 [ 8876.249511] ? hrtimer_interrupt+0x128/0x2c0 [ 8876.254633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8876.261121] ? apic_timer_interrupt+0xf/0x20 [ 8876.263640] [ 8876.266699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8876.274083] do_raw_spin_lock+0x177/0x190 [ 8876.276788] _raw_spin_lock+0x21/0x30 [ 8876.282025] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8876.287275] ? do_raw_spin_unlock+0x75/0x190 [ 8876.288894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8876.295542] ? do_wait_intr+0xf0/0xf0 [ 8876.297526] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8876.303954] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8876.310969] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8876.318003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8876.324325] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8876.327828] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8876.333058] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8876.337155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8876.342505] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8876.345505] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8876.351027] lmv_intent_open+0x338/0x1150 [lmv] [ 8876.358029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8876.361932] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8876.366692] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8876.370994] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8876.375050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8876.380399] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8876.385467] ? do_raw_spin_unlock+0x75/0x190 [ 8876.390488] ? _raw_spin_unlock+0x12/0x30 [ 8876.392011] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8876.398375] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8876.403485] ll_atomic_open+0x36a/0xd70 [lustre] [ 8876.408120] lookup_open+0x356/0xb40 [ 8876.411576] do_last+0x313/0xfc0 [ 8876.412785] ? nd_jump_root+0xe5/0x160 [ 8876.419456] ? path_init+0x437/0x520 [ 8876.420827] path_openat+0xf7/0x500 [ 8876.425975] ? do_raw_spin_unlock+0x75/0x190 [ 8876.428759] do_filp_open+0x99/0x140 [ 8876.431268] ? getname_flags+0x6e/0x330 [ 8876.432726] ? __check_object_size+0xff/0x256 [ 8876.438794] ? do_raw_spin_unlock+0x75/0x190 [ 8876.443726] ? _raw_spin_unlock+0x12/0x30 [ 8876.446998] do_sys_openat2+0x2b4/0x410 [ 8876.451344] do_sys_open+0x73/0xa0 [ 8876.454859] __x64_sys_openat+0x24/0x30 [ 8876.457087] do_syscall_64+0xc1/0x440 [ 8876.458265] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8876.467778] RIP: 0033:0x7fa749552332 [ 8876.472270] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8876.486533] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8876.493118] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8876.498890] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8876.504999] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8876.508156] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8876.512135] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8881.634310] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6964s! [ 8881.645457] Showing busy workqueues and worker pools: [ 8881.648757] workqueue events: flags=0x0 [ 8881.650128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8881.664228] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8881.683244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8881.691353] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8881.702230] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.710723] pending: kfree_rcu_monitor [ 8881.718951] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8881.723859] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8881.730223] workqueue events_long: flags=0x0 [ 8881.739466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.743746] pending: lu_objects_flush [obdclass] [ 8881.749244] workqueue events_unbound: flags=0x2 [ 8881.751012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8881.760689] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8881.774524] workqueue events_power_efficient: flags=0x80 [ 8881.778657] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.785142] pending: check_lifetime [ 8881.787313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.794910] pending: neigh_periodic_work [ 8881.803883] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.808785] pending: neigh_periodic_work [ 8881.817237] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.823372] pending: do_cache_clean [ 8881.824882] workqueue mm_percpu_wq: flags=0x8 [ 8881.828893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.833142] pending: vmstat_update [ 8881.836266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.840890] pending: vmstat_update [ 8881.843160] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8881.847853] pending: vmstat_update [ 8881.851190] workqueue writeback: flags=0x4a [ 8881.858090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8881.865332] pending: wb_workfn, wb_workfn [ 8881.867844] workqueue xprtiod: flags=0xa [ 8881.872100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8881.880823] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8881.884032] workqueue ipv6_addrconf: flags=0x40008 [ 8881.886832] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8881.892729] pending: addrconf_verify_work [ 8881.897848] workqueue knbd0-recv: flags=0x1a [ 8881.905324] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8881.915831] in-flight: 651:recv_work [ 8881.919179] workqueue ptlrpc_pinger: flags=0x2 [ 8881.924090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8881.929833] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8881.940038] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6964s workers=4 idle: 37 31846 28897 [ 8881.952480] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8871s workers=2 idle: 653 [ 8884.060463] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 8884.068628] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8884.105619] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8884.117165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8884.123647] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8884.130089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8884.136127] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8884.149302] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8884.151883] RAX: 0000000000006070 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8884.154309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8884.156421] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8884.159223] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8884.161880] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8884.164394] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8884.167629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8884.170711] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8884.173624] Call Trace: [ 8884.174567] [ 8884.175262] ? show_regs.cold.9+0x22/0x2f [ 8884.176598] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8884.178442] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8884.180225] ? watchdog+0x40/0x40 [ 8884.181151] ? hrtimer_interrupt+0x128/0x2c0 [ 8884.182920] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8884.185510] ? apic_timer_interrupt+0xf/0x20 [ 8884.187088] [ 8884.188055] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8884.191029] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8884.193053] do_raw_spin_lock+0x177/0x190 [ 8884.194762] _raw_spin_lock+0x21/0x30 [ 8884.196016] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8884.198152] process_one_work+0x2c8/0x700 [ 8884.199463] worker_thread+0x296/0x6e0 [ 8884.201104] ? rescuer_thread+0x570/0x570 [ 8884.202440] kthread+0x1d1/0x200 [ 8884.203537] ? set_kthread_struct+0x70/0x70 [ 8884.205139] ret_from_fork+0x1f/0x30 [ 8892.059014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8892.062121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8892.107115] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8892.116193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8892.118845] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8892.120505] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8892.127740] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8892.129600] RAX: 000000000000384f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8892.131572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8892.135531] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8892.141564] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8892.143490] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8892.145159] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8892.148373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8892.149816] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8892.161367] Call Trace: [ 8892.163148] [ 8892.163656] ? show_regs.cold.9+0x22/0x2f [ 8892.165071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8892.167839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8892.170410] ? watchdog+0x40/0x40 [ 8892.171341] ? hrtimer_interrupt+0x128/0x2c0 [ 8892.173599] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8892.177271] ? apic_timer_interrupt+0xf/0x20 [ 8892.178333] [ 8892.178852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8892.180161] do_raw_spin_lock+0x177/0x190 [ 8892.185580] _raw_spin_lock+0x21/0x30 [ 8892.187482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8892.190011] ? do_raw_spin_unlock+0x75/0x190 [ 8892.192236] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8892.196520] ? do_wait_intr+0xf0/0xf0 [ 8892.198725] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8892.210508] mdc_close+0x53b/0xd70 [mdc] [ 8892.211562] lmv_close+0x18d/0x490 [lmv] [ 8892.212508] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8892.221477] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8892.222829] ll_md_close+0x274/0x800 [lustre] [ 8892.226348] ll_file_release+0x128/0xc50 [lustre] [ 8892.228518] __fput+0x10d/0x440 [ 8892.229386] ____fput+0x12/0x20 [ 8892.231022] task_work_run+0xc6/0x110 [ 8892.232731] exit_to_usermode_loop+0x1dd/0x1f0 [ 8892.238085] do_syscall_64+0x3d6/0x440 [ 8892.239600] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8892.241623] RIP: 0033:0x7fa749552cd8 [ 8892.242615] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8892.250743] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8892.257698] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8892.261753] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8892.265770] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8892.267420] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8892.272044] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8896.058407] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8896.063839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8896.095733] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8896.103628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8896.109602] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8896.112738] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8896.125449] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8896.130543] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8896.134707] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8896.138435] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8896.143885] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8896.148894] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8896.152781] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8896.158379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8896.161921] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8896.167395] Call Trace: [ 8896.168711] [ 8896.170178] ? show_regs.cold.9+0x22/0x2f [ 8896.172490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8896.174961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8896.177878] ? watchdog+0x40/0x40 [ 8896.179985] ? hrtimer_interrupt+0x128/0x2c0 [ 8896.182781] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8896.186304] ? apic_timer_interrupt+0xf/0x20 [ 8896.188833] [ 8896.190878] ? native_safe_halt+0x2e/0x50 [ 8896.193604] kvm_wait+0xcd/0xe0 [ 8896.195867] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8896.199112] do_raw_spin_lock+0x177/0x190 [ 8896.200533] _raw_spin_lock+0x21/0x30 [ 8896.203411] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8896.207164] obd_mod_free+0x46/0x140 [ptlrpc] [ 8896.210174] mdc_commit_open+0x91/0xb0 [mdc] [ 8896.212514] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8896.216395] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8896.222175] after_reply+0xe2c/0x1780 [ptlrpc] [ 8896.226783] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8896.229804] ? __next_timer_interrupt+0x160/0x160 [ 8896.232301] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8896.235542] ? do_wait_intr+0xf0/0xf0 [ 8896.238132] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8896.242636] kthread+0x1d1/0x200 [ 8896.245554] ? set_kthread_struct+0x70/0x70 [ 8896.249462] ret_from_fork+0x1f/0x30 [ 8908.059999] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8908.068227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.113473] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.124205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.136953] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8908.145344] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8908.167403] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8908.173809] RAX: 0000000000005928 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8908.178964] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8908.184805] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8908.195378] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8908.208330] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8908.217954] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8908.224237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.230159] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8908.236987] Call Trace: [ 8908.239670] [ 8908.240846] ? show_regs.cold.9+0x22/0x2f [ 8908.246345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.253370] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.257958] ? watchdog+0x40/0x40 [ 8908.263955] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.269253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.275931] ? apic_timer_interrupt+0xf/0x20 [ 8908.283285] [ 8908.285252] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8908.289522] do_raw_spin_lock+0x177/0x190 [ 8908.291759] _raw_spin_lock+0x21/0x30 [ 8908.293879] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8908.300123] ? do_raw_spin_unlock+0x75/0x190 [ 8908.303260] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8908.314675] ? do_wait_intr+0xf0/0xf0 [ 8908.317836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8908.325854] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8908.336928] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8908.344397] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.350270] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8908.361431] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8908.365003] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8908.376114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.383196] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8908.393903] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8908.398100] lmv_intent_open+0x338/0x1150 [lmv] [ 8908.405631] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.407678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.422739] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8908.428019] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8908.429792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.444367] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8908.452026] ? do_raw_spin_unlock+0x75/0x190 [ 8908.454861] ? _raw_spin_unlock+0x12/0x30 [ 8908.461118] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8908.463121] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8908.466943] ll_atomic_open+0x36a/0xd70 [lustre] [ 8908.472300] lookup_open+0x356/0xb40 [ 8908.474389] do_last+0x313/0xfc0 [ 8908.479199] ? nd_jump_root+0xe5/0x160 [ 8908.483386] ? path_init+0x437/0x520 [ 8908.488215] path_openat+0xf7/0x500 [ 8908.494227] ? do_raw_spin_unlock+0x75/0x190 [ 8908.507451] do_filp_open+0x99/0x140 [ 8908.508809] ? getname_flags+0x6e/0x330 [ 8908.511738] ? __check_object_size+0xff/0x256 [ 8908.515705] ? do_raw_spin_unlock+0x75/0x190 [ 8908.520687] ? _raw_spin_unlock+0x12/0x30 [ 8908.523032] do_sys_openat2+0x2b4/0x410 [ 8908.524058] do_sys_open+0x73/0xa0 [ 8908.528302] __x64_sys_openat+0x24/0x30 [ 8908.530375] do_syscall_64+0xc1/0x440 [ 8908.532298] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8908.534675] RIP: 0033:0x7fa749552332 [ 8908.540796] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8908.559188] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8908.564235] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8908.567585] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8908.571788] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8908.576270] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8908.586675] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8912.064658] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8912.080174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8912.137114] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8912.156766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8912.166162] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8912.176261] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8912.187675] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8912.206846] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8912.214859] RAX: 0000000000004fb5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8912.218942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8912.227149] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8912.235788] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8912.240857] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8912.245460] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8912.252653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8912.256696] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8912.259989] Call Trace: [ 8912.261075] [ 8912.261876] ? show_regs.cold.9+0x22/0x2f [ 8912.263697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8912.267704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8912.271687] ? watchdog+0x40/0x40 [ 8912.273269] ? hrtimer_interrupt+0x128/0x2c0 [ 8912.276872] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8912.278944] ? apic_timer_interrupt+0xf/0x20 [ 8912.283658] [ 8912.284637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8912.288315] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8912.294076] do_raw_spin_lock+0x177/0x190 [ 8912.296053] _raw_spin_lock+0x21/0x30 [ 8912.302675] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8912.306679] process_one_work+0x2c8/0x700 [ 8912.308836] worker_thread+0x296/0x6e0 [ 8912.310348] ? rescuer_thread+0x570/0x570 [ 8912.314396] kthread+0x1d1/0x200 [ 8912.318012] ? set_kthread_struct+0x70/0x70 [ 8912.320097] ret_from_fork+0x1f/0x30 [ 8912.353913] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6995s! [ 8912.356749] Showing busy workqueues and worker pools: [ 8912.361029] workqueue events: flags=0x0 [ 8912.362424] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8912.370449] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8912.383988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8912.392456] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8912.401228] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.405466] pending: kfree_rcu_monitor [ 8912.407378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8912.410245] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8912.414907] workqueue events_long: flags=0x0 [ 8912.416775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.419735] pending: lu_objects_flush [obdclass] [ 8912.423139] workqueue events_unbound: flags=0x2 [ 8912.426250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8912.429007] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8912.432935] workqueue events_power_efficient: flags=0x80 [ 8912.435485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.438619] pending: check_lifetime [ 8912.440327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.443291] pending: neigh_periodic_work [ 8912.445950] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.449508] pending: neigh_periodic_work [ 8912.451495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.454962] pending: do_cache_clean [ 8912.456990] workqueue mm_percpu_wq: flags=0x8 [ 8912.460451] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.463639] pending: vmstat_update [ 8912.465234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.470441] pending: vmstat_update [ 8912.473079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8912.478289] pending: vmstat_update [ 8912.479579] workqueue writeback: flags=0x4a [ 8912.482984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8912.488494] pending: wb_workfn, wb_workfn [ 8912.490117] workqueue xprtiod: flags=0xa [ 8912.494876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8912.502171] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8912.506825] workqueue ipv6_addrconf: flags=0x40008 [ 8912.510677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8912.513887] pending: addrconf_verify_work [ 8912.516711] workqueue knbd0-recv: flags=0x1a [ 8912.517730] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8912.520585] in-flight: 651:recv_work [ 8912.524068] workqueue ptlrpc_pinger: flags=0x2 [ 8912.525598] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8912.528255] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8912.534285] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6995s workers=4 idle: 37 31846 28897 [ 8912.538532] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8902s workers=2 idle: 653 [ 8920.059017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8920.063471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8920.087790] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8920.094124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8920.098195] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8920.101619] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8920.110259] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8920.113361] RAX: 00000000000012e5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8920.116668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8920.119691] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8920.123129] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8920.126294] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8920.129324] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8920.132628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8920.135425] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8920.138587] Call Trace: [ 8920.139721] [ 8920.140505] ? show_regs.cold.9+0x22/0x2f [ 8920.142365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8920.145676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8920.148070] ? watchdog+0x40/0x40 [ 8920.149573] ? hrtimer_interrupt+0x128/0x2c0 [ 8920.151507] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8920.154063] ? apic_timer_interrupt+0xf/0x20 [ 8920.155598] [ 8920.156690] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8920.159332] do_raw_spin_lock+0x177/0x190 [ 8920.161308] _raw_spin_lock+0x21/0x30 [ 8920.163072] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8920.165654] ? do_raw_spin_unlock+0x75/0x190 [ 8920.167818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8920.170330] ? do_wait_intr+0xf0/0xf0 [ 8920.171640] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8920.174434] mdc_close+0x53b/0xd70 [mdc] [ 8920.176456] lmv_close+0x18d/0x490 [lmv] [ 8920.178188] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8920.180932] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8920.183136] ll_md_close+0x274/0x800 [lustre] [ 8920.185601] ll_file_release+0x128/0xc50 [lustre] [ 8920.188343] __fput+0x10d/0x440 [ 8920.189487] ____fput+0x12/0x20 [ 8920.191321] task_work_run+0xc6/0x110 [ 8920.192836] exit_to_usermode_loop+0x1dd/0x1f0 [ 8920.195245] do_syscall_64+0x3d6/0x440 [ 8920.196955] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8920.199404] RIP: 0033:0x7fa749552cd8 [ 8920.200775] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8920.209310] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8920.212575] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8920.215607] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8920.219059] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8920.222250] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8920.225434] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8924.060725] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8924.075447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.135385] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.144518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.157229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8924.166763] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8924.178899] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8924.182601] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8924.186695] RDX: 0000000000007b35 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8924.193661] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8924.200027] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8924.203194] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8924.209908] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8924.217951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.221910] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8924.227953] Call Trace: [ 8924.230554] [ 8924.233448] ? show_regs.cold.9+0x22/0x2f [ 8924.242082] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.248133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.251309] ? watchdog+0x40/0x40 [ 8924.254267] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.255794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.257298] ? apic_timer_interrupt+0xf/0x20 [ 8924.261837] [ 8924.262839] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8924.265407] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8924.268232] do_raw_spin_lock+0x177/0x190 [ 8924.273456] _raw_spin_lock+0x21/0x30 [ 8924.275733] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8924.280291] obd_mod_free+0x46/0x140 [ptlrpc] [ 8924.285654] mdc_commit_open+0x91/0xb0 [mdc] [ 8924.289595] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8924.293439] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8924.296945] after_reply+0xe2c/0x1780 [ptlrpc] [ 8924.300101] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8924.306204] ? __next_timer_interrupt+0x160/0x160 [ 8924.308688] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8924.315889] ? do_wait_intr+0xf0/0xf0 [ 8924.318144] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8924.321484] kthread+0x1d1/0x200 [ 8924.323669] ? set_kthread_struct+0x70/0x70 [ 8924.325580] ret_from_fork+0x1f/0x30 [ 8940.054035] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8940.058389] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8940.059903] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8940.088781] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8940.088797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8940.088802] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8940.088818] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8940.088823] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8940.088829] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8940.088832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8940.088834] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8940.088836] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8940.088839] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8940.088842] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8940.088845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8940.088848] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8940.088856] Call Trace: [ 8940.088862] [ 8940.088867] ? show_regs.cold.9+0x22/0x2f [ 8940.088879] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8940.088885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8940.088889] ? watchdog+0x40/0x40 [ 8940.088893] ? hrtimer_interrupt+0x128/0x2c0 [ 8940.088897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8940.088904] ? apic_timer_interrupt+0xf/0x20 [ 8940.088908] [ 8940.088911] ? native_safe_halt+0x2e/0x50 [ 8940.088915] kvm_wait+0xcd/0xe0 [ 8940.088922] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8940.088932] do_raw_spin_lock+0x177/0x190 [ 8940.088936] _raw_spin_lock+0x21/0x30 [ 8940.088940] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8940.090522] ? do_raw_spin_unlock+0x75/0x190 [ 8940.090530] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8940.090730] ? do_wait_intr+0xf0/0xf0 [ 8940.090738] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8940.090949] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8940.092622] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8940.092839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8940.092953] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8940.092994] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8940.093040] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8940.099858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8940.100901] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8940.100907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8940.100922] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8940.101713] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8940.101727] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8940.101731] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8940.101735] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8940.101738] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8940.101740] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8940.101742] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8940.101745] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8940.101748] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8940.101751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8940.101753] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8940.101762] Call Trace: [ 8940.101769] [ 8940.101775] ? show_regs.cold.9+0x22/0x2f [ 8940.101785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8940.101791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8940.101796] ? watchdog+0x40/0x40 [ 8940.101800] ? hrtimer_interrupt+0x128/0x2c0 [ 8940.101805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8940.101812] ? apic_timer_interrupt+0xf/0x20 [ 8940.101817] [ 8940.101820] ? native_safe_halt+0x2e/0x50 [ 8940.101824] kvm_wait+0xcd/0xe0 [ 8940.101831] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8940.101837] do_raw_spin_lock+0x177/0x190 [ 8940.101841] _raw_spin_lock+0x21/0x30 [ 8940.101845] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8940.102534] process_one_work+0x2c8/0x700 [ 8940.102543] worker_thread+0x296/0x6e0 [ 8940.102547] ? rescuer_thread+0x570/0x570 [ 8940.102550] kthread+0x1d1/0x200 [ 8940.102555] ? set_kthread_struct+0x70/0x70 [ 8940.102558] ret_from_fork+0x1f/0x30 [ 8940.552585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8940.554614] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8940.557566] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8940.559347] lmv_intent_open+0x338/0x1150 [lmv] [ 8940.560949] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8940.562941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8940.564983] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8940.566546] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8940.568710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8940.570949] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8940.572908] ? do_raw_spin_unlock+0x75/0x190 [ 8940.575473] ? _raw_spin_unlock+0x12/0x30 [ 8940.577354] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8940.579623] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8940.581691] ll_atomic_open+0x36a/0xd70 [lustre] [ 8940.589546] lookup_open+0x356/0xb40 [ 8940.593030] do_last+0x313/0xfc0 [ 8940.594100] ? nd_jump_root+0xe5/0x160 [ 8940.596822] ? path_init+0x437/0x520 [ 8940.599667] path_openat+0xf7/0x500 [ 8940.601418] ? do_raw_spin_unlock+0x75/0x190 [ 8940.602980] do_filp_open+0x99/0x140 [ 8940.604502] ? getname_flags+0x6e/0x330 [ 8940.608493] ? __check_object_size+0xff/0x256 [ 8940.610554] ? do_raw_spin_unlock+0x75/0x190 [ 8940.615489] ? _raw_spin_unlock+0x12/0x30 [ 8940.620106] do_sys_openat2+0x2b4/0x410 [ 8940.625728] do_sys_open+0x73/0xa0 [ 8940.626892] __x64_sys_openat+0x24/0x30 [ 8940.628178] do_syscall_64+0xc1/0x440 [ 8940.636431] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8940.642792] RIP: 0033:0x7fa749552332 [ 8940.646583] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8940.661406] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8940.666498] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8940.670455] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8940.673582] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8940.676822] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8940.680748] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8942.562121] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7025s! [ 8942.570933] Showing busy workqueues and worker pools: [ 8942.577498] workqueue events: flags=0x0 [ 8942.582809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8942.589137] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8942.601889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8942.606467] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8942.614985] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.621674] pending: kfree_rcu_monitor [ 8942.630934] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8942.642646] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8942.647891] workqueue events_long: flags=0x0 [ 8942.650874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.656828] pending: lu_objects_flush [obdclass] [ 8942.659735] workqueue events_unbound: flags=0x2 [ 8942.662742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8942.666688] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8942.671881] workqueue events_power_efficient: flags=0x80 [ 8942.675910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.682693] pending: check_lifetime [ 8942.685767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.690943] pending: neigh_periodic_work [ 8942.694870] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.698602] pending: neigh_periodic_work [ 8942.699966] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.706741] pending: do_cache_clean [ 8942.707910] workqueue mm_percpu_wq: flags=0x8 [ 8942.716758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.718733] pending: vmstat_update [ 8942.725864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.732523] pending: vmstat_update [ 8942.735137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8942.739132] pending: vmstat_update [ 8942.740420] workqueue writeback: flags=0x4a [ 8942.745774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8942.749857] pending: wb_workfn, wb_workfn [ 8942.753431] workqueue xprtiod: flags=0xa [ 8942.755942] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8942.761747] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8942.769773] workqueue ipv6_addrconf: flags=0x40008 [ 8942.774843] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8942.780116] pending: addrconf_verify_work [ 8942.785008] workqueue knbd0-recv: flags=0x1a [ 8942.787648] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8942.790514] in-flight: 651:recv_work [ 8942.794108] workqueue ptlrpc_pinger: flags=0x2 [ 8942.796934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8942.800357] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8942.803852] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7025s workers=4 idle: 37 31846 28897 [ 8942.808322] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8932s workers=2 idle: 653 [ 8948.059151] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8948.065982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8948.099352] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8948.111585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8948.119399] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8948.124273] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8948.138944] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8948.145805] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8948.151250] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8948.158336] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8948.163715] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8948.174626] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8948.180675] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8948.184265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8948.190982] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8948.197362] Call Trace: [ 8948.198661] [ 8948.199414] ? show_regs.cold.9+0x22/0x2f [ 8948.205259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8948.208468] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8948.214800] ? watchdog+0x40/0x40 [ 8948.217795] ? hrtimer_interrupt+0x128/0x2c0 [ 8948.223301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8948.227888] ? apic_timer_interrupt+0xf/0x20 [ 8948.229866] [ 8948.230599] ? native_safe_halt+0x2e/0x50 [ 8948.233622] kvm_wait+0xcd/0xe0 [ 8948.234825] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8948.239530] do_raw_spin_lock+0x177/0x190 [ 8948.242225] _raw_spin_lock+0x21/0x30 [ 8948.245642] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8948.249262] ? do_raw_spin_unlock+0x75/0x190 [ 8948.251706] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8948.254497] ? do_wait_intr+0xf0/0xf0 [ 8948.259547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8948.266652] mdc_close+0x53b/0xd70 [mdc] [ 8948.273574] lmv_close+0x18d/0x490 [lmv] [ 8948.275839] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8948.280289] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8948.290543] ll_md_close+0x274/0x800 [lustre] [ 8948.294734] ll_file_release+0x128/0xc50 [lustre] [ 8948.297522] __fput+0x10d/0x440 [ 8948.299696] ____fput+0x12/0x20 [ 8948.304127] task_work_run+0xc6/0x110 [ 8948.307341] exit_to_usermode_loop+0x1dd/0x1f0 [ 8948.310844] do_syscall_64+0x3d6/0x440 [ 8948.313378] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8948.316965] RIP: 0033:0x7fa749552cd8 [ 8948.319301] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8948.336533] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8948.344394] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8948.351290] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8948.357766] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8948.363437] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8948.369312] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8952.059162] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8952.073760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.109369] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.115180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.119246] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8952.121386] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8952.132164] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8952.139775] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8952.146590] RDX: 0000000000000758 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8952.155188] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8952.164929] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8952.170201] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8952.176581] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8952.186946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.193063] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8952.198783] Call Trace: [ 8952.200742] [ 8952.201680] ? show_regs.cold.9+0x22/0x2f [ 8952.203705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.205639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.207730] ? watchdog+0x40/0x40 [ 8952.212218] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.215892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.221818] ? apic_timer_interrupt+0xf/0x20 [ 8952.225571] [ 8952.226650] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8952.232112] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8952.233358] do_raw_spin_lock+0x177/0x190 [ 8952.237390] _raw_spin_lock+0x21/0x30 [ 8952.240783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8952.245905] obd_mod_free+0x46/0x140 [ptlrpc] [ 8952.248768] mdc_commit_open+0x91/0xb0 [mdc] [ 8952.251892] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8952.255851] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8952.259679] after_reply+0xe2c/0x1780 [ptlrpc] [ 8952.264955] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8952.268306] ? __next_timer_interrupt+0x160/0x160 [ 8952.271790] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8952.274575] ? do_wait_intr+0xf0/0xf0 [ 8952.276590] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8952.279243] kthread+0x1d1/0x200 [ 8952.280813] ? set_kthread_struct+0x70/0x70 [ 8952.284188] ret_from_fork+0x1f/0x30 [ 8968.056903] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 8968.057063] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8968.067233] Modules linked in: [ 8968.069160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8968.069228] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8968.069234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8968.069240] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8968.069515] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8968.069525] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8968.069529] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8968.069533] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8968.069535] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8968.069537] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8968.069539] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8968.069541] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8968.069544] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8968.069547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8968.069549] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8968.069557] Call Trace: [ 8968.069563] [ 8968.069568] ? show_regs.cold.9+0x22/0x2f [ 8968.069577] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8968.069582] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8968.069586] ? watchdog+0x40/0x40 [ 8968.069590] ? hrtimer_interrupt+0x128/0x2c0 [ 8968.069594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8968.069874] ? apic_timer_interrupt+0xf/0x20 [ 8968.069879] [ 8968.069882] ? native_safe_halt+0x2e/0x50 [ 8968.069886] kvm_wait+0xcd/0xe0 [ 8968.069893] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8968.069900] do_raw_spin_lock+0x177/0x190 [ 8968.069904] _raw_spin_lock+0x21/0x30 [ 8968.069908] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8968.070119] process_one_work+0x2c8/0x700 [ 8968.070125] worker_thread+0x296/0x6e0 [ 8968.070128] ? rescuer_thread+0x570/0x570 [ 8968.070131] kthread+0x1d1/0x200 [ 8968.070135] ? set_kthread_struct+0x70/0x70 [ 8968.070138] ret_from_fork+0x1f/0x30 [ 8968.482232] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8968.529082] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8968.535943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8968.545209] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8968.550400] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8968.579752] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8968.585093] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8968.597034] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 8968.604453] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 8968.608307] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 8968.612336] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 8968.619196] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 8968.625384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8968.627284] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 8968.638002] Call Trace: [ 8968.644062] [ 8968.644810] ? show_regs.cold.9+0x22/0x2f [ 8968.653286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8968.662555] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8968.670281] ? watchdog+0x40/0x40 [ 8968.676155] ? hrtimer_interrupt+0x128/0x2c0 [ 8968.685186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8968.687115] ? apic_timer_interrupt+0xf/0x20 [ 8968.689340] [ 8968.691128] ? native_safe_halt+0x2e/0x50 [ 8968.693507] kvm_wait+0xcd/0xe0 [ 8968.694641] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8968.706005] do_raw_spin_lock+0x177/0x190 [ 8968.709339] _raw_spin_lock+0x21/0x30 [ 8968.711685] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8968.716060] ? do_raw_spin_unlock+0x75/0x190 [ 8968.718727] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8968.726679] ? do_wait_intr+0xf0/0xf0 [ 8968.732214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8968.741503] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8968.750138] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8968.760759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8968.764239] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8968.767779] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8968.771155] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8968.773988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8968.778271] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8968.783785] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8968.791351] lmv_intent_open+0x338/0x1150 [lmv] [ 8968.795180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8968.801755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8968.809389] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8968.811449] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8968.815307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8968.819271] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8968.821676] ? do_raw_spin_unlock+0x75/0x190 [ 8968.823630] ? _raw_spin_unlock+0x12/0x30 [ 8968.825977] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8968.830301] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8968.834269] ll_atomic_open+0x36a/0xd70 [lustre] [ 8968.839627] lookup_open+0x356/0xb40 [ 8968.854106] do_last+0x313/0xfc0 [ 8968.855349] ? nd_jump_root+0xe5/0x160 [ 8968.856648] ? path_init+0x437/0x520 [ 8968.870730] path_openat+0xf7/0x500 [ 8968.879213] ? do_raw_spin_unlock+0x75/0x190 [ 8968.884096] do_filp_open+0x99/0x140 [ 8968.889645] ? getname_flags+0x6e/0x330 [ 8968.892137] ? __check_object_size+0xff/0x256 [ 8968.895255] ? do_raw_spin_unlock+0x75/0x190 [ 8968.902228] ? _raw_spin_unlock+0x12/0x30 [ 8968.904989] do_sys_openat2+0x2b4/0x410 [ 8968.907023] do_sys_open+0x73/0xa0 [ 8968.913139] __x64_sys_openat+0x24/0x30 [ 8968.919138] do_syscall_64+0xc1/0x440 [ 8968.925084] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8968.933328] RIP: 0033:0x7fa749552332 [ 8968.934647] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8968.977241] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8968.988171] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 8968.995075] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 8969.012772] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 8969.021918] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 8969.024543] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 8973.287821] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7056s! [ 8973.291387] Showing busy workqueues and worker pools: [ 8973.294316] workqueue events: flags=0x0 [ 8973.297297] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8973.303139] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 8973.314489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8973.318974] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8973.324357] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.329316] pending: kfree_rcu_monitor [ 8973.333173] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8973.338014] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 8973.344220] workqueue events_long: flags=0x0 [ 8973.345501] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.351096] pending: lu_objects_flush [obdclass] [ 8973.354321] workqueue events_unbound: flags=0x2 [ 8973.358024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 8973.360375] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8973.367938] workqueue events_power_efficient: flags=0x80 [ 8973.370943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.376075] pending: check_lifetime [ 8973.377933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.382266] pending: neigh_periodic_work [ 8973.386179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.390164] pending: neigh_periodic_work [ 8973.392819] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.398883] pending: do_cache_clean [ 8973.402130] workqueue mm_percpu_wq: flags=0x8 [ 8973.405361] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.410717] pending: vmstat_update [ 8973.415644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.419030] pending: vmstat_update [ 8973.423172] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8973.429295] pending: vmstat_update [ 8973.432221] workqueue writeback: flags=0x4a [ 8973.434944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8973.438916] pending: wb_workfn, wb_workfn [ 8973.444041] workqueue xprtiod: flags=0xa [ 8973.448826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8973.458189] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8973.465609] workqueue ipv6_addrconf: flags=0x40008 [ 8973.473142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8973.478033] pending: addrconf_verify_work [ 8973.481820] workqueue knbd0-recv: flags=0x1a [ 8973.484225] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8973.488554] in-flight: 651:recv_work [ 8973.490873] workqueue ptlrpc_pinger: flags=0x2 [ 8973.494305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8973.498145] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8973.501239] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7056s workers=4 idle: 37 31846 28897 [ 8973.505174] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8963s workers=2 idle: 653 [ 8976.060268] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 8976.066231] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8976.103186] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8976.114341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8976.119713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.122205] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8976.128785] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8976.132216] RAX: 0000000000007e9f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8976.134506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 8976.137329] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8976.139274] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 8976.141857] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 8976.144271] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 8976.147405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8976.149508] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 8976.152752] Call Trace: [ 8976.154315] [ 8976.155326] ? show_regs.cold.9+0x22/0x2f [ 8976.156850] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8976.168974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8976.176861] ? watchdog+0x40/0x40 [ 8976.183785] ? hrtimer_interrupt+0x128/0x2c0 [ 8976.191132] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8976.196191] ? apic_timer_interrupt+0xf/0x20 [ 8976.200238] [ 8976.202120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.207367] do_raw_spin_lock+0x177/0x190 [ 8976.209233] _raw_spin_lock+0x21/0x30 [ 8976.214053] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8976.218228] ? do_raw_spin_unlock+0x75/0x190 [ 8976.223141] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8976.225483] ? do_wait_intr+0xf0/0xf0 [ 8976.227161] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8976.229940] mdc_close+0x53b/0xd70 [mdc] [ 8976.231223] lmv_close+0x18d/0x490 [lmv] [ 8976.233035] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8976.236069] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8976.238284] ll_md_close+0x274/0x800 [lustre] [ 8976.240154] ll_file_release+0x128/0xc50 [lustre] [ 8976.242917] __fput+0x10d/0x440 [ 8976.244947] ____fput+0x12/0x20 [ 8976.246944] task_work_run+0xc6/0x110 [ 8976.249158] exit_to_usermode_loop+0x1dd/0x1f0 [ 8976.251011] do_syscall_64+0x3d6/0x440 [ 8976.252906] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8976.256282] RIP: 0033:0x7fa749552cd8 [ 8976.258266] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8976.279167] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8976.288628] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 8976.294028] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 8976.304305] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 8976.311017] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 8976.321106] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8980.058058] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 8980.061471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8980.086503] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8980.094451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8980.097810] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8980.103004] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8980.118920] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8980.125325] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 8980.136712] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 8980.144012] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 8980.153397] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 8980.160271] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8980.168030] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 8980.174185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8980.180322] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 8980.188055] Call Trace: [ 8980.192480] [ 8980.193306] ? show_regs.cold.9+0x22/0x2f [ 8980.195081] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8980.203234] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8980.205333] ? watchdog+0x40/0x40 [ 8980.210231] ? hrtimer_interrupt+0x128/0x2c0 [ 8980.214744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8980.217655] ? apic_timer_interrupt+0xf/0x20 [ 8980.221057] [ 8980.224698] ? native_safe_halt+0x2e/0x50 [ 8980.229662] kvm_wait+0xcd/0xe0 [ 8980.234081] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8980.243980] do_raw_spin_lock+0x177/0x190 [ 8980.250364] _raw_spin_lock+0x21/0x30 [ 8980.253775] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8980.262029] obd_mod_free+0x46/0x140 [ptlrpc] [ 8980.266734] mdc_commit_open+0x91/0xb0 [mdc] [ 8980.270606] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8980.274100] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8980.279127] after_reply+0xe2c/0x1780 [ptlrpc] [ 8980.283562] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8980.286638] ? __next_timer_interrupt+0x160/0x160 [ 8980.291188] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8980.295583] ? do_wait_intr+0xf0/0xf0 [ 8980.298115] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8980.304798] kthread+0x1d1/0x200 [ 8980.308281] ? set_kthread_struct+0x70/0x70 [ 8980.309906] ret_from_fork+0x1f/0x30 [ 8993.842333] rcu: INFO: rcu_sched self-detected stall on CPU [ 8993.844456] rcu: 2-...!: (3992267 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 8993.851411] (t=7080153 jiffies g=258701 q=3531) [ 8993.853212] rcu: rcu_sched kthread starved for 7076948 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8993.861451] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8993.866064] rcu: RCU grace-period kthread stack dump: [ 8993.868921] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8993.876200] Call Trace: [ 8993.876952] __schedule+0x351/0xcb0 [ 8993.878143] schedule+0xc0/0x180 [ 8993.882446] schedule_timeout+0xb4/0x190 [ 8993.884449] ? do_raw_spin_unlock+0x75/0x190 [ 8993.886444] ? __next_timer_interrupt+0x160/0x160 [ 8993.889400] ? prepare_to_swait_event+0xf3/0x140 [ 8993.894310] rcu_gp_kthread+0x8dd/0x15b0 [ 8993.896546] ? force_qs_rnp+0x2c0/0x2c0 [ 8993.901960] kthread+0x1d1/0x200 [ 8993.904257] ? set_kthread_struct+0x70/0x70 [ 8993.908291] ret_from_fork+0x1f/0x30 [ 8993.912962] rcu: Stack dump where RCU GP kthread last ran: [ 8993.914752] Sending NMI from CPU 2 to CPUs 3: [ 8993.918118] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 8993.921020] NMI backtrace for cpu 2 [ 8993.926333] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8993.938973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8993.941950] Call Trace: [ 8993.942522] [ 8993.943087] ? dump_stack+0xbb/0x10e [ 8993.950946] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8993.954537] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8993.956327] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8993.960245] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8993.963227] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8993.964653] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8993.968506] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8993.972908] ? update_process_times+0x6d/0xc0 [ 8993.978094] ? tick_sched_handle.isra.18+0x27/0x90 [ 8993.980144] ? tick_sched_timer+0x53/0xb0 [ 8993.983980] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8993.989836] ? tick_sched_handle.isra.18+0x90/0x90 [ 8993.991046] ? hrtimer_interrupt+0x128/0x2c0 [ 8993.992199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8993.996072] ? apic_timer_interrupt+0xf/0x20 [ 8993.999537] [ 8994.003781] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8994.006372] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8994.010632] ? do_raw_spin_lock+0x177/0x190 [ 8994.014479] ? _raw_spin_lock+0x21/0x30 [ 8994.017880] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8994.020250] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8994.022202] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8994.023488] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8994.025061] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8994.027660] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8994.029762] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8994.032080] ? __next_timer_interrupt+0x160/0x160 [ 8994.034507] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8994.036898] ? do_wait_intr+0xf0/0xf0 [ 8994.038160] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8994.040569] ? kthread+0x1d1/0x200 [ 8994.041800] ? set_kthread_struct+0x70/0x70 [ 8994.043791] ? ret_from_fork+0x1f/0x30 [ 8996.057044] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 8996.059252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8996.106456] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8996.111617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8996.117710] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8996.128769] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8996.130222] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8996.143477] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8996.146476] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 8996.148914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 8996.151721] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8996.154601] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 8996.157331] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 8996.160226] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 8996.163914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8996.165695] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 8996.167789] Call Trace: [ 8996.168670] [ 8996.169329] ? show_regs.cold.9+0x22/0x2f [ 8996.170538] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8996.171895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8996.173627] ? watchdog+0x40/0x40 [ 8996.174746] ? hrtimer_interrupt+0x128/0x2c0 [ 8996.176228] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8996.177904] ? apic_timer_interrupt+0xf/0x20 [ 8996.179370] [ 8996.179960] ? native_safe_halt+0x2e/0x50 [ 8996.187671] kvm_wait+0xcd/0xe0 [ 8996.189021] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8996.194028] do_raw_spin_lock+0x177/0x190 [ 8996.201654] _raw_spin_lock+0x21/0x30 [ 8996.202753] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8996.207702] process_one_work+0x2c8/0x700 [ 8996.211395] worker_thread+0x296/0x6e0 [ 8996.215537] ? rescuer_thread+0x570/0x570 [ 8996.216785] kthread+0x1d1/0x200 [ 8996.221379] ? set_kthread_struct+0x70/0x70 [ 8996.224998] ret_from_fork+0x1f/0x30 [ 9000.055782] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9000.061417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9000.113915] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9000.120830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9000.124868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9000.127291] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9000.140603] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9000.143916] RAX: 0000000000005306 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9000.147808] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9000.159859] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9000.164597] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9000.173076] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9000.184053] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9000.190618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9000.197292] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9000.206352] Call Trace: [ 9000.207186] [ 9000.207877] ? show_regs.cold.9+0x22/0x2f [ 9000.214174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9000.224941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9000.227601] ? watchdog+0x40/0x40 [ 9000.229953] ? hrtimer_interrupt+0x128/0x2c0 [ 9000.234855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9000.240871] ? apic_timer_interrupt+0xf/0x20 [ 9000.242542] [ 9000.246114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9000.252256] do_raw_spin_lock+0x177/0x190 [ 9000.256217] _raw_spin_lock+0x21/0x30 [ 9000.259486] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9000.262175] ? do_raw_spin_unlock+0x75/0x190 [ 9000.266079] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9000.269076] ? do_wait_intr+0xf0/0xf0 [ 9000.271638] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9000.276161] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9000.282451] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9000.288150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.292644] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9000.297261] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9000.301390] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9000.306871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.309261] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9000.312171] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9000.314466] lmv_intent_open+0x338/0x1150 [lmv] [ 9000.316387] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.319067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.323248] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9000.325422] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9000.327572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.331488] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9000.335462] ? do_raw_spin_unlock+0x75/0x190 [ 9000.339085] ? _raw_spin_unlock+0x12/0x30 [ 9000.342906] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9000.347662] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9000.352789] ll_atomic_open+0x36a/0xd70 [lustre] [ 9000.355546] lookup_open+0x356/0xb40 [ 9000.357423] do_last+0x313/0xfc0 [ 9000.362915] ? nd_jump_root+0xe5/0x160 [ 9000.366640] ? path_init+0x437/0x520 [ 9000.370597] path_openat+0xf7/0x500 [ 9000.375114] ? do_raw_spin_unlock+0x75/0x190 [ 9000.378404] do_filp_open+0x99/0x140 [ 9000.382399] ? getname_flags+0x6e/0x330 [ 9000.387805] ? __check_object_size+0xff/0x256 [ 9000.392632] ? do_raw_spin_unlock+0x75/0x190 [ 9000.395337] ? _raw_spin_unlock+0x12/0x30 [ 9000.399419] do_sys_openat2+0x2b4/0x410 [ 9000.403252] do_sys_open+0x73/0xa0 [ 9000.405264] __x64_sys_openat+0x24/0x30 [ 9000.409403] do_syscall_64+0xc1/0x440 [ 9000.412312] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9000.417517] RIP: 0033:0x7fa749552332 [ 9000.420285] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9000.432477] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9000.436150] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9000.440195] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9000.443253] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9000.446574] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9000.453463] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9004.002037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7086s! [ 9004.008702] Showing busy workqueues and worker pools: [ 9004.012540] workqueue events: flags=0x0 [ 9004.013655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9004.016500] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9004.024560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9004.030687] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9004.036637] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.039900] pending: kfree_rcu_monitor [ 9004.042062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9004.054134] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9004.063965] workqueue events_long: flags=0x0 [ 9004.068474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.078703] pending: lu_objects_flush [obdclass] [ 9004.083033] workqueue events_unbound: flags=0x2 [ 9004.086416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9004.093715] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9004.102686] workqueue events_power_efficient: flags=0x80 [ 9004.105979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.110659] pending: check_lifetime [ 9004.112981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.115228] pending: neigh_periodic_work [ 9004.122789] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.128365] pending: neigh_periodic_work [ 9004.131503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.138419] pending: do_cache_clean [ 9004.140601] workqueue mm_percpu_wq: flags=0x8 [ 9004.143431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.150255] pending: vmstat_update [ 9004.153139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.159446] pending: vmstat_update [ 9004.168201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9004.170719] pending: vmstat_update [ 9004.179380] workqueue writeback: flags=0x4a [ 9004.180840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9004.186535] pending: wb_workfn, wb_workfn [ 9004.193264] workqueue xprtiod: flags=0xa [ 9004.201328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9004.208603] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9004.216434] workqueue ipv6_addrconf: flags=0x40008 [ 9004.220566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9004.227607] pending: addrconf_verify_work [ 9004.232190] workqueue knbd0-recv: flags=0x1a [ 9004.235562] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9004.239173] in-flight: 651:recv_work [ 9004.245037] workqueue ptlrpc_pinger: flags=0x2 [ 9004.249637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9004.254352] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9004.258776] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7087s workers=4 idle: 37 31846 28897 [ 9004.265393] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8994s workers=2 idle: 653 [ 9020.058031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9020.061704] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9020.063388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9020.063466] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9020.063472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9020.063476] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9020.063489] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9020.063493] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9020.063497] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9020.063499] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9020.063501] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9020.063504] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9020.063506] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9020.063509] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9020.063512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9020.063515] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9020.063523] Call Trace: [ 9020.063530] [ 9020.063535] ? show_regs.cold.9+0x22/0x2f [ 9020.063545] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9020.063550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9020.063554] ? watchdog+0x40/0x40 [ 9020.063557] ? hrtimer_interrupt+0x128/0x2c0 [ 9020.063561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9020.063567] ? apic_timer_interrupt+0xf/0x20 [ 9020.063572] [ 9020.063574] ? native_safe_halt+0x2e/0x50 [ 9020.063578] kvm_wait+0xcd/0xe0 [ 9020.063585] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9020.063592] do_raw_spin_lock+0x177/0x190 [ 9020.063597] _raw_spin_lock+0x21/0x30 [ 9020.063602] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9020.064038] obd_mod_free+0x46/0x140 [ptlrpc] [ 9020.064299] mdc_commit_open+0x91/0xb0 [mdc] [ 9020.064337] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9020.064536] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9020.064737] after_reply+0xe2c/0x1780 [ptlrpc] [ 9020.065635] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9020.065865] ? __next_timer_interrupt+0x160/0x160 [ 9020.065874] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9020.070211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9020.070293] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9020.070300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9020.070304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.070320] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9020.070324] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9020.070329] RAX: 00000000000032ad RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9020.070700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9020.070703] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9020.070705] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9020.070707] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9020.070710] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9020.070713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9020.070715] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9020.070723] Call Trace: [ 9020.070727] [ 9020.070733] ? show_regs.cold.9+0x22/0x2f [ 9020.070742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9020.070748] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9020.070752] ? watchdog+0x40/0x40 [ 9020.070755] ? hrtimer_interrupt+0x128/0x2c0 [ 9020.070759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9020.070766] ? apic_timer_interrupt+0xf/0x20 [ 9020.070770] [ 9020.070773] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.070779] do_raw_spin_lock+0x177/0x190 [ 9020.070782] _raw_spin_lock+0x21/0x30 [ 9020.070787] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9020.109825] ? do_wait_intr+0xf0/0xf0 [ 9020.109849] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9020.115536] ? do_raw_spin_unlock+0x75/0x190 [ 9020.115558] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9020.115966] ? do_wait_intr+0xf0/0xf0 [ 9020.115976] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9020.116798] mdc_close+0x53b/0xd70 [mdc] [ 9020.116837] lmv_close+0x18d/0x490 [lmv] [ 9020.116865] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9020.117774] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9020.117861] ll_md_close+0x274/0x800 [lustre] [ 9020.118028] ll_file_release+0x128/0xc50 [lustre] [ 9020.118257] __fput+0x10d/0x440 [ 9020.118266] ____fput+0x12/0x20 [ 9020.118269] task_work_run+0xc6/0x110 [ 9020.118276] exit_to_usermode_loop+0x1dd/0x1f0 [ 9020.118284] do_syscall_64+0x3d6/0x440 [ 9020.118287] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9020.118296] RIP: 0033:0x7fa749552cd8 [ 9020.118302] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9020.118306] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9020.118311] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9020.118313] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9020.118315] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9020.118317] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9020.118319] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9020.440418] kthread+0x1d1/0x200 [ 9020.441814] ? set_kthread_struct+0x70/0x70 [ 9020.443828] ret_from_fork+0x1f/0x30 [ 9024.057727] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9024.069751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9024.105195] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9024.113555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9024.118969] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9024.123341] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9024.127529] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9024.137962] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9024.142633] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9024.146061] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9024.150556] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9024.159724] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9024.163358] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9024.168911] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9024.171713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9024.174337] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9024.177304] Call Trace: [ 9024.178330] [ 9024.179281] ? show_regs.cold.9+0x22/0x2f [ 9024.183528] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9024.187996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9024.193948] ? watchdog+0x40/0x40 [ 9024.196375] ? hrtimer_interrupt+0x128/0x2c0 [ 9024.198372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9024.200601] ? apic_timer_interrupt+0xf/0x20 [ 9024.208743] [ 9024.211812] ? native_safe_halt+0x2e/0x50 [ 9024.216460] kvm_wait+0xcd/0xe0 [ 9024.218993] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9024.227518] do_raw_spin_lock+0x177/0x190 [ 9024.229874] _raw_spin_lock+0x21/0x30 [ 9024.232037] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9024.235474] process_one_work+0x2c8/0x700 [ 9024.238037] worker_thread+0x296/0x6e0 [ 9024.240192] ? rescuer_thread+0x570/0x570 [ 9024.241651] kthread+0x1d1/0x200 [ 9024.242742] ? set_kthread_struct+0x70/0x70 [ 9024.252074] ret_from_fork+0x1f/0x30 [ 9032.054384] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9032.060296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9032.088930] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9032.096211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9032.102920] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.105155] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9032.118133] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9032.121223] RAX: 000000000000788e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9032.131156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9032.139072] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9032.144618] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9032.151067] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9032.155057] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9032.158548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9032.160889] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9032.166705] Call Trace: [ 9032.168262] [ 9032.170104] ? show_regs.cold.9+0x22/0x2f [ 9032.172713] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9032.174303] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9032.176354] ? watchdog+0x40/0x40 [ 9032.180101] ? hrtimer_interrupt+0x128/0x2c0 [ 9032.182810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9032.184037] ? apic_timer_interrupt+0xf/0x20 [ 9032.187826] [ 9032.189148] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.195702] do_raw_spin_lock+0x177/0x190 [ 9032.197910] _raw_spin_lock+0x21/0x30 [ 9032.199844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9032.204272] ? do_raw_spin_unlock+0x75/0x190 [ 9032.207369] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9032.215253] ? do_wait_intr+0xf0/0xf0 [ 9032.219060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9032.223539] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9032.225304] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9032.228161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9032.231910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9032.234407] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9032.240787] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9032.242389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9032.250146] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9032.254652] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9032.257128] lmv_intent_open+0x338/0x1150 [lmv] [ 9032.261532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9032.263422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9032.267329] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9032.269955] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9032.271890] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9032.274786] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9032.277075] ? do_raw_spin_unlock+0x75/0x190 [ 9032.278334] ? _raw_spin_unlock+0x12/0x30 [ 9032.279927] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9032.281220] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9032.283864] ll_atomic_open+0x36a/0xd70 [lustre] [ 9032.285271] lookup_open+0x356/0xb40 [ 9032.286448] do_last+0x313/0xfc0 [ 9032.287362] ? nd_jump_root+0xe5/0x160 [ 9032.291213] ? path_init+0x437/0x520 [ 9032.293528] path_openat+0xf7/0x500 [ 9032.297206] ? do_raw_spin_unlock+0x75/0x190 [ 9032.299968] do_filp_open+0x99/0x140 [ 9032.300939] ? getname_flags+0x6e/0x330 [ 9032.301858] ? __check_object_size+0xff/0x256 [ 9032.304811] ? do_raw_spin_unlock+0x75/0x190 [ 9032.307769] ? _raw_spin_unlock+0x12/0x30 [ 9032.311214] do_sys_openat2+0x2b4/0x410 [ 9032.312339] do_sys_open+0x73/0xa0 [ 9032.315845] __x64_sys_openat+0x24/0x30 [ 9032.318953] do_syscall_64+0xc1/0x440 [ 9032.320182] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9032.323146] RIP: 0033:0x7fa749552332 [ 9032.330929] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9032.349503] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9032.355925] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9032.366842] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9032.372840] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9032.376811] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9032.385980] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9034.724898] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7117s! [ 9034.735565] Showing busy workqueues and worker pools: [ 9034.741188] workqueue events: flags=0x0 [ 9034.744867] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9034.753202] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9034.762083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9034.776230] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9034.784088] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.792602] pending: kfree_rcu_monitor [ 9034.793970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9034.802202] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9034.814484] workqueue events_long: flags=0x0 [ 9034.818882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.828683] pending: lu_objects_flush [obdclass] [ 9034.837223] workqueue events_unbound: flags=0x2 [ 9034.839919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9034.848151] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9034.858889] workqueue events_power_efficient: flags=0x80 [ 9034.861654] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.867211] pending: check_lifetime [ 9034.868738] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.874680] pending: neigh_periodic_work [ 9034.876256] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.879895] pending: neigh_periodic_work [ 9034.886640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.890229] pending: do_cache_clean [ 9034.897243] workqueue mm_percpu_wq: flags=0x8 [ 9034.900697] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.910134] pending: vmstat_update [ 9034.917050] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.922200] pending: vmstat_update [ 9034.930174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9034.938090] pending: vmstat_update [ 9034.942983] workqueue writeback: flags=0x4a [ 9034.945249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9034.953325] pending: wb_workfn, wb_workfn [ 9034.960614] workqueue xprtiod: flags=0xa [ 9034.964828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9034.971069] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9034.982127] workqueue ipv6_addrconf: flags=0x40008 [ 9034.990903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9035.002664] pending: addrconf_verify_work [ 9035.011601] workqueue knbd0-recv: flags=0x1a [ 9035.015688] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9035.020168] in-flight: 651:recv_work [ 9035.023198] workqueue ptlrpc_pinger: flags=0x2 [ 9035.031496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9035.042772] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9035.045724] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7117s workers=4 idle: 37 31846 28897 [ 9035.057475] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9024s workers=2 idle: 653 [ 9048.058042] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9048.063012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9048.069817] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9048.088781] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9048.095158] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata [ 9048.100587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9048.100597] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9048.100652] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9048.100656] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9048.100661] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9048.100664] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9048.100666] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9048.100669] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9048.100672] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9048.100675] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9048.100677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9048.100680] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9048.100687] Call Trace: [ 9048.100701] [ 9048.100707] ? show_regs.cold.9+0x22/0x2f [ 9048.100716] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9048.100722] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9048.100726] ? watchdog+0x40/0x40 [ 9048.100731] ? hrtimer_interrupt+0x128/0x2c0 [ 9048.100735] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9048.100741] ? apic_timer_interrupt+0xf/0x20 [ 9048.100745] [ 9048.100749] ? native_safe_halt+0x2e/0x50 [ 9048.100753] kvm_wait+0xcd/0xe0 [ 9048.100759] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9048.100766] do_raw_spin_lock+0x177/0x190 [ 9048.100770] _raw_spin_lock+0x21/0x30 [ 9048.100776] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9048.124672] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9048.124693] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9048.130053] obd_mod_free+0x46/0x140 [ptlrpc] [ 9048.131435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9048.131481] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9048.143781] mdc_commit_open+0x91/0xb0 [mdc] [ 9048.146478] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9048.146488] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9048.146493] RAX: 0000000000004b09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9048.146494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9048.146496] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9048.146497] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9048.146499] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9048.146501] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9048.146503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9048.146505] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9048.146511] Call Trace: [ 9048.149958] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9048.153940] [ 9048.153951] ? show_regs.cold.9+0x22/0x2f [ 9048.158631] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9048.161605] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9048.161623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9048.161628] ? watchdog+0x40/0x40 [ 9048.165557] after_reply+0xe2c/0x1780 [ptlrpc] [ 9048.165945] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9048.166158] ? __next_timer_interrupt+0x160/0x160 [ 9048.166170] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9048.166508] ? do_wait_intr+0xf0/0xf0 [ 9048.166518] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9048.169957] ? hrtimer_interrupt+0x128/0x2c0 [ 9048.169975] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9048.173485] kthread+0x1d1/0x200 [ 9048.177547] ? apic_timer_interrupt+0xf/0x20 [ 9048.177568] [ 9048.177574] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9048.178672] ? set_kthread_struct+0x70/0x70 [ 9048.178688] ret_from_fork+0x1f/0x30 [ 9048.332441] do_raw_spin_lock+0x177/0x190 [ 9048.333889] _raw_spin_lock+0x21/0x30 [ 9048.335037] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9048.337548] ? do_raw_spin_unlock+0x75/0x190 [ 9048.339063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9048.340392] ? do_wait_intr+0xf0/0xf0 [ 9048.342874] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9048.348469] mdc_close+0x53b/0xd70 [mdc] [ 9048.349866] lmv_close+0x18d/0x490 [lmv] [ 9048.353941] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9048.357579] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9048.359858] ll_md_close+0x274/0x800 [lustre] [ 9048.361573] ll_file_release+0x128/0xc50 [lustre] [ 9048.363733] __fput+0x10d/0x440 [ 9048.366699] ____fput+0x12/0x20 [ 9048.368611] task_work_run+0xc6/0x110 [ 9048.370339] exit_to_usermode_loop+0x1dd/0x1f0 [ 9048.372325] do_syscall_64+0x3d6/0x440 [ 9048.373723] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9048.375878] RIP: 0033:0x7fa749552cd8 [ 9048.379823] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9048.387383] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9048.389635] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9048.397964] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9048.400609] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9048.407931] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9048.410938] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9052.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9052.069547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9052.112444] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9052.128348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9052.139400] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9052.148588] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9052.152473] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9052.167198] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9052.173966] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9052.178702] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9052.183652] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9052.189985] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9052.196235] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9052.201998] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9052.208734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9052.211319] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9052.216204] Call Trace: [ 9052.217727] [ 9052.218713] ? show_regs.cold.9+0x22/0x2f [ 9052.224027] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9052.226927] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9052.229238] ? watchdog+0x40/0x40 [ 9052.230625] ? hrtimer_interrupt+0x128/0x2c0 [ 9052.232411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9052.234668] ? apic_timer_interrupt+0xf/0x20 [ 9052.236422] [ 9052.237436] ? native_safe_halt+0x2e/0x50 [ 9052.238925] kvm_wait+0xcd/0xe0 [ 9052.240070] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9052.242708] do_raw_spin_lock+0x177/0x190 [ 9052.244590] _raw_spin_lock+0x21/0x30 [ 9052.245940] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9052.248276] process_one_work+0x2c8/0x700 [ 9052.250042] worker_thread+0x296/0x6e0 [ 9052.251699] ? rescuer_thread+0x570/0x570 [ 9052.253575] kthread+0x1d1/0x200 [ 9052.254939] ? set_kthread_struct+0x70/0x70 [ 9052.256582] ret_from_fork+0x1f/0x30 [ 9060.055242] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9060.059734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9060.098955] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9060.106092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9060.111814] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9060.118217] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9060.130540] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9060.137283] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9060.140104] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9060.143730] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9060.147567] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9060.151907] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9060.155519] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9060.159933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9060.164132] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9060.167687] Call Trace: [ 9060.168855] [ 9060.169706] ? show_regs.cold.9+0x22/0x2f [ 9060.171460] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9060.174169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9060.176172] ? watchdog+0x40/0x40 [ 9060.178018] ? hrtimer_interrupt+0x128/0x2c0 [ 9060.180696] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9060.183634] ? apic_timer_interrupt+0xf/0x20 [ 9060.184892] [ 9060.185701] ? native_safe_halt+0x2e/0x50 [ 9060.187451] kvm_wait+0xcd/0xe0 [ 9060.188484] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9060.190169] do_raw_spin_lock+0x177/0x190 [ 9060.191588] _raw_spin_lock+0x21/0x30 [ 9060.192935] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9060.194927] ? do_raw_spin_unlock+0x75/0x190 [ 9060.197642] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9060.200501] ? do_wait_intr+0xf0/0xf0 [ 9060.201678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9060.203520] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9060.210211] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9060.213187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9060.215409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9060.217684] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9060.220637] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9060.222372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9060.224996] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9060.227547] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9060.230056] lmv_intent_open+0x338/0x1150 [lmv] [ 9060.231760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9060.234930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9060.239394] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9060.243441] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9060.247263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9060.253487] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9060.256754] ? do_raw_spin_unlock+0x75/0x190 [ 9060.258996] ? _raw_spin_unlock+0x12/0x30 [ 9060.260441] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9060.265002] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9060.267259] ll_atomic_open+0x36a/0xd70 [lustre] [ 9060.271885] lookup_open+0x356/0xb40 [ 9060.274212] do_last+0x313/0xfc0 [ 9060.276201] ? nd_jump_root+0xe5/0x160 [ 9060.279447] ? path_init+0x437/0x520 [ 9060.281178] path_openat+0xf7/0x500 [ 9060.282569] ? do_raw_spin_unlock+0x75/0x190 [ 9060.287506] do_filp_open+0x99/0x140 [ 9060.289595] ? getname_flags+0x6e/0x330 [ 9060.293992] ? __check_object_size+0xff/0x256 [ 9060.296085] ? do_raw_spin_unlock+0x75/0x190 [ 9060.300561] ? _raw_spin_unlock+0x12/0x30 [ 9060.302256] do_sys_openat2+0x2b4/0x410 [ 9060.308190] do_sys_open+0x73/0xa0 [ 9060.311861] __x64_sys_openat+0x24/0x30 [ 9060.318548] do_syscall_64+0xc1/0x440 [ 9060.321388] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9060.327915] RIP: 0033:0x7fa749552332 [ 9060.330651] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9060.341730] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9060.351603] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9060.359198] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9060.367660] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9060.375656] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9060.386445] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9065.443961] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7148s! [ 9065.447727] Showing busy workqueues and worker pools: [ 9065.451413] workqueue events: flags=0x0 [ 9065.453482] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9065.459509] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9065.473410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9065.480550] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9065.488411] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.492973] pending: kfree_rcu_monitor [ 9065.497991] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9065.503459] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9065.517021] workqueue events_long: flags=0x0 [ 9065.519084] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.522492] pending: lu_objects_flush [obdclass] [ 9065.527737] workqueue events_unbound: flags=0x2 [ 9065.530042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9065.536676] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9065.543379] workqueue events_power_efficient: flags=0x80 [ 9065.545812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.552036] pending: check_lifetime [ 9065.553901] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.560292] pending: neigh_periodic_work [ 9065.562229] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.568057] pending: neigh_periodic_work [ 9065.570180] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.577874] pending: do_cache_clean [ 9065.579162] workqueue mm_percpu_wq: flags=0x8 [ 9065.582175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.590788] pending: vmstat_update [ 9065.592156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.603003] pending: vmstat_update [ 9065.605784] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9065.613377] pending: vmstat_update [ 9065.616227] workqueue writeback: flags=0x4a [ 9065.618422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9065.626558] pending: wb_workfn, wb_workfn [ 9065.631928] workqueue xprtiod: flags=0xa [ 9065.636077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9065.639890] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9065.645868] workqueue ipv6_addrconf: flags=0x40008 [ 9065.649677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9065.653974] pending: addrconf_verify_work [ 9065.656087] workqueue knbd0-recv: flags=0x1a [ 9065.658323] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9065.659985] in-flight: 651:recv_work [ 9065.661031] workqueue ptlrpc_pinger: flags=0x2 [ 9065.665826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9065.670697] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9065.673572] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7148s workers=4 idle: 37 31846 28897 [ 9065.677920] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9055s workers=2 idle: 653 [ 9076.058266] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9076.060581] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9076.061680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.061750] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.061759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.061762] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9076.061776] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9076.061781] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9076.061786] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9076.061788] RDX: 0000000000001de3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9076.061790] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9076.061793] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9076.061795] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9076.061798] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9076.061801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.061803] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9076.061812] Call Trace: [ 9076.061818] [ 9076.061824] ? show_regs.cold.9+0x22/0x2f [ 9076.061834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.061840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9076.061845] ? watchdog+0x40/0x40 [ 9076.061848] ? hrtimer_interrupt+0x128/0x2c0 [ 9076.061853] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9076.061860] ? apic_timer_interrupt+0xf/0x20 [ 9076.061865] [ 9076.061869] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9076.061874] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9076.061879] do_raw_spin_lock+0x177/0x190 [ 9076.061884] _raw_spin_lock+0x21/0x30 [ 9076.061891] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9076.062183] obd_mod_free+0x46/0x140 [ptlrpc] [ 9076.062385] mdc_commit_open+0x91/0xb0 [mdc] [ 9076.062421] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9076.062617] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9076.062814] after_reply+0xe2c/0x1780 [ptlrpc] [ 9076.063024] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9076.063215] ? __next_timer_interrupt+0x160/0x160 [ 9076.063224] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9076.063885] ? do_wait_intr+0xf0/0xf0 [ 9076.063895] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9076.064130] kthread+0x1d1/0x200 [ 9076.064135] ? set_kthread_struct+0x70/0x70 [ 9076.064137] ret_from_fork+0x1f/0x30 [ 9076.381529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.416714] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.422740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.427910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.430985] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9076.440641] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9076.444383] RAX: 0000000000006f83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9076.447571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9076.451268] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9076.454044] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9076.457536] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9076.460532] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9076.464433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.467327] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9076.470797] Call Trace: [ 9076.472019] [ 9076.473533] ? show_regs.cold.9+0x22/0x2f [ 9076.475688] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.478987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9076.481879] ? watchdog+0x40/0x40 [ 9076.483569] ? hrtimer_interrupt+0x128/0x2c0 [ 9076.486257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9076.488715] ? apic_timer_interrupt+0xf/0x20 [ 9076.491258] [ 9076.492468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.495207] do_raw_spin_lock+0x177/0x190 [ 9076.497416] _raw_spin_lock+0x21/0x30 [ 9076.499429] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9076.502366] ? do_raw_spin_unlock+0x75/0x190 [ 9076.504636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9076.507551] ? do_wait_intr+0xf0/0xf0 [ 9076.509518] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9076.512512] mdc_close+0x53b/0xd70 [mdc] [ 9076.514619] lmv_close+0x18d/0x490 [lmv] [ 9076.516615] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9076.519549] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9076.521460] ll_md_close+0x274/0x800 [lustre] [ 9076.523441] ll_file_release+0x128/0xc50 [lustre] [ 9076.525299] __fput+0x10d/0x440 [ 9076.526528] ____fput+0x12/0x20 [ 9076.527970] task_work_run+0xc6/0x110 [ 9076.530379] exit_to_usermode_loop+0x1dd/0x1f0 [ 9076.532497] do_syscall_64+0x3d6/0x440 [ 9076.534361] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9076.536730] RIP: 0033:0x7fa749552cd8 [ 9076.538423] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9076.548536] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9076.552432] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9076.555637] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9076.558928] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9076.563458] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9076.566921] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9080.070355] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9080.078153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9080.105554] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9080.112062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9080.120809] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9080.127906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9080.135545] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9080.152424] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9080.158381] RAX: 0000000000006d8c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9080.163182] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9080.167233] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9080.175485] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9080.184168] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9080.190872] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9080.201404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9080.205368] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9080.213589] Call Trace: [ 9080.215070] [ 9080.220045] ? show_regs.cold.9+0x22/0x2f [ 9080.223140] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9080.227453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9080.231360] ? watchdog+0x40/0x40 [ 9080.233718] ? hrtimer_interrupt+0x128/0x2c0 [ 9080.237562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9080.242462] ? apic_timer_interrupt+0xf/0x20 [ 9080.249058] [ 9080.250232] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9080.252543] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9080.258327] do_raw_spin_lock+0x177/0x190 [ 9080.262589] _raw_spin_lock+0x21/0x30 [ 9080.266957] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9080.271100] process_one_work+0x2c8/0x700 [ 9080.272623] worker_thread+0x296/0x6e0 [ 9080.274537] ? rescuer_thread+0x570/0x570 [ 9080.276935] kthread+0x1d1/0x200 [ 9080.278440] ? set_kthread_struct+0x70/0x70 [ 9080.280787] ret_from_fork+0x1f/0x30 [ 9092.054467] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9092.062139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9092.089208] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9092.095127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9092.099835] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9092.102100] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9092.111362] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9092.115347] RAX: 000000000000462e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9092.118179] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9092.121491] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9092.125081] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9092.128515] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9092.131736] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9092.135438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9092.138110] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9092.141453] Call Trace: [ 9092.142773] [ 9092.143819] ? show_regs.cold.9+0x22/0x2f [ 9092.146061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9092.148232] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9092.149977] ? watchdog+0x40/0x40 [ 9092.151899] ? hrtimer_interrupt+0x128/0x2c0 [ 9092.153913] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9092.155929] ? apic_timer_interrupt+0xf/0x20 [ 9092.158063] [ 9092.159567] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9092.163339] do_raw_spin_lock+0x177/0x190 [ 9092.166131] _raw_spin_lock+0x21/0x30 [ 9092.168176] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9092.171549] ? do_raw_spin_unlock+0x75/0x190 [ 9092.173529] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9092.176066] ? do_wait_intr+0xf0/0xf0 [ 9092.177564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9092.180160] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9092.182500] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9092.186344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9092.189550] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9092.192146] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9092.194363] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9092.196621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9092.199356] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9092.201895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9092.204125] lmv_intent_open+0x338/0x1150 [lmv] [ 9092.206274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9092.209093] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9092.212108] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9092.214268] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9092.216892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9092.219519] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9092.221905] ? do_raw_spin_unlock+0x75/0x190 [ 9092.224136] ? _raw_spin_unlock+0x12/0x30 [ 9092.226164] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9092.229203] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9092.231619] ll_atomic_open+0x36a/0xd70 [lustre] [ 9092.234566] lookup_open+0x356/0xb40 [ 9092.236232] do_last+0x313/0xfc0 [ 9092.238165] ? nd_jump_root+0xe5/0x160 [ 9092.239888] ? path_init+0x437/0x520 [ 9092.241713] path_openat+0xf7/0x500 [ 9092.243312] ? do_raw_spin_unlock+0x75/0x190 [ 9092.245071] do_filp_open+0x99/0x140 [ 9092.247067] ? getname_flags+0x6e/0x330 [ 9092.249227] ? __check_object_size+0xff/0x256 [ 9092.251141] ? do_raw_spin_unlock+0x75/0x190 [ 9092.253119] ? _raw_spin_unlock+0x12/0x30 [ 9092.255158] do_sys_openat2+0x2b4/0x410 [ 9092.257207] do_sys_open+0x73/0xa0 [ 9092.259859] __x64_sys_openat+0x24/0x30 [ 9092.262081] do_syscall_64+0xc1/0x440 [ 9092.263868] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9092.266196] RIP: 0033:0x7fa749552332 [ 9092.268755] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9092.277015] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9092.280789] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9092.284280] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9092.287359] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9092.290532] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9092.293560] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9096.160352] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7179s! [ 9096.164585] Showing busy workqueues and worker pools: [ 9096.166950] workqueue events: flags=0x0 [ 9096.168821] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9096.171818] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9096.176103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9096.179102] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9096.182614] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.184784] pending: kfree_rcu_monitor [ 9096.186430] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9096.189084] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9096.192574] workqueue events_long: flags=0x0 [ 9096.194240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.196779] pending: lu_objects_flush [obdclass] [ 9096.198970] workqueue events_unbound: flags=0x2 [ 9096.201149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9096.203509] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9096.216996] workqueue events_power_efficient: flags=0x80 [ 9096.226072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.229917] pending: check_lifetime [ 9096.231755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.243478] pending: neigh_periodic_work [ 9096.252259] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.263547] pending: neigh_periodic_work [ 9096.267284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.273563] pending: do_cache_clean [ 9096.275471] workqueue mm_percpu_wq: flags=0x8 [ 9096.282298] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.290387] pending: vmstat_update [ 9096.298934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.311474] pending: vmstat_update [ 9096.319038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9096.324897] pending: vmstat_update [ 9096.329481] workqueue writeback: flags=0x4a [ 9096.336913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9096.345286] pending: wb_workfn, wb_workfn [ 9096.348360] workqueue xprtiod: flags=0xa [ 9096.351092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9096.359666] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9096.366782] workqueue ipv6_addrconf: flags=0x40008 [ 9096.369568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9096.373457] pending: addrconf_verify_work [ 9096.378629] workqueue knbd0-recv: flags=0x1a [ 9096.382295] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9096.390411] in-flight: 651:recv_work [ 9096.393519] workqueue ptlrpc_pinger: flags=0x2 [ 9096.403860] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9096.406892] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9096.413837] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7179s workers=4 idle: 37 31846 28897 [ 9096.432872] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9086s workers=2 idle: 653 [ 9104.058011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9104.059022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9104.060966] Modules linked in: [ 9104.063619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9104.063689] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9104.063708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9104.063712] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9104.063725] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9104.063729] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9104.064932] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9104.065000] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9104.065005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9104.065008] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9104.065019] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9104.065024] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9104.065028] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9104.065030] RDX: 0000000000002c4c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9104.065032] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9104.065034] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9104.065036] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9104.065038] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9104.065041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9104.065043] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9104.065054] Call Trace: [ 9104.065061] [ 9104.065066] ? show_regs.cold.9+0x22/0x2f [ 9104.065075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9104.065082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9104.065086] ? watchdog+0x40/0x40 [ 9104.065090] ? hrtimer_interrupt+0x128/0x2c0 [ 9104.065094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9104.065101] ? apic_timer_interrupt+0xf/0x20 [ 9104.065106] [ 9104.065109] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9104.065115] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9104.065120] do_raw_spin_lock+0x177/0x190 [ 9104.065124] _raw_spin_lock+0x21/0x30 [ 9104.065130] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9104.065393] obd_mod_free+0x46/0x140 [ptlrpc] [ 9104.065736] mdc_commit_open+0x91/0xb0 [mdc] [ 9104.065779] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9104.065984] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9104.066175] after_reply+0xe2c/0x1780 [ptlrpc] [ 9104.066429] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9104.066643] ? __next_timer_interrupt+0x160/0x160 [ 9104.066655] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9104.066847] ? do_wait_intr+0xf0/0xf0 [ 9104.066856] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9104.067051] kthread+0x1d1/0x200 [ 9104.067057] ? set_kthread_struct+0x70/0x70 [ 9104.067062] ret_from_fork+0x1f/0x30 [ 9104.129835] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9104.609180] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9104.611675] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9104.623769] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9104.637961] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9104.650929] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9104.663902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9104.665851] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9104.673865] Call Trace: [ 9104.676827] [ 9104.680030] ? show_regs.cold.9+0x22/0x2f [ 9104.683818] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9104.687678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9104.689749] ? watchdog+0x40/0x40 [ 9104.693673] ? hrtimer_interrupt+0x128/0x2c0 [ 9104.699605] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9104.702199] ? apic_timer_interrupt+0xf/0x20 [ 9104.705627] [ 9104.706894] ? native_safe_halt+0x2e/0x50 [ 9104.709765] kvm_wait+0xcd/0xe0 [ 9104.714761] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9104.717639] do_raw_spin_lock+0x177/0x190 [ 9104.722015] _raw_spin_lock+0x21/0x30 [ 9104.726683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9104.733166] ? do_raw_spin_unlock+0x75/0x190 [ 9104.738840] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9104.748856] ? do_wait_intr+0xf0/0xf0 [ 9104.750804] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9104.762511] mdc_close+0x53b/0xd70 [mdc] [ 9104.773861] lmv_close+0x18d/0x490 [lmv] [ 9104.775342] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9104.790175] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9104.797238] ll_md_close+0x274/0x800 [lustre] [ 9104.804995] ll_file_release+0x128/0xc50 [lustre] [ 9104.812998] __fput+0x10d/0x440 [ 9104.818882] ____fput+0x12/0x20 [ 9104.821996] task_work_run+0xc6/0x110 [ 9104.826276] exit_to_usermode_loop+0x1dd/0x1f0 [ 9104.837237] do_syscall_64+0x3d6/0x440 [ 9104.840709] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9104.845064] RIP: 0033:0x7fa749552cd8 [ 9104.846270] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9104.873824] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9104.889889] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9104.897096] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9104.910237] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9104.919346] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9104.939980] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9108.057157] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9108.060062] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9108.079393] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9108.083558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9108.087046] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9108.090803] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9108.093369] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9108.101834] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9108.105124] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9108.108518] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9108.111731] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9108.114690] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9108.118005] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9108.121891] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9108.125152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9108.127827] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9108.130916] Call Trace: [ 9108.131975] [ 9108.132539] ? show_regs.cold.9+0x22/0x2f [ 9108.134359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9108.136293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9108.138229] ? watchdog+0x40/0x40 [ 9108.139879] ? hrtimer_interrupt+0x128/0x2c0 [ 9108.141376] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9108.143935] ? apic_timer_interrupt+0xf/0x20 [ 9108.145759] [ 9108.146824] ? native_safe_halt+0x2e/0x50 [ 9108.148519] kvm_wait+0xcd/0xe0 [ 9108.150025] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9108.152070] do_raw_spin_lock+0x177/0x190 [ 9108.153410] _raw_spin_lock+0x21/0x30 [ 9108.154559] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9108.157178] process_one_work+0x2c8/0x700 [ 9108.158953] worker_thread+0x296/0x6e0 [ 9108.160480] ? rescuer_thread+0x570/0x570 [ 9108.162271] kthread+0x1d1/0x200 [ 9108.163854] ? set_kthread_struct+0x70/0x70 [ 9108.165766] ret_from_fork+0x1f/0x30 [ 9124.054083] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9124.061433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9124.108554] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9124.117582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9124.123411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9124.128646] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9124.146292] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9124.149252] RAX: 0000000000003387 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9124.154116] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9124.157460] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9124.161014] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9124.166429] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9124.170431] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9124.173465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9124.178251] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9124.183753] Call Trace: [ 9124.185010] [ 9124.185942] ? show_regs.cold.9+0x22/0x2f [ 9124.191126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9124.194396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9124.197545] ? watchdog+0x40/0x40 [ 9124.199119] ? hrtimer_interrupt+0x128/0x2c0 [ 9124.205084] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9124.207287] ? apic_timer_interrupt+0xf/0x20 [ 9124.209831] [ 9124.212897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9124.216175] do_raw_spin_lock+0x177/0x190 [ 9124.221059] _raw_spin_lock+0x21/0x30 [ 9124.222047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9124.227728] ? do_raw_spin_unlock+0x75/0x190 [ 9124.230492] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9124.235656] ? do_wait_intr+0xf0/0xf0 [ 9124.238014] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9124.243667] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9124.246296] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9124.251250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.254544] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9124.260502] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9124.263148] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9124.268327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.273381] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9124.277127] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9124.282227] lmv_intent_open+0x338/0x1150 [lmv] [ 9124.283895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.286945] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.294099] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9124.298314] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9124.300998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.306801] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9124.309499] ? do_raw_spin_unlock+0x75/0x190 [ 9124.314185] ? _raw_spin_unlock+0x12/0x30 [ 9124.318764] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9124.322390] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9124.329272] ll_atomic_open+0x36a/0xd70 [lustre] [ 9124.331975] lookup_open+0x356/0xb40 [ 9124.337436] do_last+0x313/0xfc0 [ 9124.339052] ? nd_jump_root+0xe5/0x160 [ 9124.342090] ? path_init+0x437/0x520 [ 9124.343925] path_openat+0xf7/0x500 [ 9124.348718] ? do_raw_spin_unlock+0x75/0x190 [ 9124.350463] do_filp_open+0x99/0x140 [ 9124.354944] ? getname_flags+0x6e/0x330 [ 9124.356426] ? __check_object_size+0xff/0x256 [ 9124.359247] ? do_raw_spin_unlock+0x75/0x190 [ 9124.363512] ? _raw_spin_unlock+0x12/0x30 [ 9124.365379] do_sys_openat2+0x2b4/0x410 [ 9124.367412] do_sys_open+0x73/0xa0 [ 9124.371452] __x64_sys_openat+0x24/0x30 [ 9124.376726] do_syscall_64+0xc1/0x440 [ 9124.379830] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9124.382737] RIP: 0033:0x7fa749552332 [ 9124.384251] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9124.399370] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9124.405548] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9124.412110] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9124.422080] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9124.427531] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9124.435778] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9126.884909] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7209s! [ 9126.890924] Showing busy workqueues and worker pools: [ 9126.893745] workqueue events: flags=0x0 [ 9126.895942] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9126.899635] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9126.906166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9126.910919] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9126.915722] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9126.919513] pending: kfree_rcu_monitor [ 9126.921922] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9126.926541] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9126.931894] workqueue events_long: flags=0x0 [ 9126.934773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9126.938851] pending: lu_objects_flush [obdclass] [ 9126.941076] workqueue events_unbound: flags=0x2 [ 9126.944125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9126.947721] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9126.953526] workqueue events_power_efficient: flags=0x80 [ 9126.956213] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9126.959969] pending: check_lifetime [ 9126.962855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9126.966691] pending: neigh_periodic_work [ 9126.969048] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9126.972483] pending: neigh_periodic_work [ 9126.974785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9126.978825] pending: do_cache_clean [ 9126.981515] workqueue mm_percpu_wq: flags=0x8 [ 9126.984636] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9126.988980] pending: vmstat_update [ 9126.991527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9126.995996] pending: vmstat_update [ 9126.998076] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9127.002092] pending: vmstat_update [ 9127.004753] workqueue writeback: flags=0x4a [ 9127.006595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9127.009973] pending: wb_workfn, wb_workfn [ 9127.012901] workqueue xprtiod: flags=0xa [ 9127.015596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9127.019803] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9127.024491] workqueue ipv6_addrconf: flags=0x40008 [ 9127.027198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9127.031910] pending: addrconf_verify_work [ 9127.035183] workqueue knbd0-recv: flags=0x1a [ 9127.038893] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9127.044341] in-flight: 651:recv_work [ 9127.046865] workqueue ptlrpc_pinger: flags=0x2 [ 9127.048703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9127.053435] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9127.056950] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7209s workers=4 idle: 37 31846 28897 [ 9127.061704] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9116s workers=2 idle: 653 [ 9132.060682] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9132.061496] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9132.074851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9132.075051] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9132.075058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9132.075061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9132.075076] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9132.075080] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9132.075084] RAX: 000000000000791f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9132.075087] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9132.075089] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9132.075091] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9132.075093] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9132.075097] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9132.075100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9132.075102] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9132.075110] Call Trace: [ 9132.075115] [ 9132.075121] ? show_regs.cold.9+0x22/0x2f [ 9132.075141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9132.075146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9132.075150] ? watchdog+0x40/0x40 [ 9132.075154] ? hrtimer_interrupt+0x128/0x2c0 [ 9132.075158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9132.075166] ? apic_timer_interrupt+0xf/0x20 [ 9132.075171] [ 9132.075174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9132.075180] do_raw_spin_lock+0x177/0x190 [ 9132.075186] _raw_spin_lock+0x21/0x30 [ 9132.075191] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9132.076813] ? do_raw_spin_unlock+0x75/0x190 [ 9132.076823] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9132.077060] ? do_wait_intr+0xf0/0xf0 [ 9132.077068] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9132.077276] mdc_close+0x53b/0xd70 [mdc] [ 9132.077314] lmv_close+0x18d/0x490 [lmv] [ 9132.081100] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9132.081530] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9132.081622] ll_md_close+0x274/0x800 [lustre] [ 9132.081811] ll_file_release+0x128/0xc50 [lustre] [ 9132.081903] __fput+0x10d/0x440 [ 9132.081912] ____fput+0x12/0x20 [ 9132.081917] task_work_run+0xc6/0x110 [ 9132.081924] exit_to_usermode_loop+0x1dd/0x1f0 [ 9132.081931] do_syscall_64+0x3d6/0x440 [ 9132.081936] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9132.081945] RIP: 0033:0x7fa749552cd8 [ 9132.081952] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9132.081956] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9132.081961] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9132.081964] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9132.081977] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9132.081979] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9132.081981] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9132.547319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9132.571334] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9132.576965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9132.581782] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9132.584870] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9132.594190] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9132.598084] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9132.601251] RDX: 0000000000007600 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9132.605368] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9132.608614] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9132.611946] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9132.615894] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9132.620157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9132.622255] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9132.627121] Call Trace: [ 9132.628552] [ 9132.629657] ? show_regs.cold.9+0x22/0x2f [ 9132.632084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9132.634417] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9132.636715] ? watchdog+0x40/0x40 [ 9132.638188] ? hrtimer_interrupt+0x128/0x2c0 [ 9132.640892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9132.643382] ? apic_timer_interrupt+0xf/0x20 [ 9132.645785] [ 9132.646809] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9132.649731] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9132.652165] do_raw_spin_lock+0x177/0x190 [ 9132.654445] _raw_spin_lock+0x21/0x30 [ 9132.656782] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9132.659523] obd_mod_free+0x46/0x140 [ptlrpc] [ 9132.662454] mdc_commit_open+0x91/0xb0 [mdc] [ 9132.665061] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9132.668153] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9132.671547] after_reply+0xe2c/0x1780 [ptlrpc] [ 9132.674203] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9132.677152] ? __next_timer_interrupt+0x160/0x160 [ 9132.679481] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9132.682077] ? do_wait_intr+0xf0/0xf0 [ 9132.683765] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9132.685756] kthread+0x1d1/0x200 [ 9132.687835] ? set_kthread_struct+0x70/0x70 [ 9132.690322] ret_from_fork+0x1f/0x30 [ 9136.058308] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9136.062887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9136.120567] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9136.135397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9136.141107] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9136.147403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9136.158639] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9136.172710] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9136.181587] RAX: 000000000000608b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9136.189940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9136.198593] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9136.210085] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9136.214147] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9136.219544] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9136.226124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9136.228366] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9136.232168] Call Trace: [ 9136.233917] [ 9136.239433] ? show_regs.cold.9+0x22/0x2f [ 9136.243930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9136.253910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9136.255609] ? watchdog+0x40/0x40 [ 9136.256846] ? hrtimer_interrupt+0x128/0x2c0 [ 9136.263890] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9136.270143] ? apic_timer_interrupt+0xf/0x20 [ 9136.275381] [ 9136.276866] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9136.281296] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9136.290096] do_raw_spin_lock+0x177/0x190 [ 9136.293459] _raw_spin_lock+0x21/0x30 [ 9136.296328] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9136.303157] process_one_work+0x2c8/0x700 [ 9136.305449] worker_thread+0x296/0x6e0 [ 9136.306849] ? rescuer_thread+0x570/0x570 [ 9136.312455] kthread+0x1d1/0x200 [ 9136.315956] ? set_kthread_struct+0x70/0x70 [ 9136.322270] ret_from_fork+0x1f/0x30 [ 9152.056386] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9152.059855] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9152.082968] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9152.089738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9152.093910] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9152.096386] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9152.104405] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9152.107869] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9152.111163] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9152.114279] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9152.116863] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9152.119258] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9152.122555] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9152.125308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9152.127495] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9152.130021] Call Trace: [ 9152.130948] [ 9152.131746] ? show_regs.cold.9+0x22/0x2f [ 9152.140442] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9152.143013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9152.148063] ? watchdog+0x40/0x40 [ 9152.152017] ? hrtimer_interrupt+0x128/0x2c0 [ 9152.155977] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9152.159430] ? apic_timer_interrupt+0xf/0x20 [ 9152.165280] [ 9152.167483] ? native_safe_halt+0x2e/0x50 [ 9152.169508] kvm_wait+0xcd/0xe0 [ 9152.171989] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9152.175471] do_raw_spin_lock+0x177/0x190 [ 9152.180906] _raw_spin_lock+0x21/0x30 [ 9152.186025] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9152.190877] ? do_raw_spin_unlock+0x75/0x190 [ 9152.198071] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9152.202573] ? do_wait_intr+0xf0/0xf0 [ 9152.209874] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9152.215197] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9152.219042] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9152.223190] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.235869] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9152.242987] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9152.247906] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9152.254853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.260053] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9152.268134] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9152.273978] lmv_intent_open+0x338/0x1150 [lmv] [ 9152.278513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.286114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.294560] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9152.298499] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9152.302895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.311433] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9152.315050] ? do_raw_spin_unlock+0x75/0x190 [ 9152.319809] ? _raw_spin_unlock+0x12/0x30 [ 9152.322389] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9152.332381] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9152.344607] ll_atomic_open+0x36a/0xd70 [lustre] [ 9152.350063] lookup_open+0x356/0xb40 [ 9152.353854] do_last+0x313/0xfc0 [ 9152.365550] ? nd_jump_root+0xe5/0x160 [ 9152.367582] ? path_init+0x437/0x520 [ 9152.371363] path_openat+0xf7/0x500 [ 9152.375182] ? do_raw_spin_unlock+0x75/0x190 [ 9152.379043] do_filp_open+0x99/0x140 [ 9152.388267] ? getname_flags+0x6e/0x330 [ 9152.392062] ? __check_object_size+0xff/0x256 [ 9152.393606] ? do_raw_spin_unlock+0x75/0x190 [ 9152.395039] ? _raw_spin_unlock+0x12/0x30 [ 9152.402311] do_sys_openat2+0x2b4/0x410 [ 9152.406091] do_sys_open+0x73/0xa0 [ 9152.409908] __x64_sys_openat+0x24/0x30 [ 9152.414180] do_syscall_64+0xc1/0x440 [ 9152.418020] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9152.422989] RIP: 0033:0x7fa749552332 [ 9152.424586] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9152.435016] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9152.447868] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9152.452515] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9152.461313] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9152.468842] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9152.477303] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9157.090022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7239s! [ 9157.099643] Showing busy workqueues and worker pools: [ 9157.103076] workqueue events: flags=0x0 [ 9157.105197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9157.111994] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9157.122089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9157.128248] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9157.140699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.153981] pending: kfree_rcu_monitor [ 9157.159064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9157.165155] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9157.180357] workqueue events_long: flags=0x0 [ 9157.181945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.196910] pending: lu_objects_flush [obdclass] [ 9157.200505] workqueue events_unbound: flags=0x2 [ 9157.203410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9157.207471] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9157.214572] workqueue events_power_efficient: flags=0x80 [ 9157.218666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.227746] pending: check_lifetime [ 9157.231947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.240491] pending: neigh_periodic_work [ 9157.248352] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.260656] pending: neigh_periodic_work [ 9157.267739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.275696] pending: do_cache_clean [ 9157.279682] workqueue mm_percpu_wq: flags=0x8 [ 9157.290997] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.296660] pending: vmstat_update [ 9157.302101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.311739] pending: vmstat_update [ 9157.314333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9157.319152] pending: vmstat_update [ 9157.325857] workqueue writeback: flags=0x4a [ 9157.331207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9157.334549] pending: wb_workfn, wb_workfn [ 9157.340084] workqueue xprtiod: flags=0xa [ 9157.344801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9157.350024] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9157.356775] workqueue ipv6_addrconf: flags=0x40008 [ 9157.361829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9157.370841] pending: addrconf_verify_work [ 9157.372436] workqueue knbd0-recv: flags=0x1a [ 9157.380084] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9157.386215] in-flight: 651:recv_work [ 9157.387658] workqueue ptlrpc_pinger: flags=0x2 [ 9157.395779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9157.405085] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9157.413017] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7240s workers=4 idle: 37 31846 28897 [ 9157.420711] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9147s workers=2 idle: 653 [ 9160.061253] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9160.064369] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9160.070284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9160.070366] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9160.070374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9160.070377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9160.070392] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9160.070396] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9160.070401] RAX: 0000000000003a05 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9160.070404] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9160.070406] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9160.070408] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9160.070411] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9160.070414] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9160.070417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9160.070419] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9160.070428] Call Trace: [ 9160.070434] [ 9160.070440] ? show_regs.cold.9+0x22/0x2f [ 9160.070450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9160.070457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9160.070462] ? watchdog+0x40/0x40 [ 9160.070466] ? hrtimer_interrupt+0x128/0x2c0 [ 9160.070470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9160.071138] ? apic_timer_interrupt+0xf/0x20 [ 9160.071143] [ 9160.071146] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9160.071154] do_raw_spin_lock+0x177/0x190 [ 9160.071159] _raw_spin_lock+0x21/0x30 [ 9160.071164] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9160.071568] ? do_raw_spin_unlock+0x75/0x190 [ 9160.071575] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9160.071984] ? do_wait_intr+0xf0/0xf0 [ 9160.071992] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9160.072422] mdc_close+0x53b/0xd70 [mdc] [ 9160.072465] lmv_close+0x18d/0x490 [lmv] [ 9160.072513] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9160.072642] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9160.073224] ll_md_close+0x274/0x800 [lustre] [ 9160.073386] ll_file_release+0x128/0xc50 [lustre] [ 9160.073469] __fput+0x10d/0x440 [ 9160.073489] ____fput+0x12/0x20 [ 9160.073494] task_work_run+0xc6/0x110 [ 9160.073501] exit_to_usermode_loop+0x1dd/0x1f0 [ 9160.073509] do_syscall_64+0x3d6/0x440 [ 9160.073514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9160.073522] RIP: 0033:0x7fa749552cd8 [ 9160.073529] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9160.073533] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9160.073538] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9160.073540] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9160.073542] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9160.073544] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9160.073547] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9160.456007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9160.493285] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9160.505001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9160.511568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9160.517164] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9160.531062] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9160.535256] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9160.544565] RDX: 0000000000005d75 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9160.547926] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9160.551018] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9160.555300] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9160.560799] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9160.565829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9160.570569] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9160.581280] Call Trace: [ 9160.582158] [ 9160.582899] ? show_regs.cold.9+0x22/0x2f [ 9160.590564] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9160.594798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9160.598170] ? watchdog+0x40/0x40 [ 9160.599311] ? hrtimer_interrupt+0x128/0x2c0 [ 9160.603278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9160.611375] ? apic_timer_interrupt+0xf/0x20 [ 9160.614386] [ 9160.616880] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9160.626142] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9160.627910] do_raw_spin_lock+0x177/0x190 [ 9160.634559] _raw_spin_lock+0x21/0x30 [ 9160.635768] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9160.646508] obd_mod_free+0x46/0x140 [ptlrpc] [ 9160.648193] mdc_commit_open+0x91/0xb0 [mdc] [ 9160.653696] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9160.655661] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9160.659865] after_reply+0xe2c/0x1780 [ptlrpc] [ 9160.663345] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9160.669350] ? __next_timer_interrupt+0x160/0x160 [ 9160.671882] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9160.675033] ? do_wait_intr+0xf0/0xf0 [ 9160.678446] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9160.679862] kthread+0x1d1/0x200 [ 9160.683395] ? set_kthread_struct+0x70/0x70 [ 9160.685412] ret_from_fork+0x1f/0x30 [ 9164.057129] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9164.067559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9164.107935] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9164.117626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9164.133536] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9164.138890] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9164.144594] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9164.162803] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9164.171328] RAX: 0000000000007f85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9164.180796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9164.190035] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9164.196630] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9164.203695] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9164.212227] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9164.219372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9164.226252] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9164.232194] Call Trace: [ 9164.234979] [ 9164.238515] ? show_regs.cold.9+0x22/0x2f [ 9164.244416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9164.248353] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9164.257091] ? watchdog+0x40/0x40 [ 9164.258183] ? hrtimer_interrupt+0x128/0x2c0 [ 9164.261989] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9164.264754] ? apic_timer_interrupt+0xf/0x20 [ 9164.270373] [ 9164.272970] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9164.275915] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9164.289395] do_raw_spin_lock+0x177/0x190 [ 9164.293085] _raw_spin_lock+0x21/0x30 [ 9164.299947] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9164.308506] process_one_work+0x2c8/0x700 [ 9164.314170] worker_thread+0x296/0x6e0 [ 9164.316472] ? rescuer_thread+0x570/0x570 [ 9164.320170] kthread+0x1d1/0x200 [ 9164.324356] ? set_kthread_struct+0x70/0x70 [ 9164.326688] ret_from_fork+0x1f/0x30 [ 9173.845041] rcu: INFO: rcu_sched self-detected stall on CPU [ 9173.847484] rcu: 2-...!: (4105005 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 9173.851713] (t=7260156 jiffies g=258701 q=3598) [ 9173.853581] rcu: rcu_sched kthread starved for 7256951 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9173.858379] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9173.862634] rcu: RCU grace-period kthread stack dump: [ 9173.864810] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9173.869687] Call Trace: [ 9173.870576] __schedule+0x351/0xcb0 [ 9173.872123] schedule+0xc0/0x180 [ 9173.873099] schedule_timeout+0xb4/0x190 [ 9173.874309] ? do_raw_spin_unlock+0x75/0x190 [ 9173.875671] ? __next_timer_interrupt+0x160/0x160 [ 9173.878076] ? prepare_to_swait_event+0xf3/0x140 [ 9173.880361] rcu_gp_kthread+0x8dd/0x15b0 [ 9173.882063] ? force_qs_rnp+0x2c0/0x2c0 [ 9173.883620] kthread+0x1d1/0x200 [ 9173.885032] ? set_kthread_struct+0x70/0x70 [ 9173.886817] ret_from_fork+0x1f/0x30 [ 9173.888511] rcu: Stack dump where RCU GP kthread last ran: [ 9173.891162] Sending NMI from CPU 2 to CPUs 1: [ 9173.893968] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 9173.894922] NMI backtrace for cpu 2 [ 9173.899539] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9173.904636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9173.908575] Call Trace: [ 9173.909920] [ 9173.911253] ? dump_stack+0xbb/0x10e [ 9173.914094] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9173.919255] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9173.922074] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9173.924656] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9173.928079] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9173.930469] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9173.932791] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9173.935400] ? update_process_times+0x6d/0xc0 [ 9173.937445] ? tick_sched_handle.isra.18+0x27/0x90 [ 9173.939715] ? tick_sched_timer+0x53/0xb0 [ 9173.941496] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9173.943397] ? tick_sched_handle.isra.18+0x90/0x90 [ 9173.945814] ? hrtimer_interrupt+0x128/0x2c0 [ 9173.947646] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9173.949728] ? apic_timer_interrupt+0xf/0x20 [ 9173.951618] [ 9173.952597] ? native_safe_halt+0x2e/0x50 [ 9173.954397] ? kvm_wait+0xcd/0xe0 [ 9173.955748] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9173.958099] ? do_raw_spin_lock+0x177/0x190 [ 9173.959940] ? _raw_spin_lock+0x21/0x30 [ 9173.962010] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9173.964629] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9173.967212] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9173.969336] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9173.971832] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9173.974750] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9173.977929] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9173.981355] ? __next_timer_interrupt+0x160/0x160 [ 9173.983581] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9173.986083] ? do_wait_intr+0xf0/0xf0 [ 9173.987683] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9173.990389] ? kthread+0x1d1/0x200 [ 9173.991816] ? set_kthread_struct+0x70/0x70 [ 9173.994274] ? ret_from_fork+0x1f/0x30 [ 9184.054012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9184.057993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9184.089023] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9184.097125] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9184.103356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9184.107291] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9184.119848] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9184.124520] RAX: 00000000000061e5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9184.129616] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9184.134164] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9184.138851] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9184.143910] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9184.148748] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9184.154623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9184.159606] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9184.164180] Call Trace: [ 9184.166213] [ 9184.167009] ? show_regs.cold.9+0x22/0x2f [ 9184.170157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9184.173133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9184.176723] ? watchdog+0x40/0x40 [ 9184.178960] ? hrtimer_interrupt+0x128/0x2c0 [ 9184.182622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9184.186029] ? apic_timer_interrupt+0xf/0x20 [ 9184.188753] [ 9184.190603] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9184.195695] do_raw_spin_lock+0x177/0x190 [ 9184.198980] _raw_spin_lock+0x21/0x30 [ 9184.201574] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9184.206614] ? do_raw_spin_unlock+0x75/0x190 [ 9184.210105] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9184.213559] ? do_wait_intr+0xf0/0xf0 [ 9184.216113] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9184.221072] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9184.225520] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9184.229171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9184.232294] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9184.235949] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9184.239258] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9184.242229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9184.246565] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9184.250595] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9184.253971] lmv_intent_open+0x338/0x1150 [lmv] [ 9184.257156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9184.261005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9184.263681] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9184.265715] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9184.268550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9184.271143] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9184.273633] ? do_raw_spin_unlock+0x75/0x190 [ 9184.276957] ? _raw_spin_unlock+0x12/0x30 [ 9184.281294] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9184.284003] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9184.288912] ll_atomic_open+0x36a/0xd70 [lustre] [ 9184.290586] lookup_open+0x356/0xb40 [ 9184.291956] do_last+0x313/0xfc0 [ 9184.293119] ? nd_jump_root+0xe5/0x160 [ 9184.294967] ? path_init+0x437/0x520 [ 9184.296338] path_openat+0xf7/0x500 [ 9184.299129] ? do_raw_spin_unlock+0x75/0x190 [ 9184.300574] do_filp_open+0x99/0x140 [ 9184.301770] ? getname_flags+0x6e/0x330 [ 9184.306302] ? __check_object_size+0xff/0x256 [ 9184.308452] ? do_raw_spin_unlock+0x75/0x190 [ 9184.312088] ? _raw_spin_unlock+0x12/0x30 [ 9184.313533] do_sys_openat2+0x2b4/0x410 [ 9184.315664] do_sys_open+0x73/0xa0 [ 9184.317267] __x64_sys_openat+0x24/0x30 [ 9184.320694] do_syscall_64+0xc1/0x440 [ 9184.324959] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9184.339908] RIP: 0033:0x7fa749552332 [ 9184.340997] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9184.351848] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9184.355510] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9184.364595] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9184.380596] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9184.385963] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9184.392173] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9187.810066] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7270s! [ 9187.813368] Showing busy workqueues and worker pools: [ 9187.815413] workqueue events: flags=0x0 [ 9187.817673] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9187.819946] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9187.824877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9187.827905] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9187.831233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.833774] pending: kfree_rcu_monitor [ 9187.835318] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9187.837859] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9187.841486] workqueue events_long: flags=0x0 [ 9187.842939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.845653] pending: lu_objects_flush [obdclass] [ 9187.847826] workqueue events_unbound: flags=0x2 [ 9187.849230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9187.852025] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9187.855510] workqueue events_power_efficient: flags=0x80 [ 9187.858026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.860366] pending: check_lifetime [ 9187.863200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.866690] pending: neigh_periodic_work [ 9187.870696] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.873799] pending: neigh_periodic_work [ 9187.875692] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.879716] pending: do_cache_clean [ 9187.881181] workqueue mm_percpu_wq: flags=0x8 [ 9187.883347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.886982] pending: vmstat_update [ 9187.888622] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.892202] pending: vmstat_update [ 9187.893985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9187.897095] pending: vmstat_update [ 9187.899170] workqueue writeback: flags=0x4a [ 9187.901662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9187.904476] pending: wb_workfn, wb_workfn [ 9187.907187] workqueue xprtiod: flags=0xa [ 9187.909441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9187.912595] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9187.916783] workqueue ipv6_addrconf: flags=0x40008 [ 9187.919345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9187.922115] pending: addrconf_verify_work [ 9187.924080] workqueue knbd0-recv: flags=0x1a [ 9187.925962] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9187.929294] in-flight: 651:recv_work [ 9187.930591] workqueue ptlrpc_pinger: flags=0x2 [ 9187.933015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9187.935918] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9187.938666] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7270s workers=4 idle: 37 31846 28897 [ 9187.941753] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9177s workers=2 idle: 653 [ 9188.059011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9188.063396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9188.089379] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9188.093039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9188.096328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9188.099234] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9188.108963] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9188.114205] RAX: 00000000000002e9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9188.117782] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9188.121510] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9188.124994] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9188.128969] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9188.135310] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9188.140548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9188.144663] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9188.149860] Call Trace: [ 9188.151142] [ 9188.152159] ? show_regs.cold.9+0x22/0x2f [ 9188.154658] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9188.156976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9188.159189] ? watchdog+0x40/0x40 [ 9188.160660] ? hrtimer_interrupt+0x128/0x2c0 [ 9188.162598] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9188.166505] ? apic_timer_interrupt+0xf/0x20 [ 9188.168487] [ 9188.169560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9188.172324] do_raw_spin_lock+0x177/0x190 [ 9188.174306] _raw_spin_lock+0x21/0x30 [ 9188.175970] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9188.179525] ? do_raw_spin_unlock+0x75/0x190 [ 9188.181792] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9188.184603] ? do_wait_intr+0xf0/0xf0 [ 9188.186574] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9188.189239] mdc_close+0x53b/0xd70 [mdc] [ 9188.190771] lmv_close+0x18d/0x490 [lmv] [ 9188.192947] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9188.196473] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9188.199300] ll_md_close+0x274/0x800 [lustre] [ 9188.201663] ll_file_release+0x128/0xc50 [lustre] [ 9188.204875] __fput+0x10d/0x440 [ 9188.206689] ____fput+0x12/0x20 [ 9188.208811] task_work_run+0xc6/0x110 [ 9188.210617] exit_to_usermode_loop+0x1dd/0x1f0 [ 9188.212753] do_syscall_64+0x3d6/0x440 [ 9188.214584] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9188.217182] RIP: 0033:0x7fa749552cd8 [ 9188.218866] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9188.228215] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9188.232036] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9188.235299] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9188.238503] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9188.241968] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9188.245318] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9200.058198] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9200.058850] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9200.061661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.061744] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.061752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.061756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9200.061771] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9200.061787] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9200.061791] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9200.061793] RDX: 0000000000007d97 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9200.061796] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9200.061798] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9200.061800] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9200.061803] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9200.061806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.061808] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9200.061816] Call Trace: [ 9200.061821] [ 9200.061828] ? show_regs.cold.9+0x22/0x2f [ 9200.061839] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.061845] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.061849] ? watchdog+0x40/0x40 [ 9200.061853] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.061857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.061867] ? apic_timer_interrupt+0xf/0x20 [ 9200.061871] [ 9200.061875] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9200.061881] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9200.061886] do_raw_spin_lock+0x177/0x190 [ 9200.061891] _raw_spin_lock+0x21/0x30 [ 9200.061898] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9200.062195] obd_mod_free+0x46/0x140 [ptlrpc] [ 9200.062392] mdc_commit_open+0x91/0xb0 [mdc] [ 9200.062446] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9200.062660] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9200.063248] after_reply+0xe2c/0x1780 [ptlrpc] [ 9200.063445] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9200.063663] ? __next_timer_interrupt+0x160/0x160 [ 9200.063674] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9200.063894] ? do_wait_intr+0xf0/0xf0 [ 9200.063903] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9200.064111] kthread+0x1d1/0x200 [ 9200.064117] ? set_kthread_struct+0x70/0x70 [ 9200.064121] ret_from_fork+0x1f/0x30 [ 9200.393082] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.451832] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.471172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.483403] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9200.485560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9200.500284] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9200.521672] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9200.539609] RAX: 0000000000007f81 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9200.548938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9200.552429] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9200.572420] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9200.579241] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9200.582053] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9200.590365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.602417] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9200.610895] Call Trace: [ 9200.612063] [ 9200.612761] ? show_regs.cold.9+0x22/0x2f [ 9200.618534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.620245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.626680] ? watchdog+0x40/0x40 [ 9200.627797] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.628965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.636342] ? apic_timer_interrupt+0xf/0x20 [ 9200.640814] [ 9200.642424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9200.644447] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9200.648783] do_raw_spin_lock+0x177/0x190 [ 9200.650115] _raw_spin_lock+0x21/0x30 [ 9200.654051] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9200.664625] process_one_work+0x2c8/0x700 [ 9200.666649] worker_thread+0x296/0x6e0 [ 9200.673208] ? rescuer_thread+0x570/0x570 [ 9200.674950] kthread+0x1d1/0x200 [ 9200.678345] ? set_kthread_struct+0x70/0x70 [ 9200.683049] ret_from_fork+0x1f/0x30 [ 9212.054009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9212.059005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9212.121180] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9212.136256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9212.149780] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9212.164367] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9212.187674] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9212.208020] RAX: 00000000000052c8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9212.216004] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9212.221761] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9212.224229] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9212.236584] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9212.243451] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9212.250815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9212.254623] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9212.264107] Call Trace: [ 9212.265268] [ 9212.266002] ? show_regs.cold.9+0x22/0x2f [ 9212.271233] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9212.273334] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9212.274863] ? watchdog+0x40/0x40 [ 9212.283590] ? hrtimer_interrupt+0x128/0x2c0 [ 9212.286445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9212.293994] ? apic_timer_interrupt+0xf/0x20 [ 9212.295361] [ 9212.304874] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9212.314268] do_raw_spin_lock+0x177/0x190 [ 9212.321995] _raw_spin_lock+0x21/0x30 [ 9212.324125] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9212.328936] ? do_raw_spin_unlock+0x75/0x190 [ 9212.332978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9212.334819] ? do_wait_intr+0xf0/0xf0 [ 9212.336077] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9212.341175] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9212.344058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9212.356552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9212.360730] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9212.364628] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9212.367494] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9212.369381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9212.373088] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9212.378153] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9212.384374] lmv_intent_open+0x338/0x1150 [lmv] [ 9212.394812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9212.397746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9212.399699] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9212.401393] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9212.403639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9212.405229] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9212.411869] ? do_raw_spin_unlock+0x75/0x190 [ 9212.422397] ? _raw_spin_unlock+0x12/0x30 [ 9212.423769] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9212.425551] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9212.433184] ll_atomic_open+0x36a/0xd70 [lustre] [ 9212.440893] lookup_open+0x356/0xb40 [ 9212.443182] do_last+0x313/0xfc0 [ 9212.446700] ? nd_jump_root+0xe5/0x160 [ 9212.447975] ? path_init+0x437/0x520 [ 9212.448848] path_openat+0xf7/0x500 [ 9212.452135] ? do_raw_spin_unlock+0x75/0x190 [ 9212.454178] do_filp_open+0x99/0x140 [ 9212.459391] ? getname_flags+0x6e/0x330 [ 9212.461445] ? __check_object_size+0xff/0x256 [ 9212.463188] ? do_raw_spin_unlock+0x75/0x190 [ 9212.465651] ? _raw_spin_unlock+0x12/0x30 [ 9212.467441] do_sys_openat2+0x2b4/0x410 [ 9212.473109] do_sys_open+0x73/0xa0 [ 9212.475906] __x64_sys_openat+0x24/0x30 [ 9212.478501] do_syscall_64+0xc1/0x440 [ 9212.479757] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9212.483580] RIP: 0033:0x7fa749552332 [ 9212.489181] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9212.506569] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9212.508774] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9212.515918] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9212.520200] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9212.524618] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9212.527109] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9216.059236] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9216.066945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9216.105514] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9216.115536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9216.124008] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9216.135842] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9216.148535] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9216.159264] RAX: 0000000000007f0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9216.166949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9216.172963] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9216.182235] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9216.184179] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9216.193690] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9216.199073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9216.207870] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9216.214562] Call Trace: [ 9216.218390] [ 9216.219138] ? show_regs.cold.9+0x22/0x2f [ 9216.231016] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9216.240557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9216.242877] ? watchdog+0x40/0x40 [ 9216.246732] ? hrtimer_interrupt+0x128/0x2c0 [ 9216.251193] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9216.255642] ? apic_timer_interrupt+0xf/0x20 [ 9216.263703] [ 9216.264742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9216.269279] do_raw_spin_lock+0x177/0x190 [ 9216.272276] _raw_spin_lock+0x21/0x30 [ 9216.273475] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9216.277273] ? do_raw_spin_unlock+0x75/0x190 [ 9216.284066] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9216.285717] ? do_wait_intr+0xf0/0xf0 [ 9216.287953] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9216.291646] mdc_close+0x53b/0xd70 [mdc] [ 9216.298057] lmv_close+0x18d/0x490 [lmv] [ 9216.299413] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9216.306011] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9216.307733] ll_md_close+0x274/0x800 [lustre] [ 9216.312339] ll_file_release+0x128/0xc50 [lustre] [ 9216.316187] __fput+0x10d/0x440 [ 9216.324853] ____fput+0x12/0x20 [ 9216.326955] task_work_run+0xc6/0x110 [ 9216.333857] exit_to_usermode_loop+0x1dd/0x1f0 [ 9216.336930] do_syscall_64+0x3d6/0x440 [ 9216.338633] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9216.343119] RIP: 0033:0x7fa749552cd8 [ 9216.355700] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9216.379857] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9216.383621] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9216.395392] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9216.399582] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9216.405508] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9216.413092] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9218.042633] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7300s! [ 9218.052799] Showing busy workqueues and worker pools: [ 9218.064439] workqueue events: flags=0x0 [ 9218.070443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9218.083427] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9218.092607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9218.099489] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9218.106863] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.115526] pending: kfree_rcu_monitor [ 9218.118630] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9218.131553] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9218.143629] workqueue events_long: flags=0x0 [ 9218.145230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.153346] pending: lu_objects_flush [obdclass] [ 9218.163557] workqueue events_unbound: flags=0x2 [ 9218.167687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9218.176405] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9218.185376] workqueue events_power_efficient: flags=0x80 [ 9218.190988] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.199432] pending: check_lifetime [ 9218.208313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.218452] pending: neigh_periodic_work [ 9218.223603] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.232010] pending: neigh_periodic_work [ 9218.233631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.239408] pending: do_cache_clean [ 9218.246369] workqueue mm_percpu_wq: flags=0x8 [ 9218.252849] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.255230] pending: vmstat_update [ 9218.267450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.272027] pending: vmstat_update [ 9218.273961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9218.286132] pending: vmstat_update [ 9218.289439] workqueue writeback: flags=0x4a [ 9218.291047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9218.302622] pending: wb_workfn, wb_workfn [ 9218.311953] workqueue xprtiod: flags=0xa [ 9218.313427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9218.319530] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9218.326588] workqueue ipv6_addrconf: flags=0x40008 [ 9218.328682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9218.337184] pending: addrconf_verify_work [ 9218.340780] workqueue knbd0-recv: flags=0x1a [ 9218.342297] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9218.346947] in-flight: 651:recv_work [ 9218.349703] workqueue ptlrpc_pinger: flags=0x2 [ 9218.352964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9218.355309] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9218.359206] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7301s workers=4 idle: 37 31846 28897 [ 9218.364677] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9208s workers=2 idle: 653 [ 9228.057129] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9228.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9228.062499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 9228.074655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 9228.080932] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [ 9228.117159] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.117212] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.117220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.117223] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9228.117238] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9228.117251] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.117256] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9228.117258] RDX: 0000000000003a75 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9228.117260] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9228.117263] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9228.117265] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9228.117268] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9228.117271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.117273] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9228.117293] Call Trace: [ 9228.117299] [ 9228.117305] ? show_regs.cold.9+0x22/0x2f [ 9228.117314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.117321] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.117325] ? watchdog+0x40/0x40 [ 9228.117329] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.117334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.117343] ? apic_timer_interrupt+0xf/0x20 [ 9228.117347] [ 9228.117350] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9228.117354] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9228.117359] do_raw_spin_lock+0x177/0x190 [ 9228.117363] _raw_spin_lock+0x21/0x30 [ 9228.117370] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9228.117913] obd_mod_free+0x46/0x140 [ptlrpc] [ 9228.118173] mdc_commit_open+0x91/0xb0 [mdc] [ 9228.118221] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9228.118676] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9228.118867] after_reply+0xe2c/0x1780 [ptlrpc] [ 9228.119130] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9228.119699] ? __next_timer_interrupt+0x160/0x160 [ 9228.119709] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9228.119921] ? do_wait_intr+0xf0/0xf0 [ 9228.119930] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9228.145872] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.145900] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.145907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.145914] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9228.146331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.146342] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.146346] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9228.146351] RAX: 0000000000004a07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9228.146353] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9228.146355] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.146357] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9228.146359] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9228.146362] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9228.146499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.146503] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9228.146511] Call Trace: [ 9228.146517] [ 9228.146523] ? show_regs.cold.9+0x22/0x2f [ 9228.146533] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.146539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.146544] ? watchdog+0x40/0x40 [ 9228.146625] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.146632] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.146641] ? apic_timer_interrupt+0xf/0x20 [ 9228.146645] [ 9228.146648] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.146654] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9228.146659] do_raw_spin_lock+0x177/0x190 [ 9228.146664] _raw_spin_lock+0x21/0x30 [ 9228.146669] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9228.146884] process_one_work+0x2c8/0x700 [ 9228.146892] worker_thread+0x296/0x6e0 [ 9228.146896] ? rescuer_thread+0x570/0x570 [ 9228.146901] kthread+0x1d1/0x200 [ 9228.146905] ? set_kthread_struct+0x70/0x70 [ 9228.146909] ret_from_fork+0x1f/0x30 [ 9228.468986] kthread+0x1d1/0x200 [ 9228.470152] ? set_kthread_struct+0x70/0x70 [ 9228.471571] ret_from_fork+0x1f/0x30 [ 9244.055154] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9244.059030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9244.063611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9244.063943] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9244.063950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9244.063953] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9244.063968] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9244.063972] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9244.063977] RAX: 000000000000410e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9244.063979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9244.063981] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9244.063983] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9244.063986] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9244.063988] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9244.063991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9244.063993] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9244.064002] Call Trace: [ 9244.066123] [ 9244.066129] ? show_regs.cold.9+0x22/0x2f [ 9244.066144] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9244.066157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9244.066162] ? watchdog+0x40/0x40 [ 9244.066166] ? hrtimer_interrupt+0x128/0x2c0 [ 9244.066170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9244.066250] ? apic_timer_interrupt+0xf/0x20 [ 9244.066255] [ 9244.066259] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9244.066269] do_raw_spin_lock+0x177/0x190 [ 9244.066274] _raw_spin_lock+0x21/0x30 [ 9244.066278] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9244.066565] ? do_raw_spin_unlock+0x75/0x190 [ 9244.066571] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9244.066864] ? do_wait_intr+0xf0/0xf0 [ 9244.066872] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9244.067081] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9244.067972] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9244.068283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.068400] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9244.068441] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9244.068471] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9244.068500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.068589] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9244.068882] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9244.068912] lmv_intent_open+0x338/0x1150 [lmv] [ 9244.068944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.069047] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.069130] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9244.069155] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9244.069408] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.069492] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9244.069576] ? do_raw_spin_unlock+0x75/0x190 [ 9244.069583] ? _raw_spin_unlock+0x12/0x30 [ 9244.069589] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9244.069619] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9244.069642] ll_atomic_open+0x36a/0xd70 [lustre] [ 9244.069919] lookup_open+0x356/0xb40 [ 9244.069928] do_last+0x313/0xfc0 [ 9244.069933] ? nd_jump_root+0xe5/0x160 [ 9244.069938] ? path_init+0x437/0x520 [ 9244.069942] path_openat+0xf7/0x500 [ 9244.069946] ? do_raw_spin_unlock+0x75/0x190 [ 9244.069950] do_filp_open+0x99/0x140 [ 9244.069956] ? getname_flags+0x6e/0x330 [ 9244.069960] ? __check_object_size+0xff/0x256 [ 9244.069968] ? do_raw_spin_unlock+0x75/0x190 [ 9244.069973] ? _raw_spin_unlock+0x12/0x30 [ 9244.069978] do_sys_openat2+0x2b4/0x410 [ 9244.069982] do_sys_open+0x73/0xa0 [ 9244.069986] __x64_sys_openat+0x24/0x30 [ 9244.069989] do_syscall_64+0xc1/0x440 [ 9244.069996] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9244.070058] RIP: 0033:0x7fa749552332 [ 9244.070067] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9244.070072] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9244.070077] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9244.070080] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9244.070082] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9244.070084] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9244.070087] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9244.454598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9244.503653] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9244.507677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9244.514024] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9244.517726] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9244.524968] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9244.527957] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9244.530538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9244.535167] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9244.537706] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9244.540117] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9244.542502] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9244.545671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9244.548009] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9244.550537] Call Trace: [ 9244.551558] [ 9244.552395] ? show_regs.cold.9+0x22/0x2f [ 9244.553910] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9244.555776] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9244.557326] ? watchdog+0x40/0x40 [ 9244.558631] ? hrtimer_interrupt+0x128/0x2c0 [ 9244.561496] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9244.563308] ? apic_timer_interrupt+0xf/0x20 [ 9244.565127] [ 9244.565880] ? native_safe_halt+0x2e/0x50 [ 9244.567452] kvm_wait+0xcd/0xe0 [ 9244.568464] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9244.571474] do_raw_spin_lock+0x177/0x190 [ 9244.573480] _raw_spin_lock+0x21/0x30 [ 9244.575123] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9244.577944] ? do_raw_spin_unlock+0x75/0x190 [ 9244.579818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9244.581804] ? do_wait_intr+0xf0/0xf0 [ 9244.582819] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9244.584519] mdc_close+0x53b/0xd70 [mdc] [ 9244.586480] lmv_close+0x18d/0x490 [lmv] [ 9244.588094] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9244.590115] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9244.591900] ll_md_close+0x274/0x800 [lustre] [ 9244.593849] ll_file_release+0x128/0xc50 [lustre] [ 9244.596135] __fput+0x10d/0x440 [ 9244.597288] ____fput+0x12/0x20 [ 9244.598576] task_work_run+0xc6/0x110 [ 9244.600287] exit_to_usermode_loop+0x1dd/0x1f0 [ 9244.602552] do_syscall_64+0x3d6/0x440 [ 9244.604397] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9244.607052] RIP: 0033:0x7fa749552cd8 [ 9244.608594] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9244.616341] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9244.619486] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9244.622648] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9244.625669] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9244.628685] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9244.631812] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9248.740135] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7331s! [ 9248.744796] Showing busy workqueues and worker pools: [ 9248.748700] workqueue events: flags=0x0 [ 9248.750631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9248.755946] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9248.764236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9248.770306] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9248.779047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.783808] pending: kfree_rcu_monitor [ 9248.785670] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9248.789957] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9248.797037] workqueue events_long: flags=0x0 [ 9248.799269] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.802931] pending: lu_objects_flush [obdclass] [ 9248.805771] workqueue events_unbound: flags=0x2 [ 9248.808131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9248.812028] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9248.818295] workqueue events_power_efficient: flags=0x80 [ 9248.821304] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.824520] pending: check_lifetime [ 9248.826546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.830366] pending: neigh_periodic_work [ 9248.832583] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.837799] pending: neigh_periodic_work [ 9248.840273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.843135] pending: do_cache_clean [ 9248.844320] workqueue mm_percpu_wq: flags=0x8 [ 9248.846496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.849826] pending: vmstat_update [ 9248.851985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.855370] pending: vmstat_update [ 9248.857624] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9248.860938] pending: vmstat_update [ 9248.863519] workqueue writeback: flags=0x4a [ 9248.865537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9248.870939] pending: wb_workfn, wb_workfn [ 9248.873406] workqueue xprtiod: flags=0xa [ 9248.874697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9248.877465] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9248.884503] workqueue ipv6_addrconf: flags=0x40008 [ 9248.889310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9248.908472] pending: addrconf_verify_work [ 9248.919770] workqueue knbd0-recv: flags=0x1a [ 9248.923304] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9248.932077] in-flight: 651:recv_work [ 9248.935586] workqueue ptlrpc_pinger: flags=0x2 [ 9248.942547] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9248.950561] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9248.953866] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7331s workers=4 idle: 37 31846 28897 [ 9248.960964] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9238s workers=2 idle: 653 [ 9256.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9256.059527] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9256.059878] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.059982] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.059987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.059999] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9256.060285] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9256.060295] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9256.060299] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.060304] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9256.060306] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9256.060309] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.060311] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9256.060314] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9256.060318] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9256.060321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.060323] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9256.060331] Call Trace: [ 9256.060348] [ 9256.060353] ? show_regs.cold.9+0x22/0x2f [ 9256.060363] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.060370] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.060375] ? watchdog+0x40/0x40 [ 9256.060379] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.060383] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.060390] ? apic_timer_interrupt+0xf/0x20 [ 9256.060394] [ 9256.060397] ? native_safe_halt+0x2e/0x50 [ 9256.060401] kvm_wait+0xcd/0xe0 [ 9256.060409] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9256.060416] do_raw_spin_lock+0x177/0x190 [ 9256.060421] _raw_spin_lock+0x21/0x30 [ 9256.060425] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9256.060633] process_one_work+0x2c8/0x700 [ 9256.060640] worker_thread+0x296/0x6e0 [ 9256.060644] ? rescuer_thread+0x570/0x570 [ 9256.060647] kthread+0x1d1/0x200 [ 9256.060651] ? set_kthread_struct+0x70/0x70 [ 9256.060655] ret_from_fork+0x1f/0x30 [ 9256.201945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.224612] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.229959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.234283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9256.237270] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9256.246429] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.249340] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9256.252951] RDX: 00000000000001bb RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9256.256291] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9256.259698] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9256.263318] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9256.266738] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9256.270707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.273659] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9256.277707] Call Trace: [ 9256.278984] [ 9256.279957] ? show_regs.cold.9+0x22/0x2f [ 9256.282090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.284190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.286273] ? watchdog+0x40/0x40 [ 9256.288094] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.290090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.292501] ? apic_timer_interrupt+0xf/0x20 [ 9256.294664] [ 9256.296189] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9256.298817] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9256.302109] do_raw_spin_lock+0x177/0x190 [ 9256.304357] _raw_spin_lock+0x21/0x30 [ 9256.306068] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9256.308592] obd_mod_free+0x46/0x140 [ptlrpc] [ 9256.311087] mdc_commit_open+0x91/0xb0 [mdc] [ 9256.313610] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9256.316041] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9256.318934] after_reply+0xe2c/0x1780 [ptlrpc] [ 9256.321212] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9256.323968] ? __next_timer_interrupt+0x160/0x160 [ 9256.326511] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9256.328710] ? do_wait_intr+0xf0/0xf0 [ 9256.330703] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9256.333031] kthread+0x1d1/0x200 [ 9256.334606] ? set_kthread_struct+0x70/0x70 [ 9256.336720] ret_from_fork+0x1f/0x30 [ 9272.059555] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9272.063353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9272.113928] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9272.124177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9272.136305] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9272.141775] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9272.168524] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9272.179611] RAX: 000000000000750a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9272.190540] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9272.202579] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9272.208643] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9272.220624] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9272.231438] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9272.240367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9272.246222] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9272.251929] Call Trace: [ 9272.253501] [ 9272.255402] ? show_regs.cold.9+0x22/0x2f [ 9272.261086] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9272.266159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9272.271980] ? watchdog+0x40/0x40 [ 9272.277744] ? hrtimer_interrupt+0x128/0x2c0 [ 9272.282199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9272.291148] ? apic_timer_interrupt+0xf/0x20 [ 9272.298249] [ 9272.299168] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9272.302888] do_raw_spin_lock+0x177/0x190 [ 9272.305583] _raw_spin_lock+0x21/0x30 [ 9272.307360] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9272.310061] ? do_raw_spin_unlock+0x75/0x190 [ 9272.314961] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9272.325835] ? do_wait_intr+0xf0/0xf0 [ 9272.329051] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9272.336794] mdc_close+0x53b/0xd70 [mdc] [ 9272.339759] lmv_close+0x18d/0x490 [lmv] [ 9272.344747] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9272.348242] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9272.349844] ll_md_close+0x274/0x800 [lustre] [ 9272.353916] ll_file_release+0x128/0xc50 [lustre] [ 9272.362641] __fput+0x10d/0x440 [ 9272.364813] ____fput+0x12/0x20 [ 9272.367441] task_work_run+0xc6/0x110 [ 9272.374415] exit_to_usermode_loop+0x1dd/0x1f0 [ 9272.377442] do_syscall_64+0x3d6/0x440 [ 9272.381008] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9272.384965] RIP: 0033:0x7fa749552cd8 [ 9272.387044] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9272.401922] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9272.410068] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9272.414637] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9272.422862] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9272.427734] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9272.433437] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9276.055079] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9276.060789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9276.088505] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9276.094990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9276.098458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9276.100352] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9276.107395] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9276.110289] RAX: 0000000000004a14 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9276.112812] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9276.115471] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9276.119792] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9276.126817] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9276.129641] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9276.132624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9276.138040] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9276.143840] Call Trace: [ 9276.148176] [ 9276.148917] ? show_regs.cold.9+0x22/0x2f [ 9276.150726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9276.156748] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9276.160860] ? watchdog+0x40/0x40 [ 9276.161902] ? hrtimer_interrupt+0x128/0x2c0 [ 9276.164483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9276.166291] ? apic_timer_interrupt+0xf/0x20 [ 9276.174074] [ 9276.174863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9276.181365] do_raw_spin_lock+0x177/0x190 [ 9276.184252] _raw_spin_lock+0x21/0x30 [ 9276.186802] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9276.189635] ? do_raw_spin_unlock+0x75/0x190 [ 9276.194795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9276.201489] ? do_wait_intr+0xf0/0xf0 [ 9276.204267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9276.209985] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9276.212082] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9276.216820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9276.223112] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9276.228120] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9276.233064] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9276.238095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9276.242512] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9276.248582] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9276.251225] lmv_intent_open+0x338/0x1150 [lmv] [ 9276.253215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9276.255860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9276.259097] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9276.261975] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9276.267125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9276.273016] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9276.275740] ? do_raw_spin_unlock+0x75/0x190 [ 9276.279789] ? _raw_spin_unlock+0x12/0x30 [ 9276.283437] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9276.287467] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9276.290769] ll_atomic_open+0x36a/0xd70 [lustre] [ 9276.294890] lookup_open+0x356/0xb40 [ 9276.296192] do_last+0x313/0xfc0 [ 9276.301335] ? nd_jump_root+0xe5/0x160 [ 9276.304768] ? path_init+0x437/0x520 [ 9276.306324] path_openat+0xf7/0x500 [ 9276.307473] ? do_raw_spin_unlock+0x75/0x190 [ 9276.311735] do_filp_open+0x99/0x140 [ 9276.315101] ? getname_flags+0x6e/0x330 [ 9276.318237] ? __check_object_size+0xff/0x256 [ 9276.322048] ? do_raw_spin_unlock+0x75/0x190 [ 9276.326956] ? _raw_spin_unlock+0x12/0x30 [ 9276.329797] do_sys_openat2+0x2b4/0x410 [ 9276.334272] do_sys_open+0x73/0xa0 [ 9276.336450] __x64_sys_openat+0x24/0x30 [ 9276.341380] do_syscall_64+0xc1/0x440 [ 9276.345290] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9276.350299] RIP: 0033:0x7fa749552332 [ 9276.354341] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9276.370565] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9276.377311] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9276.382991] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9276.386382] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9276.389232] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9276.391999] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9279.457583] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7362s! [ 9279.467514] Showing busy workqueues and worker pools: [ 9279.471661] workqueue events: flags=0x0 [ 9279.474207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9279.479789] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9279.486400] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9279.491614] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9279.499167] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.510607] pending: kfree_rcu_monitor [ 9279.516868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9279.519902] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9279.524312] workqueue events_long: flags=0x0 [ 9279.525991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.531010] pending: lu_objects_flush [obdclass] [ 9279.534177] workqueue events_unbound: flags=0x2 [ 9279.537065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9279.541033] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9279.547084] workqueue events_power_efficient: flags=0x80 [ 9279.549875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.555151] pending: check_lifetime [ 9279.556906] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.560202] pending: neigh_periodic_work [ 9279.563915] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.572140] pending: neigh_periodic_work [ 9279.579149] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.581557] pending: do_cache_clean [ 9279.589905] workqueue mm_percpu_wq: flags=0x8 [ 9279.594060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.599650] pending: vmstat_update [ 9279.600982] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.610559] pending: vmstat_update [ 9279.614190] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9279.619166] pending: vmstat_update [ 9279.622514] workqueue writeback: flags=0x4a [ 9279.624080] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9279.628621] pending: wb_workfn, wb_workfn [ 9279.635901] workqueue xprtiod: flags=0xa [ 9279.637343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9279.643561] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9279.650605] workqueue ipv6_addrconf: flags=0x40008 [ 9279.654549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9279.663968] pending: addrconf_verify_work [ 9279.666651] workqueue knbd0-recv: flags=0x1a [ 9279.669624] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9279.672851] in-flight: 651:recv_work [ 9279.673944] workqueue ptlrpc_pinger: flags=0x2 [ 9279.675196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9279.679259] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9279.683738] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7362s workers=4 idle: 37 31846 28897 [ 9279.687987] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9269s workers=2 idle: 653 [ 9284.057096] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9284.059890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.061556] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9284.114666] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.124078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.124158] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.124165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.124169] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9284.124194] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9284.124199] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9284.124204] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9284.124206] RDX: 0000000000002f74 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9284.124209] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9284.124211] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9284.124213] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9284.124217] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9284.124220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.124222] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9284.124231] Call Trace: [ 9284.124237] [ 9284.124242] ? show_regs.cold.9+0x22/0x2f [ 9284.124252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.124258] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.124262] ? watchdog+0x40/0x40 [ 9284.124267] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.124272] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.124280] ? apic_timer_interrupt+0xf/0x20 [ 9284.124285] [ 9284.124288] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9284.124293] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9284.124298] do_raw_spin_lock+0x177/0x190 [ 9284.124314] _raw_spin_lock+0x21/0x30 [ 9284.124320] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9284.124583] obd_mod_free+0x46/0x140 [ptlrpc] [ 9284.124834] mdc_commit_open+0x91/0xb0 [mdc] [ 9284.124874] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9284.125098] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9284.125323] after_reply+0xe2c/0x1780 [ptlrpc] [ 9284.125531] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9284.125832] ? __next_timer_interrupt+0x160/0x160 [ 9284.125842] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9284.126111] ? do_wait_intr+0xf0/0xf0 [ 9284.126121] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9284.126331] kthread+0x1d1/0x200 [ 9284.126337] ? set_kthread_struct+0x70/0x70 [ 9284.126341] ret_from_fork+0x1f/0x30 [ 9284.452543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.459932] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9284.466966] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9284.470956] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9284.488917] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9284.494899] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9284.506793] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9284.521815] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9284.532579] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9284.538267] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9284.544381] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9284.552833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.560419] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9284.573685] Call Trace: [ 9284.580642] [ 9284.581791] ? show_regs.cold.9+0x22/0x2f [ 9284.583330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.593695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.598840] ? watchdog+0x40/0x40 [ 9284.603929] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.605975] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.609790] ? apic_timer_interrupt+0xf/0x20 [ 9284.611277] [ 9284.614160] ? native_safe_halt+0x2e/0x50 [ 9284.616597] kvm_wait+0xcd/0xe0 [ 9284.621934] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9284.626788] do_raw_spin_lock+0x177/0x190 [ 9284.632541] _raw_spin_lock+0x21/0x30 [ 9284.633791] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9284.641458] process_one_work+0x2c8/0x700 [ 9284.643841] worker_thread+0x296/0x6e0 [ 9284.646553] ? rescuer_thread+0x570/0x570 [ 9284.648168] kthread+0x1d1/0x200 [ 9284.652836] ? set_kthread_struct+0x70/0x70 [ 9284.656130] ret_from_fork+0x1f/0x30 [ 9300.059139] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9300.064692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9300.113059] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9300.121071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9300.125151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9300.128153] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9300.151535] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9300.159383] RAX: 0000000000007692 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9300.168342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9300.172484] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9300.176673] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9300.182360] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9300.192353] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9300.200151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9300.201964] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9300.211643] Call Trace: [ 9300.215550] [ 9300.216757] ? show_regs.cold.9+0x22/0x2f [ 9300.221417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9300.231005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9300.232330] ? watchdog+0x40/0x40 [ 9300.239268] ? hrtimer_interrupt+0x128/0x2c0 [ 9300.243398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9300.249450] ? apic_timer_interrupt+0xf/0x20 [ 9300.255890] [ 9300.260274] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9300.272337] do_raw_spin_lock+0x177/0x190 [ 9300.276480] _raw_spin_lock+0x21/0x30 [ 9300.280471] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9300.288623] ? do_raw_spin_unlock+0x75/0x190 [ 9300.290467] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9300.299342] ? do_wait_intr+0xf0/0xf0 [ 9300.300487] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9300.312307] mdc_close+0x53b/0xd70 [mdc] [ 9300.322705] lmv_close+0x18d/0x490 [lmv] [ 9300.329704] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9300.347459] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9300.353134] ll_md_close+0x274/0x800 [lustre] [ 9300.356487] ll_file_release+0x128/0xc50 [lustre] [ 9300.358613] __fput+0x10d/0x440 [ 9300.361883] ____fput+0x12/0x20 [ 9300.366166] task_work_run+0xc6/0x110 [ 9300.370048] exit_to_usermode_loop+0x1dd/0x1f0 [ 9300.372363] do_syscall_64+0x3d6/0x440 [ 9300.377580] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9300.380622] RIP: 0033:0x7fa749552cd8 [ 9300.384878] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9300.402355] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9300.409405] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9300.415847] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9300.421032] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9300.428994] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9300.435407] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9304.054509] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9304.061611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9304.111033] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9304.118900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9304.136800] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9304.150240] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9304.181224] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9304.188491] RAX: 0000000000003912 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9304.198845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9304.206424] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9304.214198] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9304.222217] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9304.232623] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9304.239722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9304.247634] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9304.256389] Call Trace: [ 9304.260826] [ 9304.261560] ? show_regs.cold.9+0x22/0x2f [ 9304.262719] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9304.266870] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9304.273857] ? watchdog+0x40/0x40 [ 9304.275628] ? hrtimer_interrupt+0x128/0x2c0 [ 9304.285333] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9304.288777] ? apic_timer_interrupt+0xf/0x20 [ 9304.291777] [ 9304.292748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9304.298214] do_raw_spin_lock+0x177/0x190 [ 9304.300423] _raw_spin_lock+0x21/0x30 [ 9304.305553] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9304.310627] ? do_raw_spin_unlock+0x75/0x190 [ 9304.313034] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9304.320020] ? do_wait_intr+0xf0/0xf0 [ 9304.323732] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9304.328343] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9304.332313] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9304.334944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9304.339085] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9304.343337] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9304.345658] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9304.348232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9304.352618] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9304.356273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9304.363527] lmv_intent_open+0x338/0x1150 [lmv] [ 9304.368837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9304.373192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9304.376034] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9304.378098] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9304.380549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9304.383825] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9304.386363] ? do_raw_spin_unlock+0x75/0x190 [ 9304.388355] ? _raw_spin_unlock+0x12/0x30 [ 9304.389984] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9304.392559] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9304.395070] ll_atomic_open+0x36a/0xd70 [lustre] [ 9304.397467] lookup_open+0x356/0xb40 [ 9304.398869] do_last+0x313/0xfc0 [ 9304.402074] ? nd_jump_root+0xe5/0x160 [ 9304.404039] ? path_init+0x437/0x520 [ 9304.406293] path_openat+0xf7/0x500 [ 9304.408219] ? do_raw_spin_unlock+0x75/0x190 [ 9304.416019] do_filp_open+0x99/0x140 [ 9304.418829] ? getname_flags+0x6e/0x330 [ 9304.420863] ? __check_object_size+0xff/0x256 [ 9304.425834] ? do_raw_spin_unlock+0x75/0x190 [ 9304.427811] ? _raw_spin_unlock+0x12/0x30 [ 9304.431043] do_sys_openat2+0x2b4/0x410 [ 9304.433644] do_sys_open+0x73/0xa0 [ 9304.435882] __x64_sys_openat+0x24/0x30 [ 9304.439182] do_syscall_64+0xc1/0x440 [ 9304.442065] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9304.443708] RIP: 0033:0x7fa749552332 [ 9304.446203] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9304.454401] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9304.461595] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9304.464839] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9304.467762] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9304.470470] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9304.474065] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9310.177944] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7393s! [ 9310.181277] Showing busy workqueues and worker pools: [ 9310.182495] workqueue events: flags=0x0 [ 9310.191210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9310.200495] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9310.215098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9310.221243] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9310.227062] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.234029] pending: kfree_rcu_monitor [ 9310.236935] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9310.240790] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9310.250580] workqueue events_long: flags=0x0 [ 9310.253421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.259964] pending: lu_objects_flush [obdclass] [ 9310.267945] workqueue events_unbound: flags=0x2 [ 9310.275739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9310.282485] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9310.290221] workqueue events_power_efficient: flags=0x80 [ 9310.295472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.300276] pending: check_lifetime [ 9310.302890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.308319] pending: neigh_periodic_work [ 9310.309674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.313045] pending: neigh_periodic_work [ 9310.318853] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.327868] pending: do_cache_clean [ 9310.329509] workqueue mm_percpu_wq: flags=0x8 [ 9310.332184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.339590] pending: vmstat_update [ 9310.341389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.344892] pending: vmstat_update [ 9310.349994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9310.353493] pending: vmstat_update [ 9310.355432] workqueue writeback: flags=0x4a [ 9310.360027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9310.368151] pending: wb_workfn, wb_workfn [ 9310.375582] workqueue xprtiod: flags=0xa [ 9310.379151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9310.384115] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9310.388765] workqueue ipv6_addrconf: flags=0x40008 [ 9310.392163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9310.398119] pending: addrconf_verify_work [ 9310.400381] workqueue knbd0-recv: flags=0x1a [ 9310.405826] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9310.409137] in-flight: 651:recv_work [ 9310.411410] workqueue ptlrpc_pinger: flags=0x2 [ 9310.414972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9310.419374] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9310.425984] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7393s workers=4 idle: 37 31846 28897 [ 9310.432221] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9300s workers=2 idle: 653 [ 9312.058030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9312.060682] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9312.060996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.061080] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.061086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.061089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9312.061104] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9312.061108] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9312.061113] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9312.061116] RDX: 0000000000000082 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9312.061118] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9312.061120] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9312.061122] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9312.061125] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9312.061128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.061130] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9312.061138] Call Trace: [ 9312.061145] [ 9312.061151] ? show_regs.cold.9+0x22/0x2f [ 9312.061162] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.061169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.061173] ? watchdog+0x40/0x40 [ 9312.061177] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.061181] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.061191] ? apic_timer_interrupt+0xf/0x20 [ 9312.061196] [ 9312.061199] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9312.061226] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9312.061236] do_raw_spin_lock+0x177/0x190 [ 9312.061243] _raw_spin_lock+0x21/0x30 [ 9312.061248] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9312.062000] obd_mod_free+0x46/0x140 [ptlrpc] [ 9312.062234] mdc_commit_open+0x91/0xb0 [mdc] [ 9312.062293] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9312.062551] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9312.062773] after_reply+0xe2c/0x1780 [ptlrpc] [ 9312.063000] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9312.063234] ? __next_timer_interrupt+0x160/0x160 [ 9312.063244] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9312.063624] ? do_wait_intr+0xf0/0xf0 [ 9312.063637] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9312.063879] kthread+0x1d1/0x200 [ 9312.063887] ? set_kthread_struct+0x70/0x70 [ 9312.063890] ret_from_fork+0x1f/0x30 [ 9312.422441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.462331] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.466460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.469974] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9312.474030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.476304] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.484556] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9312.488423] RAX: 0000000000007f82 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9312.491279] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9312.494258] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9312.497039] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9312.500710] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9312.504128] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9312.513592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.516642] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9312.522086] Call Trace: [ 9312.523866] [ 9312.524699] ? show_regs.cold.9+0x22/0x2f [ 9312.529169] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.531588] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.534386] ? watchdog+0x40/0x40 [ 9312.536226] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.539373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.546082] ? apic_timer_interrupt+0xf/0x20 [ 9312.551732] [ 9312.554418] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.558493] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9312.565669] do_raw_spin_lock+0x177/0x190 [ 9312.568891] _raw_spin_lock+0x21/0x30 [ 9312.570134] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9312.576085] process_one_work+0x2c8/0x700 [ 9312.579094] worker_thread+0x296/0x6e0 [ 9312.582931] ? rescuer_thread+0x570/0x570 [ 9312.585626] kthread+0x1d1/0x200 [ 9312.588172] ? set_kthread_struct+0x70/0x70 [ 9312.590491] ret_from_fork+0x1f/0x30 [ 9328.060051] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9328.065659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9328.102721] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9328.113423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9328.120185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9328.124745] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9328.147191] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9328.154188] RAX: 0000000000005587 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9328.163001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9328.169817] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9328.177661] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9328.185368] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9328.192632] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9328.199901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9328.205904] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9328.211487] Call Trace: [ 9328.213611] [ 9328.214210] ? show_regs.cold.9+0x22/0x2f [ 9328.216927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9328.218637] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9328.220438] ? watchdog+0x40/0x40 [ 9328.224490] ? hrtimer_interrupt+0x128/0x2c0 [ 9328.226066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9328.227171] ? apic_timer_interrupt+0xf/0x20 [ 9328.228240] [ 9328.228846] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9328.231825] do_raw_spin_lock+0x177/0x190 [ 9328.233173] _raw_spin_lock+0x21/0x30 [ 9328.234539] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9328.238729] ? do_raw_spin_unlock+0x75/0x190 [ 9328.245378] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9328.250924] ? do_wait_intr+0xf0/0xf0 [ 9328.257652] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9328.266925] mdc_close+0x53b/0xd70 [mdc] [ 9328.270648] lmv_close+0x18d/0x490 [lmv] [ 9328.276110] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9328.281372] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9328.288438] ll_md_close+0x274/0x800 [lustre] [ 9328.290103] ll_file_release+0x128/0xc50 [lustre] [ 9328.302860] __fput+0x10d/0x440 [ 9328.315066] ____fput+0x12/0x20 [ 9328.320749] task_work_run+0xc6/0x110 [ 9328.323739] exit_to_usermode_loop+0x1dd/0x1f0 [ 9328.328935] do_syscall_64+0x3d6/0x440 [ 9328.331796] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9328.336981] RIP: 0033:0x7fa749552cd8 [ 9328.339124] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9328.367702] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9328.371583] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9328.377759] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9328.382135] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9328.389829] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9328.400735] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9336.054073] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9336.056924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9336.078911] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9336.084300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9336.088652] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9336.091306] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9336.102149] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9336.106314] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9336.109602] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9336.113155] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9336.116457] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9336.120492] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9336.127128] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9336.130682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9336.133507] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9336.137983] Call Trace: [ 9336.138944] [ 9336.139987] ? show_regs.cold.9+0x22/0x2f [ 9336.141444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9336.145036] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9336.147214] ? watchdog+0x40/0x40 [ 9336.148144] ? hrtimer_interrupt+0x128/0x2c0 [ 9336.155772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9336.160115] ? apic_timer_interrupt+0xf/0x20 [ 9336.164976] [ 9336.165772] ? native_safe_halt+0x2e/0x50 [ 9336.167922] kvm_wait+0xcd/0xe0 [ 9336.169652] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9336.171793] do_raw_spin_lock+0x177/0x190 [ 9336.173267] _raw_spin_lock+0x21/0x30 [ 9336.175747] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9336.179978] ? do_raw_spin_unlock+0x75/0x190 [ 9336.183805] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9336.186291] ? do_wait_intr+0xf0/0xf0 [ 9336.187745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9336.195795] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9336.199400] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9336.201788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9336.205159] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9336.207772] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9336.210766] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9336.212753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9336.218665] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9336.225893] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9336.228756] lmv_intent_open+0x338/0x1150 [lmv] [ 9336.230493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9336.232528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9336.234343] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9336.235708] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9336.237260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9336.238882] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9336.242911] ? do_raw_spin_unlock+0x75/0x190 [ 9336.247217] ? _raw_spin_unlock+0x12/0x30 [ 9336.248864] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9336.251086] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9336.258482] ll_atomic_open+0x36a/0xd70 [lustre] [ 9336.261099] lookup_open+0x356/0xb40 [ 9336.264963] do_last+0x313/0xfc0 [ 9336.266674] ? nd_jump_root+0xe5/0x160 [ 9336.274644] ? path_init+0x437/0x520 [ 9336.277652] path_openat+0xf7/0x500 [ 9336.279363] ? do_raw_spin_unlock+0x75/0x190 [ 9336.280515] do_filp_open+0x99/0x140 [ 9336.284772] ? getname_flags+0x6e/0x330 [ 9336.286932] ? __check_object_size+0xff/0x256 [ 9336.288835] ? do_raw_spin_unlock+0x75/0x190 [ 9336.295650] ? _raw_spin_unlock+0x12/0x30 [ 9336.303809] do_sys_openat2+0x2b4/0x410 [ 9336.307784] do_sys_open+0x73/0xa0 [ 9336.310951] __x64_sys_openat+0x24/0x30 [ 9336.316720] do_syscall_64+0xc1/0x440 [ 9336.320407] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9336.322713] RIP: 0033:0x7fa749552332 [ 9336.328511] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9336.343818] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9336.347615] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9336.351686] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9336.354765] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9336.357514] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9336.360890] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9340.057179] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9340.059548] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9340.060935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.061004] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.061011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.061017] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9340.062012] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9340.062023] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9340.062027] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9340.062032] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9340.062260] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9340.062264] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9340.062266] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9340.062268] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9340.062272] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9340.062357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.062360] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9340.062368] Call Trace: [ 9340.062373] [ 9340.062379] ? show_regs.cold.9+0x22/0x2f [ 9340.062391] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.062397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.062402] ? watchdog+0x40/0x40 [ 9340.062406] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.062410] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.062417] ? apic_timer_interrupt+0xf/0x20 [ 9340.062422] [ 9340.062425] ? native_safe_halt+0x2e/0x50 [ 9340.062429] kvm_wait+0xcd/0xe0 [ 9340.062436] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9340.062444] do_raw_spin_lock+0x177/0x190 [ 9340.062448] _raw_spin_lock+0x21/0x30 [ 9340.062452] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9340.062678] process_one_work+0x2c8/0x700 [ 9340.062686] worker_thread+0x296/0x6e0 [ 9340.062690] ? rescuer_thread+0x570/0x570 [ 9340.062694] kthread+0x1d1/0x200 [ 9340.062699] ? set_kthread_struct+0x70/0x70 [ 9340.062703] ret_from_fork+0x1f/0x30 [ 9340.190012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.210720] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.215898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.219936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9340.221947] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9340.232492] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9340.235441] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9340.238171] RDX: 0000000000003c1a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9340.243272] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9340.248490] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9340.251828] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9340.256483] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9340.259795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.263004] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9340.266756] Call Trace: [ 9340.267788] [ 9340.268621] ? show_regs.cold.9+0x22/0x2f [ 9340.270689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.273563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.275504] ? watchdog+0x40/0x40 [ 9340.277533] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.279445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.281115] ? apic_timer_interrupt+0xf/0x20 [ 9340.283214] [ 9340.283963] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9340.286258] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9340.288848] do_raw_spin_lock+0x177/0x190 [ 9340.294133] _raw_spin_lock+0x21/0x30 [ 9340.295833] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9340.298714] obd_mod_free+0x46/0x140 [ptlrpc] [ 9340.301003] mdc_commit_open+0x91/0xb0 [mdc] [ 9340.302523] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9340.306047] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9340.308115] after_reply+0xe2c/0x1780 [ptlrpc] [ 9340.310669] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9340.312785] ? __next_timer_interrupt+0x160/0x160 [ 9340.314788] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9340.316697] ? do_wait_intr+0xf0/0xf0 [ 9340.317930] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9340.319624] kthread+0x1d1/0x200 [ 9340.320694] ? set_kthread_struct+0x70/0x70 [ 9340.321989] ret_from_fork+0x1f/0x30 [ 9340.899033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7423s! [ 9340.902143] Showing busy workqueues and worker pools: [ 9340.904183] workqueue events: flags=0x0 [ 9340.905788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9340.908722] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9340.913699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9340.916139] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9340.920888] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.923679] pending: kfree_rcu_monitor [ 9340.925569] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9340.928218] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9340.931983] workqueue events_long: flags=0x0 [ 9340.934515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.936858] pending: lu_objects_flush [obdclass] [ 9340.938935] workqueue events_unbound: flags=0x2 [ 9340.940908] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9340.943468] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9340.946853] workqueue events_power_efficient: flags=0x80 [ 9340.949266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.952051] pending: check_lifetime [ 9340.953579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.956165] pending: neigh_periodic_work [ 9340.958727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.961584] pending: neigh_periodic_work [ 9340.963471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.966672] pending: do_cache_clean [ 9340.968773] workqueue mm_percpu_wq: flags=0x8 [ 9340.970717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.973374] pending: vmstat_update [ 9340.974763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.976968] pending: vmstat_update [ 9340.978345] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9340.981455] pending: vmstat_update [ 9340.982857] workqueue writeback: flags=0x4a [ 9340.984506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9340.986742] pending: wb_workfn, wb_workfn [ 9340.988378] workqueue xprtiod: flags=0xa [ 9340.989734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9340.991759] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9340.994654] workqueue ipv6_addrconf: flags=0x40008 [ 9340.996593] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9340.998671] pending: addrconf_verify_work [ 9340.999834] workqueue knbd0-recv: flags=0x1a [ 9341.000937] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9341.003468] in-flight: 651:recv_work [ 9341.005134] workqueue ptlrpc_pinger: flags=0x2 [ 9341.006895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9341.008891] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9341.010886] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7423s workers=4 idle: 37 31846 28897 [ 9341.013967] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9330s workers=2 idle: 653 [ 9353.848701] rcu: INFO: rcu_sched self-detected stall on CPU [ 9353.853787] rcu: 2-...!: (4200076 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 9353.870951] (t=7440159 jiffies g=258701 q=3615) [ 9353.872688] rcu: rcu_sched kthread starved for 7436954 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9353.893787] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9353.910365] rcu: RCU grace-period kthread stack dump: [ 9353.924530] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9353.947738] Call Trace: [ 9353.956301] __schedule+0x351/0xcb0 [ 9353.959900] schedule+0xc0/0x180 [ 9353.965282] schedule_timeout+0xb4/0x190 [ 9353.970801] ? do_raw_spin_unlock+0x75/0x190 [ 9353.978275] ? __next_timer_interrupt+0x160/0x160 [ 9353.985261] ? prepare_to_swait_event+0xf3/0x140 [ 9353.998470] rcu_gp_kthread+0x8dd/0x15b0 [ 9354.007405] ? force_qs_rnp+0x2c0/0x2c0 [ 9354.018356] kthread+0x1d1/0x200 [ 9354.025655] ? set_kthread_struct+0x70/0x70 [ 9354.037607] ret_from_fork+0x1f/0x30 [ 9354.047505] rcu: Stack dump where RCU GP kthread last ran: [ 9354.054656] Sending NMI from CPU 2 to CPUs 3: [ 9354.067894] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 9354.074624] NMI backtrace for cpu 2 [ 9354.078903] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9354.100512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9354.107863] Call Trace: [ 9354.110921] [ 9354.112674] ? dump_stack+0xbb/0x10e [ 9354.115322] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9354.120739] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9354.129466] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9354.138482] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9354.143016] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9354.150880] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9354.153855] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9354.157181] ? update_process_times+0x6d/0xc0 [ 9354.159378] ? tick_sched_handle.isra.18+0x27/0x90 [ 9354.162142] ? tick_sched_timer+0x53/0xb0 [ 9354.164649] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9354.167700] ? tick_sched_handle.isra.18+0x90/0x90 [ 9354.170442] ? hrtimer_interrupt+0x128/0x2c0 [ 9354.172490] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9354.176439] ? apic_timer_interrupt+0xf/0x20 [ 9354.179017] [ 9354.180225] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9354.187088] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9354.192510] ? do_raw_spin_lock+0x177/0x190 [ 9354.194899] ? _raw_spin_lock+0x21/0x30 [ 9354.197072] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9354.200620] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9354.203937] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9354.206504] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9354.210137] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9354.213356] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9354.215904] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9354.220384] ? __next_timer_interrupt+0x160/0x160 [ 9354.222821] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9354.224431] ? do_wait_intr+0xf0/0xf0 [ 9354.226114] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9354.228785] ? kthread+0x1d1/0x200 [ 9354.230904] ? set_kthread_struct+0x70/0x70 [ 9354.233031] ? ret_from_fork+0x1f/0x30 [ 9368.057191] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9368.057288] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9368.061682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.061759] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.061767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.061773] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9368.062797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.062811] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9368.062815] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9368.062819] RAX: 0000000000005216 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9368.062822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9368.062824] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9368.062825] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9368.062828] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9368.062842] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9368.062845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.062848] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9368.062856] Call Trace: [ 9368.062862] [ 9368.062868] ? show_regs.cold.9+0x22/0x2f [ 9368.062879] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.062885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.062890] ? watchdog+0x40/0x40 [ 9368.063619] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.063624] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.063632] ? apic_timer_interrupt+0xf/0x20 [ 9368.063637] [ 9368.063640] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.063645] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9368.063651] do_raw_spin_lock+0x177/0x190 [ 9368.063656] _raw_spin_lock+0x21/0x30 [ 9368.063661] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9368.064113] process_one_work+0x2c8/0x700 [ 9368.064120] worker_thread+0x296/0x6e0 [ 9368.064124] ? rescuer_thread+0x570/0x570 [ 9368.064127] kthread+0x1d1/0x200 [ 9368.064131] ? set_kthread_struct+0x70/0x70 [ 9368.064135] ret_from_fork+0x1f/0x30 [ 9368.290115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.311889] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.316954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.321024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.323714] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9368.343004] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9368.347133] RAX: 000000000000648b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9368.354880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9368.360837] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9368.366954] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9368.378412] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9368.380914] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9368.390181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.399931] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9368.408060] Call Trace: [ 9368.411112] [ 9368.415041] ? show_regs.cold.9+0x22/0x2f [ 9368.419071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.428655] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.433999] ? watchdog+0x40/0x40 [ 9368.438904] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.445049] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.451141] ? apic_timer_interrupt+0xf/0x20 [ 9368.454175] [ 9368.459805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.468041] do_raw_spin_lock+0x177/0x190 [ 9368.473076] _raw_spin_lock+0x21/0x30 [ 9368.479100] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9368.488051] ? do_raw_spin_unlock+0x75/0x190 [ 9368.496812] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9368.507024] ? do_wait_intr+0xf0/0xf0 [ 9368.512134] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9368.522313] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9368.530684] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9368.536280] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9368.539704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9368.543746] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9368.548782] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9368.551104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9368.556159] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9368.569070] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9368.575328] lmv_intent_open+0x338/0x1150 [lmv] [ 9368.583099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9368.591332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9368.601116] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9368.602704] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9368.604275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9368.607355] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9368.608555] ? do_raw_spin_unlock+0x75/0x190 [ 9368.610843] ? _raw_spin_unlock+0x12/0x30 [ 9368.613138] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9368.616130] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9368.618046] ll_atomic_open+0x36a/0xd70 [lustre] [ 9368.620044] lookup_open+0x356/0xb40 [ 9368.622202] do_last+0x313/0xfc0 [ 9368.624101] ? nd_jump_root+0xe5/0x160 [ 9368.628858] ? path_init+0x437/0x520 [ 9368.630005] path_openat+0xf7/0x500 [ 9368.631423] ? do_raw_spin_unlock+0x75/0x190 [ 9368.632931] do_filp_open+0x99/0x140 [ 9368.635141] ? getname_flags+0x6e/0x330 [ 9368.637467] ? __check_object_size+0xff/0x256 [ 9368.639598] ? do_raw_spin_unlock+0x75/0x190 [ 9368.641140] ? _raw_spin_unlock+0x12/0x30 [ 9368.642884] do_sys_openat2+0x2b4/0x410 [ 9368.644128] do_sys_open+0x73/0xa0 [ 9368.648432] __x64_sys_openat+0x24/0x30 [ 9368.650332] do_syscall_64+0xc1/0x440 [ 9368.653020] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9368.658008] RIP: 0033:0x7fa749552332 [ 9368.660502] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9368.668829] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9368.674052] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9368.680314] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9368.686115] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9368.693302] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9368.701008] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9371.111220] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7453s! [ 9371.114175] Showing busy workqueues and worker pools: [ 9371.122170] workqueue events: flags=0x0 [ 9371.126407] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9371.137919] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9371.159188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9371.161537] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9371.172055] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.179550] pending: kfree_rcu_monitor [ 9371.183178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9371.188909] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9371.199823] workqueue events_long: flags=0x0 [ 9371.204747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.208273] pending: lu_objects_flush [obdclass] [ 9371.211548] workqueue events_unbound: flags=0x2 [ 9371.214345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9371.217948] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9371.224151] workqueue events_power_efficient: flags=0x80 [ 9371.227033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.231454] pending: check_lifetime [ 9371.232909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.236035] pending: neigh_periodic_work [ 9371.237939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.241871] pending: neigh_periodic_work [ 9371.243549] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.246732] pending: do_cache_clean [ 9371.248277] workqueue mm_percpu_wq: flags=0x8 [ 9371.250579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.255020] pending: vmstat_update [ 9371.259028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.261264] pending: vmstat_update [ 9371.263916] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9371.267476] pending: vmstat_update [ 9371.269253] workqueue writeback: flags=0x4a [ 9371.270787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9371.273179] pending: wb_workfn, wb_workfn [ 9371.280646] workqueue xprtiod: flags=0xa [ 9371.282113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9371.291347] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9371.296645] workqueue ipv6_addrconf: flags=0x40008 [ 9371.308571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9371.312600] pending: addrconf_verify_work [ 9371.314034] workqueue knbd0-recv: flags=0x1a [ 9371.315512] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9371.318467] in-flight: 651:recv_work [ 9371.320171] workqueue ptlrpc_pinger: flags=0x2 [ 9371.322459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9371.324981] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9371.327931] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7454s workers=4 idle: 37 31846 28897 [ 9371.331659] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9361s workers=2 idle: 653 [ 9380.058052] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9380.059216] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9380.060900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.061166] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.061174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.061177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9380.061193] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9380.061197] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9380.061202] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9380.061204] RDX: 000000000000588c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9380.061206] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9380.061209] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9380.061211] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9380.061215] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9380.061218] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.061220] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9380.061229] Call Trace: [ 9380.061235] [ 9380.061241] ? show_regs.cold.9+0x22/0x2f [ 9380.061251] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.061258] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.061262] ? watchdog+0x40/0x40 [ 9380.061266] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.061270] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.061279] ? apic_timer_interrupt+0xf/0x20 [ 9380.061284] [ 9380.061287] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9380.061292] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9380.061297] do_raw_spin_lock+0x177/0x190 [ 9380.061302] _raw_spin_lock+0x21/0x30 [ 9380.061309] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9380.061780] obd_mod_free+0x46/0x140 [ptlrpc] [ 9380.061979] mdc_commit_open+0x91/0xb0 [mdc] [ 9380.062015] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9380.062227] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9380.062534] after_reply+0xe2c/0x1780 [ptlrpc] [ 9380.062752] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9380.062956] ? __next_timer_interrupt+0x160/0x160 [ 9380.062968] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9380.063241] ? do_wait_intr+0xf0/0xf0 [ 9380.063250] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9380.063524] kthread+0x1d1/0x200 [ 9380.063529] ? set_kthread_struct+0x70/0x70 [ 9380.063533] ret_from_fork+0x1f/0x30 [ 9380.320098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.364229] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.370431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.383590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9380.386894] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9380.397455] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9380.399921] RAX: 0000000000004e99 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9380.409382] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9380.415997] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9380.420786] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9380.423502] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9380.433444] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9380.442744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.444802] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9380.455695] Call Trace: [ 9380.456935] [ 9380.458506] ? show_regs.cold.9+0x22/0x2f [ 9380.461154] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.463844] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.473752] ? watchdog+0x40/0x40 [ 9380.476991] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.480569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.486840] ? apic_timer_interrupt+0xf/0x20 [ 9380.490502] [ 9380.491864] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9380.495935] do_raw_spin_lock+0x177/0x190 [ 9380.499911] _raw_spin_lock+0x21/0x30 [ 9380.503491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9380.508174] ? do_raw_spin_unlock+0x75/0x190 [ 9380.510478] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9380.515832] ? do_wait_intr+0xf0/0xf0 [ 9380.517811] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9380.524010] mdc_close+0x53b/0xd70 [mdc] [ 9380.528573] lmv_close+0x18d/0x490 [lmv] [ 9380.530905] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9380.538576] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9380.540315] ll_md_close+0x274/0x800 [lustre] [ 9380.541954] ll_file_release+0x128/0xc50 [lustre] [ 9380.549466] __fput+0x10d/0x440 [ 9380.555924] ____fput+0x12/0x20 [ 9380.562266] task_work_run+0xc6/0x110 [ 9380.565867] exit_to_usermode_loop+0x1dd/0x1f0 [ 9380.571001] do_syscall_64+0x3d6/0x440 [ 9380.574286] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9380.582900] RIP: 0033:0x7fa749552cd8 [ 9380.595081] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9380.605530] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9380.609385] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9380.625279] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9380.639509] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9380.644143] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9380.655737] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9396.054034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9396.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9396.057587] Modules linked in: [ 9396.064045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9396.064123] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9396.064132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9396.064139] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9396.064637] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9396.064649] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9396.064653] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9396.064657] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9396.064660] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9396.064662] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9396.064664] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9396.064666] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9396.064670] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9396.064673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9396.064675] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9396.064685] Call Trace: [ 9396.064690] [ 9396.064695] ? show_regs.cold.9+0x22/0x2f [ 9396.064705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9396.064711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9396.064715] ? watchdog+0x40/0x40 [ 9396.064719] ? hrtimer_interrupt+0x128/0x2c0 [ 9396.064724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9396.064732] ? apic_timer_interrupt+0xf/0x20 [ 9396.064736] [ 9396.064739] ? native_safe_halt+0x2e/0x50 [ 9396.064742] kvm_wait+0xcd/0xe0 [ 9396.064750] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9396.064758] do_raw_spin_lock+0x177/0x190 [ 9396.064763] _raw_spin_lock+0x21/0x30 [ 9396.064767] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9396.064970] process_one_work+0x2c8/0x700 [ 9396.064978] worker_thread+0x296/0x6e0 [ 9396.064982] ? rescuer_thread+0x570/0x570 [ 9396.064986] kthread+0x1d1/0x200 [ 9396.064990] ? set_kthread_struct+0x70/0x70 [ 9396.064994] ret_from_fork+0x1f/0x30 [ 9396.277625] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9396.331220] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9396.346278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9396.357486] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9396.362951] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9396.386396] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9396.393565] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9396.396809] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9396.399816] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9396.403709] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9396.406731] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9396.409839] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9396.413788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9396.416012] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9396.419039] Call Trace: [ 9396.420123] [ 9396.420956] ? show_regs.cold.9+0x22/0x2f [ 9396.422787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9396.424686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9396.426714] ? watchdog+0x40/0x40 [ 9396.428029] ? hrtimer_interrupt+0x128/0x2c0 [ 9396.429812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9396.431665] ? apic_timer_interrupt+0xf/0x20 [ 9396.433493] [ 9396.434440] ? native_safe_halt+0x2e/0x50 [ 9396.435972] kvm_wait+0xcd/0xe0 [ 9396.437160] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9396.438851] do_raw_spin_lock+0x177/0x190 [ 9396.440348] _raw_spin_lock+0x21/0x30 [ 9396.441704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9396.443917] ? do_raw_spin_unlock+0x75/0x190 [ 9396.445661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9396.447914] ? do_wait_intr+0xf0/0xf0 [ 9396.449173] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9396.451467] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9396.453465] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9396.455960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9396.458066] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9396.460296] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9396.461769] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9396.463842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9396.465916] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9396.468635] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9396.470615] lmv_intent_open+0x338/0x1150 [lmv] [ 9396.472403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9396.474580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9396.476706] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9396.478200] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9396.480126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9396.482507] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9396.484108] ? do_raw_spin_unlock+0x75/0x190 [ 9396.485676] ? _raw_spin_unlock+0x12/0x30 [ 9396.486943] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9396.488998] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9396.491162] ll_atomic_open+0x36a/0xd70 [lustre] [ 9396.493012] lookup_open+0x356/0xb40 [ 9396.494516] do_last+0x313/0xfc0 [ 9396.495773] ? nd_jump_root+0xe5/0x160 [ 9396.497041] ? path_init+0x437/0x520 [ 9396.498615] path_openat+0xf7/0x500 [ 9396.499763] ? do_raw_spin_unlock+0x75/0x190 [ 9396.501509] do_filp_open+0x99/0x140 [ 9396.502810] ? getname_flags+0x6e/0x330 [ 9396.504349] ? __check_object_size+0xff/0x256 [ 9396.505965] ? do_raw_spin_unlock+0x75/0x190 [ 9396.507682] ? _raw_spin_unlock+0x12/0x30 [ 9396.509395] do_sys_openat2+0x2b4/0x410 [ 9396.510905] do_sys_open+0x73/0xa0 [ 9396.512316] __x64_sys_openat+0x24/0x30 [ 9396.514021] do_syscall_64+0xc1/0x440 [ 9396.515423] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9396.516829] RIP: 0033:0x7fa749552332 [ 9396.517817] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9396.524674] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9396.527181] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9396.530176] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9396.532759] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9396.535446] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9396.539040] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9401.833164] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7484s! [ 9401.839261] Showing busy workqueues and worker pools: [ 9401.842283] workqueue events: flags=0x0 [ 9401.848767] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9401.856500] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9401.873581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9401.881647] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9401.891446] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.897405] pending: kfree_rcu_monitor [ 9401.900130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9401.903363] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9401.907168] workqueue events_long: flags=0x0 [ 9401.908700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.911848] pending: lu_objects_flush [obdclass] [ 9401.914480] workqueue events_unbound: flags=0x2 [ 9401.916142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9401.918542] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9401.922042] workqueue events_power_efficient: flags=0x80 [ 9401.923801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.925905] pending: check_lifetime [ 9401.927468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.930090] pending: neigh_periodic_work [ 9401.931731] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.936699] pending: neigh_periodic_work [ 9401.939916] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.946274] pending: do_cache_clean [ 9401.948314] workqueue mm_percpu_wq: flags=0x8 [ 9401.950574] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.957157] pending: vmstat_update [ 9401.959936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.967540] pending: vmstat_update [ 9401.970269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9401.974393] pending: vmstat_update [ 9401.975580] workqueue writeback: flags=0x4a [ 9401.977819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9401.980072] pending: wb_workfn, wb_workfn [ 9402.005286] workqueue xprtiod: flags=0xa [ 9402.013015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9402.020218] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9402.028823] workqueue ipv6_addrconf: flags=0x40008 [ 9402.038174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9402.049255] pending: addrconf_verify_work [ 9402.057243] workqueue knbd0-recv: flags=0x1a [ 9402.069587] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9402.075789] in-flight: 651:recv_work [ 9402.079921] workqueue ptlrpc_pinger: flags=0x2 [ 9402.082737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9402.085593] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9402.094272] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7484s workers=4 idle: 37 31846 28897 [ 9402.101343] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9392s workers=2 idle: 653 [ 9408.059253] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9408.059985] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9408.070795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9408.070924] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9408.070932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9408.070938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.070959] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9408.070966] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9408.070971] RAX: 0000000000002a4a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9408.070974] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9408.070976] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9408.070978] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9408.070981] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9408.070984] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9408.070987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9408.070989] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9408.070997] Call Trace: [ 9408.071002] [ 9408.071008] ? show_regs.cold.9+0x22/0x2f [ 9408.071017] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9408.071024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9408.071029] ? watchdog+0x40/0x40 [ 9408.071033] ? hrtimer_interrupt+0x128/0x2c0 [ 9408.071037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9408.071046] ? apic_timer_interrupt+0xf/0x20 [ 9408.071051] [ 9408.071054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.071061] do_raw_spin_lock+0x177/0x190 [ 9408.071065] _raw_spin_lock+0x21/0x30 [ 9408.071070] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9408.071752] ? do_raw_spin_unlock+0x75/0x190 [ 9408.071759] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9408.072033] ? do_wait_intr+0xf0/0xf0 [ 9408.072041] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9408.072301] mdc_close+0x53b/0xd70 [mdc] [ 9408.072356] lmv_close+0x18d/0x490 [lmv] [ 9408.072397] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9408.072565] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9408.072738] ll_md_close+0x274/0x800 [lustre] [ 9408.072896] ll_file_release+0x128/0xc50 [lustre] [ 9408.072998] __fput+0x10d/0x440 [ 9408.073007] ____fput+0x12/0x20 [ 9408.073012] task_work_run+0xc6/0x110 [ 9408.073019] exit_to_usermode_loop+0x1dd/0x1f0 [ 9408.073027] do_syscall_64+0x3d6/0x440 [ 9408.073032] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9408.073039] RIP: 0033:0x7fa749552cd8 [ 9408.073047] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9408.073051] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9408.073056] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9408.073059] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9408.073061] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9408.073063] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9408.073066] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9408.381468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9408.416596] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9408.425728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9408.436557] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9408.442440] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9408.460851] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9408.468968] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9408.476163] RDX: 0000000000001d2f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9408.481613] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9408.484057] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9408.490264] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9408.497467] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9408.502132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9408.503953] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9408.506463] Call Trace: [ 9408.507085] [ 9408.507852] ? show_regs.cold.9+0x22/0x2f [ 9408.509387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9408.511706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9408.516700] ? watchdog+0x40/0x40 [ 9408.521487] ? hrtimer_interrupt+0x128/0x2c0 [ 9408.525663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9408.532733] ? apic_timer_interrupt+0xf/0x20 [ 9408.536892] [ 9408.537774] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9408.539781] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9408.545989] do_raw_spin_lock+0x177/0x190 [ 9408.549948] _raw_spin_lock+0x21/0x30 [ 9408.551574] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9408.557262] obd_mod_free+0x46/0x140 [ptlrpc] [ 9408.559779] mdc_commit_open+0x91/0xb0 [mdc] [ 9408.565407] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9408.568022] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9408.574845] after_reply+0xe2c/0x1780 [ptlrpc] [ 9408.584466] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9408.586181] ? __next_timer_interrupt+0x160/0x160 [ 9408.587817] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9408.592055] ? do_wait_intr+0xf0/0xf0 [ 9408.597688] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9408.599748] kthread+0x1d1/0x200 [ 9408.605189] ? set_kthread_struct+0x70/0x70 [ 9408.606804] ret_from_fork+0x1f/0x30 [ 9424.057032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9424.066920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9424.112542] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9424.125971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9424.139412] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9424.150362] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9424.162244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9424.183216] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9424.189292] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9424.195437] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9424.204227] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9424.209677] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9424.214989] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9424.224829] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9424.227950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9424.230790] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9424.234245] Call Trace: [ 9424.237888] [ 9424.241078] ? show_regs.cold.9+0x22/0x2f [ 9424.245554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9424.258429] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9424.271030] ? watchdog+0x40/0x40 [ 9424.274368] ? hrtimer_interrupt+0x128/0x2c0 [ 9424.277158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9424.280070] ? apic_timer_interrupt+0xf/0x20 [ 9424.285078] [ 9424.288089] ? native_safe_halt+0x2e/0x50 [ 9424.295524] kvm_wait+0xcd/0xe0 [ 9424.296657] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9424.303923] do_raw_spin_lock+0x177/0x190 [ 9424.312924] _raw_spin_lock+0x21/0x30 [ 9424.317034] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9424.322411] process_one_work+0x2c8/0x700 [ 9424.327884] worker_thread+0x296/0x6e0 [ 9424.330059] ? rescuer_thread+0x570/0x570 [ 9424.332623] kthread+0x1d1/0x200 [ 9424.334016] ? set_kthread_struct+0x70/0x70 [ 9424.338382] ret_from_fork+0x1f/0x30 [ 9428.061815] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9428.078308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9428.151292] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9428.167302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9428.183637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9428.190864] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9428.202865] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9428.210564] RAX: 0000000000007807 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9428.215204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9428.226192] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9428.236094] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9428.246410] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9428.254043] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9428.258941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9428.271465] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9428.280016] Call Trace: [ 9428.281049] [ 9428.282048] ? show_regs.cold.9+0x22/0x2f [ 9428.291064] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9428.303876] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9428.311989] ? watchdog+0x40/0x40 [ 9428.320819] ? hrtimer_interrupt+0x128/0x2c0 [ 9428.322314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9428.337584] ? apic_timer_interrupt+0xf/0x20 [ 9428.352772] [ 9428.356275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9428.370248] do_raw_spin_lock+0x177/0x190 [ 9428.371619] _raw_spin_lock+0x21/0x30 [ 9428.381625] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9428.393070] ? do_raw_spin_unlock+0x75/0x190 [ 9428.402365] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9428.413546] ? do_wait_intr+0xf0/0xf0 [ 9428.420379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9428.431254] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9428.444189] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9428.453181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9428.463561] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9428.470178] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9428.476978] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9428.489153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9428.498020] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9428.506368] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9428.512351] lmv_intent_open+0x338/0x1150 [lmv] [ 9428.520885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9428.529897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9428.538397] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9428.544306] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9428.547435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9428.554256] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9428.557994] ? do_raw_spin_unlock+0x75/0x190 [ 9428.559645] ? _raw_spin_unlock+0x12/0x30 [ 9428.564642] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9428.575867] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9428.580430] ll_atomic_open+0x36a/0xd70 [lustre] [ 9428.590422] lookup_open+0x356/0xb40 [ 9428.593372] do_last+0x313/0xfc0 [ 9428.595438] ? nd_jump_root+0xe5/0x160 [ 9428.599984] ? path_init+0x437/0x520 [ 9428.609220] path_openat+0xf7/0x500 [ 9428.611942] ? do_raw_spin_unlock+0x75/0x190 [ 9428.625596] do_filp_open+0x99/0x140 [ 9428.627392] ? getname_flags+0x6e/0x330 [ 9428.637519] ? __check_object_size+0xff/0x256 [ 9428.641181] ? do_raw_spin_unlock+0x75/0x190 [ 9428.646482] ? _raw_spin_unlock+0x12/0x30 [ 9428.651191] do_sys_openat2+0x2b4/0x410 [ 9428.654288] do_sys_open+0x73/0xa0 [ 9428.659077] __x64_sys_openat+0x24/0x30 [ 9428.663219] do_syscall_64+0xc1/0x440 [ 9428.669161] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9428.676143] RIP: 0033:0x7fa749552332 [ 9428.678206] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9428.696905] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9428.702919] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9428.713455] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9428.717054] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9428.723830] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9428.726482] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9432.552037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7515s! [ 9432.555713] Showing busy workqueues and worker pools: [ 9432.557587] workqueue events: flags=0x0 [ 9432.559051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9432.561795] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9432.566194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9432.568959] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9432.572833] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.575857] pending: kfree_rcu_monitor [ 9432.577392] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9432.580191] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9432.585170] workqueue events_long: flags=0x0 [ 9432.587044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.590146] pending: lu_objects_flush [obdclass] [ 9432.592799] workqueue events_unbound: flags=0x2 [ 9432.595385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9432.601494] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9432.606632] workqueue events_power_efficient: flags=0x80 [ 9432.609172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.611948] pending: check_lifetime [ 9432.613635] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.616625] pending: neigh_periodic_work [ 9432.618370] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.621343] pending: neigh_periodic_work [ 9432.624046] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.629342] pending: do_cache_clean [ 9432.631009] workqueue mm_percpu_wq: flags=0x8 [ 9432.632915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.636000] pending: vmstat_update [ 9432.637417] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.640317] pending: vmstat_update [ 9432.642112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9432.647272] pending: vmstat_update [ 9432.649050] workqueue writeback: flags=0x4a [ 9432.652041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9432.654771] pending: wb_workfn, wb_workfn [ 9432.656694] workqueue xprtiod: flags=0xa [ 9432.658305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9432.661007] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9432.664283] workqueue ipv6_addrconf: flags=0x40008 [ 9432.666178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9432.668599] pending: addrconf_verify_work [ 9432.674325] workqueue knbd0-recv: flags=0x1a [ 9432.676984] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9432.680492] in-flight: 651:recv_work [ 9432.683781] workqueue ptlrpc_pinger: flags=0x2 [ 9432.685093] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9432.687470] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9432.690382] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7515s workers=4 idle: 37 31846 28897 [ 9432.694880] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9422s workers=2 idle: 653 [ 9436.059885] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9436.064994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9436.065494] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9436.104993] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9436.105009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9436.105012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9436.105025] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9436.105029] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9436.105036] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9436.105038] RDX: 000000000000349a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9436.105041] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9436.105042] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9436.105045] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9436.105047] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9436.105050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9436.105052] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9436.105060] Call Trace: [ 9436.105066] [ 9436.105072] ? show_regs.cold.9+0x22/0x2f [ 9436.105082] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9436.105088] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9436.105093] ? watchdog+0x40/0x40 [ 9436.105097] ? hrtimer_interrupt+0x128/0x2c0 [ 9436.105101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9436.105109] ? apic_timer_interrupt+0xf/0x20 [ 9436.105114] [ 9436.105117] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9436.105121] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9436.105126] do_raw_spin_lock+0x177/0x190 [ 9436.105130] _raw_spin_lock+0x21/0x30 [ 9436.105137] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9436.105413] obd_mod_free+0x46/0x140 [ptlrpc] [ 9436.111332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9436.111697] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9436.111705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9436.111708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9436.111722] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9436.111727] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9436.111732] RAX: 0000000000002e1c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9436.111734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9436.111737] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9436.111739] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9436.111741] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9436.111745] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9436.111748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9436.111750] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9436.111759] Call Trace: [ 9436.111765] [ 9436.111771] ? show_regs.cold.9+0x22/0x2f [ 9436.111781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9436.111789] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9436.111794] ? watchdog+0x40/0x40 [ 9436.111798] ? hrtimer_interrupt+0x128/0x2c0 [ 9436.111802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9436.111811] ? apic_timer_interrupt+0xf/0x20 [ 9436.111816] [ 9436.111819] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9436.111825] do_raw_spin_lock+0x177/0x190 [ 9436.111831] _raw_spin_lock+0x21/0x30 [ 9436.111837] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9436.112196] ? do_raw_spin_unlock+0x75/0x190 [ 9436.112201] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9436.112600] ? do_wait_intr+0xf0/0xf0 [ 9436.112608] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9436.112835] mdc_close+0x53b/0xd70 [mdc] [ 9436.112939] lmv_close+0x18d/0x490 [lmv] [ 9436.112984] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9436.113159] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9436.113255] ll_md_close+0x274/0x800 [lustre] [ 9436.113339] ll_file_release+0x128/0xc50 [lustre] [ 9436.113734] __fput+0x10d/0x440 [ 9436.113744] ____fput+0x12/0x20 [ 9436.113748] task_work_run+0xc6/0x110 [ 9436.113755] exit_to_usermode_loop+0x1dd/0x1f0 [ 9436.113762] do_syscall_64+0x3d6/0x440 [ 9436.113766] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9436.113772] RIP: 0033:0x7fa749552cd8 [ 9436.113780] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9436.113784] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9436.113789] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9436.113791] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9436.113794] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9436.113796] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9436.113798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9436.674677] mdc_commit_open+0x91/0xb0 [mdc] [ 9436.676618] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9436.677951] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9436.679415] after_reply+0xe2c/0x1780 [ptlrpc] [ 9436.688096] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9436.690591] ? __next_timer_interrupt+0x160/0x160 [ 9436.692713] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9436.694622] ? do_wait_intr+0xf0/0xf0 [ 9436.696456] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9436.700692] kthread+0x1d1/0x200 [ 9436.702051] ? set_kthread_struct+0x70/0x70 [ 9436.704215] ret_from_fork+0x1f/0x30 [ 9452.074378] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9452.089328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9452.132616] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9452.147543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9452.159898] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9452.168331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9452.172295] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9452.189309] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9452.196652] RAX: 0000000000007686 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9452.203842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9452.212054] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9452.216326] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9452.222629] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9452.228230] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9452.235902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9452.240567] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9452.247222] Call Trace: [ 9452.250649] [ 9452.251491] ? show_regs.cold.9+0x22/0x2f [ 9452.255824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9452.259732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9452.265954] ? watchdog+0x40/0x40 [ 9452.267246] ? hrtimer_interrupt+0x128/0x2c0 [ 9452.271275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9452.274875] ? apic_timer_interrupt+0xf/0x20 [ 9452.278611] [ 9452.279799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9452.281766] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9452.283663] do_raw_spin_lock+0x177/0x190 [ 9452.284941] _raw_spin_lock+0x21/0x30 [ 9452.286502] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9452.288652] process_one_work+0x2c8/0x700 [ 9452.290060] worker_thread+0x296/0x6e0 [ 9452.291536] ? rescuer_thread+0x570/0x570 [ 9452.292800] kthread+0x1d1/0x200 [ 9452.294022] ? set_kthread_struct+0x70/0x70 [ 9452.295636] ret_from_fork+0x1f/0x30 [ 9460.056150] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9460.058633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9460.090585] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9460.097691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9460.103879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9460.109394] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9460.123006] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9460.125763] RAX: 0000000000007693 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9460.134545] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9460.139975] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9460.142830] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9460.149330] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9460.154950] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9460.157084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9460.162754] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9460.168208] Call Trace: [ 9460.169655] [ 9460.170344] ? show_regs.cold.9+0x22/0x2f [ 9460.174859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9460.180143] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9460.181484] ? watchdog+0x40/0x40 [ 9460.182707] ? hrtimer_interrupt+0x128/0x2c0 [ 9460.185958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9460.187279] ? apic_timer_interrupt+0xf/0x20 [ 9460.190479] [ 9460.191153] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9460.192922] do_raw_spin_lock+0x177/0x190 [ 9460.198472] _raw_spin_lock+0x21/0x30 [ 9460.201947] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9460.206045] ? do_raw_spin_unlock+0x75/0x190 [ 9460.209565] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9460.212444] ? do_wait_intr+0xf0/0xf0 [ 9460.216250] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9460.218214] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9460.220118] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9460.226032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.227733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9460.233306] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9460.234779] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9460.242929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.244551] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9460.249925] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9460.253531] lmv_intent_open+0x338/0x1150 [lmv] [ 9460.256802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.260047] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.265073] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9460.266523] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9460.271856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.273455] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9460.278342] ? do_raw_spin_unlock+0x75/0x190 [ 9460.279542] ? _raw_spin_unlock+0x12/0x30 [ 9460.284715] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9460.286108] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9460.288320] ll_atomic_open+0x36a/0xd70 [lustre] [ 9460.293590] lookup_open+0x356/0xb40 [ 9460.298342] do_last+0x313/0xfc0 [ 9460.299213] ? nd_jump_root+0xe5/0x160 [ 9460.300994] ? path_init+0x437/0x520 [ 9460.304971] path_openat+0xf7/0x500 [ 9460.306307] ? do_raw_spin_unlock+0x75/0x190 [ 9460.309747] do_filp_open+0x99/0x140 [ 9460.311698] ? getname_flags+0x6e/0x330 [ 9460.315225] ? __check_object_size+0xff/0x256 [ 9460.317908] ? do_raw_spin_unlock+0x75/0x190 [ 9460.323181] ? _raw_spin_unlock+0x12/0x30 [ 9460.327911] do_sys_openat2+0x2b4/0x410 [ 9460.330832] do_sys_open+0x73/0xa0 [ 9460.334362] __x64_sys_openat+0x24/0x30 [ 9460.336655] do_syscall_64+0xc1/0x440 [ 9460.338591] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9460.342195] RIP: 0033:0x7fa749552332 [ 9460.343252] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9460.354654] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9460.364039] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9460.366559] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9460.376669] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9460.382032] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9460.387850] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9462.753083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7545s! [ 9462.755784] Showing busy workqueues and worker pools: [ 9462.758867] workqueue events: flags=0x0 [ 9462.760781] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9462.762718] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9462.766789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9462.769752] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9462.772754] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.775548] pending: kfree_rcu_monitor [ 9462.776760] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9462.781189] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9462.786756] workqueue events_long: flags=0x0 [ 9462.788773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.790803] pending: lu_objects_flush [obdclass] [ 9462.792620] workqueue events_unbound: flags=0x2 [ 9462.794405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9462.797165] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9462.800101] workqueue events_power_efficient: flags=0x80 [ 9462.801770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.804497] pending: check_lifetime [ 9462.807018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.809788] pending: neigh_periodic_work [ 9462.811498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.813602] pending: neigh_periodic_work [ 9462.815718] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.818825] pending: do_cache_clean [ 9462.820741] workqueue mm_percpu_wq: flags=0x8 [ 9462.822037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.824454] pending: vmstat_update [ 9462.830686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.839129] pending: vmstat_update [ 9462.844367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9462.852205] pending: vmstat_update [ 9462.855845] workqueue writeback: flags=0x4a [ 9462.856968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9462.863750] pending: wb_workfn, wb_workfn [ 9462.867864] workqueue xprtiod: flags=0xa [ 9462.871196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9462.875436] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9462.880488] workqueue ipv6_addrconf: flags=0x40008 [ 9462.884181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9462.888192] pending: addrconf_verify_work [ 9462.891099] workqueue knbd0-recv: flags=0x1a [ 9462.894559] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9462.898079] in-flight: 651:recv_work [ 9462.900684] workqueue ptlrpc_pinger: flags=0x2 [ 9462.903790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9462.906598] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9462.910565] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7545s workers=4 idle: 37 31846 28897 [ 9462.914360] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9452s workers=2 idle: 653 [ 9464.059019] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9464.064038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9464.065058] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9464.093174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9464.095728] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9464.143821] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9464.143835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9464.143838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9464.143853] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9464.143857] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9464.143872] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9464.143875] RDX: 0000000000005254 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9464.143877] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9464.143879] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9464.143881] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9464.143885] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9464.143888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9464.143891] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9464.154104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9464.154116] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9464.154133] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9464.154138] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9464.154144] RAX: 0000000000000094 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9464.154157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9464.154160] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9464.154162] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9464.154165] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9464.154168] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9464.154171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9464.154174] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9464.154181] Call Trace: [ 9464.154187] [ 9464.154193] ? show_regs.cold.9+0x22/0x2f [ 9464.154203] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9464.154209] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9464.154213] ? watchdog+0x40/0x40 [ 9464.154217] ? hrtimer_interrupt+0x128/0x2c0 [ 9464.154221] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9464.154229] ? apic_timer_interrupt+0xf/0x20 [ 9464.154234] [ 9464.154237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9464.154244] do_raw_spin_lock+0x177/0x190 [ 9464.154249] _raw_spin_lock+0x21/0x30 [ 9464.154254] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9464.154539] ? do_raw_spin_unlock+0x75/0x190 [ 9464.154545] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9464.154743] ? do_wait_intr+0xf0/0xf0 [ 9464.154751] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9464.168209] Call Trace: [ 9464.168221] [ 9464.168226] ? show_regs.cold.9+0x22/0x2f [ 9464.168240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9464.168246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9464.168250] ? watchdog+0x40/0x40 [ 9464.168254] ? hrtimer_interrupt+0x128/0x2c0 [ 9464.168258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9464.168265] ? apic_timer_interrupt+0xf/0x20 [ 9464.168270] [ 9464.168272] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9464.168278] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9464.168283] do_raw_spin_lock+0x177/0x190 [ 9464.168287] _raw_spin_lock+0x21/0x30 [ 9464.168294] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9464.169757] obd_mod_free+0x46/0x140 [ptlrpc] [ 9464.170104] mdc_commit_open+0x91/0xb0 [mdc] [ 9464.170155] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9464.170692] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9464.171004] after_reply+0xe2c/0x1780 [ptlrpc] [ 9464.171240] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9464.175024] ? __next_timer_interrupt+0x160/0x160 [ 9464.175035] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9464.175251] ? do_wait_intr+0xf0/0xf0 [ 9464.175261] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9464.183625] mdc_close+0x53b/0xd70 [mdc] [ 9464.183677] lmv_close+0x18d/0x490 [lmv] [ 9464.183706] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9464.183824] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9464.183919] ll_md_close+0x274/0x800 [lustre] [ 9464.184005] ll_file_release+0x128/0xc50 [lustre] [ 9464.184095] __fput+0x10d/0x440 [ 9464.184106] ____fput+0x12/0x20 [ 9464.184111] task_work_run+0xc6/0x110 [ 9464.184119] exit_to_usermode_loop+0x1dd/0x1f0 [ 9464.184128] do_syscall_64+0x3d6/0x440 [ 9464.184133] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9464.184142] RIP: 0033:0x7fa749552cd8 [ 9464.184159] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9464.184163] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9464.184169] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9464.184172] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9464.184174] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9464.184176] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9464.184179] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9464.700701] kthread+0x1d1/0x200 [ 9464.701875] ? set_kthread_struct+0x70/0x70 [ 9464.716825] ret_from_fork+0x1f/0x30 [ 9480.057055] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9480.062557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9480.104560] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9480.115554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9480.127479] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9480.131951] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9480.135359] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9480.152071] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9480.154758] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9480.157055] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9480.159734] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9480.176544] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9480.180888] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9480.189202] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9480.195224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9480.198098] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9480.201421] Call Trace: [ 9480.202960] [ 9480.203923] ? show_regs.cold.9+0x22/0x2f [ 9480.205336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9480.207547] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9480.209072] ? watchdog+0x40/0x40 [ 9480.210297] ? hrtimer_interrupt+0x128/0x2c0 [ 9480.212431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9480.214387] ? apic_timer_interrupt+0xf/0x20 [ 9480.219984] [ 9480.221947] ? native_safe_halt+0x2e/0x50 [ 9480.227068] kvm_wait+0xcd/0xe0 [ 9480.228822] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9480.232556] do_raw_spin_lock+0x177/0x190 [ 9480.236436] _raw_spin_lock+0x21/0x30 [ 9480.239937] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9480.242808] process_one_work+0x2c8/0x700 [ 9480.249880] worker_thread+0x296/0x6e0 [ 9480.251528] ? rescuer_thread+0x570/0x570 [ 9480.253380] kthread+0x1d1/0x200 [ 9480.254892] ? set_kthread_struct+0x70/0x70 [ 9480.259117] ret_from_fork+0x1f/0x30 [ 9488.054022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9488.056423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9488.086794] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9488.097743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9488.101840] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9488.103964] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9488.116178] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9488.119398] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9488.122525] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9488.128709] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9488.132523] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9488.136081] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9488.146203] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9488.157625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9488.164457] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9488.168347] Call Trace: [ 9488.169564] [ 9488.171605] ? show_regs.cold.9+0x22/0x2f [ 9488.174810] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9488.176296] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9488.183750] ? watchdog+0x40/0x40 [ 9488.185762] ? hrtimer_interrupt+0x128/0x2c0 [ 9488.187552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9488.193652] ? apic_timer_interrupt+0xf/0x20 [ 9488.195265] [ 9488.196401] ? native_safe_halt+0x2e/0x50 [ 9488.199635] kvm_wait+0xcd/0xe0 [ 9488.201771] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9488.207931] do_raw_spin_lock+0x177/0x190 [ 9488.212366] _raw_spin_lock+0x21/0x30 [ 9488.213941] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9488.220153] ? do_raw_spin_unlock+0x75/0x190 [ 9488.224671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9488.226909] ? do_wait_intr+0xf0/0xf0 [ 9488.233400] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9488.237936] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9488.246901] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9488.254926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9488.258805] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9488.260950] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9488.270603] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9488.275494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9488.285296] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9488.292114] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9488.296397] lmv_intent_open+0x338/0x1150 [lmv] [ 9488.304544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9488.316331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9488.324786] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9488.326473] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9488.329030] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9488.331040] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9488.333488] ? do_raw_spin_unlock+0x75/0x190 [ 9488.335436] ? _raw_spin_unlock+0x12/0x30 [ 9488.337444] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9488.340074] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9488.342693] ll_atomic_open+0x36a/0xd70 [lustre] [ 9488.345084] lookup_open+0x356/0xb40 [ 9488.347105] do_last+0x313/0xfc0 [ 9488.348258] ? nd_jump_root+0xe5/0x160 [ 9488.349579] ? path_init+0x437/0x520 [ 9488.356030] path_openat+0xf7/0x500 [ 9488.358542] ? do_raw_spin_unlock+0x75/0x190 [ 9488.359998] do_filp_open+0x99/0x140 [ 9488.362936] ? getname_flags+0x6e/0x330 [ 9488.365925] ? __check_object_size+0xff/0x256 [ 9488.369549] ? do_raw_spin_unlock+0x75/0x190 [ 9488.371859] ? _raw_spin_unlock+0x12/0x30 [ 9488.374003] do_sys_openat2+0x2b4/0x410 [ 9488.377064] do_sys_open+0x73/0xa0 [ 9488.379513] __x64_sys_openat+0x24/0x30 [ 9488.388975] do_syscall_64+0xc1/0x440 [ 9488.390124] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9488.401816] RIP: 0033:0x7fa749552332 [ 9488.403332] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9488.427910] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9488.433115] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9488.441786] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9488.446098] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9488.448202] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9488.461078] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9492.058101] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9492.059762] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9492.063180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9492.063256] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9492.063264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9492.063267] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9492.063279] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9492.063284] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9492.063288] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9492.063290] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9492.063292] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9492.063294] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9492.063296] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9492.063299] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9492.063303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9492.063305] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9492.063314] Call Trace: [ 9492.063324] [ 9492.063330] ? show_regs.cold.9+0x22/0x2f [ 9492.063344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9492.063350] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9492.063354] ? watchdog+0x40/0x40 [ 9492.063359] ? hrtimer_interrupt+0x128/0x2c0 [ 9492.063363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9492.063380] ? apic_timer_interrupt+0xf/0x20 [ 9492.063385] [ 9492.063387] ? native_safe_halt+0x2e/0x50 [ 9492.063391] kvm_wait+0xcd/0xe0 [ 9492.063398] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9492.063406] do_raw_spin_lock+0x177/0x190 [ 9492.063410] _raw_spin_lock+0x21/0x30 [ 9492.063415] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9492.069181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9492.069255] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9492.069261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9492.069264] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9492.069279] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9492.069283] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9492.069288] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9492.069290] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9492.069292] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9492.069294] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9492.069297] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9492.069300] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9492.069304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9492.069306] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9492.069314] Call Trace: [ 9492.069323] [ 9492.069329] ? show_regs.cold.9+0x22/0x2f [ 9492.069338] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9492.069345] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9492.069349] ? watchdog+0x40/0x40 [ 9492.069353] ? hrtimer_interrupt+0x128/0x2c0 [ 9492.069356] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9492.069363] ? apic_timer_interrupt+0xf/0x20 [ 9492.069367] [ 9492.069426] ? native_safe_halt+0x2e/0x50 [ 9492.069432] kvm_wait+0xcd/0xe0 [ 9492.069440] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9492.069448] do_raw_spin_lock+0x177/0x190 [ 9492.069452] _raw_spin_lock+0x21/0x30 [ 9492.069456] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9492.069963] ? do_raw_spin_unlock+0x75/0x190 [ 9492.069969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9492.070174] ? do_wait_intr+0xf0/0xf0 [ 9492.070182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9492.070395] mdc_close+0x53b/0xd70 [mdc] [ 9492.070446] lmv_close+0x18d/0x490 [lmv] [ 9492.070477] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9492.070801] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9492.070884] ll_md_close+0x274/0x800 [lustre] [ 9492.070967] ll_file_release+0x128/0xc50 [lustre] [ 9492.093634] obd_mod_free+0x46/0x140 [ptlrpc] [ 9492.098490] __fput+0x10d/0x440 [ 9492.098511] ____fput+0x12/0x20 [ 9492.098516] task_work_run+0xc6/0x110 [ 9492.098524] exit_to_usermode_loop+0x1dd/0x1f0 [ 9492.098532] do_syscall_64+0x3d6/0x440 [ 9492.098537] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9492.098546] RIP: 0033:0x7fa749552cd8 [ 9492.098554] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9492.098559] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9492.098564] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9492.098632] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9492.098636] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9492.098638] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9492.098640] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9492.330882] mdc_commit_open+0x91/0xb0 [mdc] [ 9492.333184] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9492.335471] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9492.337618] after_reply+0xe2c/0x1780 [ptlrpc] [ 9492.339840] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9492.341913] ? __next_timer_interrupt+0x160/0x160 [ 9492.343785] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9492.345652] ? do_wait_intr+0xf0/0xf0 [ 9492.347148] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9492.349131] kthread+0x1d1/0x200 [ 9492.350136] ? set_kthread_struct+0x70/0x70 [ 9492.351310] ret_from_fork+0x1f/0x30 [ 9492.961025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7575s! [ 9492.963694] Showing busy workqueues and worker pools: [ 9492.965985] workqueue events: flags=0x0 [ 9492.967480] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9492.970502] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9492.976824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9492.979284] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9492.983019] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9492.985944] pending: kfree_rcu_monitor [ 9492.987587] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9492.990903] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9492.994218] workqueue events_long: flags=0x0 [ 9492.995435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9492.998146] pending: lu_objects_flush [obdclass] [ 9493.000360] workqueue events_unbound: flags=0x2 [ 9493.001786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9493.005324] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9493.010884] workqueue events_power_efficient: flags=0x80 [ 9493.013114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9493.015834] pending: check_lifetime [ 9493.017565] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9493.021562] pending: neigh_periodic_work [ 9493.023421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9493.026446] pending: neigh_periodic_work [ 9493.028164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9493.031735] pending: do_cache_clean [ 9493.033477] workqueue mm_percpu_wq: flags=0x8 [ 9493.035620] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9493.038475] pending: vmstat_update [ 9493.039943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9493.042670] pending: vmstat_update [ 9493.043858] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9493.046787] pending: vmstat_update [ 9493.048960] workqueue writeback: flags=0x4a [ 9493.050858] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9493.054375] pending: wb_workfn, wb_workfn [ 9493.056710] workqueue xprtiod: flags=0xa [ 9493.058787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9493.061929] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9493.066938] workqueue ipv6_addrconf: flags=0x40008 [ 9493.069063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9493.071927] pending: addrconf_verify_work [ 9493.073524] workqueue knbd0-recv: flags=0x1a [ 9493.075526] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9493.078799] in-flight: 651:recv_work [ 9493.080233] workqueue ptlrpc_pinger: flags=0x2 [ 9493.082013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9493.084208] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9493.086699] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7575s workers=4 idle: 37 31846 28897 [ 9493.089916] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9482s workers=2 idle: 653 [ 9508.057015] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9508.061840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9508.084604] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9508.089606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9508.093282] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9508.096622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9508.101508] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9508.113254] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9508.118479] RAX: 0000000000001441 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9508.125582] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9508.135212] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9508.142351] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9508.149293] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9508.156769] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9508.164721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9508.168409] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9508.172579] Call Trace: [ 9508.173333] [ 9508.173991] ? show_regs.cold.9+0x22/0x2f [ 9508.175305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9508.182605] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9508.185045] ? watchdog+0x40/0x40 [ 9508.192988] ? hrtimer_interrupt+0x128/0x2c0 [ 9508.195980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9508.200331] ? apic_timer_interrupt+0xf/0x20 [ 9508.201872] [ 9508.203567] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9508.206059] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9508.211826] do_raw_spin_lock+0x177/0x190 [ 9508.215735] _raw_spin_lock+0x21/0x30 [ 9508.217434] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9508.222765] process_one_work+0x2c8/0x700 [ 9508.227903] worker_thread+0x296/0x6e0 [ 9508.231703] ? rescuer_thread+0x570/0x570 [ 9508.237446] kthread+0x1d1/0x200 [ 9508.239713] ? set_kthread_struct+0x70/0x70 [ 9508.245039] ret_from_fork+0x1f/0x30 [ 9520.055867] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9520.058156] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9520.059074] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9520.059083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9520.059251] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9520.059265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9520.059268] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.059282] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9520.059286] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9520.059290] RAX: 0000000000005847 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9520.059293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9520.059295] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9520.059297] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9520.059299] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9520.059303] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9520.059305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9520.059312] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9520.059319] Call Trace: [ 9520.059324] [ 9520.059330] ? show_regs.cold.9+0x22/0x2f [ 9520.059345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9520.059351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9520.059356] ? watchdog+0x40/0x40 [ 9520.059361] ? hrtimer_interrupt+0x128/0x2c0 [ 9520.059368] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9520.059377] ? apic_timer_interrupt+0xf/0x20 [ 9520.059381] [ 9520.059385] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.059392] do_raw_spin_lock+0x177/0x190 [ 9520.059397] _raw_spin_lock+0x21/0x30 [ 9520.059402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9520.060115] ? do_raw_spin_unlock+0x75/0x190 [ 9520.060128] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9520.060440] ? do_wait_intr+0xf0/0xf0 [ 9520.060449] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9520.060831] mdc_close+0x53b/0xd70 [mdc] [ 9520.060888] lmv_close+0x18d/0x490 [lmv] [ 9520.060925] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9520.061183] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9520.061475] ll_md_close+0x274/0x800 [lustre] [ 9520.061564] ll_file_release+0x128/0xc50 [lustre] [ 9520.061831] __fput+0x10d/0x440 [ 9520.061851] ____fput+0x12/0x20 [ 9520.061860] task_work_run+0xc6/0x110 [ 9520.061873] exit_to_usermode_loop+0x1dd/0x1f0 [ 9520.061881] do_syscall_64+0x3d6/0x440 [ 9520.061885] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9520.061898] RIP: 0033:0x7fa749552cd8 [ 9520.061954] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9520.061960] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9520.061966] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9520.061972] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9520.061974] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9520.061976] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9520.061978] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9520.065332] Modules linked in: [ 9520.067789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9520.067862] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9520.067870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9520.067873] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9520.067885] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9520.067889] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9520.067894] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9520.067897] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9520.067899] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9520.067901] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9520.067903] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9520.067907] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9520.067909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9520.067912] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9520.067920] Call Trace: [ 9520.067925] [ 9520.067931] ? show_regs.cold.9+0x22/0x2f [ 9520.067941] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9520.067947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9520.067950] ? watchdog+0x40/0x40 [ 9520.067954] ? hrtimer_interrupt+0x128/0x2c0 [ 9520.067957] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9520.067965] ? apic_timer_interrupt+0xf/0x20 [ 9520.070679] lustre(O) [ 9520.085063] [ 9520.085075] ? native_safe_halt+0x2e/0x50 [ 9520.085086] kvm_wait+0xcd/0xe0 [ 9520.085094] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9520.085101] do_raw_spin_lock+0x177/0x190 [ 9520.085105] _raw_spin_lock+0x21/0x30 [ 9520.085111] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9520.093463] osp(O) ofd(O) [ 9520.096311] obd_mod_free+0x46/0x140 [ptlrpc] [ 9520.096718] mdc_commit_open+0x91/0xb0 [mdc] [ 9520.096756] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9520.096954] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9520.097158] after_reply+0xe2c/0x1780 [ptlrpc] [ 9520.099796] lod(O) mdt(O) mdd(O) mgs(O) [ 9520.105315] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9520.109738] lquota(O) [ 9520.111911] ? __next_timer_interrupt+0x160/0x160 [ 9520.111930] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9520.122829] lfsck(O) [ 9520.125150] ? do_wait_intr+0xf0/0xf0 [ 9520.125181] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9520.125593] kthread+0x1d1/0x200 [ 9520.125600] ? set_kthread_struct+0x70/0x70 [ 9520.125603] ret_from_fork+0x1f/0x30 [ 9520.588582] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9520.613740] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9520.621482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9520.625796] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.630316] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9520.638318] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9520.645000] RAX: 0000000000000105 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9520.647329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9520.651198] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9520.654555] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9520.657580] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9520.665748] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9520.670772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9520.679215] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9520.685507] Call Trace: [ 9520.688624] [ 9520.689421] ? show_regs.cold.9+0x22/0x2f [ 9520.691000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9520.694642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9520.698168] ? watchdog+0x40/0x40 [ 9520.702967] ? hrtimer_interrupt+0x128/0x2c0 [ 9520.705178] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9520.713944] ? apic_timer_interrupt+0xf/0x20 [ 9520.718881] [ 9520.719700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.724007] do_raw_spin_lock+0x177/0x190 [ 9520.725225] _raw_spin_lock+0x21/0x30 [ 9520.728934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9520.730750] ? do_raw_spin_unlock+0x75/0x190 [ 9520.731979] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9520.736687] ? do_wait_intr+0xf0/0xf0 [ 9520.741799] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9520.749379] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9520.754251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9520.760396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.763595] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9520.772445] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9520.777458] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9520.784639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.793120] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9520.803898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9520.807375] lmv_intent_open+0x338/0x1150 [lmv] [ 9520.809046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.816020] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.825602] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9520.832741] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9520.840537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.845655] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9520.852609] ? do_raw_spin_unlock+0x75/0x190 [ 9520.859565] ? _raw_spin_unlock+0x12/0x30 [ 9520.864036] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9520.865391] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9520.866639] ll_atomic_open+0x36a/0xd70 [lustre] [ 9520.877826] lookup_open+0x356/0xb40 [ 9520.879122] do_last+0x313/0xfc0 [ 9520.881738] ? nd_jump_root+0xe5/0x160 [ 9520.890813] ? path_init+0x437/0x520 [ 9520.898601] path_openat+0xf7/0x500 [ 9520.902773] ? do_raw_spin_unlock+0x75/0x190 [ 9520.908714] do_filp_open+0x99/0x140 [ 9520.913579] ? getname_flags+0x6e/0x330 [ 9520.918379] ? __check_object_size+0xff/0x256 [ 9520.922133] ? do_raw_spin_unlock+0x75/0x190 [ 9520.924679] ? _raw_spin_unlock+0x12/0x30 [ 9520.927050] do_sys_openat2+0x2b4/0x410 [ 9520.929179] do_sys_open+0x73/0xa0 [ 9520.931185] __x64_sys_openat+0x24/0x30 [ 9520.932648] do_syscall_64+0xc1/0x440 [ 9520.935179] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9520.938736] RIP: 0033:0x7fa749552332 [ 9520.940506] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9520.952265] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9520.958551] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9520.967121] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9520.975137] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9520.981726] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9520.986040] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9523.168758] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7606s! [ 9523.181288] Showing busy workqueues and worker pools: [ 9523.183145] workqueue events: flags=0x0 [ 9523.188812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9523.198996] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9523.206360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9523.208794] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9523.213798] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.217368] pending: kfree_rcu_monitor [ 9523.218703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9523.221420] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9523.226365] workqueue events_long: flags=0x0 [ 9523.228128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.230515] pending: lu_objects_flush [obdclass] [ 9523.232652] workqueue events_unbound: flags=0x2 [ 9523.235170] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9523.237715] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9523.241905] workqueue events_power_efficient: flags=0x80 [ 9523.243929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.246699] pending: check_lifetime [ 9523.248109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.250535] pending: neigh_periodic_work [ 9523.252364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.254949] pending: neigh_periodic_work [ 9523.258029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.260731] pending: do_cache_clean [ 9523.262491] workqueue mm_percpu_wq: flags=0x8 [ 9523.263966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.267615] pending: vmstat_update [ 9523.269778] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.272768] pending: vmstat_update [ 9523.274033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9523.277404] pending: vmstat_update [ 9523.279328] workqueue writeback: flags=0x4a [ 9523.281099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9523.284180] pending: wb_workfn, wb_workfn [ 9523.286309] workqueue xprtiod: flags=0xa [ 9523.287750] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9523.290484] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9523.293807] workqueue ipv6_addrconf: flags=0x40008 [ 9523.296323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9523.299323] pending: addrconf_verify_work [ 9523.301338] workqueue knbd0-recv: flags=0x1a [ 9523.303175] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9523.306325] in-flight: 651:recv_work [ 9523.308208] workqueue ptlrpc_pinger: flags=0x2 [ 9523.310183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9523.312782] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9523.315512] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7606s workers=4 idle: 37 31846 28897 [ 9523.318923] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9513s workers=2 idle: 653 [ 9533.856188] rcu: INFO: rcu_sched self-detected stall on CPU [ 9533.864750] rcu: 2-...!: (4305689 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 9533.876391] (t=7620167 jiffies g=258701 q=3632) [ 9533.885025] rcu: rcu_sched kthread starved for 7616962 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9533.898138] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9533.905507] rcu: RCU grace-period kthread stack dump: [ 9533.908636] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9533.917392] Call Trace: [ 9533.921786] __schedule+0x351/0xcb0 [ 9533.923093] schedule+0xc0/0x180 [ 9533.924190] schedule_timeout+0xb4/0x190 [ 9533.930417] ? do_raw_spin_unlock+0x75/0x190 [ 9533.932183] ? __next_timer_interrupt+0x160/0x160 [ 9533.944612] ? prepare_to_swait_event+0xf3/0x140 [ 9533.949618] rcu_gp_kthread+0x8dd/0x15b0 [ 9533.951508] ? force_qs_rnp+0x2c0/0x2c0 [ 9533.957243] kthread+0x1d1/0x200 [ 9533.961180] ? set_kthread_struct+0x70/0x70 [ 9533.966156] ret_from_fork+0x1f/0x30 [ 9533.969112] rcu: Stack dump where RCU GP kthread last ran: [ 9533.971064] Sending NMI from CPU 2 to CPUs 3: [ 9533.979404] NMI backtrace for cpu 3 [ 9533.979412] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9533.979414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9533.979416] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9533.979417] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9533.979419] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 [ 9533.979422] RAX: 0000000000007d88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9533.979424] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9533.979425] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9533.979427] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9533.979428] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9533.979429] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9533.979431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9533.979432] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9533.979433] Call Trace: [ 9533.979434] [ 9533.979435] ? show_regs.cold.9+0x22/0x2f [ 9533.979436] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9533.979437] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9533.979438] ? nmi_handle+0xbd/0x210 [ 9533.979439] ? default_do_nmi+0x4b/0x1b0 [ 9533.979440] ? do_nmi+0x280/0x380 [ 9533.979441] ? end_repeat_nmi+0x16/0x1f [ 9533.979442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9533.979443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9533.979444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9533.979446] [ 9533.979447] do_raw_spin_lock+0x177/0x190 [ 9533.979447] _raw_spin_lock+0x21/0x30 [ 9533.979448] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9533.979449] ? do_raw_spin_unlock+0x75/0x190 [ 9533.979450] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9533.979452] ? do_wait_intr+0xf0/0xf0 [ 9533.979453] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9533.979454] mdc_close+0x53b/0xd70 [mdc] [ 9533.979455] lmv_close+0x18d/0x490 [lmv] [ 9533.979456] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9533.979457] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9533.979458] ll_md_close+0x274/0x800 [lustre] [ 9533.979459] ll_file_release+0x128/0xc50 [lustre] [ 9533.979460] __fput+0x10d/0x440 [ 9533.979461] ____fput+0x12/0x20 [ 9533.979462] task_work_run+0xc6/0x110 [ 9533.979463] exit_to_usermode_loop+0x1dd/0x1f0 [ 9533.979465] do_syscall_64+0x3d6/0x440 [ 9533.979466] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9533.979467] RIP: 0033:0x7fa749552cd8 [ 9533.979469] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9533.979471] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9533.979473] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9533.979474] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9533.979476] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9533.979477] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9533.979479] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9533.981184] NMI backtrace for cpu 2 [ 9534.301995] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9534.313441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9534.319238] Call Trace: [ 9534.320334] [ 9534.321276] ? dump_stack+0xbb/0x10e [ 9534.322590] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9534.325269] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9534.327260] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9534.331411] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9534.340025] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9534.343306] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9534.345354] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9534.348081] ? update_process_times+0x6d/0xc0 [ 9534.350040] ? tick_sched_handle.isra.18+0x27/0x90 [ 9534.352273] ? tick_sched_timer+0x53/0xb0 [ 9534.354494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9534.356812] ? tick_sched_handle.isra.18+0x90/0x90 [ 9534.359153] ? hrtimer_interrupt+0x128/0x2c0 [ 9534.363777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9534.365436] ? apic_timer_interrupt+0xf/0x20 [ 9534.368162] [ 9534.369614] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9534.372559] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9534.374755] ? do_raw_spin_lock+0x177/0x190 [ 9534.377561] ? _raw_spin_lock+0x21/0x30 [ 9534.381452] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9534.387033] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9534.393696] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9534.398219] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9534.403056] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9534.406590] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9534.409842] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9534.414644] ? __next_timer_interrupt+0x160/0x160 [ 9534.426756] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9534.439820] ? do_wait_intr+0xf0/0xf0 [ 9534.444189] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9534.448928] ? kthread+0x1d1/0x200 [ 9534.450138] ? set_kthread_struct+0x70/0x70 [ 9534.459939] ? ret_from_fork+0x1f/0x30 [ 9536.057945] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9536.062363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9536.081616] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9536.085734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9536.088164] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9536.091465] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.094147] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9536.102472] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9536.105908] RAX: 0000000000003a85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9536.109670] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9536.112218] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9536.115376] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9536.117388] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9536.120240] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9536.124406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9536.127836] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9536.131403] Call Trace: [ 9536.132414] [ 9536.133371] ? show_regs.cold.9+0x22/0x2f [ 9536.135200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9536.137238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9536.139859] ? watchdog+0x40/0x40 [ 9536.141901] ? hrtimer_interrupt+0x128/0x2c0 [ 9536.143524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9536.146792] ? apic_timer_interrupt+0xf/0x20 [ 9536.149402] [ 9536.150507] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.157203] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9536.160476] do_raw_spin_lock+0x177/0x190 [ 9536.163085] _raw_spin_lock+0x21/0x30 [ 9536.164592] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9536.168165] process_one_work+0x2c8/0x700 [ 9536.171216] worker_thread+0x296/0x6e0 [ 9536.172736] ? rescuer_thread+0x570/0x570 [ 9536.175337] kthread+0x1d1/0x200 [ 9536.176917] ? set_kthread_struct+0x70/0x70 [ 9536.179757] ret_from_fork+0x1f/0x30 [ 9548.054032] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9548.056975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9548.116822] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9548.120964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9548.123912] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9548.125623] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9548.143166] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9548.156353] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9548.168354] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9548.178590] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9548.190135] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9548.195755] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9548.205558] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9548.218900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9548.223457] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9548.229563] Call Trace: [ 9548.230440] [ 9548.237111] ? show_regs.cold.9+0x22/0x2f [ 9548.241397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9548.246031] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9548.253254] ? watchdog+0x40/0x40 [ 9548.256425] ? hrtimer_interrupt+0x128/0x2c0 [ 9548.258013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9548.268829] ? apic_timer_interrupt+0xf/0x20 [ 9548.276683] [ 9548.279122] ? native_safe_halt+0x2e/0x50 [ 9548.281190] kvm_wait+0xcd/0xe0 [ 9548.283242] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9548.288916] do_raw_spin_lock+0x177/0x190 [ 9548.291504] _raw_spin_lock+0x21/0x30 [ 9548.296061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9548.307944] ? do_raw_spin_unlock+0x75/0x190 [ 9548.312466] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9548.319326] ? do_wait_intr+0xf0/0xf0 [ 9548.324325] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9548.331376] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9548.338887] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9548.350854] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9548.356930] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9548.362286] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9548.366220] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9548.368985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9548.372072] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9548.376155] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9548.380797] lmv_intent_open+0x338/0x1150 [lmv] [ 9548.384221] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9548.391955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9548.397887] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9548.404013] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9548.405772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9548.412254] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9548.416119] ? do_raw_spin_unlock+0x75/0x190 [ 9548.420601] ? _raw_spin_unlock+0x12/0x30 [ 9548.422434] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9548.426064] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9548.430035] ll_atomic_open+0x36a/0xd70 [lustre] [ 9548.436019] lookup_open+0x356/0xb40 [ 9548.440121] do_last+0x313/0xfc0 [ 9548.441320] ? nd_jump_root+0xe5/0x160 [ 9548.442664] ? path_init+0x437/0x520 [ 9548.451757] path_openat+0xf7/0x500 [ 9548.453044] ? do_raw_spin_unlock+0x75/0x190 [ 9548.460253] do_filp_open+0x99/0x140 [ 9548.464623] ? getname_flags+0x6e/0x330 [ 9548.470204] ? __check_object_size+0xff/0x256 [ 9548.475867] ? do_raw_spin_unlock+0x75/0x190 [ 9548.478130] ? _raw_spin_unlock+0x12/0x30 [ 9548.480190] do_sys_openat2+0x2b4/0x410 [ 9548.482639] do_sys_open+0x73/0xa0 [ 9548.485519] __x64_sys_openat+0x24/0x30 [ 9548.487029] do_syscall_64+0xc1/0x440 [ 9548.489095] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9548.497522] RIP: 0033:0x7fa749552332 [ 9548.503076] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9548.516734] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9548.522580] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9548.528638] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9548.537636] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9548.541722] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9548.546523] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9553.377671] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7636s! [ 9553.385394] Showing busy workqueues and worker pools: [ 9553.393582] workqueue events: flags=0x0 [ 9553.397511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9553.403071] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9553.417695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9553.424175] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9553.428025] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.434618] pending: kfree_rcu_monitor [ 9553.438629] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9553.442671] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9553.453203] workqueue events_long: flags=0x0 [ 9553.454532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.464071] pending: lu_objects_flush [obdclass] [ 9553.467503] workqueue events_unbound: flags=0x2 [ 9553.472014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9553.475041] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9553.481274] workqueue events_power_efficient: flags=0x80 [ 9553.487913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.490096] pending: check_lifetime [ 9553.500554] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.508952] pending: neigh_periodic_work [ 9553.513184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.518096] pending: neigh_periodic_work [ 9553.519356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.530278] pending: do_cache_clean [ 9553.533238] workqueue mm_percpu_wq: flags=0x8 [ 9553.538051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.544024] pending: vmstat_update [ 9553.551442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.563267] pending: vmstat_update [ 9553.567253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9553.572858] pending: vmstat_update [ 9553.575441] workqueue writeback: flags=0x4a [ 9553.584855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9553.593516] pending: wb_workfn, wb_workfn [ 9553.595185] workqueue xprtiod: flags=0xa [ 9553.598677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9553.607517] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9553.614396] workqueue ipv6_addrconf: flags=0x40008 [ 9553.617807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9553.620911] pending: addrconf_verify_work [ 9553.623199] workqueue knbd0-recv: flags=0x1a [ 9553.626045] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9553.629854] in-flight: 651:recv_work [ 9553.631776] workqueue ptlrpc_pinger: flags=0x2 [ 9553.634233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9553.642262] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9553.654254] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7636s workers=4 idle: 37 31846 28897 [ 9553.673258] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9543s workers=2 idle: 653 [ 9560.059660] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9560.060300] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9560.064823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9560.064901] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9560.064909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9560.064911] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9560.064925] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9560.064929] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9560.064933] RAX: 0000000000007e8c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9560.064936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9560.064937] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9560.064939] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9560.064941] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9560.064943] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9560.064945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9560.064946] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9560.064954] Call Trace: [ 9560.064958] [ 9560.064962] ? show_regs.cold.9+0x22/0x2f [ 9560.064969] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9560.064974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9560.064976] ? watchdog+0x40/0x40 [ 9560.064979] ? hrtimer_interrupt+0x128/0x2c0 [ 9560.064981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9560.064988] ? apic_timer_interrupt+0xf/0x20 [ 9560.064991] [ 9560.064993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9560.064997] do_raw_spin_lock+0x177/0x190 [ 9560.065001] _raw_spin_lock+0x21/0x30 [ 9560.065004] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9560.065320] ? do_raw_spin_unlock+0x75/0x190 [ 9560.065324] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9560.065449] ? do_wait_intr+0xf0/0xf0 [ 9560.065454] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9560.068617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9560.068683] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9560.068688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9560.068691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9560.068703] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9560.068706] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9560.068710] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9560.068713] RDX: 0000000000000389 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9560.068715] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9560.068717] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9560.068719] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9560.068722] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9560.068725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9560.068727] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9560.068736] Call Trace: [ 9560.068741] [ 9560.068747] ? show_regs.cold.9+0x22/0x2f [ 9560.068756] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9560.068762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9560.068767] ? watchdog+0x40/0x40 [ 9560.068771] ? hrtimer_interrupt+0x128/0x2c0 [ 9560.068775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9560.068782] ? apic_timer_interrupt+0xf/0x20 [ 9560.068787] [ 9560.068790] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9560.068795] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9560.068801] do_raw_spin_lock+0x177/0x190 [ 9560.068805] _raw_spin_lock+0x21/0x30 [ 9560.068810] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9560.069029] obd_mod_free+0x46/0x140 [ptlrpc] [ 9560.069230] mdc_commit_open+0x91/0xb0 [mdc] [ 9560.069264] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9560.069462] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9560.069680] after_reply+0xe2c/0x1780 [ptlrpc] [ 9560.069877] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9560.070076] ? __next_timer_interrupt+0x160/0x160 [ 9560.070084] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9560.139868] mdc_close+0x53b/0xd70 [mdc] [ 9560.143629] ? do_wait_intr+0xf0/0xf0 [ 9560.143650] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9560.144053] kthread+0x1d1/0x200 [ 9560.144059] ? set_kthread_struct+0x70/0x70 [ 9560.144063] ret_from_fork+0x1f/0x30 [ 9560.550612] lmv_close+0x18d/0x490 [lmv] [ 9560.554997] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9560.558717] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9560.560860] ll_md_close+0x274/0x800 [lustre] [ 9560.571731] ll_file_release+0x128/0xc50 [lustre] [ 9560.578095] __fput+0x10d/0x440 [ 9560.583976] ____fput+0x12/0x20 [ 9560.588609] task_work_run+0xc6/0x110 [ 9560.589963] exit_to_usermode_loop+0x1dd/0x1f0 [ 9560.595735] do_syscall_64+0x3d6/0x440 [ 9560.597966] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9560.600138] RIP: 0033:0x7fa749552cd8 [ 9560.601989] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9560.625255] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9560.627953] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9560.630398] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9560.635586] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9560.641751] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9560.644455] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9564.057119] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9564.061726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9564.114743] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9564.149054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9564.166636] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9564.173709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.181323] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9564.203713] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9564.209981] RAX: 0000000000001984 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9564.214850] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9564.218405] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9564.221346] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9564.226517] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9564.230253] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9564.235356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9564.240578] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9564.249774] Call Trace: [ 9564.252277] [ 9564.253511] ? show_regs.cold.9+0x22/0x2f [ 9564.255369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9564.257763] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9564.262724] ? watchdog+0x40/0x40 [ 9564.265107] ? hrtimer_interrupt+0x128/0x2c0 [ 9564.266766] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9564.270613] ? apic_timer_interrupt+0xf/0x20 [ 9564.273796] [ 9564.276830] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.278227] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9564.279882] do_raw_spin_lock+0x177/0x190 [ 9564.288844] _raw_spin_lock+0x21/0x30 [ 9564.292401] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9564.298374] process_one_work+0x2c8/0x700 [ 9564.303726] worker_thread+0x296/0x6e0 [ 9564.312625] ? rescuer_thread+0x570/0x570 [ 9564.313759] kthread+0x1d1/0x200 [ 9564.318714] ? set_kthread_struct+0x70/0x70 [ 9564.321839] ret_from_fork+0x1f/0x30 [ 9580.060808] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9580.066618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9580.095069] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9580.100441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9580.104741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9580.107613] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9580.116642] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9580.126706] RAX: 0000000000005e86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9580.132411] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9580.143391] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9580.151718] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9580.160153] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9580.175576] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9580.194216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9580.200153] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9580.211758] Call Trace: [ 9580.215777] [ 9580.219396] ? show_regs.cold.9+0x22/0x2f [ 9580.222882] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9580.227867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9580.229469] ? watchdog+0x40/0x40 [ 9580.237796] ? hrtimer_interrupt+0x128/0x2c0 [ 9580.242854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9580.248539] ? apic_timer_interrupt+0xf/0x20 [ 9580.249992] [ 9580.257265] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9580.261898] do_raw_spin_lock+0x177/0x190 [ 9580.266605] _raw_spin_lock+0x21/0x30 [ 9580.268543] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9580.275080] ? do_raw_spin_unlock+0x75/0x190 [ 9580.280752] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9580.289713] ? do_wait_intr+0xf0/0xf0 [ 9580.291853] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9580.298465] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9580.304863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9580.309806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.317268] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9580.323456] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9580.325187] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9580.335782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.346289] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9580.359773] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9580.370728] lmv_intent_open+0x338/0x1150 [lmv] [ 9580.379733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.385579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.394763] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9580.401684] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9580.404163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.412233] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9580.419007] ? do_raw_spin_unlock+0x75/0x190 [ 9580.426775] ? _raw_spin_unlock+0x12/0x30 [ 9580.437400] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9580.444137] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9580.447004] ll_atomic_open+0x36a/0xd70 [lustre] [ 9580.459710] lookup_open+0x356/0xb40 [ 9580.466363] do_last+0x313/0xfc0 [ 9580.469461] ? nd_jump_root+0xe5/0x160 [ 9580.470788] ? path_init+0x437/0x520 [ 9580.474273] path_openat+0xf7/0x500 [ 9580.479691] ? do_raw_spin_unlock+0x75/0x190 [ 9580.503441] do_filp_open+0x99/0x140 [ 9580.509061] ? getname_flags+0x6e/0x330 [ 9580.521216] ? __check_object_size+0xff/0x256 [ 9580.528156] ? do_raw_spin_unlock+0x75/0x190 [ 9580.533749] ? _raw_spin_unlock+0x12/0x30 [ 9580.538633] do_sys_openat2+0x2b4/0x410 [ 9580.543421] do_sys_open+0x73/0xa0 [ 9580.548392] __x64_sys_openat+0x24/0x30 [ 9580.559005] do_syscall_64+0xc1/0x440 [ 9580.565502] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9580.573717] RIP: 0033:0x7fa749552332 [ 9580.580926] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9580.610119] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9580.618104] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9580.638123] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9580.657271] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9580.665879] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9580.683460] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9584.102917] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7666s! [ 9584.109366] Showing busy workqueues and worker pools: [ 9584.118193] workqueue events: flags=0x0 [ 9584.119677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9584.123851] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9584.138606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9584.149670] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9584.162682] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.175336] pending: kfree_rcu_monitor [ 9584.183719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9584.203102] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9584.213919] workqueue events_long: flags=0x0 [ 9584.224401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.231361] pending: lu_objects_flush [obdclass] [ 9584.241648] workqueue events_unbound: flags=0x2 [ 9584.249487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9584.259429] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9584.285032] workqueue events_power_efficient: flags=0x80 [ 9584.301338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.316076] pending: check_lifetime [ 9584.319539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.334376] pending: neigh_periodic_work [ 9584.340949] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.346460] pending: neigh_periodic_work [ 9584.349707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.357138] pending: do_cache_clean [ 9584.359497] workqueue mm_percpu_wq: flags=0x8 [ 9584.367204] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.380320] pending: vmstat_update [ 9584.387332] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.400195] pending: vmstat_update [ 9584.409266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9584.426119] pending: vmstat_update [ 9584.427057] workqueue writeback: flags=0x4a [ 9584.431940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9584.445836] pending: wb_workfn, wb_workfn [ 9584.447625] workqueue xprtiod: flags=0xa [ 9584.456547] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9584.470530] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9584.478906] workqueue ipv6_addrconf: flags=0x40008 [ 9584.492148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9584.500489] pending: addrconf_verify_work [ 9584.504060] workqueue knbd0-recv: flags=0x1a [ 9584.509349] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9584.521175] in-flight: 651:recv_work [ 9584.524344] workqueue ptlrpc_pinger: flags=0x2 [ 9584.533120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9584.544551] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9584.555780] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7667s workers=4 idle: 37 31846 28897 [ 9584.566014] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9574s workers=2 idle: 653 [ 9588.058855] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9588.067331] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9588.071199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9588.071363] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9588.071372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9588.071374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9588.071397] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9588.071402] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9588.071406] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9588.071409] RDX: 000000000000679f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9588.071411] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9588.071414] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9588.071416] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9588.071420] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9588.071422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9588.071425] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9588.071433] Call Trace: [ 9588.071446] [ 9588.071453] ? show_regs.cold.9+0x22/0x2f [ 9588.071463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9588.071471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9588.071475] ? watchdog+0x40/0x40 [ 9588.071479] ? hrtimer_interrupt+0x128/0x2c0 [ 9588.071482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9588.071490] ? apic_timer_interrupt+0xf/0x20 [ 9588.071495] [ 9588.071498] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9588.071503] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9588.071507] do_raw_spin_lock+0x177/0x190 [ 9588.071511] _raw_spin_lock+0x21/0x30 [ 9588.071519] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9588.072286] obd_mod_free+0x46/0x140 [ptlrpc] [ 9588.072488] mdc_commit_open+0x91/0xb0 [mdc] [ 9588.072523] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9588.076836] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9588.077123] after_reply+0xe2c/0x1780 [ptlrpc] [ 9588.077332] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9588.077528] ? __next_timer_interrupt+0x160/0x160 [ 9588.077540] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9588.079756] ? do_wait_intr+0xf0/0xf0 [ 9588.079771] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9588.080017] kthread+0x1d1/0x200 [ 9588.080024] ? set_kthread_struct+0x70/0x70 [ 9588.080028] ret_from_fork+0x1f/0x30 [ 9588.351200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9588.408424] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9588.414868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9588.419639] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9588.422152] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9588.441781] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9588.451977] RAX: 000000000000390f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9588.460909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9588.467650] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9588.472489] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9588.476197] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9588.479188] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9588.483949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9588.488860] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9588.496235] Call Trace: [ 9588.497032] [ 9588.497676] ? show_regs.cold.9+0x22/0x2f [ 9588.506000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9588.511469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9588.516825] ? watchdog+0x40/0x40 [ 9588.517896] ? hrtimer_interrupt+0x128/0x2c0 [ 9588.525344] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9588.528659] ? apic_timer_interrupt+0xf/0x20 [ 9588.534554] [ 9588.535448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9588.540927] do_raw_spin_lock+0x177/0x190 [ 9588.542224] _raw_spin_lock+0x21/0x30 [ 9588.543409] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9588.555033] ? do_raw_spin_unlock+0x75/0x190 [ 9588.560839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9588.568417] ? do_wait_intr+0xf0/0xf0 [ 9588.570880] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9588.579571] mdc_close+0x53b/0xd70 [mdc] [ 9588.581201] lmv_close+0x18d/0x490 [lmv] [ 9588.586217] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9588.588567] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9588.591758] ll_md_close+0x274/0x800 [lustre] [ 9588.593273] ll_file_release+0x128/0xc50 [lustre] [ 9588.602965] __fput+0x10d/0x440 [ 9588.606143] ____fput+0x12/0x20 [ 9588.611240] task_work_run+0xc6/0x110 [ 9588.612560] exit_to_usermode_loop+0x1dd/0x1f0 [ 9588.621954] do_syscall_64+0x3d6/0x440 [ 9588.624391] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9588.630466] RIP: 0033:0x7fa749552cd8 [ 9588.634457] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9588.651460] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9588.659266] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9588.666093] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9588.671496] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9588.682450] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9588.692653] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9592.057205] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9592.059647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9592.093458] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9592.118458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9592.128936] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9592.133295] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9592.134793] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9592.168777] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9592.175333] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9592.182263] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9592.191208] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9592.196971] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9592.198928] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9592.204393] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9592.212329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9592.224383] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9592.231250] Call Trace: [ 9592.232168] [ 9592.234992] ? show_regs.cold.9+0x22/0x2f [ 9592.240409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9592.245139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9592.251666] ? watchdog+0x40/0x40 [ 9592.252749] ? hrtimer_interrupt+0x128/0x2c0 [ 9592.255452] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9592.257783] ? apic_timer_interrupt+0xf/0x20 [ 9592.265899] [ 9592.269369] ? native_safe_halt+0x2e/0x50 [ 9592.273968] kvm_wait+0xcd/0xe0 [ 9592.274985] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9592.278862] do_raw_spin_lock+0x177/0x190 [ 9592.284386] _raw_spin_lock+0x21/0x30 [ 9592.288954] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9592.293400] process_one_work+0x2c8/0x700 [ 9592.298685] worker_thread+0x296/0x6e0 [ 9592.301944] ? rescuer_thread+0x570/0x570 [ 9592.305136] kthread+0x1d1/0x200 [ 9592.309126] ? set_kthread_struct+0x70/0x70 [ 9592.313720] ret_from_fork+0x1f/0x30 [ 9612.057668] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9612.063179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9612.096476] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9612.111638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9612.121834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9612.132857] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9612.147505] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9612.152237] RAX: 0000000000000c8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9612.158611] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9612.164744] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9612.171187] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9612.180237] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9612.183620] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9612.187206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9612.189427] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9612.196939] Call Trace: [ 9612.197821] [ 9612.201182] ? show_regs.cold.9+0x22/0x2f [ 9612.202867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9612.204283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9612.207079] ? watchdog+0x40/0x40 [ 9612.208155] ? hrtimer_interrupt+0x128/0x2c0 [ 9612.209202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9612.219061] ? apic_timer_interrupt+0xf/0x20 [ 9612.225003] [ 9612.225768] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9612.231207] do_raw_spin_lock+0x177/0x190 [ 9612.235234] _raw_spin_lock+0x21/0x30 [ 9612.239496] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9612.247535] ? do_raw_spin_unlock+0x75/0x190 [ 9612.254248] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9612.259110] ? do_wait_intr+0xf0/0xf0 [ 9612.260424] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9612.266676] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9612.276118] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9612.278236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9612.287824] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9612.293909] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9612.299693] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9612.301428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9612.307207] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9612.315263] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9612.320624] lmv_intent_open+0x338/0x1150 [lmv] [ 9612.326244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9612.333335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9612.335158] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9612.339085] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9612.343436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9612.351889] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9612.357762] ? do_raw_spin_unlock+0x75/0x190 [ 9612.363437] ? _raw_spin_unlock+0x12/0x30 [ 9612.367461] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9612.371087] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9612.373699] ll_atomic_open+0x36a/0xd70 [lustre] [ 9612.376896] lookup_open+0x356/0xb40 [ 9612.383723] do_last+0x313/0xfc0 [ 9612.385516] ? nd_jump_root+0xe5/0x160 [ 9612.391824] ? path_init+0x437/0x520 [ 9612.393128] path_openat+0xf7/0x500 [ 9612.394385] ? do_raw_spin_unlock+0x75/0x190 [ 9612.404843] do_filp_open+0x99/0x140 [ 9612.406857] ? getname_flags+0x6e/0x330 [ 9612.409198] ? __check_object_size+0xff/0x256 [ 9612.414861] ? do_raw_spin_unlock+0x75/0x190 [ 9612.422951] ? _raw_spin_unlock+0x12/0x30 [ 9612.425738] do_sys_openat2+0x2b4/0x410 [ 9612.427103] do_sys_open+0x73/0xa0 [ 9612.432996] __x64_sys_openat+0x24/0x30 [ 9612.434409] do_syscall_64+0xc1/0x440 [ 9612.456653] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9612.462862] RIP: 0033:0x7fa749552332 [ 9612.466798] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9612.488322] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9612.497676] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9612.503841] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9612.514878] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9612.519199] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9612.534043] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9614.818207] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7697s! [ 9614.827199] Showing busy workqueues and worker pools: [ 9614.831796] workqueue events: flags=0x0 [ 9614.834270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9614.838679] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9614.844138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9614.847869] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9614.852285] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.855353] pending: kfree_rcu_monitor [ 9614.857134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9614.860720] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9614.865504] workqueue events_long: flags=0x0 [ 9614.868111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.871695] pending: lu_objects_flush [obdclass] [ 9614.874477] workqueue events_unbound: flags=0x2 [ 9614.876909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9614.880175] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9614.885861] workqueue events_power_efficient: flags=0x80 [ 9614.889925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.895432] pending: check_lifetime [ 9614.897992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.901643] pending: neigh_periodic_work [ 9614.904009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.907841] pending: neigh_periodic_work [ 9614.910122] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.913862] pending: do_cache_clean [ 9614.918679] workqueue mm_percpu_wq: flags=0x8 [ 9614.920145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.924063] pending: vmstat_update [ 9614.926117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.929866] pending: vmstat_update [ 9614.932406] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9614.942952] pending: vmstat_update [ 9614.944365] workqueue writeback: flags=0x4a [ 9614.952800] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9614.957237] pending: wb_workfn, wb_workfn [ 9614.960081] workqueue xprtiod: flags=0xa [ 9614.964129] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9614.967930] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9614.972143] workqueue ipv6_addrconf: flags=0x40008 [ 9614.974811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9614.978251] pending: addrconf_verify_work [ 9614.980893] workqueue knbd0-recv: flags=0x1a [ 9614.983591] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9614.987420] in-flight: 651:recv_work [ 9614.989864] workqueue ptlrpc_pinger: flags=0x2 [ 9614.992475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9614.995706] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9614.999454] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7697s workers=4 idle: 37 31846 28897 [ 9615.004041] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9604s workers=2 idle: 653 [ 9616.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9616.059135] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9616.061168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9616.061250] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9616.061257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9616.061260] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9616.061272] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9616.061277] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9616.061281] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9616.061284] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9616.061287] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9616.068056] Modules linked in: [ 9616.091326] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9616.091336] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9616.091340] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9616.091342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9616.091345] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9616.091353] Call Trace: [ 9616.091545] [ 9616.091553] ? show_regs.cold.9+0x22/0x2f [ 9616.091565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9616.091573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9616.091577] ? watchdog+0x40/0x40 [ 9616.091581] ? hrtimer_interrupt+0x128/0x2c0 [ 9616.091585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9616.091594] ? apic_timer_interrupt+0xf/0x20 [ 9616.091598] [ 9616.091601] ? native_safe_halt+0x2e/0x50 [ 9616.091606] kvm_wait+0xcd/0xe0 [ 9616.091669] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9616.091679] do_raw_spin_lock+0x177/0x190 [ 9616.091683] _raw_spin_lock+0x21/0x30 [ 9616.091690] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9616.091950] obd_mod_free+0x46/0x140 [ptlrpc] [ 9616.101354] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9616.101726] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9616.101734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9616.101737] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9616.101752] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9616.101757] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9616.101762] RAX: 0000000000006f14 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9616.101764] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9616.101766] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9616.101769] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9616.101772] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9616.101775] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9616.101778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9616.101781] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9616.101791] Call Trace: [ 9616.101796] [ 9616.101802] ? show_regs.cold.9+0x22/0x2f [ 9616.101811] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9616.101818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9616.101822] ? watchdog+0x40/0x40 [ 9616.101827] ? hrtimer_interrupt+0x128/0x2c0 [ 9616.101831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9616.101840] ? apic_timer_interrupt+0xf/0x20 [ 9616.101845] [ 9616.101848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9616.101855] do_raw_spin_lock+0x177/0x190 [ 9616.101860] _raw_spin_lock+0x21/0x30 [ 9616.101866] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9616.102557] ? do_raw_spin_unlock+0x75/0x190 [ 9616.102564] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9616.117370] mdc_commit_open+0x91/0xb0 [mdc] [ 9616.117421] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9616.117699] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9616.117895] after_reply+0xe2c/0x1780 [ptlrpc] [ 9616.118274] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9616.118528] ? __next_timer_interrupt+0x160/0x160 [ 9616.118538] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9616.118735] ? do_wait_intr+0xf0/0xf0 [ 9616.118746] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9616.118953] kthread+0x1d1/0x200 [ 9616.118959] ? set_kthread_struct+0x70/0x70 [ 9616.118963] ret_from_fork+0x1f/0x30 [ 9616.451401] ? do_wait_intr+0xf0/0xf0 [ 9616.452841] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9616.456424] mdc_close+0x53b/0xd70 [mdc] [ 9616.459136] lmv_close+0x18d/0x490 [lmv] [ 9616.460994] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9616.463808] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9616.465836] ll_md_close+0x274/0x800 [lustre] [ 9616.468686] ll_file_release+0x128/0xc50 [lustre] [ 9616.470808] __fput+0x10d/0x440 [ 9616.472283] ____fput+0x12/0x20 [ 9616.473861] task_work_run+0xc6/0x110 [ 9616.475708] exit_to_usermode_loop+0x1dd/0x1f0 [ 9616.479638] do_syscall_64+0x3d6/0x440 [ 9616.481438] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9616.484154] RIP: 0033:0x7fa749552cd8 [ 9616.485448] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9616.494539] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9616.500589] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9616.505286] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9616.509417] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9616.513906] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9616.517662] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9620.058810] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9620.063634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9620.123475] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9620.131068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9620.142668] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9620.154355] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9620.158259] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9620.178157] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9620.183646] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9620.187871] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9620.193541] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9620.196748] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9620.204951] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9620.209713] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9620.216607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9620.220064] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9620.226679] Call Trace: [ 9620.228641] [ 9620.230180] ? show_regs.cold.9+0x22/0x2f [ 9620.233035] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9620.236528] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9620.240690] ? watchdog+0x40/0x40 [ 9620.244540] ? hrtimer_interrupt+0x128/0x2c0 [ 9620.249635] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9620.252897] ? apic_timer_interrupt+0xf/0x20 [ 9620.256448] [ 9620.257327] ? native_safe_halt+0x2e/0x50 [ 9620.261056] kvm_wait+0xcd/0xe0 [ 9620.262309] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9620.265682] do_raw_spin_lock+0x177/0x190 [ 9620.268565] _raw_spin_lock+0x21/0x30 [ 9620.269893] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9620.273529] process_one_work+0x2c8/0x700 [ 9620.276925] worker_thread+0x296/0x6e0 [ 9620.278231] ? rescuer_thread+0x570/0x570 [ 9620.279990] kthread+0x1d1/0x200 [ 9620.281849] ? set_kthread_struct+0x70/0x70 [ 9620.284032] ret_from_fork+0x1f/0x30 [ 9640.055376] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9640.063882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9640.098986] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9640.102623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9640.107645] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9640.111045] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9640.122031] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9640.129192] RAX: 0000000000007fb7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9640.133332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9640.138181] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9640.139975] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9640.143401] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9640.145594] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9640.152858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9640.156593] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9640.161173] Call Trace: [ 9640.162170] [ 9640.163838] ? show_regs.cold.9+0x22/0x2f [ 9640.167651] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9640.172306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9640.177615] ? watchdog+0x40/0x40 [ 9640.178503] ? hrtimer_interrupt+0x128/0x2c0 [ 9640.179699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9640.180923] ? apic_timer_interrupt+0xf/0x20 [ 9640.187571] [ 9640.190649] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9640.192829] do_raw_spin_lock+0x177/0x190 [ 9640.194269] _raw_spin_lock+0x21/0x30 [ 9640.195475] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9640.197884] ? do_raw_spin_unlock+0x75/0x190 [ 9640.199249] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9640.202347] ? do_wait_intr+0xf0/0xf0 [ 9640.203617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9640.209003] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9640.213459] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9640.220580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9640.224109] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9640.226872] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9640.229404] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9640.231962] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9640.235083] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9640.238314] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9640.243400] lmv_intent_open+0x338/0x1150 [lmv] [ 9640.245500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9640.248922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9640.252252] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9640.254272] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9640.256636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9640.259742] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9640.262259] ? do_raw_spin_unlock+0x75/0x190 [ 9640.264042] ? _raw_spin_unlock+0x12/0x30 [ 9640.267539] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9640.271926] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9640.274268] ll_atomic_open+0x36a/0xd70 [lustre] [ 9640.276329] lookup_open+0x356/0xb40 [ 9640.278307] do_last+0x313/0xfc0 [ 9640.279877] ? nd_jump_root+0xe5/0x160 [ 9640.281384] ? path_init+0x437/0x520 [ 9640.282585] path_openat+0xf7/0x500 [ 9640.284012] ? do_raw_spin_unlock+0x75/0x190 [ 9640.285657] do_filp_open+0x99/0x140 [ 9640.287457] ? getname_flags+0x6e/0x330 [ 9640.289174] ? __check_object_size+0xff/0x256 [ 9640.291481] ? do_raw_spin_unlock+0x75/0x190 [ 9640.295112] ? _raw_spin_unlock+0x12/0x30 [ 9640.297409] do_sys_openat2+0x2b4/0x410 [ 9640.299286] do_sys_open+0x73/0xa0 [ 9640.301123] __x64_sys_openat+0x24/0x30 [ 9640.303050] do_syscall_64+0xc1/0x440 [ 9640.304894] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9640.307227] RIP: 0033:0x7fa749552332 [ 9640.308845] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9640.317370] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9640.320878] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9640.325375] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9640.328878] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9640.331831] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9640.335594] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9644.058987] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9644.059116] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9644.066266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9644.066411] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9644.066418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9644.066421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9644.066437] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9644.066441] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9644.066446] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9644.066448] RDX: 0000000000002616 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9644.066450] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9644.066452] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9644.066455] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9644.066458] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9644.066461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9644.066463] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9644.066472] Call Trace: [ 9644.066478] [ 9644.066485] ? show_regs.cold.9+0x22/0x2f [ 9644.066494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9644.066501] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9644.066504] ? watchdog+0x40/0x40 [ 9644.066508] ? hrtimer_interrupt+0x128/0x2c0 [ 9644.066512] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9644.066521] ? apic_timer_interrupt+0xf/0x20 [ 9644.066525] [ 9644.066529] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9644.066534] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9644.066539] do_raw_spin_lock+0x177/0x190 [ 9644.066543] _raw_spin_lock+0x21/0x30 [ 9644.066549] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9644.066963] obd_mod_free+0x46/0x140 [ptlrpc] [ 9644.073043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [ 9644.117567] mdc_commit_open+0x91/0xb0 [mdc] [ 9644.117853] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9644.118239] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9644.118456] after_reply+0xe2c/0x1780 [ptlrpc] [ 9644.118794] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9644.119005] ? __next_timer_interrupt+0x160/0x160 [ 9644.119017] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9644.119224] ? do_wait_intr+0xf0/0xf0 [ 9644.119234] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9644.119568] kthread+0x1d1/0x200 [ 9644.119575] ? set_kthread_struct+0x70/0x70 [ 9644.119579] ret_from_fork+0x1f/0x30 [ 9644.265289] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9644.280372] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9644.285289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9644.289275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9644.292327] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9644.305283] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9644.309045] RAX: 00000000000068a6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9644.312246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9644.320872] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9644.324349] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9644.328270] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9644.333284] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9644.337150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9644.341958] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9644.350821] Call Trace: [ 9644.352528] [ 9644.353253] ? show_regs.cold.9+0x22/0x2f [ 9644.355029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9644.357612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9644.359539] ? watchdog+0x40/0x40 [ 9644.360741] ? hrtimer_interrupt+0x128/0x2c0 [ 9644.363071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9644.364688] ? apic_timer_interrupt+0xf/0x20 [ 9644.366866] [ 9644.367870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9644.370170] do_raw_spin_lock+0x177/0x190 [ 9644.374428] _raw_spin_lock+0x21/0x30 [ 9644.375767] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9644.378120] ? do_raw_spin_unlock+0x75/0x190 [ 9644.380330] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9644.383317] ? do_wait_intr+0xf0/0xf0 [ 9644.384602] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9644.386689] mdc_close+0x53b/0xd70 [mdc] [ 9644.388416] lmv_close+0x18d/0x490 [lmv] [ 9644.389734] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9644.392294] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9644.394383] ll_md_close+0x274/0x800 [lustre] [ 9644.396209] ll_file_release+0x128/0xc50 [lustre] [ 9644.398221] __fput+0x10d/0x440 [ 9644.400006] ____fput+0x12/0x20 [ 9644.401693] task_work_run+0xc6/0x110 [ 9644.403112] exit_to_usermode_loop+0x1dd/0x1f0 [ 9644.407558] do_syscall_64+0x3d6/0x440 [ 9644.409162] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9644.411342] RIP: 0033:0x7fa749552cd8 [ 9644.412895] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9644.421384] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9644.424934] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9644.427908] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9644.430810] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9644.433586] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9644.437331] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9645.024867] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7727s! [ 9645.028500] Showing busy workqueues and worker pools: [ 9645.032458] workqueue events: flags=0x0 [ 9645.038199] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9645.041507] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9645.047193] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9645.052483] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9645.057204] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.059847] pending: kfree_rcu_monitor [ 9645.062410] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9645.065483] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9645.072211] workqueue events_long: flags=0x0 [ 9645.075429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.078870] pending: lu_objects_flush [obdclass] [ 9645.081317] workqueue events_unbound: flags=0x2 [ 9645.083437] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9645.088564] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9645.098404] workqueue events_power_efficient: flags=0x80 [ 9645.105135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.118250] pending: check_lifetime [ 9645.121248] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.133015] pending: neigh_periodic_work [ 9645.141314] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.149099] pending: neigh_periodic_work [ 9645.151242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.156515] pending: do_cache_clean [ 9645.159014] workqueue mm_percpu_wq: flags=0x8 [ 9645.161240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.164507] pending: vmstat_update [ 9645.168185] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.174186] pending: vmstat_update [ 9645.177205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9645.181585] pending: vmstat_update [ 9645.182931] workqueue writeback: flags=0x4a [ 9645.195717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9645.203922] pending: wb_workfn, wb_workfn [ 9645.211318] workqueue xprtiod: flags=0xa [ 9645.215373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9645.220343] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9645.230483] workqueue ipv6_addrconf: flags=0x40008 [ 9645.234630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9645.237958] pending: addrconf_verify_work [ 9645.243298] workqueue knbd0-recv: flags=0x1a [ 9645.256463] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9645.273049] in-flight: 651:recv_work [ 9645.275477] workqueue ptlrpc_pinger: flags=0x2 [ 9645.285276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9645.291245] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9645.304026] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7728s workers=4 idle: 37 31846 28897 [ 9645.319450] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9635s workers=2 idle: 653 [ 9648.057054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9648.060264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9648.079759] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9648.084432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9648.088402] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9648.091515] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9648.093465] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9648.101455] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9648.104872] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9648.108705] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9648.111582] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9648.114671] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9648.117808] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9648.119828] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9648.122551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9648.125186] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9648.128331] Call Trace: [ 9648.129475] [ 9648.130429] ? show_regs.cold.9+0x22/0x2f [ 9648.131883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9648.134263] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9648.136051] ? watchdog+0x40/0x40 [ 9648.137346] ? hrtimer_interrupt+0x128/0x2c0 [ 9648.139350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9648.141786] ? apic_timer_interrupt+0xf/0x20 [ 9648.143197] [ 9648.143985] ? native_safe_halt+0x2e/0x50 [ 9648.146073] kvm_wait+0xcd/0xe0 [ 9648.147226] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9648.149593] do_raw_spin_lock+0x177/0x190 [ 9648.151198] _raw_spin_lock+0x21/0x30 [ 9648.152627] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9648.155617] process_one_work+0x2c8/0x700 [ 9648.157153] worker_thread+0x296/0x6e0 [ 9648.158995] ? rescuer_thread+0x570/0x570 [ 9648.160109] kthread+0x1d1/0x200 [ 9648.161307] ? set_kthread_struct+0x70/0x70 [ 9648.162962] ret_from_fork+0x1f/0x30 [ 9672.057422] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9672.058227] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9672.060711] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9672.060724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9672.060800] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9672.060807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9672.060810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9672.060825] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9672.060830] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9672.060834] RAX: 00000000000030a2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9672.060837] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9672.060840] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9672.060843] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9672.060845] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9672.060849] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9672.060852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9672.060855] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9672.060864] Call Trace: [ 9672.060869] [ 9672.060875] ? show_regs.cold.9+0x22/0x2f [ 9672.060885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9672.060892] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9672.060897] ? watchdog+0x40/0x40 [ 9672.060901] ? hrtimer_interrupt+0x128/0x2c0 [ 9672.061243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9672.061252] ? apic_timer_interrupt+0xf/0x20 [ 9672.061258] [ 9672.061261] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9672.061269] do_raw_spin_lock+0x177/0x190 [ 9672.061275] _raw_spin_lock+0x21/0x30 [ 9672.061281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9672.063260] ? do_raw_spin_unlock+0x75/0x190 [ 9672.063273] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9672.064283] ? do_wait_intr+0xf0/0xf0 [ 9672.064988] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9672.065215] mdc_close+0x53b/0xd70 [mdc] [ 9672.065261] lmv_close+0x18d/0x490 [lmv] [ 9672.065292] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9672.066098] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9672.066180] ll_md_close+0x274/0x800 [lustre] [ 9672.066264] ll_file_release+0x128/0xc50 [lustre] [ 9672.067040] __fput+0x10d/0x440 [ 9672.067050] ____fput+0x12/0x20 [ 9672.067055] task_work_run+0xc6/0x110 [ 9672.067063] exit_to_usermode_loop+0x1dd/0x1f0 [ 9672.067072] do_syscall_64+0x3d6/0x440 [ 9672.067077] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9672.067086] RIP: 0033:0x7fa749552cd8 [ 9672.067094] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9672.067098] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9672.067104] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9672.067106] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9672.067109] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9672.067111] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9672.067114] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9672.068635] Modules linked in: [ 9672.072456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9672.072538] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9672.072546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9672.072549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9672.072563] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9672.072567] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9672.072572] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9672.072575] RDX: 0000000000003d10 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9672.072577] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9672.072580] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9672.072583] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9672.072586] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9672.072589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9672.072591] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9672.072600] Call Trace: [ 9672.072606] [ 9672.072613] ? show_regs.cold.9+0x22/0x2f [ 9672.072624] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9672.072630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9672.072635] ? watchdog+0x40/0x40 [ 9672.072640] ? hrtimer_interrupt+0x128/0x2c0 [ 9672.072644] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9672.072653] ? apic_timer_interrupt+0xf/0x20 [ 9672.072658] [ 9672.072662] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9672.072667] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9672.072671] do_raw_spin_lock+0x177/0x190 [ 9672.072675] _raw_spin_lock+0x21/0x30 [ 9672.072681] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9672.072959] obd_mod_free+0x46/0x140 [ptlrpc] [ 9672.073162] mdc_commit_open+0x91/0xb0 [mdc] [ 9672.073200] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9672.106278] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9672.106522] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9672.106530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9672.106533] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9672.106548] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9672.106553] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9672.106558] RAX: 0000000000007f93 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9672.106561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9672.106563] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9672.106565] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9672.106568] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9672.106571] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9672.106574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9672.106577] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9672.106586] Call Trace: [ 9672.106592] [ 9672.106598] ? show_regs.cold.9+0x22/0x2f [ 9672.106608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9672.106614] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9672.106618] ? watchdog+0x40/0x40 [ 9672.106623] ? hrtimer_interrupt+0x128/0x2c0 [ 9672.106628] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9672.106637] ? apic_timer_interrupt+0xf/0x20 [ 9672.106642] [ 9672.106644] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9672.106651] do_raw_spin_lock+0x177/0x190 [ 9672.106657] _raw_spin_lock+0x21/0x30 [ 9672.106662] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9672.106937] ? do_raw_spin_unlock+0x75/0x190 [ 9672.106943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9672.107138] ? do_wait_intr+0xf0/0xf0 [ 9672.107146] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9672.107487] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9672.107719] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9672.107936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.108048] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9672.108087] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9672.108115] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9672.108144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.108232] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9672.108637] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9672.108667] lmv_intent_open+0x338/0x1150 [lmv] [ 9672.108701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.108790] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.108876] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9672.108901] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9672.108986] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.109065] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9672.109146] ? do_raw_spin_unlock+0x75/0x190 [ 9672.109153] ? _raw_spin_unlock+0x12/0x30 [ 9672.109158] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9672.109186] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9672.109208] ll_atomic_open+0x36a/0xd70 [lustre] [ 9672.113964] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9672.114270] after_reply+0xe2c/0x1780 [ptlrpc] [ 9672.114633] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9672.114825] ? __next_timer_interrupt+0x160/0x160 [ 9672.114837] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9672.115044] ? do_wait_intr+0xf0/0xf0 [ 9672.115055] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9672.115250] kthread+0x1d1/0x200 [ 9672.115256] ? set_kthread_struct+0x70/0x70 [ 9672.115260] ret_from_fork+0x1f/0x30 [ 9672.879004] lookup_open+0x356/0xb40 [ 9672.882169] do_last+0x313/0xfc0 [ 9672.886876] ? nd_jump_root+0xe5/0x160 [ 9672.890899] ? path_init+0x437/0x520 [ 9672.894179] path_openat+0xf7/0x500 [ 9672.896503] ? do_raw_spin_unlock+0x75/0x190 [ 9672.899808] do_filp_open+0x99/0x140 [ 9672.902536] ? getname_flags+0x6e/0x330 [ 9672.905577] ? __check_object_size+0xff/0x256 [ 9672.914735] ? do_raw_spin_unlock+0x75/0x190 [ 9672.918561] ? _raw_spin_unlock+0x12/0x30 [ 9672.920614] do_sys_openat2+0x2b4/0x410 [ 9672.922824] do_sys_open+0x73/0xa0 [ 9672.927709] __x64_sys_openat+0x24/0x30 [ 9672.933478] do_syscall_64+0xc1/0x440 [ 9672.935808] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9672.939004] RIP: 0033:0x7fa749552332 [ 9672.940297] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9672.958596] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9672.971878] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9672.976452] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9672.983846] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9672.997150] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9673.010050] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9675.745040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7758s! [ 9675.747577] Showing busy workqueues and worker pools: [ 9675.749581] workqueue events: flags=0x0 [ 9675.750808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9675.755048] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9675.760742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9675.763046] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9675.766103] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.769398] pending: kfree_rcu_monitor [ 9675.770929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9675.773783] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9675.777882] workqueue events_long: flags=0x0 [ 9675.779727] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.783110] pending: lu_objects_flush [obdclass] [ 9675.785941] workqueue events_unbound: flags=0x2 [ 9675.789226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9675.792105] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9675.796642] workqueue events_power_efficient: flags=0x80 [ 9675.798521] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.801921] pending: check_lifetime [ 9675.803394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.805788] pending: neigh_periodic_work [ 9675.807710] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.810885] pending: neigh_periodic_work [ 9675.813941] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.816993] pending: do_cache_clean [ 9675.819412] workqueue mm_percpu_wq: flags=0x8 [ 9675.821781] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.826642] pending: vmstat_update [ 9675.827971] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.831944] pending: vmstat_update [ 9675.834976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9675.842779] pending: vmstat_update [ 9675.846616] workqueue writeback: flags=0x4a [ 9675.848823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9675.854831] pending: wb_workfn, wb_workfn [ 9675.857763] workqueue xprtiod: flags=0xa [ 9675.859709] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9675.862812] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9675.867838] workqueue ipv6_addrconf: flags=0x40008 [ 9675.870376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9675.873212] pending: addrconf_verify_work [ 9675.875783] workqueue knbd0-recv: flags=0x1a [ 9675.877302] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9675.882064] in-flight: 651:recv_work [ 9675.884485] workqueue ptlrpc_pinger: flags=0x2 [ 9675.886915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9675.894814] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9675.898498] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7758s workers=4 idle: 37 31846 28897 [ 9675.902907] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9665s workers=2 idle: 653 [ 9676.057069] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9676.062741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9676.090388] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9676.097721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9676.103914] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9676.107946] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9676.110771] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9676.122206] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9676.127029] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9676.130547] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9676.134065] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9676.138807] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9676.142718] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9676.146691] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9676.151275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9676.154767] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9676.158404] Call Trace: [ 9676.159776] [ 9676.160777] ? show_regs.cold.9+0x22/0x2f [ 9676.163533] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9676.165996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9676.168696] ? watchdog+0x40/0x40 [ 9676.170447] ? hrtimer_interrupt+0x128/0x2c0 [ 9676.172748] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9676.175106] ? apic_timer_interrupt+0xf/0x20 [ 9676.177497] [ 9676.178649] ? native_safe_halt+0x2e/0x50 [ 9676.180764] kvm_wait+0xcd/0xe0 [ 9676.182496] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9676.185376] do_raw_spin_lock+0x177/0x190 [ 9676.187876] _raw_spin_lock+0x21/0x30 [ 9676.189870] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9676.192940] process_one_work+0x2c8/0x700 [ 9676.195178] worker_thread+0x296/0x6e0 [ 9676.197064] ? rescuer_thread+0x570/0x570 [ 9676.199127] kthread+0x1d1/0x200 [ 9676.200654] ? set_kthread_struct+0x70/0x70 [ 9676.202853] ret_from_fork+0x1f/0x30 [ 9700.054037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9700.057373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9700.058716] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9700.060543] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9700.060553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9700.060622] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9700.060628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9700.060631] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9700.060644] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9700.060648] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9700.060653] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9700.060655] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9700.060657] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9700.060659] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9700.060662] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9700.060665] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9700.060669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9700.060671] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9700.060680] Call Trace: [ 9700.060685] [ 9700.060691] ? show_regs.cold.9+0x22/0x2f [ 9700.060701] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9700.060708] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9700.060712] ? watchdog+0x40/0x40 [ 9700.060717] ? hrtimer_interrupt+0x128/0x2c0 [ 9700.060721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9700.060728] ? apic_timer_interrupt+0xf/0x20 [ 9700.060733] [ 9700.060736] ? native_safe_halt+0x2e/0x50 [ 9700.060740] kvm_wait+0xcd/0xe0 [ 9700.060747] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9700.060756] do_raw_spin_lock+0x177/0x190 [ 9700.060761] _raw_spin_lock+0x21/0x30 [ 9700.060765] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9700.061055] ? do_raw_spin_unlock+0x75/0x190 [ 9700.061062] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9700.061269] ? do_wait_intr+0xf0/0xf0 [ 9700.061291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9700.061498] mdc_close+0x53b/0xd70 [mdc] [ 9700.061537] lmv_close+0x18d/0x490 [lmv] [ 9700.061565] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9700.061669] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9700.061747] ll_md_close+0x274/0x800 [lustre] [ 9700.061830] ll_file_release+0x128/0xc50 [lustre] [ 9700.061910] __fput+0x10d/0x440 [ 9700.061919] ____fput+0x12/0x20 [ 9700.061924] task_work_run+0xc6/0x110 [ 9700.061930] exit_to_usermode_loop+0x1dd/0x1f0 [ 9700.061937] do_syscall_64+0x3d6/0x440 [ 9700.061941] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9700.061947] RIP: 0033:0x7fa749552cd8 [ 9700.061954] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9700.061957] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9700.061962] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9700.061964] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9700.061967] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9700.061969] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9700.061971] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9700.120117] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9700.120132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9700.120135] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9700.121769] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9700.123590] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9700.123669] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9700.123676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9700.123678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9700.123692] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9700.123696] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9700.123701] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9700.123703] RDX: 0000000000005b77 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9700.123706] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9700.123708] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9700.123710] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9700.123713] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9700.123716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9700.123720] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9700.123728] Call Trace: [ 9700.123734] [ 9700.123740] ? show_regs.cold.9+0x22/0x2f [ 9700.123749] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9700.123755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9700.123759] ? watchdog+0x40/0x40 [ 9700.123763] ? hrtimer_interrupt+0x128/0x2c0 [ 9700.123767] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9700.123775] ? apic_timer_interrupt+0xf/0x20 [ 9700.123780] [ 9700.123783] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9700.123788] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9700.123793] do_raw_spin_lock+0x177/0x190 [ 9700.143915] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 [ 9700.212138] _raw_spin_lock+0x21/0x30 [ 9700.212249] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9700.228550] ORIG_RAX: ffffffffffffff13 [ 9700.228562] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9700.228564] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9700.228567] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9700.228569] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9700.228571] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9700.228574] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9700.228578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9700.228581] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9700.228589] Call Trace: [ 9700.228596] [ 9700.228601] ? show_regs.cold.9+0x22/0x2f [ 9700.228613] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9700.228619] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9700.228624] ? watchdog+0x40/0x40 [ 9700.228628] ? hrtimer_interrupt+0x128/0x2c0 [ 9700.228633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9700.228641] ? apic_timer_interrupt+0xf/0x20 [ 9700.228645] [ 9700.228649] ? native_safe_halt+0x2e/0x50 [ 9700.228653] kvm_wait+0xcd/0xe0 [ 9700.228661] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9700.228671] do_raw_spin_lock+0x177/0x190 [ 9700.228676] _raw_spin_lock+0x21/0x30 [ 9700.228680] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9700.230419] ? do_raw_spin_unlock+0x75/0x190 [ 9700.230426] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9700.230636] ? do_wait_intr+0xf0/0xf0 [ 9700.230644] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9700.230856] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9700.241286] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9700.241581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.241691] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9700.241745] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9700.241772] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9700.241799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.241885] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9700.245388] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9700.245434] lmv_intent_open+0x338/0x1150 [lmv] [ 9700.245469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.246996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.247102] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9700.247133] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9700.247217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.265740] obd_mod_free+0x46/0x140 [ptlrpc] [ 9700.266460] mdc_commit_open+0x91/0xb0 [mdc] [ 9700.266497] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9700.266693] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9700.270426] after_reply+0xe2c/0x1780 [ptlrpc] [ 9700.270647] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9700.279542] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9700.279663] ? do_raw_spin_unlock+0x75/0x190 [ 9700.279672] ? _raw_spin_unlock+0x12/0x30 [ 9700.279678] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9700.279705] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9700.279726] ll_atomic_open+0x36a/0xd70 [lustre] [ 9700.279810] lookup_open+0x356/0xb40 [ 9700.280114] do_last+0x313/0xfc0 [ 9700.280120] ? nd_jump_root+0xe5/0x160 [ 9700.280124] ? path_init+0x437/0x520 [ 9700.280129] path_openat+0xf7/0x500 [ 9700.280134] ? do_raw_spin_unlock+0x75/0x190 [ 9700.280139] do_filp_open+0x99/0x140 [ 9700.280146] ? getname_flags+0x6e/0x330 [ 9700.280151] ? __check_object_size+0xff/0x256 [ 9700.280159] ? do_raw_spin_unlock+0x75/0x190 [ 9700.280163] ? _raw_spin_unlock+0x12/0x30 [ 9700.280168] do_sys_openat2+0x2b4/0x410 [ 9700.280172] do_sys_open+0x73/0xa0 [ 9700.280176] __x64_sys_openat+0x24/0x30 [ 9700.280180] do_syscall_64+0xc1/0x440 [ 9700.280187] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9700.280193] RIP: 0033:0x7fa749552332 [ 9700.280201] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9700.280205] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9700.280211] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9700.280213] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9700.280216] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9700.280218] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9700.280221] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9701.878233] ? __next_timer_interrupt+0x160/0x160 [ 9701.880637] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9701.890095] ? do_wait_intr+0xf0/0xf0 [ 9701.895005] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9701.899953] kthread+0x1d1/0x200 [ 9701.906201] ? set_kthread_struct+0x70/0x70 [ 9701.920920] ret_from_fork+0x1f/0x30 [ 9704.060478] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9704.080558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9704.158444] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9704.180497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9704.198067] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9704.218099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9704.228430] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9704.271349] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9704.287436] RAX: 0000000000002983 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9704.301771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9704.306584] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9704.319194] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9704.339936] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9704.358247] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9704.382542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9704.396607] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9704.407963] Call Trace: [ 9704.408822] [ 9704.415501] ? show_regs.cold.9+0x22/0x2f [ 9704.419340] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9704.425984] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9704.441130] ? watchdog+0x40/0x40 [ 9704.445897] ? hrtimer_interrupt+0x128/0x2c0 [ 9704.458792] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9704.475363] ? apic_timer_interrupt+0xf/0x20 [ 9704.476919] [ 9704.477689] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9704.483412] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9704.487170] do_raw_spin_lock+0x177/0x190 [ 9704.495644] _raw_spin_lock+0x21/0x30 [ 9704.499600] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9704.504520] process_one_work+0x2c8/0x700 [ 9704.508810] worker_thread+0x296/0x6e0 [ 9704.522752] ? rescuer_thread+0x570/0x570 [ 9704.525914] kthread+0x1d1/0x200 [ 9704.533231] ? set_kthread_struct+0x70/0x70 [ 9704.538617] ret_from_fork+0x1f/0x30 [ 9705.961077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7788s! [ 9705.970442] Showing busy workqueues and worker pools: [ 9705.973727] workqueue events: flags=0x0 [ 9705.976225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9705.980754] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9705.991391] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9705.997167] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9706.003412] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.006813] pending: kfree_rcu_monitor [ 9706.010754] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9706.016709] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9706.027403] workqueue events_long: flags=0x0 [ 9706.030116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.031897] pending: lu_objects_flush [obdclass] [ 9706.040279] workqueue events_unbound: flags=0x2 [ 9706.044490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9706.047445] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9706.054646] workqueue events_power_efficient: flags=0x80 [ 9706.064431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.071538] pending: check_lifetime [ 9706.077431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.080388] pending: neigh_periodic_work [ 9706.087746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.095306] pending: neigh_periodic_work [ 9706.102265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.108683] pending: do_cache_clean [ 9706.113764] workqueue mm_percpu_wq: flags=0x8 [ 9706.117586] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.128483] pending: vmstat_update [ 9706.130423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.140810] pending: vmstat_update [ 9706.141764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9706.147338] pending: vmstat_update [ 9706.150073] workqueue writeback: flags=0x4a [ 9706.153022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9706.154692] pending: wb_workfn, wb_workfn [ 9706.159501] workqueue xprtiod: flags=0xa [ 9706.168667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9706.170513] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9706.174757] workqueue ipv6_addrconf: flags=0x40008 [ 9706.176179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9706.186486] pending: addrconf_verify_work [ 9706.190399] workqueue knbd0-recv: flags=0x1a [ 9706.192650] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9706.196829] in-flight: 651:recv_work [ 9706.199950] workqueue ptlrpc_pinger: flags=0x2 [ 9706.201181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9706.204919] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9706.206633] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7789s workers=4 idle: 37 31846 28897 [ 9706.212245] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9696s workers=2 idle: 653 [ 9713.859037] rcu: INFO: rcu_sched self-detected stall on CPU [ 9713.861162] rcu: 2-...!: (4405359 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 9713.865439] (t=7800170 jiffies g=258701 q=3710) [ 9713.875154] rcu: rcu_sched kthread starved for 7796965 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9713.884230] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9713.893369] rcu: RCU grace-period kthread stack dump: [ 9713.898622] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9713.907311] Call Trace: [ 9713.908855] __schedule+0x351/0xcb0 [ 9713.912107] schedule+0xc0/0x180 [ 9713.914641] schedule_timeout+0xb4/0x190 [ 9713.916039] ? do_raw_spin_unlock+0x75/0x190 [ 9713.917587] ? __next_timer_interrupt+0x160/0x160 [ 9713.926796] ? prepare_to_swait_event+0xf3/0x140 [ 9713.928396] rcu_gp_kthread+0x8dd/0x15b0 [ 9713.939756] ? force_qs_rnp+0x2c0/0x2c0 [ 9713.943705] kthread+0x1d1/0x200 [ 9713.947629] ? set_kthread_struct+0x70/0x70 [ 9713.949793] ret_from_fork+0x1f/0x30 [ 9713.952813] rcu: Stack dump where RCU GP kthread last ran: [ 9713.956859] Sending NMI from CPU 2 to CPUs 3: [ 9713.963604] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 9713.964658] NMI backtrace for cpu 2 [ 9713.969862] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9713.977476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9713.985968] Call Trace: [ 9713.987230] [ 9713.991202] ? dump_stack+0xbb/0x10e [ 9713.995983] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9714.003687] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9714.008763] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9714.011370] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9714.015136] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9714.018388] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9714.022855] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9714.025294] ? update_process_times+0x6d/0xc0 [ 9714.027766] ? tick_sched_handle.isra.18+0x27/0x90 [ 9714.030184] ? tick_sched_timer+0x53/0xb0 [ 9714.033330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9714.039246] ? tick_sched_handle.isra.18+0x90/0x90 [ 9714.044379] ? hrtimer_interrupt+0x128/0x2c0 [ 9714.048275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9714.050955] ? apic_timer_interrupt+0xf/0x20 [ 9714.053864] [ 9714.055401] ? native_safe_halt+0x2e/0x50 [ 9714.058478] ? kvm_wait+0xcd/0xe0 [ 9714.062220] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9714.064135] ? do_raw_spin_lock+0x177/0x190 [ 9714.070261] ? _raw_spin_lock+0x21/0x30 [ 9714.075313] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9714.079293] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9714.086825] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9714.093078] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9714.095281] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9714.098863] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9714.101707] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9714.107927] ? __next_timer_interrupt+0x160/0x160 [ 9714.113540] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9714.115304] ? do_wait_intr+0xf0/0xf0 [ 9714.124748] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9714.136294] ? kthread+0x1d1/0x200 [ 9714.137230] ? set_kthread_struct+0x70/0x70 [ 9714.138490] ? ret_from_fork+0x1f/0x30 [ 9732.054014] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9732.058635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9732.061684] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9732.084955] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9732.084969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9732.084972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9732.084985] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9732.084989] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9732.084994] RAX: 0000000000005126 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9732.084996] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9732.084998] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9732.085000] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9732.085002] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9732.085005] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9732.085008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9732.085010] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9732.085017] Call Trace: [ 9732.085022] [ 9732.085028] ? show_regs.cold.9+0x22/0x2f [ 9732.085036] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9732.085042] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9732.085045] ? watchdog+0x40/0x40 [ 9732.085049] ? hrtimer_interrupt+0x128/0x2c0 [ 9732.085052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9732.085059] ? apic_timer_interrupt+0xf/0x20 [ 9732.085063] [ 9732.085065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9732.085073] do_raw_spin_lock+0x177/0x190 [ 9732.085077] _raw_spin_lock+0x21/0x30 [ 9732.085081] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9732.089871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9732.089946] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9732.089955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9732.089961] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9732.091084] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9732.091097] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9732.091101] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9732.091105] RAX: 00000000000057c2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9732.091107] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9732.091109] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9732.091111] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9732.091113] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9732.091116] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9732.091119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9732.091121] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9732.093250] Call Trace: [ 9732.093264] [ 9732.093270] ? show_regs.cold.9+0x22/0x2f [ 9732.093283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9732.093290] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9732.093294] ? watchdog+0x40/0x40 [ 9732.093298] ? hrtimer_interrupt+0x128/0x2c0 [ 9732.093302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9732.093313] ? apic_timer_interrupt+0xf/0x20 [ 9732.093322] [ 9732.093325] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9732.093331] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9732.093338] do_raw_spin_lock+0x177/0x190 [ 9732.093343] _raw_spin_lock+0x21/0x30 [ 9732.093348] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9732.093716] process_one_work+0x2c8/0x700 [ 9732.093724] worker_thread+0x296/0x6e0 [ 9732.093728] ? rescuer_thread+0x570/0x570 [ 9732.093732] kthread+0x1d1/0x200 [ 9732.093736] ? set_kthread_struct+0x70/0x70 [ 9732.093740] ret_from_fork+0x1f/0x30 [ 9732.400733] ? do_raw_spin_unlock+0x75/0x190 [ 9732.402749] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9732.409593] ? do_wait_intr+0xf0/0xf0 [ 9732.411896] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9732.414318] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9732.417354] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9732.422709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9732.432267] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9732.435009] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9732.438341] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9732.440565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9732.446690] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9732.459692] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9732.462819] lmv_intent_open+0x338/0x1150 [lmv] [ 9732.464115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9732.471096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9732.473585] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9732.477718] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9732.479759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9732.489653] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9732.490991] ? do_raw_spin_unlock+0x75/0x190 [ 9732.492114] ? _raw_spin_unlock+0x12/0x30 [ 9732.496563] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9732.498029] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9732.499392] ll_atomic_open+0x36a/0xd70 [lustre] [ 9732.506326] lookup_open+0x356/0xb40 [ 9732.507934] do_last+0x313/0xfc0 [ 9732.513735] ? nd_jump_root+0xe5/0x160 [ 9732.517519] ? path_init+0x437/0x520 [ 9732.518950] path_openat+0xf7/0x500 [ 9732.521751] ? do_raw_spin_unlock+0x75/0x190 [ 9732.523051] do_filp_open+0x99/0x140 [ 9732.524698] ? getname_flags+0x6e/0x330 [ 9732.526084] ? __check_object_size+0xff/0x256 [ 9732.540824] ? do_raw_spin_unlock+0x75/0x190 [ 9732.548003] ? _raw_spin_unlock+0x12/0x30 [ 9732.549016] do_sys_openat2+0x2b4/0x410 [ 9732.550123] do_sys_open+0x73/0xa0 [ 9732.560647] __x64_sys_openat+0x24/0x30 [ 9732.562589] do_syscall_64+0xc1/0x440 [ 9732.563528] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9732.565555] RIP: 0033:0x7fa749552332 [ 9732.567658] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9732.578008] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9732.585307] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9732.593979] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9732.603316] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9732.613228] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9732.619047] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9736.682035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7819s! [ 9736.686710] Showing busy workqueues and worker pools: [ 9736.689914] workqueue events: flags=0x0 [ 9736.693647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9736.697646] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9736.704122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9736.710740] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9736.716491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.720576] pending: kfree_rcu_monitor [ 9736.724692] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9736.727826] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9736.735763] workqueue events_long: flags=0x0 [ 9736.744819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.751220] pending: lu_objects_flush [obdclass] [ 9736.754540] workqueue events_unbound: flags=0x2 [ 9736.756230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9736.762978] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9736.772433] workqueue events_power_efficient: flags=0x80 [ 9736.778437] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.785542] pending: check_lifetime [ 9736.789264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.795915] pending: neigh_periodic_work [ 9736.798628] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.803444] pending: neigh_periodic_work [ 9736.810544] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.816841] pending: do_cache_clean [ 9736.819028] workqueue mm_percpu_wq: flags=0x8 [ 9736.824312] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.829676] pending: vmstat_update [ 9736.831471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.835857] pending: vmstat_update [ 9736.841974] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9736.847888] pending: vmstat_update [ 9736.849609] workqueue writeback: flags=0x4a [ 9736.855491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9736.859057] pending: wb_workfn, wb_workfn [ 9736.866068] workqueue xprtiod: flags=0xa [ 9736.872243] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9736.875658] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9736.878618] workqueue ipv6_addrconf: flags=0x40008 [ 9736.883136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9736.890959] pending: addrconf_verify_work [ 9736.895480] workqueue knbd0-recv: flags=0x1a [ 9736.902310] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9736.906353] in-flight: 651:recv_work [ 9736.911665] workqueue ptlrpc_pinger: flags=0x2 [ 9736.917246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9736.922360] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9736.928794] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7819s workers=4 idle: 37 31846 28897 [ 9736.937693] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9726s workers=2 idle: 653 [ 9740.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9740.062028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9740.064916] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9740.098947] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9740.098961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9740.098965] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.098978] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9740.098982] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9740.098987] RAX: 0000000000007bba RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9740.098989] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9740.098992] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9740.098994] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9740.099006] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9740.099010] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9740.099013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9740.099016] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9740.099024] Call Trace: [ 9740.099029] [ 9740.099037] ? show_regs.cold.9+0x22/0x2f [ 9740.099047] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9740.099053] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9740.099058] ? watchdog+0x40/0x40 [ 9740.099061] ? hrtimer_interrupt+0x128/0x2c0 [ 9740.099065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9740.099074] ? apic_timer_interrupt+0xf/0x20 [ 9740.099078] [ 9740.099081] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.099462] do_raw_spin_lock+0x177/0x190 [ 9740.099471] _raw_spin_lock+0x21/0x30 [ 9740.099477] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9740.099842] ? do_raw_spin_unlock+0x75/0x190 [ 9740.099849] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9740.100191] ? do_wait_intr+0xf0/0xf0 [ 9740.100199] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9740.100512] mdc_close+0x53b/0xd70 [mdc] [ 9740.100552] lmv_close+0x18d/0x490 [lmv] [ 9740.100583] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9740.100700] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9740.100963] ll_md_close+0x274/0x800 [lustre] [ 9740.101119] ll_file_release+0x128/0xc50 [lustre] [ 9740.101215] __fput+0x10d/0x440 [ 9740.101224] ____fput+0x12/0x20 [ 9740.101228] task_work_run+0xc6/0x110 [ 9740.101235] exit_to_usermode_loop+0x1dd/0x1f0 [ 9740.101242] do_syscall_64+0x3d6/0x440 [ 9740.101247] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9740.101254] RIP: 0033:0x7fa749552cd8 [ 9740.101262] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9740.101266] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9740.101271] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9740.101274] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9740.101276] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9740.101278] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9740.101281] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9740.392143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9740.424771] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9740.433441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9740.442407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9740.451591] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9740.473226] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9740.479374] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9740.481946] RDX: 0000000000004473 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9740.491379] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9740.502727] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9740.518244] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9740.536206] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9740.556708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9740.558670] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9740.567613] Call Trace: [ 9740.570669] [ 9740.576680] ? show_regs.cold.9+0x22/0x2f [ 9740.582656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9740.593346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9740.596220] ? watchdog+0x40/0x40 [ 9740.598767] ? hrtimer_interrupt+0x128/0x2c0 [ 9740.612287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9740.621705] ? apic_timer_interrupt+0xf/0x20 [ 9740.632773] [ 9740.635241] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9740.659982] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9740.661744] do_raw_spin_lock+0x177/0x190 [ 9740.673887] _raw_spin_lock+0x21/0x30 [ 9740.675240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9740.688467] obd_mod_free+0x46/0x140 [ptlrpc] [ 9740.695060] mdc_commit_open+0x91/0xb0 [mdc] [ 9740.697663] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9740.705990] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9740.720337] after_reply+0xe2c/0x1780 [ptlrpc] [ 9740.733199] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9740.743250] ? __next_timer_interrupt+0x160/0x160 [ 9740.746945] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9740.750617] ? do_wait_intr+0xf0/0xf0 [ 9740.754912] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9740.759703] kthread+0x1d1/0x200 [ 9740.761285] ? set_kthread_struct+0x70/0x70 [ 9740.771641] ret_from_fork+0x1f/0x30 [ 9760.057087] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9760.059270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9760.076243] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9760.081059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9760.084360] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9760.087128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9760.089249] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9760.096545] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9760.099232] RAX: 0000000000001970 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9760.101778] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9760.104298] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9760.107105] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9760.109880] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9760.112664] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9760.115591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9760.117661] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9760.120485] Call Trace: [ 9760.121392] [ 9760.122318] ? show_regs.cold.9+0x22/0x2f [ 9760.123867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9760.125987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9760.127582] ? watchdog+0x40/0x40 [ 9760.129216] ? hrtimer_interrupt+0x128/0x2c0 [ 9760.130886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9760.133203] ? apic_timer_interrupt+0xf/0x20 [ 9760.134929] [ 9760.135634] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9760.137890] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9760.140187] do_raw_spin_lock+0x177/0x190 [ 9760.141615] _raw_spin_lock+0x21/0x30 [ 9760.143083] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9760.144935] process_one_work+0x2c8/0x700 [ 9760.146346] worker_thread+0x296/0x6e0 [ 9760.148071] ? rescuer_thread+0x570/0x570 [ 9760.149946] kthread+0x1d1/0x200 [ 9760.151350] ? set_kthread_struct+0x70/0x70 [ 9760.153270] ret_from_fork+0x1f/0x30 [ 9764.056532] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9764.080923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9764.136237] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9764.144540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9764.154100] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9764.160380] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9764.186679] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9764.192669] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9764.207355] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9764.212333] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9764.229305] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9764.245296] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9764.265505] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9764.279421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9764.282351] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9764.305071] Call Trace: [ 9764.305703] [ 9764.306247] ? show_regs.cold.9+0x22/0x2f [ 9764.307413] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9764.323417] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9764.329447] ? watchdog+0x40/0x40 [ 9764.339052] ? hrtimer_interrupt+0x128/0x2c0 [ 9764.346462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9764.354396] ? apic_timer_interrupt+0xf/0x20 [ 9764.364732] [ 9764.368768] ? native_safe_halt+0x2e/0x50 [ 9764.373858] kvm_wait+0xcd/0xe0 [ 9764.377574] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9764.388361] do_raw_spin_lock+0x177/0x190 [ 9764.402378] _raw_spin_lock+0x21/0x30 [ 9764.410404] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9764.428226] ? do_raw_spin_unlock+0x75/0x190 [ 9764.438284] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9764.442917] ? do_wait_intr+0xf0/0xf0 [ 9764.445371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9764.453844] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9764.455738] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9764.468385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9764.475517] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9764.479858] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9764.486468] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9764.491742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9764.498741] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9764.502866] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9764.521297] lmv_intent_open+0x338/0x1150 [lmv] [ 9764.527346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9764.535697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9764.544321] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9764.549728] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9764.555391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9764.565781] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9764.568835] ? do_raw_spin_unlock+0x75/0x190 [ 9764.570233] ? _raw_spin_unlock+0x12/0x30 [ 9764.576780] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9764.586181] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9764.590185] ll_atomic_open+0x36a/0xd70 [lustre] [ 9764.593298] lookup_open+0x356/0xb40 [ 9764.596241] do_last+0x313/0xfc0 [ 9764.598425] ? nd_jump_root+0xe5/0x160 [ 9764.601810] ? path_init+0x437/0x520 [ 9764.607600] path_openat+0xf7/0x500 [ 9764.613144] ? do_raw_spin_unlock+0x75/0x190 [ 9764.616235] do_filp_open+0x99/0x140 [ 9764.622213] ? getname_flags+0x6e/0x330 [ 9764.625662] ? __check_object_size+0xff/0x256 [ 9764.630774] ? do_raw_spin_unlock+0x75/0x190 [ 9764.633374] ? _raw_spin_unlock+0x12/0x30 [ 9764.641928] do_sys_openat2+0x2b4/0x410 [ 9764.644240] do_sys_open+0x73/0xa0 [ 9764.646300] __x64_sys_openat+0x24/0x30 [ 9764.649337] do_syscall_64+0xc1/0x440 [ 9764.656830] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9764.663440] RIP: 0033:0x7fa749552332 [ 9764.668074] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9764.680235] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9764.692458] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9764.697749] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9764.706392] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9764.717285] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9764.724091] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9767.396237] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7850s! [ 9767.401321] Showing busy workqueues and worker pools: [ 9767.417388] workqueue events: flags=0x0 [ 9767.418538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9767.438770] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9767.456866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9767.466204] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9767.477991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.484930] pending: kfree_rcu_monitor [ 9767.488676] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9767.492348] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9767.496875] workqueue events_long: flags=0x0 [ 9767.498205] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.506005] pending: lu_objects_flush [obdclass] [ 9767.512965] workqueue events_unbound: flags=0x2 [ 9767.514509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9767.528308] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9767.545305] workqueue events_power_efficient: flags=0x80 [ 9767.552113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.559808] pending: check_lifetime [ 9767.571453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.581218] pending: neigh_periodic_work [ 9767.585118] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.600603] pending: neigh_periodic_work [ 9767.606107] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.620980] pending: do_cache_clean [ 9767.632102] workqueue mm_percpu_wq: flags=0x8 [ 9767.636858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.650657] pending: vmstat_update [ 9767.659921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.668376] pending: vmstat_update [ 9767.677479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9767.697354] pending: vmstat_update [ 9767.700624] workqueue writeback: flags=0x4a [ 9767.714752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9767.721022] pending: wb_workfn, wb_workfn [ 9767.730457] workqueue xprtiod: flags=0xa [ 9767.734095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9767.746920] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9767.760910] workqueue ipv6_addrconf: flags=0x40008 [ 9767.765615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9767.773697] pending: addrconf_verify_work [ 9767.781155] workqueue knbd0-recv: flags=0x1a [ 9767.782717] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9767.789089] in-flight: 651:recv_work [ 9767.790483] workqueue ptlrpc_pinger: flags=0x2 [ 9767.797946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9767.802526] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9767.813425] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7850s workers=4 idle: 37 31846 28897 [ 9767.825982] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9757s workers=2 idle: 653 [ 9768.061103] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9768.062624] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9768.077018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9768.077100] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9768.077108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9768.077112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9768.077129] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9768.077134] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9768.077139] RAX: 0000000000004791 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9768.077141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9768.077144] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9768.077146] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9768.077148] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9768.077151] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9768.077155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9768.077157] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9768.077170] Call Trace: [ 9768.077175] [ 9768.077181] ? show_regs.cold.9+0x22/0x2f [ 9768.077191] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9768.077197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9768.077201] ? watchdog+0x40/0x40 [ 9768.077205] ? hrtimer_interrupt+0x128/0x2c0 [ 9768.077209] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9768.077216] ? apic_timer_interrupt+0xf/0x20 [ 9768.077221] [ 9768.077223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9768.077229] do_raw_spin_lock+0x177/0x190 [ 9768.077234] _raw_spin_lock+0x21/0x30 [ 9768.077239] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9768.077513] ? do_raw_spin_unlock+0x75/0x190 [ 9768.077518] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9768.078011] ? do_wait_intr+0xf0/0xf0 [ 9768.078020] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9768.078582] mdc_close+0x53b/0xd70 [mdc] [ 9768.078625] lmv_close+0x18d/0x490 [lmv] [ 9768.090609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9768.090687] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9768.090696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9768.090699] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9768.090715] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9768.090719] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9768.090724] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9768.090726] RDX: 0000000000002107 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9768.090728] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9768.090730] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9768.090733] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9768.090736] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9768.090738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9768.090740] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9768.090750] Call Trace: [ 9768.090757] [ 9768.090763] ? show_regs.cold.9+0x22/0x2f [ 9768.090775] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9768.090781] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9768.090786] ? watchdog+0x40/0x40 [ 9768.090790] ? hrtimer_interrupt+0x128/0x2c0 [ 9768.090793] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9768.090801] ? apic_timer_interrupt+0xf/0x20 [ 9768.090805] [ 9768.090808] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9768.090813] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9768.090817] do_raw_spin_lock+0x177/0x190 [ 9768.090832] _raw_spin_lock+0x21/0x30 [ 9768.090839] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9768.148141] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9768.157179] obd_mod_free+0x46/0x140 [ptlrpc] [ 9768.157456] mdc_commit_open+0x91/0xb0 [mdc] [ 9768.157499] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9768.157828] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9768.158274] after_reply+0xe2c/0x1780 [ptlrpc] [ 9768.158487] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9768.165324] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9768.165457] ll_md_close+0x274/0x800 [lustre] [ 9768.179230] ? __next_timer_interrupt+0x160/0x160 [ 9768.179253] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9768.179533] ? do_wait_intr+0xf0/0xf0 [ 9768.179544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9768.179756] kthread+0x1d1/0x200 [ 9768.179762] ? set_kthread_struct+0x70/0x70 [ 9768.179766] ret_from_fork+0x1f/0x30 [ 9768.737553] ll_file_release+0x128/0xc50 [lustre] [ 9768.739210] __fput+0x10d/0x440 [ 9768.742145] ____fput+0x12/0x20 [ 9768.744255] task_work_run+0xc6/0x110 [ 9768.747482] exit_to_usermode_loop+0x1dd/0x1f0 [ 9768.752020] do_syscall_64+0x3d6/0x440 [ 9768.754383] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9768.756946] RIP: 0033:0x7fa749552cd8 [ 9768.761656] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9768.772184] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9768.778709] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9768.788228] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9768.792382] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9768.795771] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9768.802707] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9788.057008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9788.059773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9788.095885] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9788.101528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9788.106766] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9788.110732] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9788.114024] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9788.123142] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9788.128270] RAX: 000000000000449b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9788.133317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9788.139109] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9788.142699] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9788.146282] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9788.149345] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9788.152975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9788.154993] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9788.159536] Call Trace: [ 9788.160806] [ 9788.161762] ? show_regs.cold.9+0x22/0x2f [ 9788.164936] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9788.168024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9788.169667] ? watchdog+0x40/0x40 [ 9788.170828] ? hrtimer_interrupt+0x128/0x2c0 [ 9788.173051] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9788.175484] ? apic_timer_interrupt+0xf/0x20 [ 9788.177728] [ 9788.178559] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9788.180448] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9788.182890] do_raw_spin_lock+0x177/0x190 [ 9788.184660] _raw_spin_lock+0x21/0x30 [ 9788.186075] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9788.188759] process_one_work+0x2c8/0x700 [ 9788.190762] worker_thread+0x296/0x6e0 [ 9788.192256] ? rescuer_thread+0x570/0x570 [ 9788.193576] kthread+0x1d1/0x200 [ 9788.197292] ? set_kthread_struct+0x70/0x70 [ 9788.199305] ret_from_fork+0x1f/0x30 [ 9792.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9792.056687] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9792.076996] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9792.081877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9792.085900] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9792.088241] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9792.096488] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9792.101554] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9792.105512] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9792.108798] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9792.112160] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9792.115220] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9792.118497] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9792.122179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9792.125016] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9792.128205] Call Trace: [ 9792.129210] [ 9792.130230] ? show_regs.cold.9+0x22/0x2f [ 9792.132211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9792.134388] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9792.136487] ? watchdog+0x40/0x40 [ 9792.137984] ? hrtimer_interrupt+0x128/0x2c0 [ 9792.139602] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9792.140984] ? apic_timer_interrupt+0xf/0x20 [ 9792.143089] [ 9792.143831] ? native_safe_halt+0x2e/0x50 [ 9792.145638] kvm_wait+0xcd/0xe0 [ 9792.146629] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9792.148851] do_raw_spin_lock+0x177/0x190 [ 9792.150652] _raw_spin_lock+0x21/0x30 [ 9792.152414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9792.158705] ? do_raw_spin_unlock+0x75/0x190 [ 9792.165089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9792.168905] ? do_wait_intr+0xf0/0xf0 [ 9792.170662] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9792.173676] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9792.176363] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9792.179053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9792.181280] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9792.183247] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9792.185557] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9792.187638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9792.190063] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9792.193444] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9792.196338] lmv_intent_open+0x338/0x1150 [lmv] [ 9792.198778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9792.201360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9792.204250] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9792.206228] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9792.208503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9792.210962] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9792.213384] ? do_raw_spin_unlock+0x75/0x190 [ 9792.215504] ? _raw_spin_unlock+0x12/0x30 [ 9792.217235] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9792.219681] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9792.222085] ll_atomic_open+0x36a/0xd70 [lustre] [ 9792.224437] lookup_open+0x356/0xb40 [ 9792.226443] do_last+0x313/0xfc0 [ 9792.227892] ? nd_jump_root+0xe5/0x160 [ 9792.229918] ? path_init+0x437/0x520 [ 9792.231835] path_openat+0xf7/0x500 [ 9792.233212] ? do_raw_spin_unlock+0x75/0x190 [ 9792.235251] do_filp_open+0x99/0x140 [ 9792.236994] ? getname_flags+0x6e/0x330 [ 9792.238810] ? __check_object_size+0xff/0x256 [ 9792.240875] ? do_raw_spin_unlock+0x75/0x190 [ 9792.243329] ? _raw_spin_unlock+0x12/0x30 [ 9792.245487] do_sys_openat2+0x2b4/0x410 [ 9792.247341] do_sys_open+0x73/0xa0 [ 9792.248930] __x64_sys_openat+0x24/0x30 [ 9792.251076] do_syscall_64+0xc1/0x440 [ 9792.252820] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9792.255613] RIP: 0033:0x7fa749552332 [ 9792.257015] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9792.265473] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9792.268790] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9792.272351] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9792.275712] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9792.278852] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9792.282459] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9796.058070] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [ 9796.059031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [ 9796.068943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9796.069017] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9796.069024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9796.069027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9796.070117] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9796.075888] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [ 9796.123155] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 [ 9796.137655] crct10dif_pclmul [ 9796.147179] ORIG_RAX: ffffffffffffff13 [ 9796.147190] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9796.147192] RDX: 000000000000193a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9796.147194] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9796.147196] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9796.147198] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9796.147201] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9796.147204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9796.147206] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9796.147214] Call Trace: [ 9796.147221] [ 9796.147226] ? show_regs.cold.9+0x22/0x2f [ 9796.147239] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9796.147245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9796.147250] ? watchdog+0x40/0x40 [ 9796.147254] ? hrtimer_interrupt+0x128/0x2c0 [ 9796.147258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9796.147266] ? apic_timer_interrupt+0xf/0x20 [ 9796.147270] [ 9796.147273] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9796.147279] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9796.147284] do_raw_spin_lock+0x177/0x190 [ 9796.147288] _raw_spin_lock+0x21/0x30 [ 9796.147295] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9796.151850] crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9796.151878] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9796.151886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9796.151889] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9796.151903] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9796.151908] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9796.151913] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9796.151915] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9796.151918] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9796.151919] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9796.151922] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9796.151925] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9796.151928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9796.151930] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9796.151939] Call Trace: [ 9796.151944] [ 9796.151951] ? show_regs.cold.9+0x22/0x2f [ 9796.151962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9796.151968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9796.151972] ? watchdog+0x40/0x40 [ 9796.151977] ? hrtimer_interrupt+0x128/0x2c0 [ 9796.151980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9796.151987] ? apic_timer_interrupt+0xf/0x20 [ 9796.151991] [ 9796.151994] ? native_safe_halt+0x2e/0x50 [ 9796.151998] kvm_wait+0xcd/0xe0 [ 9796.152005] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9796.152015] do_raw_spin_lock+0x177/0x190 [ 9796.152019] _raw_spin_lock+0x21/0x30 [ 9796.152024] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9796.174812] obd_mod_free+0x46/0x140 [ptlrpc] [ 9796.203286] ? do_raw_spin_unlock+0x75/0x190 [ 9796.203310] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9796.207775] ? do_wait_intr+0xf0/0xf0 [ 9796.207785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9796.207988] mdc_close+0x53b/0xd70 [mdc] [ 9796.208031] lmv_close+0x18d/0x490 [lmv] [ 9796.208130] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9796.208410] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9796.208489] ll_md_close+0x274/0x800 [lustre] [ 9796.208570] ll_file_release+0x128/0xc50 [lustre] [ 9796.208646] __fput+0x10d/0x440 [ 9796.208655] ____fput+0x12/0x20 [ 9796.208659] task_work_run+0xc6/0x110 [ 9796.208666] exit_to_usermode_loop+0x1dd/0x1f0 [ 9796.208672] do_syscall_64+0x3d6/0x440 [ 9796.208677] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9796.208685] RIP: 0033:0x7fa749552cd8 [ 9796.208692] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9796.208696] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9796.208701] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9796.208703] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9796.208705] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9796.208707] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9796.208709] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9796.583909] mdc_commit_open+0x91/0xb0 [mdc] [ 9796.590550] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9796.596359] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9796.601655] after_reply+0xe2c/0x1780 [ptlrpc] [ 9796.603851] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9796.611542] ? __next_timer_interrupt+0x160/0x160 [ 9796.613297] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9796.617667] ? do_wait_intr+0xf0/0xf0 [ 9796.619134] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9796.623738] kthread+0x1d1/0x200 [ 9796.626703] ? set_kthread_struct+0x70/0x70 [ 9796.628947] ret_from_fork+0x1f/0x30 [ 9798.113047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7880s! [ 9798.115442] Showing busy workqueues and worker pools: [ 9798.119137] workqueue events: flags=0x0 [ 9798.127688] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9798.133281] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9798.140677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9798.145274] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9798.151718] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.155287] pending: kfree_rcu_monitor [ 9798.157925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9798.161964] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9798.166883] workqueue events_long: flags=0x0 [ 9798.168440] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.172536] pending: lu_objects_flush [obdclass] [ 9798.176164] workqueue events_unbound: flags=0x2 [ 9798.179247] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9798.186751] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9798.197150] workqueue events_power_efficient: flags=0x80 [ 9798.203888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.212714] pending: check_lifetime [ 9798.216899] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.221535] pending: neigh_periodic_work [ 9798.226572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.232946] pending: neigh_periodic_work [ 9798.235705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.245232] pending: do_cache_clean [ 9798.252959] workqueue mm_percpu_wq: flags=0x8 [ 9798.260872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.264964] pending: vmstat_update [ 9798.267219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.270945] pending: vmstat_update [ 9798.272884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9798.276865] pending: vmstat_update [ 9798.279071] workqueue writeback: flags=0x4a [ 9798.281925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9798.285944] pending: wb_workfn, wb_workfn [ 9798.288670] workqueue xprtiod: flags=0xa [ 9798.291271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9798.294955] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9798.300248] workqueue ipv6_addrconf: flags=0x40008 [ 9798.304146] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9798.308164] pending: addrconf_verify_work [ 9798.310972] workqueue knbd0-recv: flags=0x1a [ 9798.314900] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9798.318131] in-flight: 651:recv_work [ 9798.320627] workqueue ptlrpc_pinger: flags=0x2 [ 9798.323166] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9798.327626] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9798.331132] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7881s workers=4 idle: 37 31846 28897 [ 9798.335853] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9788s workers=2 idle: 653 [ 9816.064096] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9816.075548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9816.157976] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9816.166173] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9816.171621] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9816.173765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9816.181855] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9816.200873] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9816.210548] RAX: 0000000000005015 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9816.214825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9816.218093] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9816.221631] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9816.227618] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9816.235598] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9816.242901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9816.248042] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9816.254188] Call Trace: [ 9816.256655] [ 9816.260007] ? show_regs.cold.9+0x22/0x2f [ 9816.268105] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9816.269640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9816.274669] ? watchdog+0x40/0x40 [ 9816.280875] ? hrtimer_interrupt+0x128/0x2c0 [ 9816.283144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9816.293105] ? apic_timer_interrupt+0xf/0x20 [ 9816.297565] [ 9816.298736] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9816.310188] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9816.315483] do_raw_spin_lock+0x177/0x190 [ 9816.319500] _raw_spin_lock+0x21/0x30 [ 9816.320556] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9816.334548] process_one_work+0x2c8/0x700 [ 9816.339581] worker_thread+0x296/0x6e0 [ 9816.344529] ? rescuer_thread+0x570/0x570 [ 9816.348233] kthread+0x1d1/0x200 [ 9816.351063] ? set_kthread_struct+0x70/0x70 [ 9816.354853] ret_from_fork+0x1f/0x30 [ 9824.054012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9824.057995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 9824.058029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9824.059010] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9824.059018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9824.059082] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9824.059088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9824.059162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9824.059176] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9824.059180] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9824.059185] RAX: 0000000000003087 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9824.059187] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9824.059189] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9824.059191] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9824.059194] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9824.059197] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9824.059200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9824.059202] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9824.059210] Call Trace: [ 9824.059216] [ 9824.059221] ? show_regs.cold.9+0x22/0x2f [ 9824.059236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9824.059240] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9824.059243] ? watchdog+0x40/0x40 [ 9824.059246] ? hrtimer_interrupt+0x128/0x2c0 [ 9824.059249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9824.059255] ? apic_timer_interrupt+0xf/0x20 [ 9824.059258] [ 9824.059260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9824.059265] do_raw_spin_lock+0x177/0x190 [ 9824.059268] _raw_spin_lock+0x21/0x30 [ 9824.059271] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9824.059634] ? do_raw_spin_unlock+0x75/0x190 [ 9824.059641] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9824.059839] ? do_wait_intr+0xf0/0xf0 [ 9824.059856] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9824.060056] mdc_close+0x53b/0xd70 [mdc] [ 9824.060159] lmv_close+0x18d/0x490 [lmv] [ 9824.060183] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9824.060275] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9824.060383] ll_md_close+0x274/0x800 [lustre] [ 9824.060437] ll_file_release+0x128/0xc50 [lustre] [ 9824.060490] __fput+0x10d/0x440 [ 9824.060497] ____fput+0x12/0x20 [ 9824.060500] task_work_run+0xc6/0x110 [ 9824.060505] exit_to_usermode_loop+0x1dd/0x1f0 [ 9824.060511] do_syscall_64+0x3d6/0x440 [ 9824.060514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9824.060519] RIP: 0033:0x7fa749552cd8 [ 9824.060525] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9824.060528] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9824.060531] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9824.060533] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9824.060534] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9824.060536] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9824.060538] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9824.062951] mgc(O) [ 9824.066932] Modules linked in: lustre(O) osp(O) ofd(O) [ 9824.070683] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9824.070748] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9824.070756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9824.070759] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9824.070774] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9824.070779] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9824.070784] RAX: 0000000000006f29 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9824.070787] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9824.070789] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9824.070791] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9824.070794] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9824.070797] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9824.070801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9824.070803] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9824.070812] Call Trace: [ 9824.070818] [ 9824.070824] ? show_regs.cold.9+0x22/0x2f [ 9824.070834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9824.070840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9824.070857] ? watchdog+0x40/0x40 [ 9824.070862] ? hrtimer_interrupt+0x128/0x2c0 [ 9824.070865] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9824.070874] ? apic_timer_interrupt+0xf/0x20 [ 9824.070878] [ 9824.070881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9824.070888] do_raw_spin_lock+0x177/0x190 [ 9824.070893] _raw_spin_lock+0x21/0x30 [ 9824.070899] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9824.071323] ? do_raw_spin_unlock+0x75/0x190 [ 9824.071330] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9824.097936] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 9824.103633] ? do_wait_intr+0xf0/0xf0 [ 9824.103656] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9824.108249] ksocklnd(O) lnet(O) libcfs(O) [ 9824.110810] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9824.111115] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9824.119825] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9824.119880] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9824.119888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9824.119891] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9824.119905] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9824.119909] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9824.119914] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9824.119917] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9824.119919] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9824.119922] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9824.119925] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9824.119928] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9824.119931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9824.119934] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9824.119942] Call Trace: [ 9824.119948] [ 9824.119955] ? show_regs.cold.9+0x22/0x2f [ 9824.119964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9824.119971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9824.119975] ? watchdog+0x40/0x40 [ 9824.119980] ? hrtimer_interrupt+0x128/0x2c0 [ 9824.119984] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9824.119993] ? apic_timer_interrupt+0xf/0x20 [ 9824.119998] [ 9824.120001] ? native_safe_halt+0x2e/0x50 [ 9824.120006] kvm_wait+0xcd/0xe0 [ 9824.120014] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9824.120022] do_raw_spin_lock+0x177/0x190 [ 9824.120027] _raw_spin_lock+0x21/0x30 [ 9824.120033] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9824.120497] obd_mod_free+0x46/0x140 [ptlrpc] [ 9824.120770] mdc_commit_open+0x91/0xb0 [mdc] [ 9824.120814] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9824.121032] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9824.121468] after_reply+0xe2c/0x1780 [ptlrpc] [ 9824.123101] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9824.127704] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.127865] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9824.127908] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9824.127940] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9824.127971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.128064] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9824.128493] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9824.128524] lmv_intent_open+0x338/0x1150 [lmv] [ 9824.128624] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.128775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.128872] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9824.128896] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9824.128988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.129079] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9824.133388] ? __next_timer_interrupt+0x160/0x160 [ 9824.133416] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9824.133856] ? do_wait_intr+0xf0/0xf0 [ 9824.133866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9824.134082] kthread+0x1d1/0x200 [ 9824.134089] ? set_kthread_struct+0x70/0x70 [ 9824.134093] ret_from_fork+0x1f/0x30 [ 9824.705404] ? do_raw_spin_unlock+0x75/0x190 [ 9824.710086] ? _raw_spin_unlock+0x12/0x30 [ 9824.712006] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9824.717394] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9824.719705] ll_atomic_open+0x36a/0xd70 [lustre] [ 9824.722437] lookup_open+0x356/0xb40 [ 9824.724222] do_last+0x313/0xfc0 [ 9824.725941] ? nd_jump_root+0xe5/0x160 [ 9824.727497] ? path_init+0x437/0x520 [ 9824.729419] path_openat+0xf7/0x500 [ 9824.731190] ? do_raw_spin_unlock+0x75/0x190 [ 9824.732781] do_filp_open+0x99/0x140 [ 9824.733956] ? getname_flags+0x6e/0x330 [ 9824.735702] ? __check_object_size+0xff/0x256 [ 9824.738317] ? do_raw_spin_unlock+0x75/0x190 [ 9824.751344] ? _raw_spin_unlock+0x12/0x30 [ 9824.763261] do_sys_openat2+0x2b4/0x410 [ 9824.768592] do_sys_open+0x73/0xa0 [ 9824.776504] __x64_sys_openat+0x24/0x30 [ 9824.780767] do_syscall_64+0xc1/0x440 [ 9824.784372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9824.785687] RIP: 0033:0x7fa749552332 [ 9824.791364] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9824.807410] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9824.811261] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9824.815307] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9824.819316] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9824.822728] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9824.825652] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9828.836603] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7911s! [ 9828.839839] Showing busy workqueues and worker pools: [ 9828.842428] workqueue events: flags=0x0 [ 9828.844240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9828.850487] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9828.857234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9828.861173] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9828.866418] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.871156] pending: kfree_rcu_monitor [ 9828.872900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9828.875336] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9828.880424] workqueue events_long: flags=0x0 [ 9828.888719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.892370] pending: lu_objects_flush [obdclass] [ 9828.895591] workqueue events_unbound: flags=0x2 [ 9828.898319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9828.900715] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9828.908504] workqueue events_power_efficient: flags=0x80 [ 9828.913151] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.917446] pending: check_lifetime [ 9828.920129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.926282] pending: neigh_periodic_work [ 9828.929239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.935377] pending: neigh_periodic_work [ 9828.939083] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.941590] pending: do_cache_clean [ 9828.945588] workqueue mm_percpu_wq: flags=0x8 [ 9828.948077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.951380] pending: vmstat_update [ 9828.953281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.963348] pending: vmstat_update [ 9828.968028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9828.976550] pending: vmstat_update [ 9828.979645] workqueue writeback: flags=0x4a [ 9828.982104] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9828.988552] pending: wb_workfn, wb_workfn [ 9828.993054] workqueue xprtiod: flags=0xa [ 9829.001272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9829.008266] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9829.015317] workqueue ipv6_addrconf: flags=0x40008 [ 9829.020114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9829.022670] pending: addrconf_verify_work [ 9829.024721] workqueue knbd0-recv: flags=0x1a [ 9829.026251] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9829.030168] in-flight: 651:recv_work [ 9829.032089] workqueue ptlrpc_pinger: flags=0x2 [ 9829.034095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9829.037199] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9829.039437] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7911s workers=4 idle: 37 31846 28897 [ 9829.044302] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9818s workers=2 idle: 653 [ 9844.057125] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9844.072178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9844.098742] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9844.103074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9844.105956] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9844.107878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9844.109437] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9844.116063] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9844.118201] RAX: 0000000000004786 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9844.120506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9844.123437] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9844.129009] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9844.133767] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9844.137837] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9844.141440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9844.144116] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9844.148317] Call Trace: [ 9844.149432] [ 9844.150193] ? show_regs.cold.9+0x22/0x2f [ 9844.152803] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9844.155480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9844.157634] ? watchdog+0x40/0x40 [ 9844.158956] ? hrtimer_interrupt+0x128/0x2c0 [ 9844.162815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9844.165040] ? apic_timer_interrupt+0xf/0x20 [ 9844.166590] [ 9844.167312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9844.169548] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9844.171720] do_raw_spin_lock+0x177/0x190 [ 9844.173347] _raw_spin_lock+0x21/0x30 [ 9844.174790] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9844.178632] process_one_work+0x2c8/0x700 [ 9844.180553] worker_thread+0x296/0x6e0 [ 9844.182278] ? rescuer_thread+0x570/0x570 [ 9844.184111] kthread+0x1d1/0x200 [ 9844.188079] ? set_kthread_struct+0x70/0x70 [ 9844.190561] ret_from_fork+0x1f/0x30 [ 9852.059008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9852.066469] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9852.066732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9852.066874] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9852.066881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9852.066884] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9852.066898] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9852.066902] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9852.066912] RAX: 0000000000001886 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9852.066915] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9852.066917] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9852.066919] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9852.066922] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9852.066925] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9852.066928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9852.066930] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9852.066937] Call Trace: [ 9852.066942] [ 9852.066949] ? show_regs.cold.9+0x22/0x2f [ 9852.066959] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9852.066965] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9852.066969] ? watchdog+0x40/0x40 [ 9852.066974] ? hrtimer_interrupt+0x128/0x2c0 [ 9852.066978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9852.066985] ? apic_timer_interrupt+0xf/0x20 [ 9852.066989] [ 9852.066992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9852.066999] do_raw_spin_lock+0x177/0x190 [ 9852.067004] _raw_spin_lock+0x21/0x30 [ 9852.067015] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9852.067598] ? do_raw_spin_unlock+0x75/0x190 [ 9852.067605] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9852.067878] ? do_wait_intr+0xf0/0xf0 [ 9852.067886] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9852.069648] mdc_close+0x53b/0xd70 [mdc] [ 9852.069688] lmv_close+0x18d/0x490 [lmv] [ 9852.069716] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9852.069884] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9852.069969] ll_md_close+0x274/0x800 [lustre] [ 9852.070168] ll_file_release+0x128/0xc50 [lustre] [ 9852.070323] __fput+0x10d/0x440 [ 9852.070332] ____fput+0x12/0x20 [ 9852.070336] task_work_run+0xc6/0x110 [ 9852.070344] exit_to_usermode_loop+0x1dd/0x1f0 [ 9852.070436] do_syscall_64+0x3d6/0x440 [ 9852.070442] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9852.070450] RIP: 0033:0x7fa749552cd8 [ 9852.070458] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9852.070461] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9852.070465] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9852.070468] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9852.070479] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9852.070482] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9852.070485] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9852.526933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9852.560947] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9852.566366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9852.573489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9852.575140] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9852.584927] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9852.589126] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9852.594777] RDX: 0000000000005d6d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9852.598277] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9852.602570] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9852.606402] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9852.611278] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9852.615502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9852.618537] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9852.621398] Call Trace: [ 9852.622678] [ 9852.626239] ? show_regs.cold.9+0x22/0x2f [ 9852.629834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9852.634850] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9852.638023] ? watchdog+0x40/0x40 [ 9852.645826] ? hrtimer_interrupt+0x128/0x2c0 [ 9852.647694] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9852.650760] ? apic_timer_interrupt+0xf/0x20 [ 9852.652959] [ 9852.654010] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9852.655554] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9852.658790] do_raw_spin_lock+0x177/0x190 [ 9852.660977] _raw_spin_lock+0x21/0x30 [ 9852.662298] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9852.663792] obd_mod_free+0x46/0x140 [ptlrpc] [ 9852.666915] mdc_commit_open+0x91/0xb0 [mdc] [ 9852.669698] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9852.672596] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9852.676998] after_reply+0xe2c/0x1780 [ptlrpc] [ 9852.680853] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9852.686804] ? __next_timer_interrupt+0x160/0x160 [ 9852.690814] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9852.694673] ? do_wait_intr+0xf0/0xf0 [ 9852.696850] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9852.707966] kthread+0x1d1/0x200 [ 9852.709311] ? set_kthread_struct+0x70/0x70 [ 9852.714393] ret_from_fork+0x1f/0x30 [ 9856.057316] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9856.062733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9856.104901] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9856.118825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9856.130592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9856.134855] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9856.166480] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9856.177917] RAX: 0000000000003f85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9856.184922] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9856.197887] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9856.205584] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9856.217125] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9856.228453] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9856.236910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9856.244140] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9856.250686] Call Trace: [ 9856.251933] [ 9856.253822] ? show_regs.cold.9+0x22/0x2f [ 9856.255977] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9856.258911] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9856.261600] ? watchdog+0x40/0x40 [ 9856.263549] ? hrtimer_interrupt+0x128/0x2c0 [ 9856.265645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9856.267657] ? apic_timer_interrupt+0xf/0x20 [ 9856.269014] [ 9856.269770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9856.272423] do_raw_spin_lock+0x177/0x190 [ 9856.274031] _raw_spin_lock+0x21/0x30 [ 9856.275012] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9856.278889] ? do_raw_spin_unlock+0x75/0x190 [ 9856.280689] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9856.291225] ? do_wait_intr+0xf0/0xf0 [ 9856.293408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9856.296358] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9856.301540] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9856.304695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9856.310047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9856.316315] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9856.320631] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9856.325727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9856.328643] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9856.332769] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9856.336240] lmv_intent_open+0x338/0x1150 [lmv] [ 9856.344329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9856.349828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9856.352767] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9856.363669] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9856.368788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9856.377760] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9856.384562] ? do_raw_spin_unlock+0x75/0x190 [ 9856.391435] ? _raw_spin_unlock+0x12/0x30 [ 9856.392865] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9856.399698] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9856.404924] ll_atomic_open+0x36a/0xd70 [lustre] [ 9856.409724] lookup_open+0x356/0xb40 [ 9856.414034] do_last+0x313/0xfc0 [ 9856.416823] ? nd_jump_root+0xe5/0x160 [ 9856.421167] ? path_init+0x437/0x520 [ 9856.423431] path_openat+0xf7/0x500 [ 9856.429507] ? do_raw_spin_unlock+0x75/0x190 [ 9856.432121] do_filp_open+0x99/0x140 [ 9856.434464] ? getname_flags+0x6e/0x330 [ 9856.436871] ? __check_object_size+0xff/0x256 [ 9856.442104] ? do_raw_spin_unlock+0x75/0x190 [ 9856.444603] ? _raw_spin_unlock+0x12/0x30 [ 9856.447182] do_sys_openat2+0x2b4/0x410 [ 9856.449603] do_sys_open+0x73/0xa0 [ 9856.451437] __x64_sys_openat+0x24/0x30 [ 9856.456062] do_syscall_64+0xc1/0x440 [ 9856.458657] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9856.463323] RIP: 0033:0x7fa749552332 [ 9856.468687] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9856.484635] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9856.490778] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9856.498725] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9856.507482] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9856.516609] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9856.522689] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9859.560366] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7942s! [ 9859.580432] Showing busy workqueues and worker pools: [ 9859.587407] workqueue events: flags=0x0 [ 9859.592699] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9859.602028] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9859.618017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9859.620372] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9859.634566] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.639442] pending: kfree_rcu_monitor [ 9859.644938] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9859.653935] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9859.663867] workqueue events_long: flags=0x0 [ 9859.666427] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.672152] pending: lu_objects_flush [obdclass] [ 9859.681391] workqueue events_unbound: flags=0x2 [ 9859.689274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9859.695494] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9859.711569] workqueue events_power_efficient: flags=0x80 [ 9859.716581] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.725723] pending: check_lifetime [ 9859.732543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.736292] pending: neigh_periodic_work [ 9859.741962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.746530] pending: neigh_periodic_work [ 9859.748150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.760767] pending: do_cache_clean [ 9859.763616] workqueue mm_percpu_wq: flags=0x8 [ 9859.768717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.774794] pending: vmstat_update [ 9859.776803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.782832] pending: vmstat_update [ 9859.785689] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9859.792693] pending: vmstat_update [ 9859.795084] workqueue writeback: flags=0x4a [ 9859.798133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9859.803302] pending: wb_workfn, wb_workfn [ 9859.806949] workqueue xprtiod: flags=0xa [ 9859.809700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9859.813489] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9859.817441] workqueue ipv6_addrconf: flags=0x40008 [ 9859.820397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9859.823583] pending: addrconf_verify_work [ 9859.826276] workqueue knbd0-recv: flags=0x1a [ 9859.828492] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9859.833533] in-flight: 651:recv_work [ 9859.835819] workqueue ptlrpc_pinger: flags=0x2 [ 9859.841825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9859.845509] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9859.848680] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7942s workers=4 idle: 37 31846 28897 [ 9859.853409] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9849s workers=2 idle: 653 [ 9872.057873] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9872.063430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9872.113038] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9872.117574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9872.123907] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9872.125536] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9872.127405] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9872.146127] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9872.148489] RAX: 0000000000007f87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9872.151587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9872.154373] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9872.165393] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9872.167903] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9872.183976] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9872.187315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9872.191334] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9872.195013] Call Trace: [ 9872.196125] [ 9872.200057] ? show_regs.cold.9+0x22/0x2f [ 9872.204202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9872.206611] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9872.211324] ? watchdog+0x40/0x40 [ 9872.213564] ? hrtimer_interrupt+0x128/0x2c0 [ 9872.215594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9872.218552] ? apic_timer_interrupt+0xf/0x20 [ 9872.220860] [ 9872.222181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9872.225063] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9872.227491] do_raw_spin_lock+0x177/0x190 [ 9872.229436] _raw_spin_lock+0x21/0x30 [ 9872.231273] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9872.233592] process_one_work+0x2c8/0x700 [ 9872.237810] worker_thread+0x296/0x6e0 [ 9872.242162] ? rescuer_thread+0x570/0x570 [ 9872.244915] kthread+0x1d1/0x200 [ 9872.247283] ? set_kthread_struct+0x70/0x70 [ 9872.249502] ret_from_fork+0x1f/0x30 [ 9880.058328] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9880.059376] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9880.060944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9880.061017] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9880.061023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9880.061026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9880.061039] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9880.061043] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9880.061047] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9880.061049] RDX: 0000000000006479 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9880.061052] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9880.061054] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9880.061056] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9880.061059] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9880.061062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9880.061064] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9880.061071] Call Trace: [ 9880.061076] [ 9880.061080] ? show_regs.cold.9+0x22/0x2f [ 9880.061088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9880.061094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9880.061097] ? watchdog+0x40/0x40 [ 9880.061101] ? hrtimer_interrupt+0x128/0x2c0 [ 9880.061106] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9880.061114] ? apic_timer_interrupt+0xf/0x20 [ 9880.061117] [ 9880.061119] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9880.061122] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9880.061125] do_raw_spin_lock+0x177/0x190 [ 9880.061128] _raw_spin_lock+0x21/0x30 [ 9880.061133] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9880.061404] obd_mod_free+0x46/0x140 [ptlrpc] [ 9880.061527] mdc_commit_open+0x91/0xb0 [mdc] [ 9880.061553] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9880.061805] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9880.062008] after_reply+0xe2c/0x1780 [ptlrpc] [ 9880.062389] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9880.062514] ? __next_timer_interrupt+0x160/0x160 [ 9880.062521] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9880.063314] ? do_wait_intr+0xf0/0xf0 [ 9880.063321] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9880.063449] kthread+0x1d1/0x200 [ 9880.063453] ? set_kthread_struct+0x70/0x70 [ 9880.063455] ret_from_fork+0x1f/0x30 [ 9880.156483] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9880.170338] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9880.174357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9880.176514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9880.179332] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9880.186390] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9880.188514] RAX: 0000000000007085 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9880.190328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9880.192388] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9880.194103] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9880.195802] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9880.197690] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9880.200396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9880.201718] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9880.204296] Call Trace: [ 9880.205524] [ 9880.205992] ? show_regs.cold.9+0x22/0x2f [ 9880.207398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9880.209079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9880.210360] ? watchdog+0x40/0x40 [ 9880.211799] ? hrtimer_interrupt+0x128/0x2c0 [ 9880.213036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9880.214995] ? apic_timer_interrupt+0xf/0x20 [ 9880.215977] [ 9880.216574] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9880.217978] do_raw_spin_lock+0x177/0x190 [ 9880.218971] _raw_spin_lock+0x21/0x30 [ 9880.220021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9880.221675] ? do_raw_spin_unlock+0x75/0x190 [ 9880.223055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9880.224481] ? do_wait_intr+0xf0/0xf0 [ 9880.225458] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9880.227211] mdc_close+0x53b/0xd70 [mdc] [ 9880.228236] lmv_close+0x18d/0x490 [lmv] [ 9880.229306] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9880.231210] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9880.232511] ll_md_close+0x274/0x800 [lustre] [ 9880.234124] ll_file_release+0x128/0xc50 [lustre] [ 9880.236298] __fput+0x10d/0x440 [ 9880.237138] ____fput+0x12/0x20 [ 9880.238240] task_work_run+0xc6/0x110 [ 9880.239255] exit_to_usermode_loop+0x1dd/0x1f0 [ 9880.240325] do_syscall_64+0x3d6/0x440 [ 9880.241331] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9880.242871] RIP: 0033:0x7fa749552cd8 [ 9880.244099] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9880.250114] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9880.252508] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9880.254486] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9880.256789] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9880.258708] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9880.261265] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9884.054542] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9884.065184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9884.110532] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9884.114177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9884.117711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9884.119976] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9884.127206] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9884.130398] RAX: 00000000000011f9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9884.133351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9884.136055] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9884.138933] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9884.141742] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9884.144615] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9884.147459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9884.149916] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9884.152829] Call Trace: [ 9884.153872] [ 9884.154905] ? show_regs.cold.9+0x22/0x2f [ 9884.156310] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9884.157896] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9884.159908] ? watchdog+0x40/0x40 [ 9884.161091] ? hrtimer_interrupt+0x128/0x2c0 [ 9884.163122] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9884.165279] ? apic_timer_interrupt+0xf/0x20 [ 9884.166982] [ 9884.167982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9884.169483] do_raw_spin_lock+0x177/0x190 [ 9884.170717] _raw_spin_lock+0x21/0x30 [ 9884.171690] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9884.173960] ? do_raw_spin_unlock+0x75/0x190 [ 9884.175706] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9884.177981] ? do_wait_intr+0xf0/0xf0 [ 9884.179571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9884.181440] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9884.182881] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9884.184742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9884.187146] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9884.189058] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9884.191146] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9884.192734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9884.194953] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9884.197722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9884.199606] lmv_intent_open+0x338/0x1150 [lmv] [ 9884.201446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9884.203253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9884.205226] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9884.206424] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9884.208068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9884.210051] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9884.211740] ? do_raw_spin_unlock+0x75/0x190 [ 9884.213218] ? _raw_spin_unlock+0x12/0x30 [ 9884.214868] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9884.216674] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9884.218736] ll_atomic_open+0x36a/0xd70 [lustre] [ 9884.220366] lookup_open+0x356/0xb40 [ 9884.221812] do_last+0x313/0xfc0 [ 9884.222987] ? nd_jump_root+0xe5/0x160 [ 9884.224412] ? path_init+0x437/0x520 [ 9884.225627] path_openat+0xf7/0x500 [ 9884.226914] ? do_raw_spin_unlock+0x75/0x190 [ 9884.228689] do_filp_open+0x99/0x140 [ 9884.229861] ? getname_flags+0x6e/0x330 [ 9884.231464] ? __check_object_size+0xff/0x256 [ 9884.233056] ? do_raw_spin_unlock+0x75/0x190 [ 9884.234256] ? _raw_spin_unlock+0x12/0x30 [ 9884.236174] do_sys_openat2+0x2b4/0x410 [ 9884.237980] do_sys_open+0x73/0xa0 [ 9884.239543] __x64_sys_openat+0x24/0x30 [ 9884.241007] do_syscall_64+0xc1/0x440 [ 9884.242149] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9884.244241] RIP: 0033:0x7fa749552332 [ 9884.245810] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9884.253340] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9884.256382] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9884.259429] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9884.262440] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9884.265131] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9884.268067] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9890.273048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7973s! [ 9890.277126] Showing busy workqueues and worker pools: [ 9890.278882] workqueue events: flags=0x0 [ 9890.279963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9890.282817] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9890.288078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9890.295760] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9890.304978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.310831] pending: kfree_rcu_monitor [ 9890.312520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9890.317982] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9890.323271] workqueue events_long: flags=0x0 [ 9890.326431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.328184] pending: lu_objects_flush [obdclass] [ 9890.338037] workqueue events_unbound: flags=0x2 [ 9890.340255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9890.343887] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9890.351235] workqueue events_power_efficient: flags=0x80 [ 9890.352661] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.355902] pending: check_lifetime [ 9890.358042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.361614] pending: neigh_periodic_work [ 9890.373369] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.379391] pending: neigh_periodic_work [ 9890.383151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.392626] pending: do_cache_clean [ 9890.395610] workqueue mm_percpu_wq: flags=0x8 [ 9890.400320] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.408078] pending: vmstat_update [ 9890.410045] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.412637] pending: vmstat_update [ 9890.413941] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9890.418221] pending: vmstat_update [ 9890.420624] workqueue writeback: flags=0x4a [ 9890.424257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9890.426882] pending: wb_workfn, wb_workfn [ 9890.429312] workqueue xprtiod: flags=0xa [ 9890.432744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9890.435022] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9890.438126] workqueue ipv6_addrconf: flags=0x40008 [ 9890.440261] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9890.442826] pending: addrconf_verify_work [ 9890.444110] workqueue knbd0-recv: flags=0x1a [ 9890.445305] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9890.447631] in-flight: 651:recv_work [ 9890.451324] workqueue ptlrpc_pinger: flags=0x2 [ 9890.453000] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9890.456049] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9890.458160] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7973s workers=4 idle: 37 31846 28897 [ 9890.460304] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9880s workers=2 idle: 653 [ 9893.862117] rcu: INFO: rcu_sched self-detected stall on CPU [ 9893.864898] rcu: 2-...!: (4496896 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [ 9893.870317] (t=7980173 jiffies g=258701 q=3778) [ 9893.872753] rcu: rcu_sched kthread starved for 7976968 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 9893.878609] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9893.883899] rcu: RCU grace-period kthread stack dump: [ 9893.887085] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9893.896414] Call Trace: [ 9893.898139] __schedule+0x351/0xcb0 [ 9893.902679] schedule+0xc0/0x180 [ 9893.904628] schedule_timeout+0xb4/0x190 [ 9893.906612] ? do_raw_spin_unlock+0x75/0x190 [ 9893.908696] ? __next_timer_interrupt+0x160/0x160 [ 9893.911293] ? prepare_to_swait_event+0xf3/0x140 [ 9893.914280] rcu_gp_kthread+0x8dd/0x15b0 [ 9893.916091] ? force_qs_rnp+0x2c0/0x2c0 [ 9893.918078] kthread+0x1d1/0x200 [ 9893.919729] ? set_kthread_struct+0x70/0x70 [ 9893.922580] ret_from_fork+0x1f/0x30 [ 9893.926736] rcu: Stack dump where RCU GP kthread last ran: [ 9893.934011] NMI backtrace for cpu 2 [ 9893.934925] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9893.941736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9893.950428] Call Trace: [ 9893.952043] [ 9893.952821] ? dump_stack+0xbb/0x10e [ 9893.954142] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9893.961181] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9893.967129] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9893.973180] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9893.980186] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9893.985662] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 9893.994362] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 9893.996313] ? update_process_times+0x6d/0xc0 [ 9894.000667] ? tick_sched_handle.isra.18+0x27/0x90 [ 9894.003440] ? tick_sched_timer+0x53/0xb0 [ 9894.005202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9894.007413] ? tick_sched_handle.isra.18+0x90/0x90 [ 9894.010577] ? hrtimer_interrupt+0x128/0x2c0 [ 9894.013887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9894.016447] ? apic_timer_interrupt+0xf/0x20 [ 9894.019170] [ 9894.020521] ? native_safe_halt+0x2e/0x50 [ 9894.022720] ? kvm_wait+0xcd/0xe0 [ 9894.024534] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9894.027461] ? do_raw_spin_lock+0x177/0x190 [ 9894.033874] ? _raw_spin_lock+0x21/0x30 [ 9894.035244] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9894.042607] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9894.043985] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9894.051965] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9894.053693] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9894.057635] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9894.061871] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9894.065147] ? __next_timer_interrupt+0x160/0x160 [ 9894.073883] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9894.079589] ? do_wait_intr+0xf0/0xf0 [ 9894.084768] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9894.092680] ? kthread+0x1d1/0x200 [ 9894.097034] ? set_kthread_struct+0x70/0x70 [ 9894.099972] ? ret_from_fork+0x1f/0x30 [ 9894.105608] NMI backtrace for cpu 2 [ 9894.111673] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9894.121794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9894.130698] Call Trace: [ 9894.134550] [ 9894.139545] ? dump_stack+0xbb/0x10e [ 9894.145700] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9894.154295] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9894.160560] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9894.169258] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9894.175124] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9894.184739] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9894.186845] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9894.191794] ? update_process_times+0x6d/0xc0 [ 9894.195167] ? tick_sched_handle.isra.18+0x27/0x90 [ 9894.203875] ? tick_sched_timer+0x53/0xb0 [ 9894.206914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9894.208981] ? tick_sched_handle.isra.18+0x90/0x90 [ 9894.211533] ? hrtimer_interrupt+0x128/0x2c0 [ 9894.216857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9894.224111] ? apic_timer_interrupt+0xf/0x20 [ 9894.225701] [ 9894.226475] ? native_safe_halt+0x2e/0x50 [ 9894.229778] ? kvm_wait+0xcd/0xe0 [ 9894.232110] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9894.234668] ? do_raw_spin_lock+0x177/0x190 [ 9894.240769] ? _raw_spin_lock+0x21/0x30 [ 9894.244754] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9894.250980] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9894.257255] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9894.262655] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9894.267138] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9894.274555] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9894.282508] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9894.288683] ? __next_timer_interrupt+0x160/0x160 [ 9894.294570] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9894.302885] ? do_wait_intr+0xf0/0xf0 [ 9894.304881] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9894.308576] ? kthread+0x1d1/0x200 [ 9894.311650] ? set_kthread_struct+0x70/0x70 [ 9894.323761] ? ret_from_fork+0x1f/0x30 [ 9900.058314] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9900.062064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9900.104109] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9900.112018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9900.121496] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9900.125579] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9900.127478] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9900.137200] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9900.140161] RAX: 0000000000007f87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9900.143724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9900.148647] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9900.152837] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9900.156999] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9900.159226] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9900.164651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9900.167903] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9900.171497] Call Trace: [ 9900.172859] [ 9900.175753] ? show_regs.cold.9+0x22/0x2f [ 9900.177261] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9900.180155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9900.182655] ? watchdog+0x40/0x40 [ 9900.184762] ? hrtimer_interrupt+0x128/0x2c0 [ 9900.186353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9900.187862] ? apic_timer_interrupt+0xf/0x20 [ 9900.189900] [ 9900.190939] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9900.193471] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9900.195859] do_raw_spin_lock+0x177/0x190 [ 9900.197656] _raw_spin_lock+0x21/0x30 [ 9900.200740] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9900.204972] process_one_work+0x2c8/0x700 [ 9900.207367] worker_thread+0x296/0x6e0 [ 9900.209054] ? rescuer_thread+0x570/0x570 [ 9900.210579] kthread+0x1d1/0x200 [ 9900.212946] ? set_kthread_struct+0x70/0x70 [ 9900.215377] ret_from_fork+0x1f/0x30 [ 9908.059182] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9908.066911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9908.097281] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9908.103291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9908.112424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9908.116090] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9908.126894] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9908.132835] RAX: 0000000000007b8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9908.135585] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9908.138699] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9908.141491] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9908.144478] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9908.149309] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9908.154868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9908.158743] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9908.166298] Call Trace: [ 9908.167095] [ 9908.168320] ? show_regs.cold.9+0x22/0x2f [ 9908.169736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9908.171758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9908.173387] ? watchdog+0x40/0x40 [ 9908.176224] ? hrtimer_interrupt+0x128/0x2c0 [ 9908.177720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9908.179874] ? apic_timer_interrupt+0xf/0x20 [ 9908.182738] [ 9908.185892] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9908.189190] do_raw_spin_lock+0x177/0x190 [ 9908.190315] _raw_spin_lock+0x21/0x30 [ 9908.191245] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9908.194703] ? do_raw_spin_unlock+0x75/0x190 [ 9908.195922] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9908.198776] ? do_wait_intr+0xf0/0xf0 [ 9908.204483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9908.209539] mdc_close+0x53b/0xd70 [mdc] [ 9908.211208] lmv_close+0x18d/0x490 [lmv] [ 9908.215851] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9908.218928] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9908.221019] ll_md_close+0x274/0x800 [lustre] [ 9908.222309] ll_file_release+0x128/0xc50 [lustre] [ 9908.228024] __fput+0x10d/0x440 [ 9908.229058] ____fput+0x12/0x20 [ 9908.232049] task_work_run+0xc6/0x110 [ 9908.233448] exit_to_usermode_loop+0x1dd/0x1f0 [ 9908.237484] do_syscall_64+0x3d6/0x440 [ 9908.239700] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9908.243060] RIP: 0033:0x7fa749552cd8 [ 9908.246693] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9908.262854] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9908.267443] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9908.270900] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9908.274783] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9908.282451] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9908.289717] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9916.054015] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [ 9916.057893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9916.110511] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9916.126759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9916.137846] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9916.140405] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9916.148712] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9916.153869] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9916.160803] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9916.165926] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9916.170473] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9916.174536] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9916.179751] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9916.183503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9916.187668] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9916.190718] Call Trace: [ 9916.197461] [ 9916.198252] ? show_regs.cold.9+0x22/0x2f [ 9916.199722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9916.202244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9916.204664] ? watchdog+0x40/0x40 [ 9916.205870] ? hrtimer_interrupt+0x128/0x2c0 [ 9916.212329] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9916.216045] ? apic_timer_interrupt+0xf/0x20 [ 9916.219008] [ 9916.221659] ? native_safe_halt+0x2e/0x50 [ 9916.227508] kvm_wait+0xcd/0xe0 [ 9916.232471] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9916.237795] do_raw_spin_lock+0x177/0x190 [ 9916.241731] _raw_spin_lock+0x21/0x30 [ 9916.242990] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9916.248489] ? do_raw_spin_unlock+0x75/0x190 [ 9916.252460] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9916.257805] ? do_wait_intr+0xf0/0xf0 [ 9916.262322] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9916.267621] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9916.278951] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9916.283816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9916.289591] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9916.293874] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9916.299763] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9916.301414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9916.310685] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9916.318359] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9916.324598] lmv_intent_open+0x338/0x1150 [lmv] [ 9916.328925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9916.334564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9916.342359] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9916.346849] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9916.354410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9916.361656] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9916.367650] ? do_raw_spin_unlock+0x75/0x190 [ 9916.372656] ? _raw_spin_unlock+0x12/0x30 [ 9916.375462] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9916.380525] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9916.387989] ll_atomic_open+0x36a/0xd70 [lustre] [ 9916.391588] lookup_open+0x356/0xb40 [ 9916.396494] do_last+0x313/0xfc0 [ 9916.397412] ? nd_jump_root+0xe5/0x160 [ 9916.401572] ? path_init+0x437/0x520 [ 9916.403669] path_openat+0xf7/0x500 [ 9916.406908] ? do_raw_spin_unlock+0x75/0x190 [ 9916.411872] do_filp_open+0x99/0x140 [ 9916.417800] ? getname_flags+0x6e/0x330 [ 9916.419517] ? __check_object_size+0xff/0x256 [ 9916.422767] ? do_raw_spin_unlock+0x75/0x190 [ 9916.429736] ? _raw_spin_unlock+0x12/0x30 [ 9916.433401] do_sys_openat2+0x2b4/0x410 [ 9916.435714] do_sys_open+0x73/0xa0 [ 9916.437639] __x64_sys_openat+0x24/0x30 [ 9916.441527] do_syscall_64+0xc1/0x440 [ 9916.443563] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9916.447812] RIP: 0033:0x7fa749552332 [ 9916.450626] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9916.465441] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9916.468264] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9916.470959] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9916.474174] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9916.478622] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9916.482739] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9920.058295] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9920.064547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9920.092496] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9920.098469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9920.109632] RIP: 0010:load_balance+0x725/0x1090 [ 9920.118244] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [ 9920.132018] RSP: 0018:ffffb893c0d58de0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9920.134375] RAX: 0000000000000006 RBX: ffff8d7358e000b0 RCX: ffff8d7382133440 [ 9920.139602] RDX: 0000000000000008 RSI: ffff8d7358e00000 RDI: 0000000000000206 [ 9920.144727] RBP: ffffb893c0d58e90 R08: ffff8d7359110000 R09: 0000000000000000 [ 9920.149917] R10: ffff8d7358e00098 R11: 0000000000009418 R12: ffff8d7358e00000 [ 9920.157270] R13: ffff8d73821b3e08 R14: 0000000000000002 R15: ffff8d73821b3380 [ 9920.160687] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9920.168958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9920.171864] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9920.181425] Call Trace: [ 9920.184453] [ 9920.186583] ? show_regs.cold.9+0x22/0x2f [ 9920.194622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9920.203415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9920.211667] ? watchdog+0x40/0x40 [ 9920.213705] ? hrtimer_interrupt+0x128/0x2c0 [ 9920.219463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9920.222750] ? apic_timer_interrupt+0xf/0x20 [ 9920.229671] ? apic_timer_interrupt+0xa/0x20 [ 9920.232967] ? load_balance+0x725/0x1090 [ 9920.235663] rebalance_domains+0x348/0x580 [ 9920.238612] ? kvm_clock_get_cycles+0x2c/0x50 [ 9920.247652] run_rebalance_domains+0x68/0xa0 [ 9920.252637] __do_softirq+0x134/0x4e8 [ 9920.257860] irq_exit_rcu+0x1c6/0x1e0 [ 9920.265429] irq_exit+0x12/0x30 [ 9920.269871] smp_apic_timer_interrupt+0xe0/0x270 [ 9920.280078] apic_timer_interrupt+0xf/0x20 [ 9920.281439] [ 9920.282366] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9920.286049] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9920.296645] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9920.301547] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9920.305662] RDX: 0000000000007f7e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9920.308008] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9920.310299] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9920.313478] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9920.324792] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9920.327396] do_raw_spin_lock+0x177/0x190 [ 9920.331304] _raw_spin_lock+0x21/0x30 [ 9920.335310] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9920.340601] obd_mod_free+0x46/0x140 [ptlrpc] [ 9920.345185] mdc_commit_open+0x91/0xb0 [mdc] [ 9920.350624] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9920.354830] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9920.360515] after_reply+0xe2c/0x1780 [ptlrpc] [ 9920.364056] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9920.369083] ? __next_timer_interrupt+0x160/0x160 [ 9920.374523] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9920.376301] ? do_wait_intr+0xf0/0xf0 [ 9920.386998] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9920.391213] kthread+0x1d1/0x200 [ 9920.395852] ? set_kthread_struct+0x70/0x70 [ 9920.403202] ret_from_fork+0x1f/0x30 [ 9920.483757] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8003s! [ 9920.488990] Showing busy workqueues and worker pools: [ 9920.493734] workqueue events: flags=0x0 [ 9920.497275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9920.505516] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9920.514477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9920.520859] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9920.533569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.536176] pending: kfree_rcu_monitor [ 9920.537189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9920.539060] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9920.548077] workqueue events_long: flags=0x0 [ 9920.552245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.558876] pending: lu_objects_flush [obdclass] [ 9920.564275] workqueue events_unbound: flags=0x2 [ 9920.569466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9920.572378] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9920.576797] workqueue events_power_efficient: flags=0x80 [ 9920.579773] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.583079] pending: check_lifetime [ 9920.584990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.589549] pending: neigh_periodic_work [ 9920.595476] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.599524] pending: neigh_periodic_work [ 9920.601795] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.608728] pending: do_cache_clean [ 9920.613717] workqueue mm_percpu_wq: flags=0x8 [ 9920.615053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.619319] pending: vmstat_update [ 9920.621384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.627500] pending: vmstat_update [ 9920.630055] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9920.633306] pending: vmstat_update [ 9920.634791] workqueue writeback: flags=0x4a [ 9920.636550] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9920.638351] pending: wb_workfn, wb_workfn [ 9920.639846] workqueue xprtiod: flags=0xa [ 9920.641759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9920.644503] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9920.650403] workqueue ipv6_addrconf: flags=0x40008 [ 9920.652032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9920.660219] pending: addrconf_verify_work [ 9920.664390] workqueue knbd0-recv: flags=0x1a [ 9920.667368] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9920.670519] in-flight: 651:recv_work [ 9920.673127] workqueue ptlrpc_pinger: flags=0x2 [ 9920.675399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9920.683765] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9920.690740] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8003s workers=4 idle: 37 31846 28897 [ 9920.700459] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9910s workers=2 idle: 653 [ 9928.057571] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9928.060387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9928.106995] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9928.116977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9928.124091] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9928.131949] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9928.137131] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9928.156258] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9928.160794] RAX: 0000000000001055 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9928.168306] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9928.179048] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9928.183924] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9928.192825] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9928.198835] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9928.208535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9928.214278] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9928.220258] Call Trace: [ 9928.221770] [ 9928.223224] ? show_regs.cold.9+0x22/0x2f [ 9928.228509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9928.238084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9928.246042] ? watchdog+0x40/0x40 [ 9928.247231] ? hrtimer_interrupt+0x128/0x2c0 [ 9928.271423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9928.273177] ? apic_timer_interrupt+0xf/0x20 [ 9928.283755] [ 9928.284560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9928.291721] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9928.295577] do_raw_spin_lock+0x177/0x190 [ 9928.300287] _raw_spin_lock+0x21/0x30 [ 9928.304105] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9928.310232] process_one_work+0x2c8/0x700 [ 9928.316325] worker_thread+0x296/0x6e0 [ 9928.321217] ? rescuer_thread+0x570/0x570 [ 9928.325133] kthread+0x1d1/0x200 [ 9928.327138] ? set_kthread_struct+0x70/0x70 [ 9928.330340] ret_from_fork+0x1f/0x30 [ 9936.059034] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9936.066402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9936.103754] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9936.115573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9936.127750] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9936.131770] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9936.150987] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9936.158514] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9936.164584] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9936.175085] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9936.185327] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9936.195125] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9936.202441] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9936.209174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9936.217310] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9936.227281] Call Trace: [ 9936.229566] [ 9936.231538] ? show_regs.cold.9+0x22/0x2f [ 9936.237927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9936.239097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9936.240345] ? watchdog+0x40/0x40 [ 9936.243411] ? hrtimer_interrupt+0x128/0x2c0 [ 9936.250145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9936.256964] ? apic_timer_interrupt+0xf/0x20 [ 9936.261639] [ 9936.263986] ? native_safe_halt+0x2e/0x50 [ 9936.265787] kvm_wait+0xcd/0xe0 [ 9936.267376] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9936.271617] do_raw_spin_lock+0x177/0x190 [ 9936.273141] _raw_spin_lock+0x21/0x30 [ 9936.275105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9936.279347] ? do_raw_spin_unlock+0x75/0x190 [ 9936.283368] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9936.285465] ? do_wait_intr+0xf0/0xf0 [ 9936.289754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9936.294311] mdc_close+0x53b/0xd70 [mdc] [ 9936.296382] lmv_close+0x18d/0x490 [lmv] [ 9936.304059] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9936.315421] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9936.325080] ll_md_close+0x274/0x800 [lustre] [ 9936.333908] ll_file_release+0x128/0xc50 [lustre] [ 9936.339110] __fput+0x10d/0x440 [ 9936.340175] ____fput+0x12/0x20 [ 9936.341897] task_work_run+0xc6/0x110 [ 9936.344973] exit_to_usermode_loop+0x1dd/0x1f0 [ 9936.348448] do_syscall_64+0x3d6/0x440 [ 9936.350028] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9936.351710] RIP: 0033:0x7fa749552cd8 [ 9936.360322] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9936.381421] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9936.386426] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9936.390384] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9936.393997] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9936.397848] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9936.401421] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9948.055194] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9948.058055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9948.066712] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9948.066791] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9948.066799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9948.066802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.066818] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9948.066822] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9948.066827] RAX: 0000000000002b6d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9948.066829] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9948.066831] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9948.066833] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9948.066835] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9948.066839] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9948.066842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9948.066844] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9948.066853] Call Trace: [ 9948.066859] [ 9948.066864] ? show_regs.cold.9+0x22/0x2f [ 9948.066874] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9948.066882] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9948.066886] ? watchdog+0x40/0x40 [ 9948.066891] ? hrtimer_interrupt+0x128/0x2c0 [ 9948.066895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9948.066904] ? apic_timer_interrupt+0xf/0x20 [ 9948.066910] [ 9948.066913] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.066921] do_raw_spin_lock+0x177/0x190 [ 9948.066926] _raw_spin_lock+0x21/0x30 [ 9948.066931] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9948.067211] ? do_raw_spin_unlock+0x75/0x190 [ 9948.067217] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9948.067429] ? do_wait_intr+0xf0/0xf0 [ 9948.067438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9948.067642] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9948.078692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9948.078787] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9948.078794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9948.078797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9948.078811] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9948.078815] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9948.078820] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9948.078823] RDX: 000000000000294b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9948.078826] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9948.078828] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9948.078831] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9948.078834] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9948.078837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9948.078840] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9948.078849] Call Trace: [ 9948.078854] [ 9948.078860] ? show_regs.cold.9+0x22/0x2f [ 9948.078869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9948.078875] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9948.078880] ? watchdog+0x40/0x40 [ 9948.078884] ? hrtimer_interrupt+0x128/0x2c0 [ 9948.078889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9948.078895] ? apic_timer_interrupt+0xf/0x20 [ 9948.078900] [ 9948.078903] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9948.078908] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9948.078913] do_raw_spin_lock+0x177/0x190 [ 9948.078918] _raw_spin_lock+0x21/0x30 [ 9948.078924] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9948.079316] obd_mod_free+0x46/0x140 [ptlrpc] [ 9948.079545] mdc_commit_open+0x91/0xb0 [mdc] [ 9948.079746] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9948.080051] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9948.080282] after_reply+0xe2c/0x1780 [ptlrpc] [ 9948.080564] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9948.081014] ? __next_timer_interrupt+0x160/0x160 [ 9948.081025] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9948.082057] ? do_wait_intr+0xf0/0xf0 [ 9948.082072] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9948.126262] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9948.133705] kthread+0x1d1/0x200 [ 9948.147907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.151181] ? set_kthread_struct+0x70/0x70 [ 9948.151204] ret_from_fork+0x1f/0x30 [ 9948.600667] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9948.608162] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9948.613519] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9948.616052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.619900] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9948.623332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9948.625849] lmv_intent_open+0x338/0x1150 [lmv] [ 9948.627818] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.631923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.635901] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9948.639131] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9948.641742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.644942] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9948.648672] ? do_raw_spin_unlock+0x75/0x190 [ 9948.651182] ? _raw_spin_unlock+0x12/0x30 [ 9948.653727] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9948.656673] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9948.659563] ll_atomic_open+0x36a/0xd70 [lustre] [ 9948.662290] lookup_open+0x356/0xb40 [ 9948.664437] do_last+0x313/0xfc0 [ 9948.666216] ? nd_jump_root+0xe5/0x160 [ 9948.668429] ? path_init+0x437/0x520 [ 9948.670452] path_openat+0xf7/0x500 [ 9948.672350] ? do_raw_spin_unlock+0x75/0x190 [ 9948.674823] do_filp_open+0x99/0x140 [ 9948.676896] ? getname_flags+0x6e/0x330 [ 9948.679051] ? __check_object_size+0xff/0x256 [ 9948.681584] ? do_raw_spin_unlock+0x75/0x190 [ 9948.686430] ? _raw_spin_unlock+0x12/0x30 [ 9948.687891] do_sys_openat2+0x2b4/0x410 [ 9948.700508] do_sys_open+0x73/0xa0 [ 9948.705694] __x64_sys_openat+0x24/0x30 [ 9948.711940] do_syscall_64+0xc1/0x440 [ 9948.717431] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9948.723127] RIP: 0033:0x7fa749552332 [ 9948.724500] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9948.751111] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9948.766399] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9948.773471] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9948.782702] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9948.793591] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9948.797633] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9951.201826] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8034s! [ 9951.206211] Showing busy workqueues and worker pools: [ 9951.209671] workqueue events: flags=0x0 [ 9951.211388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9951.214087] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9951.218835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9951.221758] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9951.225632] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.228457] pending: kfree_rcu_monitor [ 9951.229901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9951.232394] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9951.235938] workqueue events_long: flags=0x0 [ 9951.241571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.243852] pending: lu_objects_flush [obdclass] [ 9951.246146] workqueue events_unbound: flags=0x2 [ 9951.252504] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9951.254673] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9951.269661] workqueue events_power_efficient: flags=0x80 [ 9951.271753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.276717] pending: check_lifetime [ 9951.278526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.281793] pending: neigh_periodic_work [ 9951.283643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.286341] pending: neigh_periodic_work [ 9951.288087] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.291482] pending: do_cache_clean [ 9951.294108] workqueue mm_percpu_wq: flags=0x8 [ 9951.296087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.299805] pending: vmstat_update [ 9951.303667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.308306] pending: vmstat_update [ 9951.309576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9951.311903] pending: vmstat_update [ 9951.313046] workqueue writeback: flags=0x4a [ 9951.319819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9951.322991] pending: wb_workfn, wb_workfn [ 9951.324268] workqueue xprtiod: flags=0xa [ 9951.327645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9951.330899] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9951.334535] workqueue ipv6_addrconf: flags=0x40008 [ 9951.339895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9951.342334] pending: addrconf_verify_work [ 9951.345969] workqueue knbd0-recv: flags=0x1a [ 9951.350025] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9951.354937] in-flight: 651:recv_work [ 9951.357085] workqueue ptlrpc_pinger: flags=0x2 [ 9951.359014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9951.361509] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9951.364299] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8034s workers=4 idle: 37 31846 28897 [ 9951.367931] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9941s workers=2 idle: 653 [ 9956.057043] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [ 9956.060651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9956.081951] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9956.086867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9956.091051] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9956.094058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9956.097025] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9956.105270] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9956.108055] RAX: 0000000000006e0d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9956.110780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9956.114132] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9956.118224] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9956.121165] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9956.123900] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9956.127137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9956.129543] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9956.132480] Call Trace: [ 9956.133610] [ 9956.134462] ? show_regs.cold.9+0x22/0x2f [ 9956.136205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9956.138000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9956.139985] ? watchdog+0x40/0x40 [ 9956.141427] ? hrtimer_interrupt+0x128/0x2c0 [ 9956.143510] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9956.145756] ? apic_timer_interrupt+0xf/0x20 [ 9956.147555] [ 9956.148389] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9956.149841] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9956.152374] do_raw_spin_lock+0x177/0x190 [ 9956.153681] _raw_spin_lock+0x21/0x30 [ 9956.155381] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9956.157961] process_one_work+0x2c8/0x700 [ 9956.159459] worker_thread+0x296/0x6e0 [ 9956.161119] ? rescuer_thread+0x570/0x570 [ 9956.162905] kthread+0x1d1/0x200 [ 9956.164579] ? set_kthread_struct+0x70/0x70 [ 9956.166032] ret_from_fork+0x1f/0x30 [ 9964.067320] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9964.077730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9964.104929] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9964.108662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9964.111561] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9964.113631] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9964.118911] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9964.121653] RAX: 000000000000698c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9964.123970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9964.126382] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9964.128698] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9964.130946] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9964.133298] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9964.135774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9964.139073] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9964.142273] Call Trace: [ 9964.143218] [ 9964.143788] ? show_regs.cold.9+0x22/0x2f [ 9964.144865] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9964.146480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9964.148153] ? watchdog+0x40/0x40 [ 9964.149490] ? hrtimer_interrupt+0x128/0x2c0 [ 9964.150937] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9964.152716] ? apic_timer_interrupt+0xf/0x20 [ 9964.154242] [ 9964.154988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9964.157090] do_raw_spin_lock+0x177/0x190 [ 9964.158732] _raw_spin_lock+0x21/0x30 [ 9964.160472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9964.162823] ? do_raw_spin_unlock+0x75/0x190 [ 9964.164537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9964.166896] ? do_wait_intr+0xf0/0xf0 [ 9964.168506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9964.171089] mdc_close+0x53b/0xd70 [mdc] [ 9964.172895] lmv_close+0x18d/0x490 [lmv] [ 9964.174637] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9964.177321] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9964.179486] ll_md_close+0x274/0x800 [lustre] [ 9964.181424] ll_file_release+0x128/0xc50 [lustre] [ 9964.183469] __fput+0x10d/0x440 [ 9964.184698] ____fput+0x12/0x20 [ 9964.185964] task_work_run+0xc6/0x110 [ 9964.187290] exit_to_usermode_loop+0x1dd/0x1f0 [ 9964.189533] do_syscall_64+0x3d6/0x440 [ 9964.190940] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9964.192972] RIP: 0033:0x7fa749552cd8 [ 9964.194553] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9964.202087] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9964.205309] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9964.208000] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9964.211217] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9964.213903] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9964.216653] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9976.054767] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [ 9976.059901] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [ 9976.060735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9976.060809] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9976.060825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9976.060828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.060841] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9976.060845] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9976.060850] RAX: 0000000000004884 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9976.060852] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [ 9976.060855] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [ 9976.060857] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [ 9976.060860] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [ 9976.060863] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [ 9976.060866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9976.060868] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [ 9976.060875] Call Trace: [ 9976.060880] [ 9976.060885] ? show_regs.cold.9+0x22/0x2f [ 9976.060893] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9976.060899] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9976.060903] ? watchdog+0x40/0x40 [ 9976.060907] ? hrtimer_interrupt+0x128/0x2c0 [ 9976.060912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9976.060919] ? apic_timer_interrupt+0xf/0x20 [ 9976.060923] [ 9976.060926] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.060933] do_raw_spin_lock+0x177/0x190 [ 9976.060938] _raw_spin_lock+0x21/0x30 [ 9976.060942] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9976.061218] ? do_raw_spin_unlock+0x75/0x190 [ 9976.061225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9976.061424] ? do_wait_intr+0xf0/0xf0 [ 9976.061432] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9976.061628] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9976.061850] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9976.062118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.062275] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9976.062312] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9976.062339] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9976.062365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.062448] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9976.062654] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9976.062682] lmv_intent_open+0x338/0x1150 [lmv] [ 9976.062712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.062796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.062874] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9976.062896] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9976.062978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.063058] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9976.063138] ? do_raw_spin_unlock+0x75/0x190 [ 9976.063143] ? _raw_spin_unlock+0x12/0x30 [ 9976.063149] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9976.063175] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9976.070204] ll_atomic_open+0x36a/0xd70 [lustre] [ 9976.070333] lookup_open+0x356/0xb40 [ 9976.070343] do_last+0x313/0xfc0 [ 9976.070348] ? nd_jump_root+0xe5/0x160 [ 9976.070353] ? path_init+0x437/0x520 [ 9976.070358] path_openat+0xf7/0x500 [ 9976.070362] ? do_raw_spin_unlock+0x75/0x190 [ 9976.070369] do_filp_open+0x99/0x140 [ 9976.070376] ? getname_flags+0x6e/0x330 [ 9976.070381] ? __check_object_size+0xff/0x256 [ 9976.070388] ? do_raw_spin_unlock+0x75/0x190 [ 9976.070394] ? _raw_spin_unlock+0x12/0x30 [ 9976.070400] do_sys_openat2+0x2b4/0x410 [ 9976.070404] do_sys_open+0x73/0xa0 [ 9976.070408] __x64_sys_openat+0x24/0x30 [ 9976.070412] do_syscall_64+0xc1/0x440 [ 9976.070418] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9976.070424] RIP: 0033:0x7fa749552332 [ 9976.070431] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9976.070435] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9976.070441] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [ 9976.070444] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [ 9976.070446] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [ 9976.070448] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [ 9976.070451] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [ 9976.516019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9976.562685] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9976.583795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9976.591020] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9976.598488] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9976.619513] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9976.627458] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [ 9976.634409] RDX: 0000000000003326 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [ 9976.643767] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [ 9976.649395] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [ 9976.655435] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9976.657953] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [ 9976.667219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9976.673723] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [ 9976.681629] Call Trace: [ 9976.686228] [ 9976.687917] ? show_regs.cold.9+0x22/0x2f [ 9976.689433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9976.694402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9976.695939] ? watchdog+0x40/0x40 [ 9976.701683] ? hrtimer_interrupt+0x128/0x2c0 [ 9976.705207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9976.708447] ? apic_timer_interrupt+0xf/0x20 [ 9976.715024] [ 9976.715791] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9976.720436] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9976.726198] do_raw_spin_lock+0x177/0x190 [ 9976.728654] _raw_spin_lock+0x21/0x30 [ 9976.735577] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9976.742805] obd_mod_free+0x46/0x140 [ptlrpc] [ 9976.746144] mdc_commit_open+0x91/0xb0 [mdc] [ 9976.757013] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9976.760482] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9976.763947] after_reply+0xe2c/0x1780 [ptlrpc] [ 9976.772661] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9976.776227] ? __next_timer_interrupt+0x160/0x160 [ 9976.783094] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9976.784954] ? do_wait_intr+0xf0/0xf0 [ 9976.788870] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9976.792239] kthread+0x1d1/0x200 [ 9976.796770] ? set_kthread_struct+0x70/0x70 [ 9976.799519] ret_from_fork+0x1f/0x30 [ 9981.417228] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8064s! [ 9981.426133] Showing busy workqueues and worker pools: [ 9981.436564] workqueue events: flags=0x0 [ 9981.446411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9981.458935] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [ 9981.478553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9981.482440] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9981.494713] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.502307] pending: kfree_rcu_monitor [ 9981.506535] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9981.510872] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [ 9981.516022] workqueue events_long: flags=0x0 [ 9981.519559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.524701] pending: lu_objects_flush [obdclass] [ 9981.533626] workqueue events_unbound: flags=0x2 [ 9981.536742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 9981.540739] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9981.546934] workqueue events_power_efficient: flags=0x80 [ 9981.550441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.554662] pending: check_lifetime [ 9981.558724] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.562308] pending: neigh_periodic_work [ 9981.569178] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.573079] pending: neigh_periodic_work [ 9981.582028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.596320] pending: do_cache_clean [ 9981.597732] workqueue mm_percpu_wq: flags=0x8 [ 9981.604017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.608195] pending: vmstat_update [ 9981.615678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.624022] pending: vmstat_update [ 9981.625175] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9981.632982] pending: vmstat_update [ 9981.634421] workqueue writeback: flags=0x4a [ 9981.641623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9981.645301] pending: wb_workfn, wb_workfn [ 9981.649636] workqueue xprtiod: flags=0xa [ 9981.653736] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9981.656745] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9981.667206] workqueue ipv6_addrconf: flags=0x40008 [ 9981.670712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9981.675753] pending: addrconf_verify_work [ 9981.679543] workqueue knbd0-recv: flags=0x1a [ 9981.682650] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9981.705580] in-flight: 651:recv_work [ 9981.723013] workqueue ptlrpc_pinger: flags=0x2 [ 9981.727526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9981.741138] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9981.763541] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8064s workers=4 idle: 37 31846 28897 [ 9981.765453] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9971s workers=2 idle: 653 [ 9984.059835] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [ 9984.064819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9984.117529] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9984.134790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9984.140727] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9984.147403] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9984.156456] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9984.179760] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9984.189052] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9984.195251] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [ 9984.212702] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9984.222645] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [ 9984.230377] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [ 9984.237229] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [ 9984.239887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9984.252818] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [ 9984.264537] Call Trace: [ 9984.272724] [ 9984.273467] ? show_regs.cold.9+0x22/0x2f [ 9984.274821] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9984.282156] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9984.286416] ? watchdog+0x40/0x40 [ 9984.288686] ? hrtimer_interrupt+0x128/0x2c0 [ 9984.289998] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9984.298098] ? apic_timer_interrupt+0xf/0x20 [ 9984.300520] [ 9984.301456] ? native_safe_halt+0x2e/0x50 [ 9984.303945] kvm_wait+0xcd/0xe0 [ 9984.311110] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9984.320954] do_raw_spin_lock+0x177/0x190 [ 9984.327015] _raw_spin_lock+0x21/0x30 [ 9984.328744] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9984.335561] process_one_work+0x2c8/0x700 [ 9984.339930] worker_thread+0x296/0x6e0 [ 9984.343024] ? rescuer_thread+0x570/0x570 [ 9984.344452] kthread+0x1d1/0x200 [ 9984.350869] ? set_kthread_struct+0x70/0x70 [ 9984.352292] ret_from_fork+0x1f/0x30 [ 9992.060227] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [ 9992.064333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9992.085935] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9992.092195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9992.095604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9992.098212] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9992.105082] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9992.107790] RAX: 0000000000004a48 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [ 9992.110319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [ 9992.112384] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9992.114665] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [ 9992.116822] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [ 9992.119338] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [ 9992.122201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9992.124034] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [ 9992.126320] Call Trace: [ 9992.127221] [ 9992.127988] ? show_regs.cold.9+0x22/0x2f [ 9992.129432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9992.131227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9992.132874] ? watchdog+0x40/0x40 [ 9992.134067] ? hrtimer_interrupt+0x128/0x2c0 [ 9992.137491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9992.139813] ? apic_timer_interrupt+0xf/0x20 [ 9992.141777] [ 9992.142507] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9992.146399] do_raw_spin_lock+0x177/0x190 [ 9992.148555] _raw_spin_lock+0x21/0x30 [ 9992.151370] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9992.155212] ? do_raw_spin_unlock+0x75/0x190 [ 9992.157653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9992.161409] ? do_wait_intr+0xf0/0xf0 [ 9992.163988] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9992.167130] mdc_close+0x53b/0xd70 [mdc] [ 9992.169303] lmv_close+0x18d/0x490 [lmv] [ 9992.171206] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9992.174050] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9992.176231] ll_md_close+0x274/0x800 [lustre] [ 9992.178369] ll_file_release+0x128/0xc50 [lustre] [ 9992.181570] __fput+0x10d/0x440 [ 9992.184090] ____fput+0x12/0x20 [ 9992.185663] task_work_run+0xc6/0x110 [ 9992.187226] exit_to_usermode_loop+0x1dd/0x1f0 [ 9992.190178] do_syscall_64+0x3d6/0x440 [ 9992.191744] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9992.197581] RIP: 0033:0x7fa749552cd8 [ 9992.200270] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9992.211432] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9992.216725] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [ 9992.220210] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [ 9992.224456] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [ 9992.226511] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [ 9992.232544] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10004.058029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10004.061637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10004.084964] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10004.091175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10004.111716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10004.119980] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10004.144770] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10004.155437] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10004.168117] RDX: 00000000000002f1 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10004.183149] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10004.187940] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10004.193610] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10004.198698] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10004.214317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10004.218299] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10004.226710] Call Trace: [10004.229777] [10004.242012] ? show_regs.cold.9+0x22/0x2f [10004.245209] ? watchdog_timer_fn.cold.6+0x62/0x10a [10004.251391] ? __hrtimer_run_queues+0x1c2/0x4d0 [10004.253921] ? watchdog+0x40/0x40 [10004.260330] ? hrtimer_interrupt+0x128/0x2c0 [10004.268871] ? smp_apic_timer_interrupt+0xa6/0x270 [10004.273169] ? apic_timer_interrupt+0xf/0x20 [10004.277531] [10004.278386] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10004.283360] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10004.287178] do_raw_spin_lock+0x177/0x190 [10004.292979] _raw_spin_lock+0x21/0x30 [10004.297619] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10004.301740] obd_mod_free+0x46/0x140 [ptlrpc] [10004.304752] mdc_commit_open+0x91/0xb0 [mdc] [10004.313284] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10004.316096] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10004.324810] after_reply+0xe2c/0x1780 [ptlrpc] [10004.328462] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10004.333311] ? __next_timer_interrupt+0x160/0x160 [10004.335891] ptlrpcd+0xa82/0xe40 [ptlrpc] [10004.344045] ? do_wait_intr+0xf0/0xf0 [10004.346166] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10004.349054] kthread+0x1d1/0x200 [10004.351310] ? set_kthread_struct+0x70/0x70 [10004.354085] ret_from_fork+0x1f/0x30 [10008.054030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10008.058685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10008.090942] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10008.095844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10008.099660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10008.102128] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10008.113266] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10008.120527] RAX: 0000000000000148 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10008.126624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10008.130807] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10008.135177] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10008.140110] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10008.144773] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10008.148487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10008.151790] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10008.157176] Call Trace: [10008.158782] [10008.160584] ? show_regs.cold.9+0x22/0x2f [10008.162294] ? watchdog_timer_fn.cold.6+0x62/0x10a [10008.164836] ? __hrtimer_run_queues+0x1c2/0x4d0 [10008.166953] ? watchdog+0x40/0x40 [10008.168876] ? hrtimer_interrupt+0x128/0x2c0 [10008.172448] ? smp_apic_timer_interrupt+0xa6/0x270 [10008.175794] ? apic_timer_interrupt+0xf/0x20 [10008.178104] [10008.179244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10008.183271] do_raw_spin_lock+0x177/0x190 [10008.185291] _raw_spin_lock+0x21/0x30 [10008.188504] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10008.191308] ? do_raw_spin_unlock+0x75/0x190 [10008.193058] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10008.195524] ? do_wait_intr+0xf0/0xf0 [10008.197073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10008.199928] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10008.206412] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10008.211303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10008.213990] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10008.217226] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10008.220173] mdc_intent_lock+0x353/0x7e0 [mdc] [10008.224122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10008.228221] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10008.232960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10008.236472] lmv_intent_open+0x338/0x1150 [lmv] [10008.239462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10008.242446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10008.246962] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10008.250179] ll_intent_lock+0xc5/0xaa0 [lustre] [10008.253638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10008.255795] ll_lookup_it+0xe0e/0x2270 [lustre] [10008.259457] ? do_raw_spin_unlock+0x75/0x190 [10008.263634] ? _raw_spin_unlock+0x12/0x30 [10008.266123] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10008.269957] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10008.274464] ll_atomic_open+0x36a/0xd70 [lustre] [10008.278199] lookup_open+0x356/0xb40 [10008.280090] do_last+0x313/0xfc0 [10008.282291] ? nd_jump_root+0xe5/0x160 [10008.284464] ? path_init+0x437/0x520 [10008.286507] path_openat+0xf7/0x500 [10008.288348] ? do_raw_spin_unlock+0x75/0x190 [10008.291016] do_filp_open+0x99/0x140 [10008.293045] ? getname_flags+0x6e/0x330 [10008.295157] ? __check_object_size+0xff/0x256 [10008.297357] ? do_raw_spin_unlock+0x75/0x190 [10008.300150] ? _raw_spin_unlock+0x12/0x30 [10008.303669] do_sys_openat2+0x2b4/0x410 [10008.305925] do_sys_open+0x73/0xa0 [10008.307917] __x64_sys_openat+0x24/0x30 [10008.309940] do_syscall_64+0xc1/0x440 [10008.312500] entry_SYSCALL_64_after_hwframe+0x49/0xae [10008.319608] RIP: 0033:0x7fa749552332 [10008.324225] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10008.340236] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10008.352299] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10008.359933] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10008.365305] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10008.368542] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10008.373070] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10012.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10012.066093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10012.129900] BUG: workqueue lockup - pool [10012.133910] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10012.133924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10012.133931] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10012.135235] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10012.135251] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10012.135255] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10012.136879] RAX: 0000000000002f0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10012.136883] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10012.136885] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10012.136888] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10012.136891] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10012.136894] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10012.136897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10012.136899] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10012.136907] Call Trace: [10012.136912] [10012.136918] ? show_regs.cold.9+0x22/0x2f [10012.136931] ? watchdog_timer_fn.cold.6+0x62/0x10a [10012.136938] ? __hrtimer_run_queues+0x1c2/0x4d0 [10012.136943] ? watchdog+0x40/0x40 [10012.136948] ? hrtimer_interrupt+0x128/0x2c0 [10012.136951] ? smp_apic_timer_interrupt+0xa6/0x270 [10012.137003] ? apic_timer_interrupt+0xf/0x20 [10012.137009] [10012.137012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10012.137018] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10012.137025] do_raw_spin_lock+0x177/0x190 [10012.137031] _raw_spin_lock+0x21/0x30 [10012.137036] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10012.138080] process_one_work+0x2c8/0x700 [10012.138090] worker_thread+0x296/0x6e0 [10012.138095] ? rescuer_thread+0x570/0x570 [10012.138099] kthread+0x1d1/0x200 [10012.138104] ? set_kthread_struct+0x70/0x70 [10012.138107] ret_from_fork+0x1f/0x30 [10012.261900] cpus=0-3 flags=0x4 nice=0 stuck for 8095s! [10012.264494] Showing busy workqueues and worker pools: [10012.266830] workqueue events: flags=0x0 [10012.268569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10012.270761] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10012.276483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10012.279699] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10012.284889] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.287851] pending: kfree_rcu_monitor [10012.289858] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10012.292808] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10012.296903] workqueue events_long: flags=0x0 [10012.298767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.301778] pending: lu_objects_flush [obdclass] [10012.304181] workqueue events_unbound: flags=0x2 [10012.306217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10012.309069] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10012.312858] workqueue events_power_efficient: flags=0x80 [10012.315037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.317931] pending: check_lifetime [10012.319539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.322516] pending: neigh_periodic_work [10012.324231] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.327751] pending: neigh_periodic_work [10012.329410] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.332004] pending: do_cache_clean [10012.333464] workqueue mm_percpu_wq: flags=0x8 [10012.335104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.337686] pending: vmstat_update [10012.339273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.342046] pending: vmstat_update [10012.343632] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10012.346627] pending: vmstat_update [10012.348496] workqueue writeback: flags=0x4a [10012.352654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10012.356882] pending: wb_workfn, wb_workfn [10012.359150] workqueue xprtiod: flags=0xa [10012.361480] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10012.364462] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10012.368772] workqueue ipv6_addrconf: flags=0x40008 [10012.371818] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10012.374623] pending: addrconf_verify_work [10012.376645] workqueue knbd0-recv: flags=0x1a [10012.378506] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10012.381636] in-flight: 651:recv_work [10012.383615] workqueue ptlrpc_pinger: flags=0x2 [10012.386594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10012.389508] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10012.392044] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8095s workers=4 idle: 37 31846 28897 [10012.394714] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10002s workers=2 idle: 653 [10020.060364] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10020.069840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10020.097418] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10020.116280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10020.129669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10020.135692] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10020.168809] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10020.175168] RAX: 0000000000005a75 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10020.178163] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10020.183158] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10020.190386] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10020.197820] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10020.205098] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10020.212491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10020.216828] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10020.220938] Call Trace: [10020.230621] [10020.231378] ? show_regs.cold.9+0x22/0x2f [10020.233524] ? watchdog_timer_fn.cold.6+0x62/0x10a [10020.240968] ? __hrtimer_run_queues+0x1c2/0x4d0 [10020.245126] ? watchdog+0x40/0x40 [10020.246361] ? hrtimer_interrupt+0x128/0x2c0 [10020.250959] ? smp_apic_timer_interrupt+0xa6/0x270 [10020.253734] ? apic_timer_interrupt+0xf/0x20 [10020.256583] [10020.257818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10020.262505] do_raw_spin_lock+0x177/0x190 [10020.264638] _raw_spin_lock+0x21/0x30 [10020.268453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10020.277388] ? do_raw_spin_unlock+0x75/0x190 [10020.279598] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10020.285088] ? do_wait_intr+0xf0/0xf0 [10020.287016] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10020.292502] mdc_close+0x53b/0xd70 [mdc] [10020.298307] lmv_close+0x18d/0x490 [lmv] [10020.300586] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10020.305000] ll_md_real_close+0xa5/0x2a0 [lustre] [10020.310926] ll_md_close+0x274/0x800 [lustre] [10020.312702] ll_file_release+0x128/0xc50 [lustre] [10020.317534] __fput+0x10d/0x440 [10020.318696] ____fput+0x12/0x20 [10020.327412] task_work_run+0xc6/0x110 [10020.328761] exit_to_usermode_loop+0x1dd/0x1f0 [10020.331792] do_syscall_64+0x3d6/0x440 [10020.333341] entry_SYSCALL_64_after_hwframe+0x49/0xae [10020.335422] RIP: 0033:0x7fa749552cd8 [10020.337552] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10020.345097] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10020.348204] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10020.351054] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10020.354427] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10020.357252] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10020.359982] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10032.058022] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10032.060676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10032.076279] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10032.079948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10032.082927] RIP: 0010:native_safe_halt+0x2e/0x50 [10032.084560] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10032.090800] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10032.093453] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10032.095939] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10032.098352] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10032.100705] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10032.103042] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10032.105404] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10032.111536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10032.113540] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10032.116020] Call Trace: [10032.116853] [10032.117604] ? show_regs.cold.9+0x22/0x2f [10032.119016] ? watchdog_timer_fn.cold.6+0x62/0x10a [10032.120688] ? __hrtimer_run_queues+0x1c2/0x4d0 [10032.122268] ? watchdog+0x40/0x40 [10032.123431] ? hrtimer_interrupt+0x128/0x2c0 [10032.124913] ? smp_apic_timer_interrupt+0xa6/0x270 [10032.126607] ? apic_timer_interrupt+0xf/0x20 [10032.128034] [10032.128825] ? native_safe_halt+0x2e/0x50 [10032.130796] kvm_wait+0xcd/0xe0 [10032.132467] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10032.134925] do_raw_spin_lock+0x177/0x190 [10032.136421] _raw_spin_lock+0x21/0x30 [10032.137984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10032.140270] obd_mod_free+0x46/0x140 [ptlrpc] [10032.143992] mdc_commit_open+0x91/0xb0 [mdc] [10032.146221] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10032.148062] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10032.150107] after_reply+0xe2c/0x1780 [ptlrpc] [10032.151971] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10032.154447] ? __next_timer_interrupt+0x160/0x160 [10032.156249] ptlrpcd+0xa82/0xe40 [ptlrpc] [10032.158071] ? do_wait_intr+0xf0/0xf0 [10032.159501] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10032.161651] kthread+0x1d1/0x200 [10032.162890] ? set_kthread_struct+0x70/0x70 [10032.164525] ret_from_fork+0x1f/0x30 [10040.054129] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10040.057507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10040.059003] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10040.134861] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10040.134877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10040.134880] RIP: 0010:native_safe_halt+0x2e/0x50 [10040.134893] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10040.134897] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10040.134903] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10040.134906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10040.134908] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10040.134910] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10040.134912] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10040.134915] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10040.134918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10040.134920] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10040.134929] Call Trace: [10040.134934] [10040.134939] ? show_regs.cold.9+0x22/0x2f [10040.134949] ? watchdog_timer_fn.cold.6+0x62/0x10a [10040.134956] ? __hrtimer_run_queues+0x1c2/0x4d0 [10040.134960] ? watchdog+0x40/0x40 [10040.134964] ? hrtimer_interrupt+0x128/0x2c0 [10040.134967] ? smp_apic_timer_interrupt+0xa6/0x270 [10040.134973] ? apic_timer_interrupt+0xf/0x20 [10040.134977] [10040.134979] ? native_safe_halt+0x2e/0x50 [10040.134982] kvm_wait+0xcd/0xe0 [10040.134990] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10040.134999] do_raw_spin_lock+0x177/0x190 [10040.135003] _raw_spin_lock+0x21/0x30 [10040.135008] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10040.136030] ? do_raw_spin_unlock+0x75/0x190 [10040.136037] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10040.136273] ? do_wait_intr+0xf0/0xf0 [10040.136283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10040.136649] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10040.137664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10040.138980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10040.139103] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10040.139145] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10040.139175] mdc_intent_lock+0x353/0x7e0 [mdc] [10040.141368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10040.150133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10040.150211] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10040.150218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10040.150225] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10040.176090] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10040.176750] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10040.177156] lmv_intent_open+0x338/0x1150 [lmv] [10040.177192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10040.177467] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10040.177555] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10040.177578] ll_intent_lock+0xc5/0xaa0 [lustre] [10040.177660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10040.177741] ll_lookup_it+0xe0e/0x2270 [lustre] [10040.196690] [10040.196700] RIP: 0010:native_safe_halt+0x2e/0x50 [10040.196716] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10040.196720] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10040.196725] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10040.196727] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10040.196729] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10040.196732] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10040.196734] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10040.196737] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10040.196739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10040.196742] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10040.196750] Call Trace: [10040.196756] [10040.196762] ? show_regs.cold.9+0x22/0x2f [10040.196772] ? watchdog_timer_fn.cold.6+0x62/0x10a [10040.196791] ? __hrtimer_run_queues+0x1c2/0x4d0 [10040.196795] ? watchdog+0x40/0x40 [10040.196799] ? hrtimer_interrupt+0x128/0x2c0 [10040.196803] ? smp_apic_timer_interrupt+0xa6/0x270 [10040.196809] ? apic_timer_interrupt+0xf/0x20 [10040.196814] [10040.196817] ? native_safe_halt+0x2e/0x50 [10040.196822] kvm_wait+0xcd/0xe0 [10040.196829] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10040.196850] do_raw_spin_lock+0x177/0x190 [10040.196857] _raw_spin_lock+0x21/0x30 [10040.196861] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10040.204154] ? do_raw_spin_unlock+0x75/0x190 [10040.204174] ? _raw_spin_unlock+0x12/0x30 [10040.204181] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10040.204217] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10040.204646] ll_atomic_open+0x36a/0xd70 [lustre] [10040.205643] lookup_open+0x356/0xb40 [10040.205659] do_last+0x313/0xfc0 [10040.205665] ? nd_jump_root+0xe5/0x160 [10040.205670] ? path_init+0x437/0x520 [10040.205675] path_openat+0xf7/0x500 [10040.205680] ? do_raw_spin_unlock+0x75/0x190 [10040.205686] do_filp_open+0x99/0x140 [10040.205693] ? getname_flags+0x6e/0x330 [10040.205697] ? __check_object_size+0xff/0x256 [10040.205704] ? do_raw_spin_unlock+0x75/0x190 [10040.205708] ? _raw_spin_unlock+0x12/0x30 [10040.205714] do_sys_openat2+0x2b4/0x410 [10040.205718] do_sys_open+0x73/0xa0 [10040.205723] __x64_sys_openat+0x24/0x30 [10040.205726] do_syscall_64+0xc1/0x440 [10040.205733] entry_SYSCALL_64_after_hwframe+0x49/0xae [10040.205739] RIP: 0033:0x7fa749552332 [10040.205747] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10040.205751] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10040.205757] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10040.205760] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10040.205762] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10040.205763] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10040.205873] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10041.144395] process_one_work+0x2c8/0x700 [10041.152415] worker_thread+0x296/0x6e0 [10041.160948] ? rescuer_thread+0x570/0x570 [10041.168907] kthread+0x1d1/0x200 [10041.173101] ? set_kthread_struct+0x70/0x70 [10041.177303] ret_from_fork+0x1f/0x30 [10042.967129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8125s! [10042.970036] Showing busy workqueues and worker pools: [10042.976742] workqueue events: flags=0x0 [10042.983549] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10042.997710] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10043.014737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10043.020793] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10043.037610] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.043849] pending: kfree_rcu_monitor [10043.047990] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10043.055398] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10043.064436] workqueue events_long: flags=0x0 [10043.069616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.078065] pending: lu_objects_flush [obdclass] [10043.084440] workqueue events_unbound: flags=0x2 [10043.095856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10043.105636] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10043.108090] workqueue events_power_efficient: flags=0x80 [10043.116602] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.125667] pending: check_lifetime [10043.132740] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.140985] pending: neigh_periodic_work [10043.145112] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.156664] pending: neigh_periodic_work [10043.165933] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.169529] pending: do_cache_clean [10043.176708] workqueue mm_percpu_wq: flags=0x8 [10043.182594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.188148] pending: vmstat_update [10043.193570] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.201351] pending: vmstat_update [10043.206088] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10043.210396] pending: vmstat_update [10043.216944] workqueue writeback: flags=0x4a [10043.220361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10043.241592] pending: wb_workfn, wb_workfn [10043.248616] workqueue xprtiod: flags=0xa [10043.256027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10043.264234] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10043.279652] workqueue ipv6_addrconf: flags=0x40008 [10043.287151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10043.298086] pending: addrconf_verify_work [10043.305255] workqueue knbd0-recv: flags=0x1a [10043.309480] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10043.321400] in-flight: 651:recv_work [10043.326108] workqueue ptlrpc_pinger: flags=0x2 [10043.329024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10043.336604] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10043.338690] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8126s workers=4 idle: 37 31846 28897 [10043.346934] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10033s workers=2 idle: 653 [10048.059486] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10048.065569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10048.117353] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10048.121859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10048.126321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10048.129792] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10048.142162] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10048.146173] RAX: 000000000000783d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10048.155870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10048.161004] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10048.164400] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10048.176378] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10048.184370] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10048.192400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10048.197794] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10048.205734] Call Trace: [10048.207103] [10048.208519] ? show_regs.cold.9+0x22/0x2f [10048.210290] ? watchdog_timer_fn.cold.6+0x62/0x10a [10048.231911] ? __hrtimer_run_queues+0x1c2/0x4d0 [10048.233566] ? watchdog+0x40/0x40 [10048.234474] ? hrtimer_interrupt+0x128/0x2c0 [10048.237291] ? smp_apic_timer_interrupt+0xa6/0x270 [10048.241460] ? apic_timer_interrupt+0xf/0x20 [10048.244177] [10048.248069] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10048.253896] do_raw_spin_lock+0x177/0x190 [10048.255761] _raw_spin_lock+0x21/0x30 [10048.261346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10048.267673] ? do_raw_spin_unlock+0x75/0x190 [10048.269656] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10048.273216] ? do_wait_intr+0xf0/0xf0 [10048.274318] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10048.277646] mdc_close+0x53b/0xd70 [mdc] [10048.279898] lmv_close+0x18d/0x490 [lmv] [10048.286991] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10048.290353] ll_md_real_close+0xa5/0x2a0 [lustre] [10048.293323] ll_md_close+0x274/0x800 [lustre] [10048.297772] ll_file_release+0x128/0xc50 [lustre] [10048.300414] __fput+0x10d/0x440 [10048.301926] ____fput+0x12/0x20 [10048.316299] task_work_run+0xc6/0x110 [10048.319472] exit_to_usermode_loop+0x1dd/0x1f0 [10048.323155] do_syscall_64+0x3d6/0x440 [10048.327327] entry_SYSCALL_64_after_hwframe+0x49/0xae [10048.336314] RIP: 0033:0x7fa749552cd8 [10048.338140] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10048.355742] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10048.360281] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10048.364178] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10048.368327] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10048.375473] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10048.380596] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10060.058081] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10060.060770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10060.076229] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10060.086759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10060.094290] RIP: 0010:native_safe_halt+0x2e/0x50 [10060.098091] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10060.115702] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10060.119865] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10060.123345] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10060.126702] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10060.130055] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10060.133739] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10060.136673] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10060.139930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10060.142157] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10060.145869] Call Trace: [10060.146841] [10060.148171] ? show_regs.cold.9+0x22/0x2f [10060.150002] ? watchdog_timer_fn.cold.6+0x62/0x10a [10060.151851] ? __hrtimer_run_queues+0x1c2/0x4d0 [10060.153649] ? watchdog+0x40/0x40 [10060.154941] ? hrtimer_interrupt+0x128/0x2c0 [10060.156495] ? smp_apic_timer_interrupt+0xa6/0x270 [10060.158545] ? apic_timer_interrupt+0xf/0x20 [10060.160249] [10060.161233] ? native_safe_halt+0x2e/0x50 [10060.162898] kvm_wait+0xcd/0xe0 [10060.163861] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10060.165374] do_raw_spin_lock+0x177/0x190 [10060.166513] _raw_spin_lock+0x21/0x30 [10060.167398] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10060.168943] obd_mod_free+0x46/0x140 [ptlrpc] [10060.170654] mdc_commit_open+0x91/0xb0 [mdc] [10060.172559] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10060.174864] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10060.177244] after_reply+0xe2c/0x1780 [ptlrpc] [10060.179786] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10060.182514] ? __next_timer_interrupt+0x160/0x160 [10060.184492] ptlrpcd+0xa82/0xe40 [ptlrpc] [10060.185916] ? do_wait_intr+0xf0/0xf0 [10060.186917] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10060.189245] kthread+0x1d1/0x200 [10060.190612] ? set_kthread_struct+0x70/0x70 [10060.192118] ret_from_fork+0x1f/0x30 [10068.057694] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10068.059113] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10068.061038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10068.061211] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10068.061218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10068.061221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10068.061246] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10068.061250] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10068.061255] RAX: 0000000000004cc2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10068.061257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10068.061259] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10068.061261] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10068.061263] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10068.061266] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10068.061269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10068.061271] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10068.061279] Call Trace: [10068.061295] [10068.061300] ? show_regs.cold.9+0x22/0x2f [10068.061311] ? watchdog_timer_fn.cold.6+0x62/0x10a [10068.061318] ? __hrtimer_run_queues+0x1c2/0x4d0 [10068.061322] ? watchdog+0x40/0x40 [10068.061327] ? hrtimer_interrupt+0x128/0x2c0 [10068.061331] ? smp_apic_timer_interrupt+0xa6/0x270 [10068.061340] ? apic_timer_interrupt+0xf/0x20 [10068.061345] [10068.061348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10068.061355] do_raw_spin_lock+0x177/0x190 [10068.061359] _raw_spin_lock+0x21/0x30 [10068.061364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10068.062461] ? do_raw_spin_unlock+0x75/0x190 [10068.067820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10068.067902] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10068.067909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10068.067915] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10068.068525] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10068.068537] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10068.068540] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10068.068545] RAX: 0000000000006cbc RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10068.068548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10068.068550] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10068.068553] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10068.068555] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10068.068559] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10068.068562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10068.068851] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10068.068860] Call Trace: [10068.068866] [10068.068872] ? show_regs.cold.9+0x22/0x2f [10068.068881] ? watchdog_timer_fn.cold.6+0x62/0x10a [10068.068888] ? __hrtimer_run_queues+0x1c2/0x4d0 [10068.068892] ? watchdog+0x40/0x40 [10068.068896] ? hrtimer_interrupt+0x128/0x2c0 [10068.068900] ? smp_apic_timer_interrupt+0xa6/0x270 [10068.068909] ? apic_timer_interrupt+0xf/0x20 [10068.068913] [10068.068916] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10068.068920] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10068.068925] do_raw_spin_lock+0x177/0x190 [10068.068929] _raw_spin_lock+0x21/0x30 [10068.068934] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10068.069470] process_one_work+0x2c8/0x700 [10068.069475] worker_thread+0x296/0x6e0 [10068.069478] ? rescuer_thread+0x570/0x570 [10068.069480] kthread+0x1d1/0x200 [10068.069484] ? set_kthread_struct+0x70/0x70 [10068.069486] ret_from_fork+0x1f/0x30 [10068.450710] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10068.452501] ? do_wait_intr+0xf0/0xf0 [10068.458691] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10068.462760] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10068.468551] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10068.479795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10068.491501] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10068.496157] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10068.500156] mdc_intent_lock+0x353/0x7e0 [mdc] [10068.503424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10068.508933] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10068.510877] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10068.513176] lmv_intent_open+0x338/0x1150 [lmv] [10068.514880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10068.516637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10068.518603] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10068.520101] ll_intent_lock+0xc5/0xaa0 [lustre] [10068.521784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10068.524034] ll_lookup_it+0xe0e/0x2270 [lustre] [10068.525722] ? do_raw_spin_unlock+0x75/0x190 [10068.527768] ? _raw_spin_unlock+0x12/0x30 [10068.529367] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10068.531085] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10068.532804] ll_atomic_open+0x36a/0xd70 [lustre] [10068.534612] lookup_open+0x356/0xb40 [10068.536051] do_last+0x313/0xfc0 [10068.537256] ? nd_jump_root+0xe5/0x160 [10068.538909] ? path_init+0x437/0x520 [10068.540543] path_openat+0xf7/0x500 [10068.541731] ? do_raw_spin_unlock+0x75/0x190 [10068.543068] do_filp_open+0x99/0x140 [10068.544462] ? getname_flags+0x6e/0x330 [10068.545779] ? __check_object_size+0xff/0x256 [10068.547495] ? do_raw_spin_unlock+0x75/0x190 [10068.549084] ? _raw_spin_unlock+0x12/0x30 [10068.550545] do_sys_openat2+0x2b4/0x410 [10068.551912] do_sys_open+0x73/0xa0 [10068.553084] __x64_sys_openat+0x24/0x30 [10068.554420] do_syscall_64+0xc1/0x440 [10068.555649] entry_SYSCALL_64_after_hwframe+0x49/0xae [10068.559784] RIP: 0033:0x7fa749552332 [10068.563790] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10068.577567] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10068.580707] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10068.583438] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10068.585924] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10068.589132] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10068.592189] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10073.576294] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8156s! [10073.581737] Showing busy workqueues and worker pools: [10073.586641] workqueue events: flags=0x0 [10073.590883] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10073.596139] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10073.605669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10073.612939] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10073.629223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.633153] pending: kfree_rcu_monitor [10073.635444] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10073.642503] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10073.656575] workqueue events_long: flags=0x0 [10073.664144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.673141] pending: lu_objects_flush [obdclass] [10073.676510] workqueue events_unbound: flags=0x2 [10073.679356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10073.685955] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10073.696014] workqueue events_power_efficient: flags=0x80 [10073.698075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.715360] pending: check_lifetime [10073.720566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.727969] pending: neigh_periodic_work [10073.731177] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.737746] pending: neigh_periodic_work [10073.742358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.747088] pending: do_cache_clean [10073.749165] workqueue mm_percpu_wq: flags=0x8 [10073.751295] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.755474] pending: vmstat_update [10073.758642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.765349] pending: vmstat_update [10073.770521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10073.777767] pending: vmstat_update [10073.784505] workqueue writeback: flags=0x4a [10073.786822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10073.794396] pending: wb_workfn, wb_workfn [10073.800473] workqueue xprtiod: flags=0xa [10073.809662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10073.816414] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10073.823229] workqueue ipv6_addrconf: flags=0x40008 [10073.824977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10073.835838] pending: addrconf_verify_work [10073.840416] workqueue knbd0-recv: flags=0x1a [10073.844894] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10073.849017] in-flight: 651:recv_work [10073.856143] workqueue ptlrpc_pinger: flags=0x2 [10073.859754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10073.865054] rcu: INFO: rcu_sched self-detected stall on CPU [10073.868997] in-flight: [10073.871307] rcu: 2-...!: (4595549 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [10073.871324] (t=8160176 jiffies g=258701 q=3895) [10073.871328] rcu: rcu_sched kthread starved for 8156971 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [10073.871333] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10073.871335] rcu: RCU grace-period kthread stack dump: [10073.871337] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10073.871345] Call Trace: [10073.871353] __schedule+0x351/0xcb0 [10073.871367] schedule+0xc0/0x180 [10073.871372] schedule_timeout+0xb4/0x190 [10073.871377] ? do_raw_spin_unlock+0x75/0x190 [10073.871384] ? __next_timer_interrupt+0x160/0x160 [10073.871392] ? prepare_to_swait_event+0xf3/0x140 [10073.871400] rcu_gp_kthread+0x8dd/0x15b0 [10073.871407] ? force_qs_rnp+0x2c0/0x2c0 [10073.871410] kthread+0x1d1/0x200 [10073.871415] ? set_kthread_struct+0x70/0x70 [10073.871419] ret_from_fork+0x1f/0x30 [10073.871428] rcu: Stack dump where RCU GP kthread last ran: [10073.871431] Sending NMI from CPU 2 to CPUs 3: [10073.872391] NMI backtrace for cpu 3 [10073.872399] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10073.872401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10073.872403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10073.872404] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10073.872407] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 [10073.872409] RAX: 0000000000002216 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10073.872411] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10073.872412] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10073.872414] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10073.872415] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10073.872416] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10073.872418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10073.872420] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10073.872421] Call Trace: [10073.872422] [10073.872423] ? show_regs.cold.9+0x22/0x2f [10073.872424] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10073.872425] ? nmi_cpu_backtrace_handler+0x15/0x40 [10073.872430] ? nmi_handle+0xbd/0x210 [10073.872431] ? default_do_nmi+0x4b/0x1b0 [10073.872432] ? do_nmi+0x280/0x380 [10073.872433] ? end_repeat_nmi+0x16/0x1f [10073.872434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10073.872436] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10073.872437] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10073.872438] [10073.872439] do_raw_spin_lock+0x177/0x190 [10073.872441] _raw_spin_lock+0x21/0x30 [10073.872442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10073.872443] ? do_raw_spin_unlock+0x75/0x190 [10073.872445] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10073.872446] ? do_wait_intr+0xf0/0xf0 [10073.872447] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10073.872448] mdc_close+0x53b/0xd70 [mdc] [10073.872449] lmv_close+0x18d/0x490 [lmv] [10073.872450] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10073.872452] ll_md_real_close+0xa5/0x2a0 [lustre] [10073.872453] ll_md_close+0x274/0x800 [lustre] [10073.872454] ll_file_release+0x128/0xc50 [lustre] [10073.872455] __fput+0x10d/0x440 [10073.872456] ____fput+0x12/0x20 [10073.872457] task_work_run+0xc6/0x110 [10073.872458] exit_to_usermode_loop+0x1dd/0x1f0 [10073.872458] do_syscall_64+0x3d6/0x440 [10073.872460] entry_SYSCALL_64_after_hwframe+0x49/0xae [10073.872461] RIP: 0033:0x7fa749552cd8 [10073.872474] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10073.872480] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10073.872483] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10073.872484] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10073.872486] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10073.872487] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10073.872488] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10073.872496] NMI backtrace for cpu 2 [10073.872503] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10073.872508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10073.872511] Call Trace: [10073.872514] [10073.872516] ? dump_stack+0xbb/0x10e [10073.872522] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10073.872528] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10073.872533] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10073.872537] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10073.872542] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10073.872546] ? rcu_dump_cpu_stacks+0x11b/0x171 [10073.872626] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10073.872632] ? update_process_times+0x6d/0xc0 [10073.872640] ? tick_sched_handle.isra.18+0x27/0x90 [10073.872646] ? tick_sched_timer+0x53/0xb0 [10073.872650] ? __hrtimer_run_queues+0x1c2/0x4d0 [10073.872654] ? tick_sched_handle.isra.18+0x90/0x90 [10073.872659] ? hrtimer_interrupt+0x128/0x2c0 [10073.872663] ? smp_apic_timer_interrupt+0xa6/0x270 [10073.872668] ? apic_timer_interrupt+0xf/0x20 [10073.872673] [10073.872676] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10073.872682] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10073.872686] ? do_raw_spin_lock+0x177/0x190 [10073.872689] ? _raw_spin_lock+0x21/0x30 [10073.872693] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10073.872989] ? obd_mod_free+0x46/0x140 [ptlrpc] [10073.873257] ? mdc_commit_open+0x91/0xb0 [mdc] [10073.873291] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10073.873497] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10073.873786] ? after_reply+0xe2c/0x1780 [ptlrpc] [10073.874071] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10073.874335] ? __next_timer_interrupt+0x160/0x160 [10073.874345] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10073.874636] ? do_wait_intr+0xf0/0xf0 [10073.874646] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10073.874925] ? kthread+0x1d1/0x200 [10073.874930] ? set_kthread_struct+0x70/0x70 [10073.874934] ? ret_from_fork+0x1f/0x30 [10074.368732] 8:ptlrpc_pinger_main [ptlrpc] [10074.371720] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8157s workers=4 idle: 37 31846 28897 [10074.375722] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10064s workers=2 idle: 653 [10096.057567] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10096.067694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10096.129763] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10096.140186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10096.145658] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10096.149100] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10096.152399] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10096.166469] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10096.171044] RAX: 0000000000002986 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10096.178995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10096.185307] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10096.192583] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10096.195355] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10096.197646] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10096.202772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10096.205456] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10096.210344] Call Trace: [10096.211228] [10096.212578] ? show_regs.cold.9+0x22/0x2f [10096.214425] ? watchdog_timer_fn.cold.6+0x62/0x10a [10096.216510] ? __hrtimer_run_queues+0x1c2/0x4d0 [10096.222165] ? watchdog+0x40/0x40 [10096.224139] ? hrtimer_interrupt+0x128/0x2c0 [10096.226488] ? smp_apic_timer_interrupt+0xa6/0x270 [10096.228779] ? apic_timer_interrupt+0xf/0x20 [10096.230544] [10096.231572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10096.234130] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10096.236248] do_raw_spin_lock+0x177/0x190 [10096.241433] _raw_spin_lock+0x21/0x30 [10096.243000] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10096.249288] process_one_work+0x2c8/0x700 [10096.250733] worker_thread+0x296/0x6e0 [10096.252628] ? rescuer_thread+0x570/0x570 [10096.253753] kthread+0x1d1/0x200 [10096.255182] ? set_kthread_struct+0x70/0x70 [10096.258485] ret_from_fork+0x1f/0x30 [10100.055858] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10100.058049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10100.060217] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10100.060230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.060310] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.060315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.060317] RIP: 0010:native_safe_halt+0x2e/0x50 [10100.060331] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10100.060335] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10100.060340] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10100.060343] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10100.060345] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10100.060347] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10100.060350] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10100.060353] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10100.060356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.060358] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10100.060368] Call Trace: [10100.060374] [10100.060380] ? show_regs.cold.9+0x22/0x2f [10100.060389] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.060396] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.060400] ? watchdog+0x40/0x40 [10100.060405] ? hrtimer_interrupt+0x128/0x2c0 [10100.060409] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.060416] ? apic_timer_interrupt+0xf/0x20 [10100.060421] [10100.060424] ? native_safe_halt+0x2e/0x50 [10100.060427] kvm_wait+0xcd/0xe0 [10100.060434] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10100.060442] do_raw_spin_lock+0x177/0x190 [10100.060446] _raw_spin_lock+0x21/0x30 [10100.060451] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10100.060713] ? do_raw_spin_unlock+0x75/0x190 [10100.060719] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10100.060910] ? do_wait_intr+0xf0/0xf0 [10100.060918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10100.061620] mdc_close+0x53b/0xd70 [mdc] [10100.061662] lmv_close+0x18d/0x490 [lmv] [10100.061701] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10100.061811] ll_md_real_close+0xa5/0x2a0 [lustre] [10100.061892] ll_md_close+0x274/0x800 [lustre] [10100.061975] ll_file_release+0x128/0xc50 [lustre] [10100.062073] __fput+0x10d/0x440 [10100.062082] ____fput+0x12/0x20 [10100.062087] task_work_run+0xc6/0x110 [10100.062095] exit_to_usermode_loop+0x1dd/0x1f0 [10100.062103] do_syscall_64+0x3d6/0x440 [10100.062108] entry_SYSCALL_64_after_hwframe+0x49/0xae [10100.063353] RIP: 0033:0x7fa749552cd8 [10100.063370] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10100.063375] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10100.063380] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10100.063382] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10100.063384] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10100.063387] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10100.063389] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10100.069290] Modules linked in: [10100.073609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.073690] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.073699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.073702] RIP: 0010:native_safe_halt+0x2e/0x50 [10100.073716] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10100.073733] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10100.073737] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10100.073739] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10100.073741] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10100.073743] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10100.073745] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10100.073748] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10100.073751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.073753] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10100.073763] Call Trace: [10100.073769] [10100.073776] ? show_regs.cold.9+0x22/0x2f [10100.073785] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.073792] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.073797] ? watchdog+0x40/0x40 [10100.073801] ? hrtimer_interrupt+0x128/0x2c0 [10100.073805] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.073811] ? apic_timer_interrupt+0xf/0x20 [10100.073815] [10100.073817] ? native_safe_halt+0x2e/0x50 [10100.073821] kvm_wait+0xcd/0xe0 [10100.073828] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10100.073836] do_raw_spin_lock+0x177/0x190 [10100.073841] _raw_spin_lock+0x21/0x30 [10100.073847] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10100.075330] obd_mod_free+0x46/0x140 [ptlrpc] [10100.075522] mdc_commit_open+0x91/0xb0 [mdc] [10100.075563] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10100.075831] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10100.076038] after_reply+0xe2c/0x1780 [ptlrpc] [10100.076237] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10100.084640] lustre(O) [10100.105695] ? __next_timer_interrupt+0x160/0x160 [10100.117624] osp(O) [10100.121583] ptlrpcd+0xa82/0xe40 [ptlrpc] [10100.121863] ? do_wait_intr+0xf0/0xf0 [10100.121874] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10100.122074] kthread+0x1d1/0x200 [10100.122080] ? set_kthread_struct+0x70/0x70 [10100.122085] ret_from_fork+0x1f/0x30 [10100.704576] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.745763] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.754447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.759056] RIP: 0010:native_safe_halt+0x2e/0x50 [10100.763078] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10100.780335] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10100.784729] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10100.789524] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [10100.799153] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10100.809204] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10100.812413] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10100.816355] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10100.821281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.831671] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10100.840963] Call Trace: [10100.844831] [10100.846193] ? show_regs.cold.9+0x22/0x2f [10100.848987] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.851630] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.855060] ? watchdog+0x40/0x40 [10100.856865] ? hrtimer_interrupt+0x128/0x2c0 [10100.859372] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.866546] ? apic_timer_interrupt+0xf/0x20 [10100.871979] [10100.875377] ? native_safe_halt+0x2e/0x50 [10100.878989] kvm_wait+0xcd/0xe0 [10100.887549] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10100.902249] do_raw_spin_lock+0x177/0x190 [10100.908017] _raw_spin_lock+0x21/0x30 [10100.915862] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10100.922088] ? do_raw_spin_unlock+0x75/0x190 [10100.934266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10100.937688] ? do_wait_intr+0xf0/0xf0 [10100.946678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10100.954815] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10100.956799] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10100.965546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10100.974987] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10100.984544] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10100.987741] mdc_intent_lock+0x353/0x7e0 [mdc] [10100.991028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10101.002058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10101.009830] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10101.016053] lmv_intent_open+0x338/0x1150 [lmv] [10101.019469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10101.022516] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10101.027799] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10101.035578] ll_intent_lock+0xc5/0xaa0 [lustre] [10101.043353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10101.051679] ll_lookup_it+0xe0e/0x2270 [lustre] [10101.058597] ? do_raw_spin_unlock+0x75/0x190 [10101.066150] ? _raw_spin_unlock+0x12/0x30 [10101.073578] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10101.078545] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10101.079881] ll_atomic_open+0x36a/0xd70 [lustre] [10101.085562] lookup_open+0x356/0xb40 [10101.090912] do_last+0x313/0xfc0 [10101.095112] ? nd_jump_root+0xe5/0x160 [10101.098189] ? path_init+0x437/0x520 [10101.103818] path_openat+0xf7/0x500 [10101.106627] ? do_raw_spin_unlock+0x75/0x190 [10101.109669] do_filp_open+0x99/0x140 [10101.114072] ? getname_flags+0x6e/0x330 [10101.118008] ? __check_object_size+0xff/0x256 [10101.126666] ? do_raw_spin_unlock+0x75/0x190 [10101.131320] ? _raw_spin_unlock+0x12/0x30 [10101.132798] do_sys_openat2+0x2b4/0x410 [10101.138531] do_sys_open+0x73/0xa0 [10101.147486] __x64_sys_openat+0x24/0x30 [10101.152127] do_syscall_64+0xc1/0x440 [10101.154146] entry_SYSCALL_64_after_hwframe+0x49/0xae [10101.160636] RIP: 0033:0x7fa749552332 [10101.166501] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10101.184676] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10101.197924] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10101.200434] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10101.213294] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10101.222225] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10101.235745] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10104.808076] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8187s! [10104.815480] Showing busy workqueues and worker pools: [10104.819268] workqueue events: flags=0x0 [10104.824019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10104.826442] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10104.836480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10104.850114] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10104.857554] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10104.861875] pending: kfree_rcu_monitor [10104.867515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10104.876801] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10104.887701] workqueue events_long: flags=0x0 [10104.896209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10104.904830] pending: lu_objects_flush [obdclass] [10104.911830] workqueue events_unbound: flags=0x2 [10104.920277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10104.927108] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10104.934544] workqueue events_power_efficient: flags=0x80 [10104.941527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10104.949122] pending: check_lifetime [10104.951385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10104.958471] pending: neigh_periodic_work [10104.963279] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10104.970531] pending: neigh_periodic_work [10104.972159] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10104.978623] pending: do_cache_clean [10104.980885] workqueue mm_percpu_wq: flags=0x8 [10104.993226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10104.995638] pending: vmstat_update [10105.008067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10105.014176] pending: vmstat_update [10105.015560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10105.027206] pending: vmstat_update [10105.030270] workqueue writeback: flags=0x4a [10105.035485] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10105.040010] pending: wb_workfn, wb_workfn [10105.049580] workqueue xprtiod: flags=0xa [10105.052435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10105.060096] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10105.067355] workqueue ipv6_addrconf: flags=0x40008 [10105.072044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10105.075958] pending: addrconf_verify_work [10105.082187] workqueue knbd0-recv: flags=0x1a [10105.084992] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10105.092071] in-flight: 651:recv_work [10105.097033] workqueue ptlrpc_pinger: flags=0x2 [10105.102856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10105.110758] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10105.116826] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8187s workers=4 idle: 37 31846 28897 [10105.130212] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10095s workers=2 idle: 653 [10124.061426] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10124.068189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10124.108974] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10124.115559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10124.122643] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10124.128455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10124.132865] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10124.141763] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10124.144153] RAX: 0000000000007f0d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10124.146563] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10124.153782] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10124.155453] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10124.159845] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10124.162986] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10124.168525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10124.173229] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10124.180510] Call Trace: [10124.181894] [10124.183113] ? show_regs.cold.9+0x22/0x2f [10124.185436] ? watchdog_timer_fn.cold.6+0x62/0x10a [10124.187743] ? __hrtimer_run_queues+0x1c2/0x4d0 [10124.189922] ? watchdog+0x40/0x40 [10124.191767] ? hrtimer_interrupt+0x128/0x2c0 [10124.194837] ? smp_apic_timer_interrupt+0xa6/0x270 [10124.196563] ? apic_timer_interrupt+0xf/0x20 [10124.199390] [10124.200823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10124.206682] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10124.213910] do_raw_spin_lock+0x177/0x190 [10124.217772] _raw_spin_lock+0x21/0x30 [10124.222283] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10124.225913] process_one_work+0x2c8/0x700 [10124.228713] worker_thread+0x296/0x6e0 [10124.230627] ? rescuer_thread+0x570/0x570 [10124.232291] kthread+0x1d1/0x200 [10124.233393] ? set_kthread_struct+0x70/0x70 [10124.240888] ret_from_fork+0x1f/0x30 [10128.058198] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10128.060338] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10128.063915] Modules linked in: lustre(O) [10128.067609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.067928] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.067934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.067938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.067955] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10128.067958] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10128.067963] RAX: 0000000000007d82 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10128.067966] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10128.067972] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10128.067974] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10128.067977] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10128.067980] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10128.067986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.067988] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10128.067995] Call Trace: [10128.068000] [10128.068007] ? show_regs.cold.9+0x22/0x2f [10128.068021] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.068027] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.068032] ? watchdog+0x40/0x40 [10128.068037] ? hrtimer_interrupt+0x128/0x2c0 [10128.068041] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.068049] ? apic_timer_interrupt+0xf/0x20 [10128.068055] [10128.068057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.068064] do_raw_spin_lock+0x177/0x190 [10128.068068] _raw_spin_lock+0x21/0x30 [10128.068074] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10128.068400] ? do_raw_spin_unlock+0x75/0x190 [10128.068406] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10128.069210] ? do_wait_intr+0xf0/0xf0 [10128.069218] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10128.069418] mdc_close+0x53b/0xd70 [mdc] [10128.069456] lmv_close+0x18d/0x490 [lmv] [10128.069596] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10128.069707] ll_md_real_close+0xa5/0x2a0 [lustre] [10128.069788] ll_md_close+0x274/0x800 [lustre] [10128.069869] ll_file_release+0x128/0xc50 [lustre] [10128.069955] __fput+0x10d/0x440 [10128.069966] ____fput+0x12/0x20 [10128.069971] task_work_run+0xc6/0x110 [10128.069978] exit_to_usermode_loop+0x1dd/0x1f0 [10128.069985] do_syscall_64+0x3d6/0x440 [10128.069990] entry_SYSCALL_64_after_hwframe+0x49/0xae [10128.069998] RIP: 0033:0x7fa749552cd8 [10128.070004] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10128.070008] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10128.070013] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10128.070016] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10128.070019] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10128.070022] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10128.070024] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10128.258676] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.277598] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.284332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.289232] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10128.291036] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10128.303580] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10128.307726] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10128.311587] RDX: 0000000000006677 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10128.315042] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10128.318605] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10128.321757] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10128.324106] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10128.327799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.330508] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10128.339182] Call Trace: [10128.340524] [10128.341702] ? show_regs.cold.9+0x22/0x2f [10128.344859] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.347665] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.349502] ? watchdog+0x40/0x40 [10128.352759] ? hrtimer_interrupt+0x128/0x2c0 [10128.354571] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.356400] ? apic_timer_interrupt+0xf/0x20 [10128.364176] [10128.369316] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10128.373050] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10128.378313] do_raw_spin_lock+0x177/0x190 [10128.381785] _raw_spin_lock+0x21/0x30 [10128.385455] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10128.388652] obd_mod_free+0x46/0x140 [ptlrpc] [10128.395261] mdc_commit_open+0x91/0xb0 [mdc] [10128.399798] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10128.404204] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10128.407473] after_reply+0xe2c/0x1780 [ptlrpc] [10128.410124] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10128.412447] ? __next_timer_interrupt+0x160/0x160 [10128.414312] ptlrpcd+0xa82/0xe40 [ptlrpc] [10128.418111] ? do_wait_intr+0xf0/0xf0 [10128.419948] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10128.422913] kthread+0x1d1/0x200 [10128.426718] ? set_kthread_struct+0x70/0x70 [10128.430476] ret_from_fork+0x1f/0x30 [10132.054030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10132.058333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10132.093923] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10132.112023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10132.117819] RIP: 0010:native_safe_halt+0x2e/0x50 [10132.121504] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10132.132328] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10132.138390] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10132.141717] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [10132.145563] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10132.148643] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10132.153524] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10132.156683] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10132.160995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10132.162962] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10132.166543] Call Trace: [10132.167704] [10132.168808] ? show_regs.cold.9+0x22/0x2f [10132.172748] ? watchdog_timer_fn.cold.6+0x62/0x10a [10132.179256] ? __hrtimer_run_queues+0x1c2/0x4d0 [10132.181472] ? watchdog+0x40/0x40 [10132.185129] ? hrtimer_interrupt+0x128/0x2c0 [10132.186344] ? smp_apic_timer_interrupt+0xa6/0x270 [10132.187672] ? apic_timer_interrupt+0xf/0x20 [10132.191490] [10132.193710] ? native_safe_halt+0x2e/0x50 [10132.195290] kvm_wait+0xcd/0xe0 [10132.197992] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10132.205491] do_raw_spin_lock+0x177/0x190 [10132.207340] _raw_spin_lock+0x21/0x30 [10132.208759] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10132.214055] ? do_raw_spin_unlock+0x75/0x190 [10132.217660] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10132.221731] ? do_wait_intr+0xf0/0xf0 [10132.224399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10132.226779] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10132.231240] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10132.234611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10132.237460] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10132.241473] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10132.242801] mdc_intent_lock+0x353/0x7e0 [mdc] [10132.245323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10132.248287] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10132.254459] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10132.258815] lmv_intent_open+0x338/0x1150 [lmv] [10132.261946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10132.266695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10132.268319] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10132.270517] ll_intent_lock+0xc5/0xaa0 [lustre] [10132.275522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10132.277659] ll_lookup_it+0xe0e/0x2270 [lustre] [10132.281816] ? do_raw_spin_unlock+0x75/0x190 [10132.285352] ? _raw_spin_unlock+0x12/0x30 [10132.289346] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10132.294592] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10132.297584] ll_atomic_open+0x36a/0xd70 [lustre] [10132.299048] lookup_open+0x356/0xb40 [10132.299985] do_last+0x313/0xfc0 [10132.304629] ? nd_jump_root+0xe5/0x160 [10132.305926] ? path_init+0x437/0x520 [10132.308903] path_openat+0xf7/0x500 [10132.311559] ? do_raw_spin_unlock+0x75/0x190 [10132.313827] do_filp_open+0x99/0x140 [10132.316796] ? getname_flags+0x6e/0x330 [10132.319552] ? __check_object_size+0xff/0x256 [10132.321587] ? do_raw_spin_unlock+0x75/0x190 [10132.322945] ? _raw_spin_unlock+0x12/0x30 [10132.325783] do_sys_openat2+0x2b4/0x410 [10132.327658] do_sys_open+0x73/0xa0 [10132.329726] __x64_sys_openat+0x24/0x30 [10132.330824] do_syscall_64+0xc1/0x440 [10132.332622] entry_SYSCALL_64_after_hwframe+0x49/0xae [10132.335876] RIP: 0033:0x7fa749552332 [10132.338222] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10132.347494] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10132.352352] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10132.357549] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10132.362548] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10132.367593] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10132.370649] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10135.521669] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8218s! [10135.526148] Showing busy workqueues and worker pools: [10135.529353] workqueue events: flags=0x0 [10135.532225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10135.537956] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10135.543676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10135.547675] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10135.553400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.557766] pending: kfree_rcu_monitor [10135.560969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10135.564890] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10135.570861] workqueue events_long: flags=0x0 [10135.573092] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.577312] pending: lu_objects_flush [obdclass] [10135.580538] workqueue events_unbound: flags=0x2 [10135.583851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10135.589191] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10135.594768] workqueue events_power_efficient: flags=0x80 [10135.597807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.601262] pending: check_lifetime [10135.603884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.607967] pending: neigh_periodic_work [10135.610140] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.612964] pending: neigh_periodic_work [10135.614931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.617882] pending: do_cache_clean [10135.619417] workqueue mm_percpu_wq: flags=0x8 [10135.622323] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.626800] pending: vmstat_update [10135.629655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.634087] pending: vmstat_update [10135.635918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10135.639936] pending: vmstat_update [10135.642397] workqueue writeback: flags=0x4a [10135.647881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10135.651903] pending: wb_workfn, wb_workfn [10135.656836] workqueue xprtiod: flags=0xa [10135.660150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10135.662535] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10135.667821] workqueue ipv6_addrconf: flags=0x40008 [10135.671782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10135.675054] pending: addrconf_verify_work [10135.677610] workqueue knbd0-recv: flags=0x1a [10135.679960] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10135.683252] in-flight: 651:recv_work [10135.687258] workqueue ptlrpc_pinger: flags=0x2 [10135.689579] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10135.693843] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10135.697519] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8218s workers=4 idle: 37 31846 28897 [10135.705207] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10125s workers=2 idle: 653 [10152.057117] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10152.067420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10152.115044] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10152.130190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10152.143168] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10152.152244] RIP: 0010:native_safe_halt+0x2e/0x50 [10152.156495] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10152.186355] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10152.198455] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10152.210563] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10152.218795] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10152.227340] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10152.238057] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10152.246965] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10152.260585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10152.269413] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10152.279101] Call Trace: [10152.279904] [10152.280409] ? show_regs.cold.9+0x22/0x2f [10152.281708] ? watchdog_timer_fn.cold.6+0x62/0x10a [10152.283097] ? __hrtimer_run_queues+0x1c2/0x4d0 [10152.284771] ? watchdog+0x40/0x40 [10152.285835] ? hrtimer_interrupt+0x128/0x2c0 [10152.286889] ? smp_apic_timer_interrupt+0xa6/0x270 [10152.288086] ? apic_timer_interrupt+0xf/0x20 [10152.289423] [10152.289962] ? native_safe_halt+0x2e/0x50 [10152.291038] kvm_wait+0xcd/0xe0 [10152.292015] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10152.294410] do_raw_spin_lock+0x177/0x190 [10152.296327] _raw_spin_lock+0x21/0x30 [10152.298101] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10152.300963] process_one_work+0x2c8/0x700 [10152.302313] worker_thread+0x296/0x6e0 [10152.304386] ? rescuer_thread+0x570/0x570 [10152.307321] kthread+0x1d1/0x200 [10152.309814] ? set_kthread_struct+0x70/0x70 [10152.312269] ret_from_fork+0x1f/0x30 [10156.059079] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10156.060534] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10156.065495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.065645] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.065652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.065655] RIP: 0010:native_safe_halt+0x2e/0x50 [10156.065667] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10156.065672] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10156.065676] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10156.065679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10156.065681] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10156.065683] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10156.065686] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10156.065692] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10156.065696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.065698] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10156.065705] Call Trace: [10156.065710] [10156.065717] ? show_regs.cold.9+0x22/0x2f [10156.065726] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.065733] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.065802] ? watchdog+0x40/0x40 [10156.065807] ? hrtimer_interrupt+0x128/0x2c0 [10156.065810] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.065817] ? apic_timer_interrupt+0xf/0x20 [10156.065821] [10156.065824] ? native_safe_halt+0x2e/0x50 [10156.065828] kvm_wait+0xcd/0xe0 [10156.065838] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10156.065845] do_raw_spin_lock+0x177/0x190 [10156.065849] _raw_spin_lock+0x21/0x30 [10156.065853] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10156.066347] ? do_raw_spin_unlock+0x75/0x190 [10156.066353] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10156.066722] ? do_wait_intr+0xf0/0xf0 [10156.066735] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10156.067246] mdc_close+0x53b/0xd70 [mdc] [10156.067286] lmv_close+0x18d/0x490 [lmv] [10156.067317] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10156.067426] ll_md_real_close+0xa5/0x2a0 [lustre] [10156.067513] ll_md_close+0x274/0x800 [lustre] [10156.067722] ll_file_release+0x128/0xc50 [lustre] [10156.067812] __fput+0x10d/0x440 [10156.067821] ____fput+0x12/0x20 [10156.067826] task_work_run+0xc6/0x110 [10156.067833] exit_to_usermode_loop+0x1dd/0x1f0 [10156.067850] do_syscall_64+0x3d6/0x440 [10156.067856] entry_SYSCALL_64_after_hwframe+0x49/0xae [10156.067869] RIP: 0033:0x7fa749552cd8 [10156.067879] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10156.067889] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10156.067894] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10156.067897] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10156.067899] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10156.067901] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10156.067903] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10156.237828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.255344] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.260160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.263287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10156.265322] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10156.274722] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10156.278518] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10156.285513] RDX: 0000000000000de2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10156.289264] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10156.292608] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10156.296613] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10156.299598] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10156.302712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.304959] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10156.307679] Call Trace: [10156.308708] [10156.309522] ? show_regs.cold.9+0x22/0x2f [10156.311297] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.313348] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.315660] ? watchdog+0x40/0x40 [10156.317189] ? hrtimer_interrupt+0x128/0x2c0 [10156.318955] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.321334] ? apic_timer_interrupt+0xf/0x20 [10156.323367] [10156.324276] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10156.327424] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10156.330248] do_raw_spin_lock+0x177/0x190 [10156.331539] _raw_spin_lock+0x21/0x30 [10156.334162] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10156.336673] obd_mod_free+0x46/0x140 [ptlrpc] [10156.339414] mdc_commit_open+0x91/0xb0 [mdc] [10156.341432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10156.344357] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10156.347837] after_reply+0xe2c/0x1780 [ptlrpc] [10156.350462] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10156.352888] ? __next_timer_interrupt+0x160/0x160 [10156.355233] ptlrpcd+0xa82/0xe40 [ptlrpc] [10156.358203] ? do_wait_intr+0xf0/0xf0 [10156.359206] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10156.365326] kthread+0x1d1/0x200 [10156.366180] ? set_kthread_struct+0x70/0x70 [10156.367227] ret_from_fork+0x1f/0x30 [10160.054033] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10160.057154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10160.076255] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10160.080965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10160.085038] RIP: 0010:native_safe_halt+0x2e/0x50 [10160.086736] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10160.095226] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10160.098785] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10160.101550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10160.104116] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10160.108533] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10160.112787] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10160.116623] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10160.120924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10160.124084] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10160.127866] Call Trace: [10160.129429] [10160.130019] ? show_regs.cold.9+0x22/0x2f [10160.131118] ? watchdog_timer_fn.cold.6+0x62/0x10a [10160.133132] ? __hrtimer_run_queues+0x1c2/0x4d0 [10160.136239] ? watchdog+0x40/0x40 [10160.138555] ? hrtimer_interrupt+0x128/0x2c0 [10160.141291] ? smp_apic_timer_interrupt+0xa6/0x270 [10160.144245] ? apic_timer_interrupt+0xf/0x20 [10160.146762] [10160.148171] ? native_safe_halt+0x2e/0x50 [10160.150371] kvm_wait+0xcd/0xe0 [10160.152525] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10160.155490] do_raw_spin_lock+0x177/0x190 [10160.158686] _raw_spin_lock+0x21/0x30 [10160.160718] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10160.163856] ? do_raw_spin_unlock+0x75/0x190 [10160.166890] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10160.171544] ? do_wait_intr+0xf0/0xf0 [10160.175132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10160.178164] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10160.181741] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10160.185130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10160.188323] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10160.191636] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10160.194375] mdc_intent_lock+0x353/0x7e0 [mdc] [10160.197254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10160.200360] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10160.204268] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10160.207142] lmv_intent_open+0x338/0x1150 [lmv] [10160.209842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10160.213316] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10160.216466] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10160.218765] ll_intent_lock+0xc5/0xaa0 [lustre] [10160.221500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10160.224511] ll_lookup_it+0xe0e/0x2270 [lustre] [10160.226229] ? do_raw_spin_unlock+0x75/0x190 [10160.228360] ? _raw_spin_unlock+0x12/0x30 [10160.230318] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10160.234009] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10160.236410] ll_atomic_open+0x36a/0xd70 [lustre] [10160.239943] lookup_open+0x356/0xb40 [10160.242020] do_last+0x313/0xfc0 [10160.243532] ? nd_jump_root+0xe5/0x160 [10160.245496] ? path_init+0x437/0x520 [10160.246700] path_openat+0xf7/0x500 [10160.248244] ? do_raw_spin_unlock+0x75/0x190 [10160.250265] do_filp_open+0x99/0x140 [10160.251849] ? getname_flags+0x6e/0x330 [10160.253860] ? __check_object_size+0xff/0x256 [10160.256190] ? do_raw_spin_unlock+0x75/0x190 [10160.258294] ? _raw_spin_unlock+0x12/0x30 [10160.260442] do_sys_openat2+0x2b4/0x410 [10160.262393] do_sys_open+0x73/0xa0 [10160.263931] __x64_sys_openat+0x24/0x30 [10160.265761] do_syscall_64+0xc1/0x440 [10160.268192] entry_SYSCALL_64_after_hwframe+0x49/0xae [10160.270783] RIP: 0033:0x7fa749552332 [10160.272624] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10160.282105] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10160.285670] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10160.288769] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10160.292463] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10160.295694] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10160.299146] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10165.733978] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8248s! [10165.737931] Showing busy workqueues and worker pools: [10165.740549] workqueue events: flags=0x0 [10165.743023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10165.754223] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10165.768618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10165.779422] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10165.787261] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.792107] pending: kfree_rcu_monitor [10165.797153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10165.803804] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10165.809118] workqueue events_long: flags=0x0 [10165.811225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.815142] pending: lu_objects_flush [obdclass] [10165.818058] workqueue events_unbound: flags=0x2 [10165.820255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10165.823818] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10165.833312] workqueue events_power_efficient: flags=0x80 [10165.836178] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.843255] pending: check_lifetime [10165.844626] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.853390] pending: neigh_periodic_work [10165.856286] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.860845] pending: neigh_periodic_work [10165.864055] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.871075] pending: do_cache_clean [10165.873330] workqueue mm_percpu_wq: flags=0x8 [10165.876177] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.880255] pending: vmstat_update [10165.883167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.891128] pending: vmstat_update [10165.893314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10165.897322] pending: vmstat_update [10165.899958] workqueue writeback: flags=0x4a [10165.901907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10165.904076] pending: wb_workfn, wb_workfn [10165.906331] workqueue xprtiod: flags=0xa [10165.908752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10165.912101] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10165.919234] workqueue ipv6_addrconf: flags=0x40008 [10165.922327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10165.927716] pending: addrconf_verify_work [10165.930527] workqueue knbd0-recv: flags=0x1a [10165.932836] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10165.937012] in-flight: 651:recv_work [10165.939466] workqueue ptlrpc_pinger: flags=0x2 [10165.943217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10165.950980] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10165.956833] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8248s workers=4 idle: 37 31846 28897 [10165.961993] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10155s workers=2 idle: 653 [10180.058118] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10180.070194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10180.149849] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10180.175368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10180.193709] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10180.195895] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10180.210754] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10180.253633] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10180.263086] RAX: 0000000000000b1d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10180.265433] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10180.279920] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10180.290163] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10180.302658] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10180.315856] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10180.318455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10180.338676] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10180.352282] Call Trace: [10180.354778] [10180.355765] ? show_regs.cold.9+0x22/0x2f [10180.358167] ? watchdog_timer_fn.cold.6+0x62/0x10a [10180.361413] ? __hrtimer_run_queues+0x1c2/0x4d0 [10180.364519] ? watchdog+0x40/0x40 [10180.367740] ? hrtimer_interrupt+0x128/0x2c0 [10180.371483] ? smp_apic_timer_interrupt+0xa6/0x270 [10180.380576] ? apic_timer_interrupt+0xf/0x20 [10180.383497] [10180.385800] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10180.388280] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10180.390710] do_raw_spin_lock+0x177/0x190 [10180.394581] _raw_spin_lock+0x21/0x30 [10180.397650] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10180.401718] process_one_work+0x2c8/0x700 [10180.405749] worker_thread+0x296/0x6e0 [10180.413411] ? rescuer_thread+0x570/0x570 [10180.416150] kthread+0x1d1/0x200 [10180.419594] ? set_kthread_struct+0x70/0x70 [10180.422864] ret_from_fork+0x1f/0x30 [10184.058057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10184.060402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.062033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10184.076318] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.076332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.076335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10184.076350] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10184.076354] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10184.076360] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10184.076363] RDX: 0000000000001559 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10184.076365] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10184.076367] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10184.076370] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10184.076373] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10184.076376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.076378] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10184.076386] Call Trace: [10184.076392] [10184.076398] ? show_regs.cold.9+0x22/0x2f [10184.076407] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.076414] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.076418] ? watchdog+0x40/0x40 [10184.076422] ? hrtimer_interrupt+0x128/0x2c0 [10184.076425] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.076431] ? apic_timer_interrupt+0xf/0x20 [10184.076435] [10184.076437] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10184.076441] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10184.076446] do_raw_spin_lock+0x177/0x190 [10184.076449] _raw_spin_lock+0x21/0x30 [10184.076456] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10184.076734] obd_mod_free+0x46/0x140 [ptlrpc] [10184.076926] mdc_commit_open+0x91/0xb0 [mdc] [10184.076969] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10184.077160] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10184.077357] after_reply+0xe2c/0x1780 [ptlrpc] [10184.077565] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10184.077759] ? __next_timer_interrupt+0x160/0x160 [10184.077768] ptlrpcd+0xa82/0xe40 [ptlrpc] [10184.077969] ? do_wait_intr+0xf0/0xf0 [10184.077977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10184.081074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.081147] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.081155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.081157] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.081171] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10184.081174] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10184.081178] RAX: 0000000000003c85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10184.081180] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10184.081183] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10184.081185] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10184.081188] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10184.081191] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10184.081193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.081195] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10184.081205] Call Trace: [10184.081210] [10184.081214] ? show_regs.cold.9+0x22/0x2f [10184.081223] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.081228] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.081233] ? watchdog+0x40/0x40 [10184.081237] ? hrtimer_interrupt+0x128/0x2c0 [10184.081242] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.081249] ? apic_timer_interrupt+0xf/0x20 [10184.081254] [10184.081257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.081263] do_raw_spin_lock+0x177/0x190 [10184.081267] _raw_spin_lock+0x21/0x30 [10184.081272] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10184.081530] ? do_raw_spin_unlock+0x75/0x190 [10184.086161] kthread+0x1d1/0x200 [10184.086180] ? set_kthread_struct+0x70/0x70 [10184.086183] ret_from_fork+0x1f/0x30 [10184.288827] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10184.291319] ? do_wait_intr+0xf0/0xf0 [10184.292872] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10184.295737] mdc_close+0x53b/0xd70 [mdc] [10184.297620] lmv_close+0x18d/0x490 [lmv] [10184.299399] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10184.301843] ll_md_real_close+0xa5/0x2a0 [lustre] [10184.304047] ll_md_close+0x274/0x800 [lustre] [10184.306486] ll_file_release+0x128/0xc50 [lustre] [10184.308348] __fput+0x10d/0x440 [10184.309660] ____fput+0x12/0x20 [10184.310835] task_work_run+0xc6/0x110 [10184.312426] exit_to_usermode_loop+0x1dd/0x1f0 [10184.314636] do_syscall_64+0x3d6/0x440 [10184.315838] entry_SYSCALL_64_after_hwframe+0x49/0xae [10184.317942] RIP: 0033:0x7fa749552cd8 [10184.319380] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10184.325964] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10184.329778] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10184.332920] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10184.336030] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10184.338897] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10184.341589] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10192.056305] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10192.067981] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10192.138337] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10192.161332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10192.180263] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10192.188356] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10192.215768] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10192.231341] RAX: 000000000000700e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10192.240938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10192.248630] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10192.254677] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10192.262381] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10192.272669] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10192.282745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10192.290870] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10192.299841] Call Trace: [10192.307301] [10192.309379] ? show_regs.cold.9+0x22/0x2f [10192.317050] ? watchdog_timer_fn.cold.6+0x62/0x10a [10192.322159] ? __hrtimer_run_queues+0x1c2/0x4d0 [10192.323817] ? watchdog+0x40/0x40 [10192.342704] ? hrtimer_interrupt+0x128/0x2c0 [10192.344213] ? smp_apic_timer_interrupt+0xa6/0x270 [10192.355679] ? apic_timer_interrupt+0xf/0x20 [10192.357169] [10192.357920] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10192.372647] do_raw_spin_lock+0x177/0x190 [10192.375636] _raw_spin_lock+0x21/0x30 [10192.377000] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10192.394293] ? do_raw_spin_unlock+0x75/0x190 [10192.395799] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10192.413741] ? do_wait_intr+0xf0/0xf0 [10192.415085] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10192.425450] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10192.431764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10192.447622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10192.457491] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10192.461362] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10192.463069] mdc_intent_lock+0x353/0x7e0 [mdc] [10192.474431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10192.478926] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10192.482583] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10192.486275] lmv_intent_open+0x338/0x1150 [lmv] [10192.489573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10192.491507] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10192.496917] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10192.501576] ll_intent_lock+0xc5/0xaa0 [lustre] [10192.511485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10192.520837] ll_lookup_it+0xe0e/0x2270 [lustre] [10192.522658] ? do_raw_spin_unlock+0x75/0x190 [10192.530495] ? _raw_spin_unlock+0x12/0x30 [10192.532837] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10192.542575] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10192.544370] ll_atomic_open+0x36a/0xd70 [lustre] [10192.553248] lookup_open+0x356/0xb40 [10192.558188] do_last+0x313/0xfc0 [10192.560835] ? nd_jump_root+0xe5/0x160 [10192.564327] ? path_init+0x437/0x520 [10192.566804] path_openat+0xf7/0x500 [10192.570425] ? do_raw_spin_unlock+0x75/0x190 [10192.573703] do_filp_open+0x99/0x140 [10192.576945] ? getname_flags+0x6e/0x330 [10192.582560] ? __check_object_size+0xff/0x256 [10192.590098] ? do_raw_spin_unlock+0x75/0x190 [10192.591930] ? _raw_spin_unlock+0x12/0x30 [10192.599960] do_sys_openat2+0x2b4/0x410 [10192.603427] do_sys_open+0x73/0xa0 [10192.604565] __x64_sys_openat+0x24/0x30 [10192.610508] do_syscall_64+0xc1/0x440 [10192.616798] entry_SYSCALL_64_after_hwframe+0x49/0xae [10192.622619] RIP: 0033:0x7fa749552332 [10192.627976] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10192.641311] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10192.656590] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10192.663942] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10192.676553] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10192.684047] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10192.691658] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10196.452682] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8279s! [10196.459544] Showing busy workqueues and worker pools: [10196.466528] workqueue events: flags=0x0 [10196.472704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10196.475189] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10196.495828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10196.507661] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10196.514743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.521307] pending: kfree_rcu_monitor [10196.525685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10196.530338] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10196.539919] workqueue events_long: flags=0x0 [10196.541865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.546775] pending: lu_objects_flush [obdclass] [10196.549883] workqueue events_unbound: flags=0x2 [10196.552958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10196.555362] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10196.559944] workqueue events_power_efficient: flags=0x80 [10196.564670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.568475] pending: check_lifetime [10196.570155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.575088] pending: neigh_periodic_work [10196.582194] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.586380] pending: neigh_periodic_work [10196.589561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.593749] pending: do_cache_clean [10196.595910] workqueue mm_percpu_wq: flags=0x8 [10196.598968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.602383] pending: vmstat_update [10196.604424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.610328] pending: vmstat_update [10196.614471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10196.621935] pending: vmstat_update [10196.627218] workqueue writeback: flags=0x4a [10196.631575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10196.637969] pending: wb_workfn, wb_workfn [10196.642120] workqueue xprtiod: flags=0xa [10196.646570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10196.655524] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10196.663404] workqueue ipv6_addrconf: flags=0x40008 [10196.666377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10196.672019] pending: addrconf_verify_work [10196.680018] workqueue knbd0-recv: flags=0x1a [10196.685897] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10196.691137] in-flight: 651:recv_work [10196.692524] workqueue ptlrpc_pinger: flags=0x2 [10196.693945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10196.697911] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10196.702437] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8279s workers=4 idle: 37 31846 28897 [10196.708469] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10186s workers=2 idle: 653 [10208.059134] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10208.065648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10208.120349] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10208.130437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10208.137518] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10208.149277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10208.156254] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10208.167600] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10208.173708] RAX: 0000000000004807 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10208.182852] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10208.185328] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10208.195150] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10208.201952] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10208.207805] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10208.214993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10208.217931] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10208.222140] Call Trace: [10208.225571] [10208.228077] ? show_regs.cold.9+0x22/0x2f [10208.232099] ? watchdog_timer_fn.cold.6+0x62/0x10a [10208.235614] ? __hrtimer_run_queues+0x1c2/0x4d0 [10208.247057] ? watchdog+0x40/0x40 [10208.252356] ? hrtimer_interrupt+0x128/0x2c0 [10208.258221] ? smp_apic_timer_interrupt+0xa6/0x270 [10208.263804] ? apic_timer_interrupt+0xf/0x20 [10208.271245] [10208.273627] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10208.278424] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10208.281973] do_raw_spin_lock+0x177/0x190 [10208.294957] _raw_spin_lock+0x21/0x30 [10208.297294] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10208.304619] process_one_work+0x2c8/0x700 [10208.306020] worker_thread+0x296/0x6e0 [10208.316048] ? rescuer_thread+0x570/0x570 [10208.319650] kthread+0x1d1/0x200 [10208.322284] ? set_kthread_struct+0x70/0x70 [10208.327946] ret_from_fork+0x1f/0x30 [10212.058117] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10212.059033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10212.063522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.063607] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.063614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.063617] RIP: 0010:native_safe_halt+0x2e/0x50 [10212.063630] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10212.063635] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10212.063640] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10212.063642] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10212.063645] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10212.063647] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10212.063650] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10212.063653] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10212.063656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.063658] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10212.063668] Call Trace: [10212.063673] [10212.063680] ? show_regs.cold.9+0x22/0x2f [10212.063690] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.063696] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.063701] ? watchdog+0x40/0x40 [10212.063706] ? hrtimer_interrupt+0x128/0x2c0 [10212.063711] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.063718] ? apic_timer_interrupt+0xf/0x20 [10212.063723] [10212.063726] ? native_safe_halt+0x2e/0x50 [10212.063730] kvm_wait+0xcd/0xe0 [10212.063739] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10212.063746] do_raw_spin_lock+0x177/0x190 [10212.063752] _raw_spin_lock+0x21/0x30 [10212.063892] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10212.064448] obd_mod_free+0x46/0x140 [ptlrpc] [10212.071117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [10212.112721] mdc_commit_open+0x91/0xb0 [mdc] [10212.117764] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.118220] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.118227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.118230] RIP: 0010:native_safe_halt+0x2e/0x50 [10212.118242] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10212.118250] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10212.118254] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10212.118256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10212.118258] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10212.118260] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10212.118263] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10212.118266] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10212.118269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.118271] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10212.118279] Call Trace: [10212.118285] [10212.118291] ? show_regs.cold.9+0x22/0x2f [10212.118301] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.118308] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.118312] ? watchdog+0x40/0x40 [10212.118316] ? hrtimer_interrupt+0x128/0x2c0 [10212.118320] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.118327] ? apic_timer_interrupt+0xf/0x20 [10212.118331] [10212.118334] ? native_safe_halt+0x2e/0x50 [10212.118338] kvm_wait+0xcd/0xe0 [10212.118345] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10212.118354] do_raw_spin_lock+0x177/0x190 [10212.118358] _raw_spin_lock+0x21/0x30 [10212.118363] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10212.118646] ? do_raw_spin_unlock+0x75/0x190 [10212.118653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10212.129862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10212.130460] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10212.138590] ? do_wait_intr+0xf0/0xf0 [10212.173914] after_reply+0xe2c/0x1780 [ptlrpc] [10212.174511] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10212.174755] ? __next_timer_interrupt+0x160/0x160 [10212.174766] ptlrpcd+0xa82/0xe40 [ptlrpc] [10212.175141] ? do_wait_intr+0xf0/0xf0 [10212.175150] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10212.175585] kthread+0x1d1/0x200 [10212.175592] ? set_kthread_struct+0x70/0x70 [10212.175601] ret_from_fork+0x1f/0x30 [10212.532784] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10212.538006] mdc_close+0x53b/0xd70 [mdc] [10212.541340] lmv_close+0x18d/0x490 [lmv] [10212.544965] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10212.548253] ll_md_real_close+0xa5/0x2a0 [lustre] [10212.553893] ll_md_close+0x274/0x800 [lustre] [10212.559019] ll_file_release+0x128/0xc50 [lustre] [10212.564081] __fput+0x10d/0x440 [10212.564975] ____fput+0x12/0x20 [10212.565801] task_work_run+0xc6/0x110 [10212.569335] exit_to_usermode_loop+0x1dd/0x1f0 [10212.572976] do_syscall_64+0x3d6/0x440 [10212.580080] entry_SYSCALL_64_after_hwframe+0x49/0xae [10212.586538] RIP: 0033:0x7fa749552cd8 [10212.589100] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10212.617738] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10212.620349] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10212.626516] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10212.632558] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10212.640730] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10212.654387] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10224.054023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10224.056255] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10224.072455] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10224.081626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10224.087072] RIP: 0010:native_safe_halt+0x2e/0x50 [10224.089238] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10224.098121] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10224.101805] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10224.104613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [10224.107662] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10224.110563] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10224.116385] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10224.124150] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10224.131278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10224.145343] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10224.156376] Call Trace: [10224.158474] [10224.159337] ? show_regs.cold.9+0x22/0x2f [10224.161174] ? watchdog_timer_fn.cold.6+0x62/0x10a [10224.166191] ? __hrtimer_run_queues+0x1c2/0x4d0 [10224.167501] ? watchdog+0x40/0x40 [10224.168453] ? hrtimer_interrupt+0x128/0x2c0 [10224.178097] ? smp_apic_timer_interrupt+0xa6/0x270 [10224.179687] ? apic_timer_interrupt+0xf/0x20 [10224.183466] [10224.184434] ? native_safe_halt+0x2e/0x50 [10224.190911] kvm_wait+0xcd/0xe0 [10224.192058] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10224.203451] do_raw_spin_lock+0x177/0x190 [10224.204750] _raw_spin_lock+0x21/0x30 [10224.207725] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10224.209666] ? do_raw_spin_unlock+0x75/0x190 [10224.214762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10224.219562] ? do_wait_intr+0xf0/0xf0 [10224.225049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10224.240311] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10224.242448] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10224.251383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10224.257247] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10224.263182] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10224.268479] mdc_intent_lock+0x353/0x7e0 [mdc] [10224.273315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10224.276590] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10224.281393] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10224.286098] lmv_intent_open+0x338/0x1150 [lmv] [10224.287623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10224.293062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10224.296124] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10224.302028] ll_intent_lock+0xc5/0xaa0 [lustre] [10224.310543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10224.314758] ll_lookup_it+0xe0e/0x2270 [lustre] [10224.318043] ? do_raw_spin_unlock+0x75/0x190 [10224.319618] ? _raw_spin_unlock+0x12/0x30 [10224.321996] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10224.330682] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10224.340441] ll_atomic_open+0x36a/0xd70 [lustre] [10224.354024] lookup_open+0x356/0xb40 [10224.357719] do_last+0x313/0xfc0 [10224.367120] ? nd_jump_root+0xe5/0x160 [10224.368514] ? path_init+0x437/0x520 [10224.369771] path_openat+0xf7/0x500 [10224.380692] ? do_raw_spin_unlock+0x75/0x190 [10224.382220] do_filp_open+0x99/0x140 [10224.384506] ? getname_flags+0x6e/0x330 [10224.386455] ? __check_object_size+0xff/0x256 [10224.388318] ? do_raw_spin_unlock+0x75/0x190 [10224.391535] ? _raw_spin_unlock+0x12/0x30 [10224.397396] do_sys_openat2+0x2b4/0x410 [10224.401231] do_sys_open+0x73/0xa0 [10224.403640] __x64_sys_openat+0x24/0x30 [10224.406630] do_syscall_64+0xc1/0x440 [10224.410019] entry_SYSCALL_64_after_hwframe+0x49/0xae [10224.414576] RIP: 0033:0x7fa749552332 [10224.417246] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10224.432214] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10224.435313] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10224.446787] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10224.454271] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10224.457765] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10224.461333] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10227.176263] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8310s! [10227.186575] Showing busy workqueues and worker pools: [10227.192526] workqueue events: flags=0x0 [10227.195135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10227.198271] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10227.210576] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10227.223841] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10227.248640] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.264604] pending: kfree_rcu_monitor [10227.267010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10227.277966] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10227.294253] workqueue events_long: flags=0x0 [10227.299334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.314198] pending: lu_objects_flush [obdclass] [10227.326451] workqueue events_unbound: flags=0x2 [10227.329925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10227.336718] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10227.350455] workqueue events_power_efficient: flags=0x80 [10227.357876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.360792] pending: check_lifetime [10227.365162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.372886] pending: neigh_periodic_work [10227.385993] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.400327] pending: neigh_periodic_work [10227.411785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.417248] pending: do_cache_clean [10227.425121] workqueue mm_percpu_wq: flags=0x8 [10227.431854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.442678] pending: vmstat_update [10227.453301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.469986] pending: vmstat_update [10227.480241] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10227.491917] pending: vmstat_update [10227.501256] workqueue writeback: flags=0x4a [10227.511278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10227.521297] pending: wb_workfn, wb_workfn [10227.526390] workqueue xprtiod: flags=0xa [10227.532102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10227.543296] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10227.566268] workqueue ipv6_addrconf: flags=0x40008 [10227.576656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10227.588287] pending: addrconf_verify_work [10227.601062] workqueue knbd0-recv: flags=0x1a [10227.611426] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10227.625743] in-flight: 651:recv_work [10227.633342] workqueue ptlrpc_pinger: flags=0x2 [10227.638700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10227.651163] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10227.660192] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8310s workers=4 idle: 37 31846 28897 [10227.677601] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10217s workers=2 idle: 653 [10236.057645] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10236.063041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10236.086082] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10236.092720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10236.097934] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10236.101618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10236.106434] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10236.121581] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10236.128129] RAX: 0000000000006e06 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10236.134267] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10236.137383] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10236.144580] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10236.151045] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10236.155605] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10236.160584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10236.164213] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10236.172032] Call Trace: [10236.173628] [10236.174490] ? show_regs.cold.9+0x22/0x2f [10236.176910] ? watchdog_timer_fn.cold.6+0x62/0x10a [10236.181224] ? __hrtimer_run_queues+0x1c2/0x4d0 [10236.184709] ? watchdog+0x40/0x40 [10236.187176] ? hrtimer_interrupt+0x128/0x2c0 [10236.193488] ? smp_apic_timer_interrupt+0xa6/0x270 [10236.197725] ? apic_timer_interrupt+0xf/0x20 [10236.201128] [10236.203277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10236.207329] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10236.211705] do_raw_spin_lock+0x177/0x190 [10236.220063] _raw_spin_lock+0x21/0x30 [10236.221487] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10236.223452] process_one_work+0x2c8/0x700 [10236.239649] worker_thread+0x296/0x6e0 [10236.245586] ? rescuer_thread+0x570/0x570 [10236.253445] kthread+0x1d1/0x200 [10236.259569] ? set_kthread_struct+0x70/0x70 [10236.266742] ret_from_fork+0x1f/0x30 [10240.059017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10240.061741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.062270] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10240.086113] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.086127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.086130] RIP: 0010:native_safe_halt+0x2e/0x50 [10240.086143] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10240.086147] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10240.086152] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10240.086155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10240.086157] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10240.086159] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10240.086161] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10240.086164] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10240.086167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.086169] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10240.086177] Call Trace: [10240.086183] [10240.086189] ? show_regs.cold.9+0x22/0x2f [10240.086200] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.086207] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.086212] ? watchdog+0x40/0x40 [10240.086216] ? hrtimer_interrupt+0x128/0x2c0 [10240.086220] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.086229] ? apic_timer_interrupt+0xf/0x20 [10240.086233] [10240.086236] ? native_safe_halt+0x2e/0x50 [10240.086240] kvm_wait+0xcd/0xe0 [10240.086248] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10240.086259] do_raw_spin_lock+0x177/0x190 [10240.086329] _raw_spin_lock+0x21/0x30 [10240.086336] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10240.086685] ? do_raw_spin_unlock+0x75/0x190 [10240.086691] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10240.086885] ? do_wait_intr+0xf0/0xf0 [10240.086893] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10240.087093] mdc_close+0x53b/0xd70 [mdc] [10240.087134] lmv_close+0x18d/0x490 [lmv] [10240.087162] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10240.087350] ll_md_real_close+0xa5/0x2a0 [lustre] [10240.087433] ll_md_close+0x274/0x800 [lustre] [10240.087520] ll_file_release+0x128/0xc50 [lustre] [10240.087687] __fput+0x10d/0x440 [10240.087698] ____fput+0x12/0x20 [10240.087703] task_work_run+0xc6/0x110 [10240.087710] exit_to_usermode_loop+0x1dd/0x1f0 [10240.087717] do_syscall_64+0x3d6/0x440 [10240.087723] entry_SYSCALL_64_after_hwframe+0x49/0xae [10240.087729] RIP: 0033:0x7fa749552cd8 [10240.087736] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10240.087741] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10240.087746] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10240.087748] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10240.087750] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10240.087752] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10240.087755] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10240.454697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.487552] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.492028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.496049] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10240.499062] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10240.506979] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10240.510359] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10240.514034] RDX: 000000000000781e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10240.516754] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10240.520881] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10240.523926] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10240.531749] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10240.547162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.554838] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10240.567124] Call Trace: [10240.569606] [10240.571053] ? show_regs.cold.9+0x22/0x2f [10240.577708] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.585670] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.587133] ? watchdog+0x40/0x40 [10240.600044] ? hrtimer_interrupt+0x128/0x2c0 [10240.609928] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.615064] ? apic_timer_interrupt+0xf/0x20 [10240.621987] [10240.625506] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10240.633662] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10240.646065] do_raw_spin_lock+0x177/0x190 [10240.647452] _raw_spin_lock+0x21/0x30 [10240.648787] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10240.665743] obd_mod_free+0x46/0x140 [ptlrpc] [10240.669932] mdc_commit_open+0x91/0xb0 [mdc] [10240.673182] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10240.686153] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10240.696056] after_reply+0xe2c/0x1780 [ptlrpc] [10240.705637] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10240.712172] ? __next_timer_interrupt+0x160/0x160 [10240.717623] ptlrpcd+0xa82/0xe40 [ptlrpc] [10240.728812] ? do_wait_intr+0xf0/0xf0 [10240.730108] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10240.741889] kthread+0x1d1/0x200 [10240.747622] ? set_kthread_struct+0x70/0x70 [10240.758356] ret_from_fork+0x1f/0x30 [10252.055085] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10252.057245] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10252.075820] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10252.082776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10252.091323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10252.094637] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10252.102964] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10252.106213] RAX: 00000000000032ae RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10252.112561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10252.115249] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10252.120004] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10252.129479] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10252.138001] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10252.142583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10252.144817] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10252.147888] Call Trace: [10252.148886] [10252.150218] ? show_regs.cold.9+0x22/0x2f [10252.151789] ? watchdog_timer_fn.cold.6+0x62/0x10a [10252.154864] ? __hrtimer_run_queues+0x1c2/0x4d0 [10252.162436] ? watchdog+0x40/0x40 [10252.164925] ? hrtimer_interrupt+0x128/0x2c0 [10252.166720] ? smp_apic_timer_interrupt+0xa6/0x270 [10252.171596] ? apic_timer_interrupt+0xf/0x20 [10252.174134] [10252.184828] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10252.193581] do_raw_spin_lock+0x177/0x190 [10252.196966] _raw_spin_lock+0x21/0x30 [10252.200958] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10252.204470] ? do_raw_spin_unlock+0x75/0x190 [10252.206864] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10252.209226] ? do_wait_intr+0xf0/0xf0 [10252.210906] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10252.213555] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10252.217373] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10252.224183] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10252.225862] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10252.228021] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10252.238516] mdc_intent_lock+0x353/0x7e0 [mdc] [10252.239946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10252.247962] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10252.250919] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10252.256662] lmv_intent_open+0x338/0x1150 [lmv] [10252.258056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10252.263178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10252.273130] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10252.274362] ll_intent_lock+0xc5/0xaa0 [lustre] [10252.275757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10252.286731] ll_lookup_it+0xe0e/0x2270 [lustre] [10252.291887] ? do_raw_spin_unlock+0x75/0x190 [10252.303300] ? _raw_spin_unlock+0x12/0x30 [10252.306088] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10252.307573] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10252.309211] ll_atomic_open+0x36a/0xd70 [lustre] [10252.322060] lookup_open+0x356/0xb40 [10252.323940] do_last+0x313/0xfc0 [10252.330676] ? nd_jump_root+0xe5/0x160 [10252.332391] ? path_init+0x437/0x520 [10252.333992] path_openat+0xf7/0x500 [10252.335044] ? do_raw_spin_unlock+0x75/0x190 [10252.336103] do_filp_open+0x99/0x140 [10252.336973] ? getname_flags+0x6e/0x330 [10252.340397] ? __check_object_size+0xff/0x256 [10252.345327] ? do_raw_spin_unlock+0x75/0x190 [10252.349628] ? _raw_spin_unlock+0x12/0x30 [10252.352828] do_sys_openat2+0x2b4/0x410 [10252.355677] do_sys_open+0x73/0xa0 [10252.358774] __x64_sys_openat+0x24/0x30 [10252.361202] do_syscall_64+0xc1/0x440 [10252.363619] entry_SYSCALL_64_after_hwframe+0x49/0xae [10252.366207] RIP: 0033:0x7fa749552332 [10252.368248] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10252.377977] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10252.382199] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10252.386033] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10252.389763] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10252.393551] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10252.397149] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10253.868772] rcu: INFO: rcu_sched self-detected stall on CPU [10253.872798] rcu: 2-...!: (4680404 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [10253.876708] (t=8340179 jiffies g=258701 q=3975) [10253.881805] rcu: rcu_sched kthread starved for 8336974 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10253.893098] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10253.901697] rcu: RCU grace-period kthread stack dump: [10253.904399] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10253.911865] Call Trace: [10253.916269] __schedule+0x351/0xcb0 [10253.917740] schedule+0xc0/0x180 [10253.920282] schedule_timeout+0xb4/0x190 [10253.924291] ? do_raw_spin_unlock+0x75/0x190 [10253.925727] ? __next_timer_interrupt+0x160/0x160 [10253.931648] ? prepare_to_swait_event+0xf3/0x140 [10253.934612] rcu_gp_kthread+0x8dd/0x15b0 [10253.939338] ? force_qs_rnp+0x2c0/0x2c0 [10253.941526] kthread+0x1d1/0x200 [10253.945501] ? set_kthread_struct+0x70/0x70 [10253.946981] ret_from_fork+0x1f/0x30 [10253.952801] rcu: Stack dump where RCU GP kthread last ran: [10253.957325] Sending NMI from CPU 2 to CPUs 0: [10253.970876] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 7.907 msecs [10253.972324] NMI backtrace for cpu 0 [10253.972330] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10253.972332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10253.972334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10253.972335] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10253.972337] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [10253.972340] RAX: 0000000000002b8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10253.972342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10253.972343] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10253.972344] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10253.972346] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10253.972347] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10253.972348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10253.972350] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10253.972351] Call Trace: [10253.972352] [10253.972353] ? show_regs.cold.9+0x22/0x2f [10253.972354] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10253.972355] ? nmi_cpu_backtrace_handler+0x15/0x40 [10253.972356] ? nmi_handle+0xbd/0x210 [10253.972358] ? default_do_nmi+0x4b/0x1b0 [10253.972359] ? do_nmi+0x280/0x380 [10253.972360] ? end_repeat_nmi+0x16/0x1f [10253.972361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10253.972362] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10253.972364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10253.972365] [10253.972366] do_raw_spin_lock+0x177/0x190 [10253.972367] _raw_spin_lock+0x21/0x30 [10253.972368] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10253.972369] ? do_raw_spin_unlock+0x75/0x190 [10253.972370] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10253.972371] ? do_wait_intr+0xf0/0xf0 [10253.972373] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10253.972374] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10253.972375] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10253.972377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10253.972378] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10253.972379] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10253.972380] mdc_intent_lock+0x353/0x7e0 [mdc] [10253.972382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10253.972383] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10253.972384] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10253.972386] lmv_intent_open+0x338/0x1150 [lmv] [10253.972387] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10253.972388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10253.972389] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10253.972390] ll_intent_lock+0xc5/0xaa0 [lustre] [10253.972392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10253.972393] ll_lookup_it+0xe0e/0x2270 [lustre] [10253.972394] ? do_raw_spin_unlock+0x75/0x190 [10253.972395] ? _raw_spin_unlock+0x12/0x30 [10253.972396] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10253.972397] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10253.972399] ll_atomic_open+0x36a/0xd70 [lustre] [10253.972400] lookup_open+0x356/0xb40 [10253.972401] do_last+0x313/0xfc0 [10253.972402] ? nd_jump_root+0xe5/0x160 [10253.972403] ? path_init+0x437/0x520 [10253.972404] path_openat+0xf7/0x500 [10253.972405] ? do_raw_spin_unlock+0x75/0x190 [10253.972406] do_filp_open+0x99/0x140 [10253.972407] ? getname_flags+0x6e/0x330 [10253.972408] ? __check_object_size+0xff/0x256 [10253.972409] ? do_raw_spin_unlock+0x75/0x190 [10253.972410] ? _raw_spin_unlock+0x12/0x30 [10253.972411] do_sys_openat2+0x2b4/0x410 [10253.972412] do_sys_open+0x73/0xa0 [10253.972413] __x64_sys_openat+0x24/0x30 [10253.972414] do_syscall_64+0xc1/0x440 [10253.972416] entry_SYSCALL_64_after_hwframe+0x49/0xae [10253.972417] RIP: 0033:0x7fa749552332 [10253.972419] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10253.972422] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10253.972424] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10253.972426] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10253.972428] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10253.972430] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10253.972431] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10253.972439] NMI backtrace for cpu 2 [10254.271818] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10254.277380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10254.281806] Call Trace: [10254.282777] [10254.285460] ? dump_stack+0xbb/0x10e [10254.289999] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10254.296070] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10254.299820] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10254.302954] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10254.304346] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10254.308145] ? rcu_dump_cpu_stacks+0x11b/0x171 [10254.311458] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10254.315993] ? update_process_times+0x6d/0xc0 [10254.317875] ? tick_sched_handle.isra.18+0x27/0x90 [10254.320104] ? tick_sched_timer+0x53/0xb0 [10254.321775] ? __hrtimer_run_queues+0x1c2/0x4d0 [10254.324233] ? tick_sched_handle.isra.18+0x90/0x90 [10254.326417] ? hrtimer_interrupt+0x128/0x2c0 [10254.328670] ? smp_apic_timer_interrupt+0xa6/0x270 [10254.331254] ? apic_timer_interrupt+0xf/0x20 [10254.333007] [10254.334108] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10254.336473] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10254.339621] ? do_raw_spin_lock+0x177/0x190 [10254.341670] ? _raw_spin_lock+0x21/0x30 [10254.343634] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10254.346439] ? obd_mod_free+0x46/0x140 [ptlrpc] [10254.350587] ? mdc_commit_open+0x91/0xb0 [mdc] [10254.352603] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10254.354943] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10254.358279] ? after_reply+0xe2c/0x1780 [ptlrpc] [10254.360874] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10254.363864] ? __next_timer_interrupt+0x160/0x160 [10254.366148] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10254.372796] ? do_wait_intr+0xf0/0xf0 [10254.374456] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10254.377495] ? kthread+0x1d1/0x200 [10254.378770] ? set_kthread_struct+0x70/0x70 [10254.380743] ? ret_from_fork+0x1f/0x30 [10257.900571] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8340s! [10257.903620] Showing busy workqueues and worker pools: [10257.906495] workqueue events: flags=0x0 [10257.908202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10257.911708] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10257.923388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10257.927947] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10257.939412] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10257.950225] pending: kfree_rcu_monitor [10257.957029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10257.961031] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10257.974685] workqueue events_long: flags=0x0 [10257.975789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10257.979759] pending: lu_objects_flush [obdclass] [10257.982479] workqueue events_unbound: flags=0x2 [10257.985869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10257.991036] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10257.998813] workqueue events_power_efficient: flags=0x80 [10258.003977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.009860] pending: check_lifetime [10258.014066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.019538] pending: neigh_periodic_work [10258.025269] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.035012] pending: neigh_periodic_work [10258.047777] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.053883] pending: do_cache_clean [10258.056893] workqueue mm_percpu_wq: flags=0x8 [10258.063152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.067184] pending: vmstat_update [10258.072334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.081320] pending: vmstat_update [10258.085089] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.090861] pending: vmstat_update [10258.093102] workqueue writeback: flags=0x4a [10258.098386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10258.109001] pending: wb_workfn, wb_workfn [10258.114616] workqueue xprtiod: flags=0xa [10258.121902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10258.125791] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10258.129884] workqueue ipv6_addrconf: flags=0x40008 [10258.133352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10258.142869] pending: addrconf_verify_work [10258.145817] workqueue knbd0-recv: flags=0x1a [10258.149202] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10258.153145] in-flight: 651:recv_work [10258.155231] workqueue ptlrpc_pinger: flags=0x2 [10258.159932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10258.165429] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10258.176367] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8341s workers=4 idle: 37 31846 28897 [10258.185747] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10248s workers=2 idle: 653 [10264.057025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10264.060013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10264.081715] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10264.086028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10264.090506] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10264.093367] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.096251] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10264.104578] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10264.108098] RAX: 000000000000561e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10264.111226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10264.114061] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10264.117619] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10264.120815] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10264.123829] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10264.127765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10264.130286] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10264.133689] Call Trace: [10264.134897] [10264.135873] ? show_regs.cold.9+0x22/0x2f [10264.137615] ? watchdog_timer_fn.cold.6+0x62/0x10a [10264.140078] ? __hrtimer_run_queues+0x1c2/0x4d0 [10264.142230] ? watchdog+0x40/0x40 [10264.143950] ? hrtimer_interrupt+0x128/0x2c0 [10264.145718] ? smp_apic_timer_interrupt+0xa6/0x270 [10264.147756] ? apic_timer_interrupt+0xf/0x20 [10264.149697] [10264.150576] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.152981] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10264.155352] do_raw_spin_lock+0x177/0x190 [10264.156643] _raw_spin_lock+0x21/0x30 [10264.158661] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10264.161238] process_one_work+0x2c8/0x700 [10264.163005] worker_thread+0x296/0x6e0 [10264.164532] ? rescuer_thread+0x570/0x570 [10264.166418] kthread+0x1d1/0x200 [10264.167589] ? set_kthread_struct+0x70/0x70 [10264.169557] ret_from_fork+0x1f/0x30 [10268.059761] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10268.064643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10268.098920] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10268.114176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10268.123616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [10268.127539] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [10268.141798] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10268.150524] RAX: 000000000000402f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10268.159497] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10268.167936] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10268.179870] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10268.186825] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10268.190911] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10268.200376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10268.203756] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10268.209912] Call Trace: [10268.218557] [10268.219368] ? show_regs.cold.9+0x22/0x2f [10268.229608] ? watchdog_timer_fn.cold.6+0x62/0x10a [10268.231420] ? __hrtimer_run_queues+0x1c2/0x4d0 [10268.240532] ? watchdog+0x40/0x40 [10268.243538] ? hrtimer_interrupt+0x128/0x2c0 [10268.248382] ? smp_apic_timer_interrupt+0xa6/0x270 [10268.255537] ? apic_timer_interrupt+0xf/0x20 [10268.262374] [10268.265937] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [10268.271068] do_raw_spin_lock+0x177/0x190 [10268.274472] _raw_spin_lock+0x21/0x30 [10268.276567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10268.280412] ? do_raw_spin_unlock+0x75/0x190 [10268.288310] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10268.296451] ? do_wait_intr+0xf0/0xf0 [10268.297694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10268.311988] mdc_close+0x53b/0xd70 [mdc] [10268.323819] lmv_close+0x18d/0x490 [lmv] [10268.325274] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10268.329538] ll_md_real_close+0xa5/0x2a0 [lustre] [10268.338746] ll_md_close+0x274/0x800 [lustre] [10268.343500] ll_file_release+0x128/0xc50 [lustre] [10268.348154] __fput+0x10d/0x440 [10268.351270] ____fput+0x12/0x20 [10268.352464] task_work_run+0xc6/0x110 [10268.357564] exit_to_usermode_loop+0x1dd/0x1f0 [10268.366469] do_syscall_64+0x3d6/0x440 [10268.367800] entry_SYSCALL_64_after_hwframe+0x49/0xae [10268.372856] RIP: 0033:0x7fa749552cd8 [10268.378349] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10268.401000] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10268.411385] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10268.413869] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10268.443477] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10268.446655] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10268.471542] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10280.065034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10280.069702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10280.099882] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10280.105282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10280.114240] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10280.120155] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10280.130127] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10280.142108] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10280.148080] RDX: 0000000000003957 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10280.150258] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10280.153265] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10280.158024] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10280.162416] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10280.166494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10280.168882] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10280.173289] Call Trace: [10280.176917] [10280.177533] ? show_regs.cold.9+0x22/0x2f [10280.179412] ? watchdog_timer_fn.cold.6+0x62/0x10a [10280.183405] ? __hrtimer_run_queues+0x1c2/0x4d0 [10280.185889] ? watchdog+0x40/0x40 [10280.188416] ? hrtimer_interrupt+0x128/0x2c0 [10280.193335] ? smp_apic_timer_interrupt+0xa6/0x270 [10280.197372] ? apic_timer_interrupt+0xf/0x20 [10280.199417] [10280.200342] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10280.203523] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10280.205489] do_raw_spin_lock+0x177/0x190 [10280.208201] _raw_spin_lock+0x21/0x30 [10280.212235] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10280.214772] obd_mod_free+0x46/0x140 [ptlrpc] [10280.217998] mdc_commit_open+0x91/0xb0 [mdc] [10280.219777] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10280.221626] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10280.224399] after_reply+0xe2c/0x1780 [ptlrpc] [10280.227687] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10280.232148] ? __next_timer_interrupt+0x160/0x160 [10280.234319] ptlrpcd+0xa82/0xe40 [ptlrpc] [10280.241165] ? do_wait_intr+0xf0/0xf0 [10280.243126] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10280.245984] kthread+0x1d1/0x200 [10280.249395] ? set_kthread_struct+0x70/0x70 [10280.251213] ret_from_fork+0x1f/0x30 [10284.054030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10284.057679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10284.078017] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10284.083602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10284.088017] RIP: 0010:native_safe_halt+0x2e/0x50 [10284.090256] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10284.098685] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10284.104139] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10284.107332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10284.111203] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10284.118573] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10284.124918] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10284.129837] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10284.133693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10284.136517] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10284.140270] Call Trace: [10284.141506] [10284.142612] ? show_regs.cold.9+0x22/0x2f [10284.144546] ? watchdog_timer_fn.cold.6+0x62/0x10a [10284.147042] ? __hrtimer_run_queues+0x1c2/0x4d0 [10284.149293] ? watchdog+0x40/0x40 [10284.151347] ? hrtimer_interrupt+0x128/0x2c0 [10284.154949] ? smp_apic_timer_interrupt+0xa6/0x270 [10284.157409] ? apic_timer_interrupt+0xf/0x20 [10284.159709] [10284.160708] ? native_safe_halt+0x2e/0x50 [10284.162843] kvm_wait+0xcd/0xe0 [10284.164195] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10284.165996] do_raw_spin_lock+0x177/0x190 [10284.167817] _raw_spin_lock+0x21/0x30 [10284.169416] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10284.171947] ? do_raw_spin_unlock+0x75/0x190 [10284.173897] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10284.176187] ? do_wait_intr+0xf0/0xf0 [10284.177281] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10284.179371] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10284.181709] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10284.183969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.185592] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10284.187709] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10284.189906] mdc_intent_lock+0x353/0x7e0 [mdc] [10284.191625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.194043] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10284.196015] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10284.197887] lmv_intent_open+0x338/0x1150 [lmv] [10284.199686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.201949] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.204466] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10284.206735] ll_intent_lock+0xc5/0xaa0 [lustre] [10284.209231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.212200] ll_lookup_it+0xe0e/0x2270 [lustre] [10284.214694] ? do_raw_spin_unlock+0x75/0x190 [10284.216645] ? _raw_spin_unlock+0x12/0x30 [10284.218840] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10284.221253] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10284.223839] ll_atomic_open+0x36a/0xd70 [lustre] [10284.226345] lookup_open+0x356/0xb40 [10284.228296] do_last+0x313/0xfc0 [10284.230271] ? nd_jump_root+0xe5/0x160 [10284.232125] ? path_init+0x437/0x520 [10284.233738] path_openat+0xf7/0x500 [10284.235413] ? do_raw_spin_unlock+0x75/0x190 [10284.237288] do_filp_open+0x99/0x140 [10284.239339] ? getname_flags+0x6e/0x330 [10284.241061] ? __check_object_size+0xff/0x256 [10284.243173] ? do_raw_spin_unlock+0x75/0x190 [10284.245054] ? _raw_spin_unlock+0x12/0x30 [10284.246740] do_sys_openat2+0x2b4/0x410 [10284.249195] do_sys_open+0x73/0xa0 [10284.250603] __x64_sys_openat+0x24/0x30 [10284.252645] do_syscall_64+0xc1/0x440 [10284.254422] entry_SYSCALL_64_after_hwframe+0x49/0xae [10284.259319] RIP: 0033:0x7fa749552332 [10284.261480] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10284.281919] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10284.289827] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10284.297435] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10284.306110] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10284.313859] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10284.322276] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10288.609089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8371s! [10288.613383] Showing busy workqueues and worker pools: [10288.616550] workqueue events: flags=0x0 [10288.618241] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10288.622751] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10288.630799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10288.636268] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10288.641071] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.647287] pending: kfree_rcu_monitor [10288.651100] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10288.655112] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10288.660193] workqueue events_long: flags=0x0 [10288.662302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.665762] pending: lu_objects_flush [obdclass] [10288.668404] workqueue events_unbound: flags=0x2 [10288.670820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10288.674219] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10288.679012] workqueue events_power_efficient: flags=0x80 [10288.682399] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.686938] pending: check_lifetime [10288.688779] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.692148] pending: neigh_periodic_work [10288.694856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.699273] pending: neigh_periodic_work [10288.701485] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.704802] pending: do_cache_clean [10288.707106] workqueue mm_percpu_wq: flags=0x8 [10288.709754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.713343] pending: vmstat_update [10288.715205] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.718601] pending: vmstat_update [10288.720601] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.723996] pending: vmstat_update [10288.726011] workqueue writeback: flags=0x4a [10288.728862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10288.731826] pending: wb_workfn, wb_workfn [10288.733989] workqueue xprtiod: flags=0xa [10288.736218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10288.739159] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10288.742642] workqueue ipv6_addrconf: flags=0x40008 [10288.744752] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10288.747639] pending: addrconf_verify_work [10288.749383] workqueue knbd0-recv: flags=0x1a [10288.751057] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10288.753784] in-flight: 651:recv_work [10288.755398] workqueue ptlrpc_pinger: flags=0x2 [10288.756990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10288.758773] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10288.761371] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8371s workers=4 idle: 37 31846 28897 [10288.764281] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10278s workers=2 idle: 653 [10292.057054] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10292.061533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10292.086061] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10292.091667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10292.096188] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10292.100079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10292.103418] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10292.115441] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10292.121665] RAX: 0000000000003d14 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10292.125199] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10292.129946] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10292.134228] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10292.137693] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10292.141183] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10292.145238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10292.147477] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10292.150732] Call Trace: [10292.151923] [10292.152821] ? show_regs.cold.9+0x22/0x2f [10292.154694] ? watchdog_timer_fn.cold.6+0x62/0x10a [10292.156920] ? __hrtimer_run_queues+0x1c2/0x4d0 [10292.158917] ? watchdog+0x40/0x40 [10292.160312] ? hrtimer_interrupt+0x128/0x2c0 [10292.162514] ? smp_apic_timer_interrupt+0xa6/0x270 [10292.164900] ? apic_timer_interrupt+0xf/0x20 [10292.166868] [10292.168142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10292.170596] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10292.172611] do_raw_spin_lock+0x177/0x190 [10292.174152] _raw_spin_lock+0x21/0x30 [10292.175984] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10292.178847] process_one_work+0x2c8/0x700 [10292.180963] worker_thread+0x296/0x6e0 [10292.182898] ? rescuer_thread+0x570/0x570 [10292.185033] kthread+0x1d1/0x200 [10292.186573] ? set_kthread_struct+0x70/0x70 [10292.189060] ret_from_fork+0x1f/0x30 [10296.064086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10296.069826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10296.119313] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10296.146599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10296.157159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.159192] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10296.166943] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10296.169822] RAX: 0000000000007201 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10296.172344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10296.176909] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10296.183874] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10296.188229] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10296.194426] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10296.201782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10296.209385] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10296.223562] Call Trace: [10296.224500] [10296.225228] ? show_regs.cold.9+0x22/0x2f [10296.226477] ? watchdog_timer_fn.cold.6+0x62/0x10a [10296.238219] ? __hrtimer_run_queues+0x1c2/0x4d0 [10296.243748] ? watchdog+0x40/0x40 [10296.248482] ? hrtimer_interrupt+0x128/0x2c0 [10296.250649] ? smp_apic_timer_interrupt+0xa6/0x270 [10296.255604] ? apic_timer_interrupt+0xf/0x20 [10296.260749] [10296.261814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.269003] do_raw_spin_lock+0x177/0x190 [10296.270385] _raw_spin_lock+0x21/0x30 [10296.278182] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10296.282033] ? do_raw_spin_unlock+0x75/0x190 [10296.283532] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10296.291035] ? do_wait_intr+0xf0/0xf0 [10296.292686] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10296.297183] mdc_close+0x53b/0xd70 [mdc] [10296.298964] lmv_close+0x18d/0x490 [lmv] [10296.301251] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10296.304652] ll_md_real_close+0xa5/0x2a0 [lustre] [10296.309169] ll_md_close+0x274/0x800 [lustre] [10296.313378] ll_file_release+0x128/0xc50 [lustre] [10296.317749] __fput+0x10d/0x440 [10296.322165] ____fput+0x12/0x20 [10296.324234] task_work_run+0xc6/0x110 [10296.332430] exit_to_usermode_loop+0x1dd/0x1f0 [10296.335004] do_syscall_64+0x3d6/0x440 [10296.336302] entry_SYSCALL_64_after_hwframe+0x49/0xae [10296.351693] RIP: 0033:0x7fa749552cd8 [10296.361088] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10296.373409] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10296.381056] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10296.390582] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10296.395338] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10296.399058] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10296.402452] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10308.058227] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10308.065402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10308.085618] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10308.088952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10308.091410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10308.093356] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10308.099235] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10308.101784] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10308.103937] RDX: 0000000000005237 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10308.106685] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10308.108827] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10308.111233] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10308.113650] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10308.116167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10308.118055] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10308.120267] Call Trace: [10308.121050] [10308.122137] ? show_regs.cold.9+0x22/0x2f [10308.124113] ? watchdog_timer_fn.cold.6+0x62/0x10a [10308.126503] ? __hrtimer_run_queues+0x1c2/0x4d0 [10308.128445] ? watchdog+0x40/0x40 [10308.129826] ? hrtimer_interrupt+0x128/0x2c0 [10308.131774] ? smp_apic_timer_interrupt+0xa6/0x270 [10308.133488] ? apic_timer_interrupt+0xf/0x20 [10308.135237] [10308.135915] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10308.137775] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10308.139754] do_raw_spin_lock+0x177/0x190 [10308.140825] _raw_spin_lock+0x21/0x30 [10308.141857] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10308.143773] obd_mod_free+0x46/0x140 [ptlrpc] [10308.145423] mdc_commit_open+0x91/0xb0 [mdc] [10308.146985] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10308.149029] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10308.151101] after_reply+0xe2c/0x1780 [ptlrpc] [10308.152900] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10308.154775] ? __next_timer_interrupt+0x160/0x160 [10308.156264] ptlrpcd+0xa82/0xe40 [ptlrpc] [10308.158776] ? do_wait_intr+0xf0/0xf0 [10308.160685] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10308.163135] kthread+0x1d1/0x200 [10308.164679] ? set_kthread_struct+0x70/0x70 [10308.167300] ret_from_fork+0x1f/0x30 [10316.054177] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10316.060339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10316.102736] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10316.109277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10316.114385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10316.116650] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10316.132675] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10316.143468] RAX: 0000000000007d12 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10316.149664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10316.156034] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10316.158198] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10316.172595] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10316.179667] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10316.188742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10316.191027] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10316.197910] Call Trace: [10316.201264] [10316.202243] ? show_regs.cold.9+0x22/0x2f [10316.205339] ? watchdog_timer_fn.cold.6+0x62/0x10a [10316.208150] ? __hrtimer_run_queues+0x1c2/0x4d0 [10316.212904] ? watchdog+0x40/0x40 [10316.214249] ? hrtimer_interrupt+0x128/0x2c0 [10316.223086] ? smp_apic_timer_interrupt+0xa6/0x270 [10316.227768] ? apic_timer_interrupt+0xf/0x20 [10316.231168] [10316.233016] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10316.235448] do_raw_spin_lock+0x177/0x190 [10316.237841] _raw_spin_lock+0x21/0x30 [10316.240584] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10316.244766] ? do_raw_spin_unlock+0x75/0x190 [10316.246397] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10316.251845] ? do_wait_intr+0xf0/0xf0 [10316.253232] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10316.260622] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10316.265307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10316.276688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10316.283435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10316.288005] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10316.291451] mdc_intent_lock+0x353/0x7e0 [mdc] [10316.293270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10316.295777] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10316.298890] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10316.303431] lmv_intent_open+0x338/0x1150 [lmv] [10316.306260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10316.311431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10316.316936] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10316.322727] ll_intent_lock+0xc5/0xaa0 [lustre] [10316.332523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10316.339166] ll_lookup_it+0xe0e/0x2270 [lustre] [10316.345482] ? do_raw_spin_unlock+0x75/0x190 [10316.351584] ? _raw_spin_unlock+0x12/0x30 [10316.354615] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10316.359651] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10316.363618] ll_atomic_open+0x36a/0xd70 [lustre] [10316.365991] lookup_open+0x356/0xb40 [10316.367690] do_last+0x313/0xfc0 [10316.368985] ? nd_jump_root+0xe5/0x160 [10316.375501] ? path_init+0x437/0x520 [10316.377773] path_openat+0xf7/0x500 [10316.380378] ? do_raw_spin_unlock+0x75/0x190 [10316.384508] do_filp_open+0x99/0x140 [10316.388302] ? getname_flags+0x6e/0x330 [10316.389825] ? __check_object_size+0xff/0x256 [10316.393484] ? do_raw_spin_unlock+0x75/0x190 [10316.395241] ? _raw_spin_unlock+0x12/0x30 [10316.398503] do_sys_openat2+0x2b4/0x410 [10316.399835] do_sys_open+0x73/0xa0 [10316.400907] __x64_sys_openat+0x24/0x30 [10316.403792] do_syscall_64+0xc1/0x440 [10316.406829] entry_SYSCALL_64_after_hwframe+0x49/0xae [10316.411106] RIP: 0033:0x7fa749552332 [10316.414109] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10316.424028] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10316.426796] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10316.430369] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10316.433834] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10316.441415] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10316.447154] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10318.823869] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8401s! [10318.834922] Showing busy workqueues and worker pools: [10318.839824] workqueue events: flags=0x0 [10318.844123] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10318.851341] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10318.857389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10318.864339] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10318.872597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.878640] pending: kfree_rcu_monitor [10318.881479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10318.887539] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10318.891850] workqueue events_long: flags=0x0 [10318.896510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.907045] pending: lu_objects_flush [obdclass] [10318.908727] workqueue events_unbound: flags=0x2 [10318.910844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10318.913959] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10318.922362] workqueue events_power_efficient: flags=0x80 [10318.930426] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.934489] pending: check_lifetime [10318.937224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.941342] pending: neigh_periodic_work [10318.943495] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.946282] pending: neigh_periodic_work [10318.952305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.955663] pending: do_cache_clean [10318.960017] workqueue mm_percpu_wq: flags=0x8 [10318.962328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.965534] pending: vmstat_update [10318.968800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.974345] pending: vmstat_update [10318.977865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.982214] pending: vmstat_update [10318.983456] workqueue writeback: flags=0x4a [10318.993087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10318.995008] pending: wb_workfn, wb_workfn [10318.999970] workqueue xprtiod: flags=0xa [10319.001943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10319.005582] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10319.014755] workqueue ipv6_addrconf: flags=0x40008 [10319.020147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10319.025944] pending: addrconf_verify_work [10319.030152] workqueue knbd0-recv: flags=0x1a [10319.032671] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10319.039011] in-flight: 651:recv_work [10319.041136] workqueue ptlrpc_pinger: flags=0x2 [10319.044492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10319.047607] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10319.052479] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8401s workers=4 idle: 37 31846 28897 [10319.058157] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10308s workers=2 idle: 653 [10320.058706] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10320.075545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10320.148715] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10320.155316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10320.159986] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10320.163852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10320.167740] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10320.191678] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10320.196051] RAX: 0000000000006ee4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10320.201284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10320.207804] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10320.213441] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10320.216981] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10320.221695] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10320.226250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10320.230666] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10320.235133] Call Trace: [10320.235979] [10320.241766] ? show_regs.cold.9+0x22/0x2f [10320.249462] ? watchdog_timer_fn.cold.6+0x62/0x10a [10320.254584] ? __hrtimer_run_queues+0x1c2/0x4d0 [10320.258442] ? watchdog+0x40/0x40 [10320.259740] ? hrtimer_interrupt+0x128/0x2c0 [10320.266389] ? smp_apic_timer_interrupt+0xa6/0x270 [10320.271733] ? apic_timer_interrupt+0xf/0x20 [10320.275478] [10320.276364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10320.278069] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10320.287867] do_raw_spin_lock+0x177/0x190 [10320.291652] _raw_spin_lock+0x21/0x30 [10320.294071] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10320.298456] process_one_work+0x2c8/0x700 [10320.303209] worker_thread+0x296/0x6e0 [10320.305973] ? rescuer_thread+0x570/0x570 [10320.307485] kthread+0x1d1/0x200 [10320.310720] ? set_kthread_struct+0x70/0x70 [10320.313802] ret_from_fork+0x1f/0x30 [10324.059018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10324.063049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10324.083878] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10324.089035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10324.093009] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.095918] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10324.104497] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10324.109338] RAX: 0000000000005a8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10324.112486] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10324.115849] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10324.119310] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10324.123072] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10324.127064] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10324.131024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10324.134603] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10324.138487] Call Trace: [10324.139604] [10324.140632] ? show_regs.cold.9+0x22/0x2f [10324.142686] ? watchdog_timer_fn.cold.6+0x62/0x10a [10324.145348] ? __hrtimer_run_queues+0x1c2/0x4d0 [10324.147354] ? watchdog+0x40/0x40 [10324.148868] ? hrtimer_interrupt+0x128/0x2c0 [10324.150894] ? smp_apic_timer_interrupt+0xa6/0x270 [10324.153147] ? apic_timer_interrupt+0xf/0x20 [10324.155294] [10324.156401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.159108] do_raw_spin_lock+0x177/0x190 [10324.160702] _raw_spin_lock+0x21/0x30 [10324.162735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10324.165551] ? do_raw_spin_unlock+0x75/0x190 [10324.167716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10324.171399] ? do_wait_intr+0xf0/0xf0 [10324.173017] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10324.176651] mdc_close+0x53b/0xd70 [mdc] [10324.178650] lmv_close+0x18d/0x490 [lmv] [10324.180545] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10324.183123] ll_md_real_close+0xa5/0x2a0 [lustre] [10324.185787] ll_md_close+0x274/0x800 [lustre] [10324.188051] ll_file_release+0x128/0xc50 [lustre] [10324.190329] __fput+0x10d/0x440 [10324.191721] ____fput+0x12/0x20 [10324.193102] task_work_run+0xc6/0x110 [10324.195056] exit_to_usermode_loop+0x1dd/0x1f0 [10324.197156] do_syscall_64+0x3d6/0x440 [10324.198998] entry_SYSCALL_64_after_hwframe+0x49/0xae [10324.200627] RIP: 0033:0x7fa749552cd8 [10324.202461] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10324.210553] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10324.214655] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10324.219349] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10324.223067] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10324.226685] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10324.230254] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10336.058015] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10336.061372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10336.080306] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10336.087701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10336.092174] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10336.095294] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10336.103836] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10336.106954] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10336.110004] RDX: 00000000000078d6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10336.113082] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10336.116326] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10336.118864] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10336.121610] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10336.124724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10336.127087] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10336.130197] Call Trace: [10336.131060] [10336.131880] ? show_regs.cold.9+0x22/0x2f [10336.133388] ? watchdog_timer_fn.cold.6+0x62/0x10a [10336.135999] ? __hrtimer_run_queues+0x1c2/0x4d0 [10336.138375] ? watchdog+0x40/0x40 [10336.139949] ? hrtimer_interrupt+0x128/0x2c0 [10336.141730] ? smp_apic_timer_interrupt+0xa6/0x270 [10336.143666] ? apic_timer_interrupt+0xf/0x20 [10336.145183] [10336.146140] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10336.148335] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10336.150373] do_raw_spin_lock+0x177/0x190 [10336.152037] _raw_spin_lock+0x21/0x30 [10336.153329] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10336.155607] obd_mod_free+0x46/0x140 [ptlrpc] [10336.157681] mdc_commit_open+0x91/0xb0 [mdc] [10336.159396] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10336.161849] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10336.164855] after_reply+0xe2c/0x1780 [ptlrpc] [10336.169259] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10336.175299] ? __next_timer_interrupt+0x160/0x160 [10336.177754] ptlrpcd+0xa82/0xe40 [ptlrpc] [10336.180075] ? do_wait_intr+0xf0/0xf0 [10336.181841] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10336.185009] kthread+0x1d1/0x200 [10336.186093] ? set_kthread_struct+0x70/0x70 [10336.188334] ret_from_fork+0x1f/0x30 [10344.054088] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10344.057745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10344.077709] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10344.083642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10344.086522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10344.089084] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10344.096853] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10344.099593] RAX: 0000000000003863 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10344.102543] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10344.105544] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10344.108394] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10344.112127] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10344.116672] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10344.120772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10344.124453] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10344.128084] Call Trace: [10344.129593] [10344.131401] ? show_regs.cold.9+0x22/0x2f [10344.136324] ? watchdog_timer_fn.cold.6+0x62/0x10a [10344.141577] ? __hrtimer_run_queues+0x1c2/0x4d0 [10344.144613] ? watchdog+0x40/0x40 [10344.147397] ? hrtimer_interrupt+0x128/0x2c0 [10344.148941] ? smp_apic_timer_interrupt+0xa6/0x270 [10344.150840] ? apic_timer_interrupt+0xf/0x20 [10344.152918] [10344.153874] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10344.156507] do_raw_spin_lock+0x177/0x190 [10344.158330] _raw_spin_lock+0x21/0x30 [10344.160517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10344.163314] ? do_raw_spin_unlock+0x75/0x190 [10344.165695] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10344.168163] ? do_wait_intr+0xf0/0xf0 [10344.169997] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10344.180287] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10344.186347] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10344.196182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10344.201924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10344.209324] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10344.217080] mdc_intent_lock+0x353/0x7e0 [mdc] [10344.219860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10344.227398] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10344.235347] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10344.242041] lmv_intent_open+0x338/0x1150 [lmv] [10344.245395] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10344.252845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10344.256205] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10344.258505] ll_intent_lock+0xc5/0xaa0 [lustre] [10344.262494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10344.266333] ll_lookup_it+0xe0e/0x2270 [lustre] [10344.268884] ? do_raw_spin_unlock+0x75/0x190 [10344.270510] ? _raw_spin_unlock+0x12/0x30 [10344.273964] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10344.275786] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10344.281456] ll_atomic_open+0x36a/0xd70 [lustre] [10344.287719] lookup_open+0x356/0xb40 [10344.289880] do_last+0x313/0xfc0 [10344.293722] ? nd_jump_root+0xe5/0x160 [10344.295515] ? path_init+0x437/0x520 [10344.297175] path_openat+0xf7/0x500 [10344.298398] ? do_raw_spin_unlock+0x75/0x190 [10344.300394] do_filp_open+0x99/0x140 [10344.302177] ? getname_flags+0x6e/0x330 [10344.303757] ? __check_object_size+0xff/0x256 [10344.306732] ? do_raw_spin_unlock+0x75/0x190 [10344.310317] ? _raw_spin_unlock+0x12/0x30 [10344.314354] do_sys_openat2+0x2b4/0x410 [10344.318392] do_sys_open+0x73/0xa0 [10344.320965] __x64_sys_openat+0x24/0x30 [10344.323965] do_syscall_64+0xc1/0x440 [10344.325748] entry_SYSCALL_64_after_hwframe+0x49/0xae [10344.328313] RIP: 0033:0x7fa749552332 [10344.332141] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10344.346187] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10344.353888] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10344.358809] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10344.361685] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10344.368916] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10344.376153] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10348.060374] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10348.066216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10348.085217] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10348.088784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10348.092060] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10348.094257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10348.096361] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10348.103897] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10348.106238] RAX: 0000000000003485 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10348.108258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10348.118808] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10348.122001] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10348.124207] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10348.130983] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10348.133963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10348.138034] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10348.141358] Call Trace: [10348.142933] [10348.143991] ? show_regs.cold.9+0x22/0x2f [10348.146916] ? watchdog_timer_fn.cold.6+0x62/0x10a [10348.151143] ? __hrtimer_run_queues+0x1c2/0x4d0 [10348.153335] ? watchdog+0x40/0x40 [10348.155171] ? hrtimer_interrupt+0x128/0x2c0 [10348.157822] ? smp_apic_timer_interrupt+0xa6/0x270 [10348.160148] ? apic_timer_interrupt+0xf/0x20 [10348.162803] [10348.163743] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10348.166654] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10348.172756] do_raw_spin_lock+0x177/0x190 [10348.174752] _raw_spin_lock+0x21/0x30 [10348.176184] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10348.180885] process_one_work+0x2c8/0x700 [10348.183201] worker_thread+0x296/0x6e0 [10348.184960] ? rescuer_thread+0x570/0x570 [10348.188317] kthread+0x1d1/0x200 [10348.190124] ? set_kthread_struct+0x70/0x70 [10348.194876] ret_from_fork+0x1f/0x30 [10349.547835] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8432s! [10349.556894] Showing busy workqueues and worker pools: [10349.568068] workqueue events: flags=0x0 [10349.572743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10349.582963] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10349.598934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10349.603216] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10349.613179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.616940] pending: kfree_rcu_monitor [10349.620001] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10349.624670] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10349.639772] workqueue events_long: flags=0x0 [10349.644120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.651322] pending: lu_objects_flush [obdclass] [10349.655258] workqueue events_unbound: flags=0x2 [10349.658303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10349.667036] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10349.686285] workqueue events_power_efficient: flags=0x80 [10349.694271] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.704748] pending: check_lifetime [10349.707304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.712526] pending: neigh_periodic_work [10349.717136] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.731878] pending: neigh_periodic_work [10349.734186] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.743110] pending: do_cache_clean [10349.746543] workqueue mm_percpu_wq: flags=0x8 [10349.752817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.760100] pending: vmstat_update [10349.761229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.773072] pending: vmstat_update [10349.774388] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.784904] pending: vmstat_update [10349.792283] workqueue writeback: flags=0x4a [10349.799296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10349.805859] pending: wb_workfn, wb_workfn [10349.810260] workqueue xprtiod: flags=0xa [10349.820518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10349.829375] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10349.838283] workqueue ipv6_addrconf: flags=0x40008 [10349.849434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10349.866900] pending: addrconf_verify_work [10349.868738] workqueue knbd0-recv: flags=0x1a [10349.873258] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10349.890188] in-flight: 651:recv_work [10349.893988] workqueue ptlrpc_pinger: flags=0x2 [10349.902114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10349.914886] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10349.921126] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8432s workers=4 idle: 37 31846 28897 [10349.929028] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10339s workers=2 idle: 653 [10352.059720] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10352.062881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10352.107628] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10352.114634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10352.120482] RIP: 0010:native_safe_halt+0x2e/0x50 [10352.122199] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10352.136042] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10352.141462] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10352.143324] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10352.145532] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10352.147264] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10352.159082] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10352.163903] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10352.170692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10352.172395] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10352.176748] Call Trace: [10352.177726] [10352.181015] ? show_regs.cold.9+0x22/0x2f [10352.184670] ? watchdog_timer_fn.cold.6+0x62/0x10a [10352.188170] ? __hrtimer_run_queues+0x1c2/0x4d0 [10352.192739] ? watchdog+0x40/0x40 [10352.193578] ? hrtimer_interrupt+0x128/0x2c0 [10352.202101] ? smp_apic_timer_interrupt+0xa6/0x270 [10352.204492] ? apic_timer_interrupt+0xf/0x20 [10352.207731] [10352.209013] ? native_safe_halt+0x2e/0x50 [10352.210545] kvm_wait+0xcd/0xe0 [10352.212450] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10352.215681] do_raw_spin_lock+0x177/0x190 [10352.216888] _raw_spin_lock+0x21/0x30 [10352.218694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10352.224008] ? do_raw_spin_unlock+0x75/0x190 [10352.226460] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10352.227967] ? do_wait_intr+0xf0/0xf0 [10352.230810] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10352.235030] mdc_close+0x53b/0xd70 [mdc] [10352.236952] lmv_close+0x18d/0x490 [lmv] [10352.238813] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10352.241610] ll_md_real_close+0xa5/0x2a0 [lustre] [10352.243350] ll_md_close+0x274/0x800 [lustre] [10352.248330] ll_file_release+0x128/0xc50 [lustre] [10352.250498] __fput+0x10d/0x440 [10352.252070] ____fput+0x12/0x20 [10352.253112] task_work_run+0xc6/0x110 [10352.254821] exit_to_usermode_loop+0x1dd/0x1f0 [10352.259840] do_syscall_64+0x3d6/0x440 [10352.264041] entry_SYSCALL_64_after_hwframe+0x49/0xae [10352.268470] RIP: 0033:0x7fa749552cd8 [10352.269839] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10352.282494] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10352.298706] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10352.305919] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10352.309927] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10352.313791] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10352.320147] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10364.058060] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10364.061988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10364.094200] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10364.100048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10364.104133] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10364.107050] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10364.117069] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10364.122092] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10364.126437] RDX: 0000000000003f04 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10364.130262] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10364.134173] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10364.138692] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10364.143673] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10364.149396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10364.153353] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10364.159216] Call Trace: [10364.161702] [10364.166450] ? show_regs.cold.9+0x22/0x2f [10364.172513] ? watchdog_timer_fn.cold.6+0x62/0x10a [10364.178419] ? __hrtimer_run_queues+0x1c2/0x4d0 [10364.181480] ? watchdog+0x40/0x40 [10364.184714] ? hrtimer_interrupt+0x128/0x2c0 [10364.187756] ? smp_apic_timer_interrupt+0xa6/0x270 [10364.192049] ? apic_timer_interrupt+0xf/0x20 [10364.194501] [10364.197311] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10364.202346] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10364.206028] do_raw_spin_lock+0x177/0x190 [10364.210667] _raw_spin_lock+0x21/0x30 [10364.213624] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10364.220373] obd_mod_free+0x46/0x140 [ptlrpc] [10364.223861] mdc_commit_open+0x91/0xb0 [mdc] [10364.226947] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10364.230215] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10364.233677] after_reply+0xe2c/0x1780 [ptlrpc] [10364.236233] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10364.239747] ? __next_timer_interrupt+0x160/0x160 [10364.242647] ptlrpcd+0xa82/0xe40 [ptlrpc] [10364.245363] ? do_wait_intr+0xf0/0xf0 [10364.247359] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10364.249743] kthread+0x1d1/0x200 [10364.251676] ? set_kthread_struct+0x70/0x70 [10364.254300] ret_from_fork+0x1f/0x30 [10376.055656] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10376.057572] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10376.068536] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10376.069087] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10376.069096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10376.069100] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10376.069117] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10376.069122] RSP: 0018:ffffb893c1493500 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [10376.069127] RAX: 0000000000006a00 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10376.069130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10376.069132] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10376.069134] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10376.069137] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10376.069140] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10376.069143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10376.069145] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10376.069153] Call Trace: [10376.069160] [10376.069175] ? show_regs.cold.9+0x22/0x2f [10376.069188] ? watchdog_timer_fn.cold.6+0x62/0x10a [10376.069195] ? __hrtimer_run_queues+0x1c2/0x4d0 [10376.069200] ? watchdog+0x40/0x40 [10376.069204] ? hrtimer_interrupt+0x128/0x2c0 [10376.069208] ? smp_apic_timer_interrupt+0xa6/0x270 [10376.069216] ? apic_timer_interrupt+0xf/0x20 [10376.069221] [10376.069224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10376.069232] do_raw_spin_lock+0x177/0x190 [10376.069237] _raw_spin_lock+0x21/0x30 [10376.069242] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10376.069543] ? do_raw_spin_unlock+0x75/0x190 [10376.069549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10376.070527] ? do_wait_intr+0xf0/0xf0 [10376.070538] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10376.079823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10376.080779] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10376.080786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10376.080793] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10376.130662] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10376.146597] [10376.146609] RIP: 0010:native_safe_halt+0x2e/0x50 [10376.153595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10376.154120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10376.154250] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10376.154291] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10376.154321] mdc_intent_lock+0x353/0x7e0 [mdc] [10376.154349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10376.154431] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10376.154654] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10376.154682] lmv_intent_open+0x338/0x1150 [lmv] [10376.154715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10376.157964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10376.158900] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10376.158937] ll_intent_lock+0xc5/0xaa0 [lustre] [10376.160044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10376.160828] ll_lookup_it+0xe0e/0x2270 [lustre] [10376.160918] ? do_raw_spin_unlock+0x75/0x190 [10376.160926] ? _raw_spin_unlock+0x12/0x30 [10376.160934] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10376.160961] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10376.160984] ll_atomic_open+0x36a/0xd70 [lustre] [10376.162669] lookup_open+0x356/0xb40 [10376.163042] do_last+0x313/0xfc0 [10376.163174] ? nd_jump_root+0xe5/0x160 [10376.163391] ? path_init+0x437/0x520 [10376.163397] path_openat+0xf7/0x500 [10376.163404] ? do_raw_spin_unlock+0x75/0x190 [10376.168417] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10376.168436] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10376.169222] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10376.169225] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10376.169228] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10376.169230] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10376.169232] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10376.169235] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10376.169243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10376.169245] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10376.169253] Call Trace: [10376.169259] [10376.169265] ? show_regs.cold.9+0x22/0x2f [10376.169284] ? watchdog_timer_fn.cold.6+0x62/0x10a [10376.169291] ? __hrtimer_run_queues+0x1c2/0x4d0 [10376.169296] ? watchdog+0x40/0x40 [10376.169301] ? hrtimer_interrupt+0x128/0x2c0 [10376.169305] ? smp_apic_timer_interrupt+0xa6/0x270 [10376.169314] ? apic_timer_interrupt+0xf/0x20 [10376.169319] [10376.169322] ? native_safe_halt+0x2e/0x50 [10376.169326] kvm_wait+0xcd/0xe0 [10376.169336] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10376.169346] do_raw_spin_lock+0x177/0x190 [10376.169350] _raw_spin_lock+0x21/0x30 [10376.169355] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10376.170421] process_one_work+0x2c8/0x700 [10376.170430] worker_thread+0x296/0x6e0 [10376.170434] ? rescuer_thread+0x570/0x570 [10376.170438] kthread+0x1d1/0x200 [10376.170442] ? set_kthread_struct+0x70/0x70 [10376.170445] ret_from_fork+0x1f/0x30 [10376.748491] do_filp_open+0x99/0x140 [10376.752019] ? getname_flags+0x6e/0x330 [10376.754132] ? __check_object_size+0xff/0x256 [10376.758125] ? do_raw_spin_unlock+0x75/0x190 [10376.760004] ? _raw_spin_unlock+0x12/0x30 [10376.761448] do_sys_openat2+0x2b4/0x410 [10376.763360] do_sys_open+0x73/0xa0 [10376.765837] __x64_sys_openat+0x24/0x30 [10376.769107] do_syscall_64+0xc1/0x440 [10376.772434] entry_SYSCALL_64_after_hwframe+0x49/0xae [10376.778528] RIP: 0033:0x7fa749552332 [10376.782545] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10376.794819] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10376.807786] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10376.814723] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10376.820432] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10376.826071] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10376.833367] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10380.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10380.064571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10380.089305] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10380.101423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10380.111187] RIP: 0010:native_safe_halt+0x2e/0x50 [10380.115089] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10380.130747] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10380.137479] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10380.142373] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10380.146346] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10380.149870] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10380.153491] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10380.157146] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10380.161466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10380.164319] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10380.169630] Call Trace: [10380.171514] [10380.173498] ? show_regs.cold.9+0x22/0x2f [10380.176000] ? watchdog_timer_fn.cold.6+0x62/0x10a [10380.182431] ? __hrtimer_run_queues+0x1c2/0x4d0 [10380.185163] ? watchdog+0x40/0x40 [10380.188325] ? hrtimer_interrupt+0x128/0x2c0 [10380.190991] ? smp_apic_timer_interrupt+0xa6/0x270 [10380.193058] ? apic_timer_interrupt+0xf/0x20 [10380.195616] [10380.196322] ? native_safe_halt+0x2e/0x50 [10380.199061] kvm_wait+0xcd/0xe0 [10380.200704] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10380.204692] do_raw_spin_lock+0x177/0x190 [10380.205891] _raw_spin_lock+0x21/0x30 [10380.206815] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10380.209285] ? do_raw_spin_unlock+0x75/0x190 [10380.210535] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10380.212715] ? do_wait_intr+0xf0/0xf0 [10380.213778] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10380.215421] mdc_close+0x53b/0xd70 [mdc] [10380.216761] lmv_close+0x18d/0x490 [lmv] [10380.218274] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10380.222534] ll_md_real_close+0xa5/0x2a0 [lustre] [10380.227117] ll_md_close+0x274/0x800 [lustre] [10380.228382] ll_file_release+0x128/0xc50 [lustre] [10380.232036] __fput+0x10d/0x440 [10380.233327] ____fput+0x12/0x20 [10380.237938] task_work_run+0xc6/0x110 [10380.243635] exit_to_usermode_loop+0x1dd/0x1f0 [10380.245946] do_syscall_64+0x3d6/0x440 [10380.247466] entry_SYSCALL_64_after_hwframe+0x49/0xae [10380.248995] RIP: 0033:0x7fa749552cd8 [10380.251600] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10380.257052] BUG: workqueue lockup - pool [10380.258521] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10380.258533] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10380.258535] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10380.258538] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10380.260703] cpus=0-3 [10380.263738] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10380.263750] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10380.298673] flags=0x4 nice=0 stuck for 8463s! [10380.300437] Showing busy workqueues and worker pools: [10380.303907] workqueue events: flags=0x0 [10380.306071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10380.311707] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10380.321521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10380.326482] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10380.330716] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.338081] pending: kfree_rcu_monitor [10380.340382] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10380.344647] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10380.348905] workqueue events_long: flags=0x0 [10380.350634] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.355437] pending: lu_objects_flush [obdclass] [10380.359696] workqueue events_unbound: flags=0x2 [10380.362417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10380.367971] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10380.371735] workqueue events_power_efficient: flags=0x80 [10380.375932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.381575] pending: check_lifetime [10380.385598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.395087] pending: neigh_periodic_work [10380.397974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.402399] pending: neigh_periodic_work [10380.404464] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.412086] pending: do_cache_clean [10380.414474] workqueue mm_percpu_wq: flags=0x8 [10380.417345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.421644] pending: vmstat_update [10380.423493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.426636] pending: vmstat_update [10380.428614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.433555] pending: vmstat_update [10380.436015] workqueue writeback: flags=0x4a [10380.437578] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10380.442708] pending: wb_workfn, wb_workfn [10380.446362] workqueue xprtiod: flags=0xa [10380.452714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10380.460441] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10380.468640] workqueue ipv6_addrconf: flags=0x40008 [10380.470182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10380.482208] pending: addrconf_verify_work [10380.485990] workqueue knbd0-recv: flags=0x1a [10380.491041] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10380.497992] in-flight: 651:recv_work [10380.503381] workqueue ptlrpc_pinger: flags=0x2 [10380.509381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10380.518712] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10380.524990] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8463s workers=4 idle: 37 31846 28897 [10380.534685] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10370s workers=2 idle: 653 [10392.058059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10392.060708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10392.081948] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10392.087223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10392.091311] RIP: 0010:native_safe_halt+0x2e/0x50 [10392.093434] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10392.101745] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10392.104743] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10392.107838] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10392.111590] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10392.114738] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10392.118330] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10392.122615] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10392.126744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10392.128507] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10392.131463] Call Trace: [10392.132182] [10392.132700] ? show_regs.cold.9+0x22/0x2f [10392.134428] ? watchdog_timer_fn.cold.6+0x62/0x10a [10392.136232] ? __hrtimer_run_queues+0x1c2/0x4d0 [10392.137777] ? watchdog+0x40/0x40 [10392.139082] ? hrtimer_interrupt+0x128/0x2c0 [10392.140561] ? smp_apic_timer_interrupt+0xa6/0x270 [10392.142448] ? apic_timer_interrupt+0xf/0x20 [10392.143942] [10392.144775] ? native_safe_halt+0x2e/0x50 [10392.146499] kvm_wait+0xcd/0xe0 [10392.147789] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10392.149728] do_raw_spin_lock+0x177/0x190 [10392.151359] _raw_spin_lock+0x21/0x30 [10392.152383] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10392.154133] obd_mod_free+0x46/0x140 [ptlrpc] [10392.155744] mdc_commit_open+0x91/0xb0 [mdc] [10392.157423] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10392.159337] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10392.161925] after_reply+0xe2c/0x1780 [ptlrpc] [10392.163700] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10392.165465] ? __next_timer_interrupt+0x160/0x160 [10392.167415] ptlrpcd+0xa82/0xe40 [ptlrpc] [10392.169030] ? do_wait_intr+0xf0/0xf0 [10392.170090] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10392.171828] kthread+0x1d1/0x200 [10392.173074] ? set_kthread_struct+0x70/0x70 [10392.174441] ret_from_fork+0x1f/0x30 [10404.057320] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10404.062121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10404.085254] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10404.090190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10404.096777] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10404.101019] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10404.103835] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10404.112987] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10404.116082] RAX: 0000000000007446 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10404.120170] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10404.124728] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10404.128319] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10404.130203] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10404.133276] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10404.136959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10404.139132] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10404.143283] Call Trace: [10404.144124] [10404.145291] ? show_regs.cold.9+0x22/0x2f [10404.147285] ? watchdog_timer_fn.cold.6+0x62/0x10a [10404.149810] ? __hrtimer_run_queues+0x1c2/0x4d0 [10404.153008] ? watchdog+0x40/0x40 [10404.154503] ? hrtimer_interrupt+0x128/0x2c0 [10404.156480] ? smp_apic_timer_interrupt+0xa6/0x270 [10404.159017] ? apic_timer_interrupt+0xf/0x20 [10404.160579] [10404.161350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10404.164979] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10404.166593] do_raw_spin_lock+0x177/0x190 [10404.171044] _raw_spin_lock+0x21/0x30 [10404.172375] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10404.175193] process_one_work+0x2c8/0x700 [10404.180990] worker_thread+0x296/0x6e0 [10404.182368] ? rescuer_thread+0x570/0x570 [10404.185001] kthread+0x1d1/0x200 [10404.186261] ? set_kthread_struct+0x70/0x70 [10404.191879] ret_from_fork+0x1f/0x30 [10408.054026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10408.056761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10408.059787] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10408.078724] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10408.078740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10408.078743] RIP: 0010:native_safe_halt+0x2e/0x50 [10408.078757] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10408.078761] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10408.078767] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10408.078769] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [10408.078771] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10408.078773] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10408.078775] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10408.078778] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10408.078793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10408.078796] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10408.078804] Call Trace: [10408.078809] [10408.078815] ? show_regs.cold.9+0x22/0x2f [10408.078825] ? watchdog_timer_fn.cold.6+0x62/0x10a [10408.078832] ? __hrtimer_run_queues+0x1c2/0x4d0 [10408.078837] ? watchdog+0x40/0x40 [10408.078842] ? hrtimer_interrupt+0x128/0x2c0 [10408.078846] ? smp_apic_timer_interrupt+0xa6/0x270 [10408.078852] ? apic_timer_interrupt+0xf/0x20 [10408.078856] [10408.078859] ? native_safe_halt+0x2e/0x50 [10408.078864] kvm_wait+0xcd/0xe0 [10408.078871] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10408.078882] do_raw_spin_lock+0x177/0x190 [10408.078887] _raw_spin_lock+0x21/0x30 [10408.078891] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10408.079378] ? do_raw_spin_unlock+0x75/0x190 [10408.079386] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10408.079683] ? do_wait_intr+0xf0/0xf0 [10408.079692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10408.080006] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10408.080226] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10408.080519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10408.086953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10408.087033] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10408.087041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10408.087044] RIP: 0010:native_safe_halt+0x2e/0x50 [10408.087057] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10408.087061] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10408.087066] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10408.087068] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10408.087070] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10408.087072] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10408.087075] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10408.087078] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10408.087081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10408.087084] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10408.087094] Call Trace: [10408.087100] [10408.087107] ? show_regs.cold.9+0x22/0x2f [10408.093400] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10408.093459] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10408.093489] mdc_intent_lock+0x353/0x7e0 [mdc] [10408.093521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10408.093767] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10408.094162] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10408.094195] lmv_intent_open+0x338/0x1150 [lmv] [10408.094228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10408.094332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10408.094416] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10408.094438] ll_intent_lock+0xc5/0xaa0 [lustre] [10408.094521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10408.094734] ll_lookup_it+0xe0e/0x2270 [lustre] [10408.098309] ? watchdog_timer_fn.cold.6+0x62/0x10a [10408.098336] ? __hrtimer_run_queues+0x1c2/0x4d0 [10408.098341] ? watchdog+0x40/0x40 [10408.098345] ? hrtimer_interrupt+0x128/0x2c0 [10408.098349] ? smp_apic_timer_interrupt+0xa6/0x270 [10408.098358] ? apic_timer_interrupt+0xf/0x20 [10408.098362] [10408.098368] ? native_safe_halt+0x2e/0x50 [10408.098373] kvm_wait+0xcd/0xe0 [10408.098384] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10408.098393] do_raw_spin_lock+0x177/0x190 [10408.098397] _raw_spin_lock+0x21/0x30 [10408.098402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10408.098998] ? do_raw_spin_unlock+0x75/0x190 [10408.099005] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10408.099210] ? do_wait_intr+0xf0/0xf0 [10408.099218] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10408.099428] mdc_close+0x53b/0xd70 [mdc] [10408.099470] lmv_close+0x18d/0x490 [lmv] [10408.099503] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10408.099753] ll_md_real_close+0xa5/0x2a0 [lustre] [10408.099848] ll_md_close+0x274/0x800 [lustre] [10408.099928] ll_file_release+0x128/0xc50 [lustre] [10408.100009] __fput+0x10d/0x440 [10408.100020] ____fput+0x12/0x20 [10408.100025] task_work_run+0xc6/0x110 [10408.100035] exit_to_usermode_loop+0x1dd/0x1f0 [10408.100045] do_syscall_64+0x3d6/0x440 [10408.100050] entry_SYSCALL_64_after_hwframe+0x49/0xae [10408.100057] RIP: 0033:0x7fa749552cd8 [10408.100066] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10408.100070] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10408.100075] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10408.100078] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10408.100080] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10408.100082] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10408.100084] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10408.520910] ? do_raw_spin_unlock+0x75/0x190 [10408.526993] ? _raw_spin_unlock+0x12/0x30 [10408.531822] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10408.534794] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10408.538550] ll_atomic_open+0x36a/0xd70 [lustre] [10408.545257] lookup_open+0x356/0xb40 [10408.546884] do_last+0x313/0xfc0 [10408.549520] ? nd_jump_root+0xe5/0x160 [10408.552423] ? path_init+0x437/0x520 [10408.554619] path_openat+0xf7/0x500 [10408.559806] ? do_raw_spin_unlock+0x75/0x190 [10408.567072] do_filp_open+0x99/0x140 [10408.568773] ? getname_flags+0x6e/0x330 [10408.570681] ? __check_object_size+0xff/0x256 [10408.572617] ? do_raw_spin_unlock+0x75/0x190 [10408.581929] ? _raw_spin_unlock+0x12/0x30 [10408.585575] do_sys_openat2+0x2b4/0x410 [10408.590585] do_sys_open+0x73/0xa0 [10408.592769] __x64_sys_openat+0x24/0x30 [10408.595679] do_syscall_64+0xc1/0x440 [10408.598060] entry_SYSCALL_64_after_hwframe+0x49/0xae [10408.601207] RIP: 0033:0x7fa749552332 [10408.603759] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10408.612589] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10408.618726] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10408.623016] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10408.627095] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10408.630689] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10408.634194] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10410.979414] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8493s! [10410.990012] Showing busy workqueues and worker pools: [10410.995153] workqueue events: flags=0x0 [10410.999597] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10411.010473] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10411.018942] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10411.022222] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10411.026743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.029351] pending: kfree_rcu_monitor [10411.031325] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10411.037304] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10411.044898] workqueue events_long: flags=0x0 [10411.050715] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.053965] pending: lu_objects_flush [obdclass] [10411.056912] workqueue events_unbound: flags=0x2 [10411.064317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10411.067886] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10411.076628] workqueue events_power_efficient: flags=0x80 [10411.080051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.085303] pending: check_lifetime [10411.088981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.096046] pending: neigh_periodic_work [10411.098138] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.103871] pending: neigh_periodic_work [10411.107812] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.115978] pending: do_cache_clean [10411.117288] workqueue mm_percpu_wq: flags=0x8 [10411.119964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.126561] pending: vmstat_update [10411.130023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.137774] pending: vmstat_update [10411.142236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.149898] pending: vmstat_update [10411.153161] workqueue writeback: flags=0x4a [10411.162045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10411.166187] pending: wb_workfn, wb_workfn [10411.168515] workqueue xprtiod: flags=0xa [10411.177260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10411.180411] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10411.184048] workqueue ipv6_addrconf: flags=0x40008 [10411.186732] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10411.189705] pending: addrconf_verify_work [10411.191888] workqueue knbd0-recv: flags=0x1a [10411.193512] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10411.197202] in-flight: 651:recv_work [10411.203483] workqueue ptlrpc_pinger: flags=0x2 [10411.206186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10411.211752] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10411.215825] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8494s workers=4 idle: 37 31846 28897 [10411.226498] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10401s workers=2 idle: 653 [10420.058142] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10420.063097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10420.082556] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10420.087703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10420.092090] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10420.094894] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10420.102633] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10420.106702] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10420.110345] RDX: 0000000000001524 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10420.113677] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10420.116849] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10420.120104] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10420.123379] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10420.126983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10420.129792] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10420.132961] Call Trace: [10420.134063] [10420.134941] ? show_regs.cold.9+0x22/0x2f [10420.137310] ? watchdog_timer_fn.cold.6+0x62/0x10a [10420.139876] ? __hrtimer_run_queues+0x1c2/0x4d0 [10420.142154] ? watchdog+0x40/0x40 [10420.143679] ? hrtimer_interrupt+0x128/0x2c0 [10420.146453] ? smp_apic_timer_interrupt+0xa6/0x270 [10420.148918] ? apic_timer_interrupt+0xf/0x20 [10420.151181] [10420.151941] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10420.154849] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10420.158254] do_raw_spin_lock+0x177/0x190 [10420.161218] _raw_spin_lock+0x21/0x30 [10420.162896] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10420.165523] obd_mod_free+0x46/0x140 [ptlrpc] [10420.167558] mdc_commit_open+0x91/0xb0 [mdc] [10420.169292] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10420.173534] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10420.176708] after_reply+0xe2c/0x1780 [ptlrpc] [10420.179228] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10420.181694] ? __next_timer_interrupt+0x160/0x160 [10420.183454] ptlrpcd+0xa82/0xe40 [ptlrpc] [10420.186017] ? do_wait_intr+0xf0/0xf0 [10420.188371] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10420.190906] kthread+0x1d1/0x200 [10420.192530] ? set_kthread_struct+0x70/0x70 [10420.194870] ret_from_fork+0x1f/0x30 [10432.059163] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10432.073427] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10432.107300] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10432.116303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10432.123633] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10432.133033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10432.139698] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10432.151356] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10432.156385] RAX: 0000000000000507 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10432.159741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10432.162029] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10432.164451] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10432.176987] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10432.188820] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10432.197042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10432.205175] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10432.216723] Call Trace: [10432.220619] [10432.221826] ? show_regs.cold.9+0x22/0x2f [10432.226723] ? watchdog_timer_fn.cold.6+0x62/0x10a [10432.229914] ? __hrtimer_run_queues+0x1c2/0x4d0 [10432.233453] ? watchdog+0x40/0x40 [10432.235466] ? hrtimer_interrupt+0x128/0x2c0 [10432.237859] ? smp_apic_timer_interrupt+0xa6/0x270 [10432.244097] ? apic_timer_interrupt+0xf/0x20 [10432.254258] [10432.256425] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10432.264285] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10432.270212] do_raw_spin_lock+0x177/0x190 [10432.273239] _raw_spin_lock+0x21/0x30 [10432.275221] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10432.284774] process_one_work+0x2c8/0x700 [10432.293145] worker_thread+0x296/0x6e0 [10432.296708] ? rescuer_thread+0x570/0x570 [10432.302946] kthread+0x1d1/0x200 [10432.305392] ? set_kthread_struct+0x70/0x70 [10432.312630] ret_from_fork+0x1f/0x30 [10433.871137] rcu: INFO: rcu_sched self-detected stall on CPU [10433.877068] rcu: 2-...!: (4793258 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [10433.886889] (t=8520182 jiffies g=258701 q=3994) [10433.888408] rcu: rcu_sched kthread starved for 8516977 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10433.896313] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10433.908255] rcu: RCU grace-period kthread stack dump: [10433.919211] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10433.929798] Call Trace: [10433.932784] __schedule+0x351/0xcb0 [10433.940247] schedule+0xc0/0x180 [10433.947148] schedule_timeout+0xb4/0x190 [10433.949560] ? do_raw_spin_unlock+0x75/0x190 [10433.951400] ? __next_timer_interrupt+0x160/0x160 [10433.953277] ? prepare_to_swait_event+0xf3/0x140 [10433.954825] rcu_gp_kthread+0x8dd/0x15b0 [10433.956067] ? force_qs_rnp+0x2c0/0x2c0 [10433.962531] kthread+0x1d1/0x200 [10433.963937] ? set_kthread_struct+0x70/0x70 [10433.968526] ret_from_fork+0x1f/0x30 [10433.970635] rcu: Stack dump where RCU GP kthread last ran: [10433.973643] Sending NMI from CPU 2 to CPUs 0: [10433.980916] NMI backtrace for cpu 0 [10433.980923] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10433.980925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10433.980927] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10433.980929] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10433.980932] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [10433.980935] RAX: 0000000000005b91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10433.980936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10433.980938] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10433.980940] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10433.980941] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10433.980942] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10433.980944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10433.980946] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10433.980947] Call Trace: [10433.980948] [10433.980950] ? show_regs.cold.9+0x22/0x2f [10433.980951] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10433.980953] ? nmi_cpu_backtrace_handler+0x15/0x40 [10433.980954] ? nmi_handle+0xbd/0x210 [10433.980955] ? default_do_nmi+0x4b/0x1b0 [10433.980956] ? do_nmi+0x280/0x380 [10433.980957] ? end_repeat_nmi+0x16/0x1f [10433.980958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10433.980959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10433.980960] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10433.980962] [10433.980962] do_raw_spin_lock+0x177/0x190 [10433.980964] _raw_spin_lock+0x21/0x30 [10433.980965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10433.980966] ? do_raw_spin_unlock+0x75/0x190 [10433.980967] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10433.980968] ? do_wait_intr+0xf0/0xf0 [10433.980969] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10433.980970] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10433.980971] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10433.980973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10433.980974] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10433.980975] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10433.980976] mdc_intent_lock+0x353/0x7e0 [mdc] [10433.980978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10433.980979] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10433.980981] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10433.980982] lmv_intent_open+0x338/0x1150 [lmv] [10433.980983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10433.980985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10433.980986] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10433.980988] ll_intent_lock+0xc5/0xaa0 [lustre] [10433.980989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10433.980990] ll_lookup_it+0xe0e/0x2270 [lustre] [10433.981002] ? do_raw_spin_unlock+0x75/0x190 [10433.981003] ? _raw_spin_unlock+0x12/0x30 [10433.981004] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10433.981006] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10433.981007] ll_atomic_open+0x36a/0xd70 [lustre] [10433.981008] lookup_open+0x356/0xb40 [10433.981009] do_last+0x313/0xfc0 [10433.981011] ? nd_jump_root+0xe5/0x160 [10433.981012] ? path_init+0x437/0x520 [10433.981013] path_openat+0xf7/0x500 [10433.981014] ? do_raw_spin_unlock+0x75/0x190 [10433.981016] do_filp_open+0x99/0x140 [10433.981017] ? getname_flags+0x6e/0x330 [10433.981018] ? __check_object_size+0xff/0x256 [10433.981020] ? do_raw_spin_unlock+0x75/0x190 [10433.981021] ? _raw_spin_unlock+0x12/0x30 [10433.981022] do_sys_openat2+0x2b4/0x410 [10433.981023] do_sys_open+0x73/0xa0 [10433.981024] __x64_sys_openat+0x24/0x30 [10433.981025] do_syscall_64+0xc1/0x440 [10433.981026] entry_SYSCALL_64_after_hwframe+0x49/0xae [10433.981027] RIP: 0033:0x7fa749552332 [10433.981028] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10433.981030] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10433.981033] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10433.981034] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10433.981036] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10433.981037] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10433.981038] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10433.987853] NMI backtrace for cpu 2 [10434.417678] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10434.431839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10434.441269] Call Trace: [10434.445530] [10434.446308] ? dump_stack+0xbb/0x10e [10434.449142] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10434.455947] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10434.463493] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10434.469784] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10434.474106] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10434.475901] ? rcu_dump_cpu_stacks+0x11b/0x171 [10434.488135] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10434.498772] ? update_process_times+0x6d/0xc0 [10434.502600] ? tick_sched_handle.isra.18+0x27/0x90 [10434.504668] ? tick_sched_timer+0x53/0xb0 [10434.506165] ? __hrtimer_run_queues+0x1c2/0x4d0 [10434.508625] ? tick_sched_handle.isra.18+0x90/0x90 [10434.510954] ? hrtimer_interrupt+0x128/0x2c0 [10434.513946] ? smp_apic_timer_interrupt+0xa6/0x270 [10434.523819] ? apic_timer_interrupt+0xf/0x20 [10434.532288] [10434.534787] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10434.538742] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10434.542742] ? do_raw_spin_lock+0x177/0x190 [10434.545037] ? _raw_spin_lock+0x21/0x30 [10434.548151] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10434.554827] ? obd_mod_free+0x46/0x140 [ptlrpc] [10434.560227] ? mdc_commit_open+0x91/0xb0 [mdc] [10434.562593] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10434.565404] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10434.567954] ? after_reply+0xe2c/0x1780 [ptlrpc] [10434.569730] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10434.582028] ? __next_timer_interrupt+0x160/0x160 [10434.584774] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10434.588789] ? do_wait_intr+0xf0/0xf0 [10434.591055] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10434.596689] ? kthread+0x1d1/0x200 [10434.598702] ? set_kthread_struct+0x70/0x70 [10434.601035] ? ret_from_fork+0x1f/0x30 [10436.059197] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10436.063895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10436.107377] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10436.115299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10436.125209] RIP: 0010:native_safe_halt+0x2e/0x50 [10436.126890] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10436.142545] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10436.154773] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10436.159098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10436.163121] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10436.166726] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10436.170090] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10436.173810] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10436.177724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10436.180168] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10436.183599] Call Trace: [10436.184683] [10436.186640] ? show_regs.cold.9+0x22/0x2f [10436.187951] ? watchdog_timer_fn.cold.6+0x62/0x10a [10436.190331] ? __hrtimer_run_queues+0x1c2/0x4d0 [10436.192103] ? watchdog+0x40/0x40 [10436.196579] ? hrtimer_interrupt+0x128/0x2c0 [10436.200166] ? smp_apic_timer_interrupt+0xa6/0x270 [10436.204334] ? apic_timer_interrupt+0xf/0x20 [10436.206604] [10436.207947] ? native_safe_halt+0x2e/0x50 [10436.209701] kvm_wait+0xcd/0xe0 [10436.211654] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10436.214147] do_raw_spin_lock+0x177/0x190 [10436.218057] _raw_spin_lock+0x21/0x30 [10436.220974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10436.224163] ? do_raw_spin_unlock+0x75/0x190 [10436.226997] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10436.231273] ? do_wait_intr+0xf0/0xf0 [10436.235255] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10436.244298] mdc_close+0x53b/0xd70 [mdc] [10436.246105] lmv_close+0x18d/0x490 [lmv] [10436.248381] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10436.255370] ll_md_real_close+0xa5/0x2a0 [lustre] [10436.257979] ll_md_close+0x274/0x800 [lustre] [10436.260385] ll_file_release+0x128/0xc50 [lustre] [10436.263137] __fput+0x10d/0x440 [10436.264606] ____fput+0x12/0x20 [10436.268029] task_work_run+0xc6/0x110 [10436.269944] exit_to_usermode_loop+0x1dd/0x1f0 [10436.272701] do_syscall_64+0x3d6/0x440 [10436.273886] entry_SYSCALL_64_after_hwframe+0x49/0xae [10436.275421] RIP: 0033:0x7fa749552cd8 [10436.279714] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10436.290404] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10436.296719] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10436.300555] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10436.303264] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10436.308127] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10436.311425] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10441.696944] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8524s! [10441.704451] Showing busy workqueues and worker pools: [10441.708205] workqueue events: flags=0x0 [10441.710349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10441.713475] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10441.723554] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10441.729378] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10441.743616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.745448] pending: kfree_rcu_monitor [10441.747458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10441.751989] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10441.757500] workqueue events_long: flags=0x0 [10441.760851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.765108] pending: lu_objects_flush [obdclass] [10441.766982] workqueue events_unbound: flags=0x2 [10441.774027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10441.780494] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10441.787465] workqueue events_power_efficient: flags=0x80 [10441.793060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.798991] pending: check_lifetime [10441.800780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.804268] pending: neigh_periodic_work [10441.807132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.810495] pending: neigh_periodic_work [10441.812519] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.819872] pending: do_cache_clean [10441.822311] workqueue mm_percpu_wq: flags=0x8 [10441.824972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.828477] pending: vmstat_update [10441.830416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.833493] pending: vmstat_update [10441.835336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.839362] pending: vmstat_update [10441.843499] workqueue writeback: flags=0x4a [10441.845486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10441.848482] pending: wb_workfn, wb_workfn [10441.850810] workqueue xprtiod: flags=0xa [10441.852486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10441.855872] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10441.864126] workqueue ipv6_addrconf: flags=0x40008 [10441.866504] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10441.870089] pending: addrconf_verify_work [10441.872143] workqueue knbd0-recv: flags=0x1a [10441.877719] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10441.881275] in-flight: 651:recv_work [10441.883053] workqueue ptlrpc_pinger: flags=0x2 [10441.885474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10441.889386] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10441.892989] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8524s workers=4 idle: 37 31846 28897 [10441.897113] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10431s workers=2 idle: 653 [10460.057690] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10460.058570] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10460.061779] Modules linked in: [10460.067174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10460.067250] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10460.067258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10460.067261] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10460.067275] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10460.067740] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 [10460.069833] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10460.069910] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10460.069916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10460.069922] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10460.070227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10460.070239] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10460.070242] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10460.070247] RAX: 0000000000003b09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10460.070250] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10460.070252] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10460.070254] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10460.070256] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10460.070260] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10460.070263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10460.070265] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10460.070273] Call Trace: [10460.070279] [10460.070285] ? show_regs.cold.9+0x22/0x2f [10460.070295] ? watchdog_timer_fn.cold.6+0x62/0x10a [10460.070302] ? __hrtimer_run_queues+0x1c2/0x4d0 [10460.070306] ? watchdog+0x40/0x40 [10460.070310] ? hrtimer_interrupt+0x128/0x2c0 [10460.070314] ? smp_apic_timer_interrupt+0xa6/0x270 [10460.070322] ? apic_timer_interrupt+0xf/0x20 [10460.070327] [10460.070330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10460.070335] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10460.070340] do_raw_spin_lock+0x177/0x190 [10460.070346] _raw_spin_lock+0x21/0x30 [10460.070350] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10460.070555] process_one_work+0x2c8/0x700 [10460.070563] worker_thread+0x296/0x6e0 [10460.070568] ? rescuer_thread+0x570/0x570 [10460.070572] kthread+0x1d1/0x200 [10460.070577] ? set_kthread_struct+0x70/0x70 [10460.070581] ret_from_fork+0x1f/0x30 [10460.338224] ORIG_RAX: ffffffffffffff13 [10460.339276] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10460.348638] RDX: 0000000000007f84 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10460.354932] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10460.358640] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10460.362989] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10460.365365] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10460.376285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10460.378317] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10460.383124] Call Trace: [10460.384169] [10460.385163] ? show_regs.cold.9+0x22/0x2f [10460.391662] ? watchdog_timer_fn.cold.6+0x62/0x10a [10460.394228] ? __hrtimer_run_queues+0x1c2/0x4d0 [10460.398712] ? watchdog+0x40/0x40 [10460.401748] ? hrtimer_interrupt+0x128/0x2c0 [10460.406354] ? smp_apic_timer_interrupt+0xa6/0x270 [10460.417467] ? apic_timer_interrupt+0xf/0x20 [10460.420623] [10460.422937] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10460.434604] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10460.445049] do_raw_spin_lock+0x177/0x190 [10460.450250] _raw_spin_lock+0x21/0x30 [10460.457844] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10460.467934] obd_mod_free+0x46/0x140 [ptlrpc] [10460.478581] mdc_commit_open+0x91/0xb0 [mdc] [10460.484191] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10460.492966] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10460.497031] after_reply+0xe2c/0x1780 [ptlrpc] [10460.499508] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10460.506045] ? __next_timer_interrupt+0x160/0x160 [10460.509110] ptlrpcd+0xa82/0xe40 [ptlrpc] [10460.512854] ? do_wait_intr+0xf0/0xf0 [10460.515055] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10460.519101] kthread+0x1d1/0x200 [10460.521029] ? set_kthread_struct+0x70/0x70 [10460.522969] ret_from_fork+0x1f/0x30 [10464.059075] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10464.065188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10464.099563] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10464.106666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10464.117074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10464.127522] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10464.145144] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10464.148747] RAX: 0000000000007b96 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10464.154283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10464.158525] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10464.162767] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10464.169307] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10464.175075] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10464.181710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10464.184760] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10464.193582] Call Trace: [10464.194453] [10464.195516] ? show_regs.cold.9+0x22/0x2f [10464.198823] ? watchdog_timer_fn.cold.6+0x62/0x10a [10464.206661] ? __hrtimer_run_queues+0x1c2/0x4d0 [10464.208814] ? watchdog+0x40/0x40 [10464.211773] ? hrtimer_interrupt+0x128/0x2c0 [10464.220258] ? smp_apic_timer_interrupt+0xa6/0x270 [10464.223931] ? apic_timer_interrupt+0xf/0x20 [10464.231162] [10464.233761] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10464.235606] do_raw_spin_lock+0x177/0x190 [10464.237578] _raw_spin_lock+0x21/0x30 [10464.239609] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10464.245342] ? do_raw_spin_unlock+0x75/0x190 [10464.252266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10464.256333] ? do_wait_intr+0xf0/0xf0 [10464.262130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10464.268904] mdc_close+0x53b/0xd70 [mdc] [10464.271218] lmv_close+0x18d/0x490 [lmv] [10464.276474] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10464.281176] ll_md_real_close+0xa5/0x2a0 [lustre] [10464.285641] ll_md_close+0x274/0x800 [lustre] [10464.288483] ll_file_release+0x128/0xc50 [lustre] [10464.289936] __fput+0x10d/0x440 [10464.292915] ____fput+0x12/0x20 [10464.293888] task_work_run+0xc6/0x110 [10464.295317] exit_to_usermode_loop+0x1dd/0x1f0 [10464.297234] do_syscall_64+0x3d6/0x440 [10464.298479] entry_SYSCALL_64_after_hwframe+0x49/0xae [10464.302092] RIP: 0033:0x7fa749552cd8 [10464.303481] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10464.311136] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10464.316198] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10464.323223] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10464.330441] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10464.337480] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10464.344986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10468.060650] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10468.070885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10468.136433] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10468.150873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10468.160110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10468.163820] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10468.182655] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10468.189813] RAX: 0000000000005b0f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10468.200079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10468.202545] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10468.216166] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10468.220073] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10468.230224] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10468.238513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10468.244108] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10468.251190] Call Trace: [10468.252133] [10468.252930] ? show_regs.cold.9+0x22/0x2f [10468.256858] ? watchdog_timer_fn.cold.6+0x62/0x10a [10468.259828] ? __hrtimer_run_queues+0x1c2/0x4d0 [10468.262173] ? watchdog+0x40/0x40 [10468.267191] ? hrtimer_interrupt+0x128/0x2c0 [10468.272819] ? smp_apic_timer_interrupt+0xa6/0x270 [10468.277664] ? apic_timer_interrupt+0xf/0x20 [10468.279279] [10468.281339] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10468.284630] do_raw_spin_lock+0x177/0x190 [10468.287620] _raw_spin_lock+0x21/0x30 [10468.289884] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10468.292993] ? do_raw_spin_unlock+0x75/0x190 [10468.297876] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10468.301090] ? do_wait_intr+0xf0/0xf0 [10468.305045] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10468.311294] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10468.316158] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10468.321040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10468.330928] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10468.334047] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10468.337500] mdc_intent_lock+0x353/0x7e0 [mdc] [10468.340375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10468.347052] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10468.350001] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10468.352287] lmv_intent_open+0x338/0x1150 [lmv] [10468.354033] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10468.360791] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10468.367801] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10468.371140] ll_intent_lock+0xc5/0xaa0 [lustre] [10468.379008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10468.384184] ll_lookup_it+0xe0e/0x2270 [lustre] [10468.391105] ? do_raw_spin_unlock+0x75/0x190 [10468.393910] ? _raw_spin_unlock+0x12/0x30 [10468.397007] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10468.401049] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10468.404517] ll_atomic_open+0x36a/0xd70 [lustre] [10468.406339] lookup_open+0x356/0xb40 [10468.411178] do_last+0x313/0xfc0 [10468.413203] ? nd_jump_root+0xe5/0x160 [10468.414829] ? path_init+0x437/0x520 [10468.417122] path_openat+0xf7/0x500 [10468.419035] ? do_raw_spin_unlock+0x75/0x190 [10468.424572] do_filp_open+0x99/0x140 [10468.431203] ? getname_flags+0x6e/0x330 [10468.433749] ? __check_object_size+0xff/0x256 [10468.437613] ? do_raw_spin_unlock+0x75/0x190 [10468.442653] ? _raw_spin_unlock+0x12/0x30 [10468.447317] do_sys_openat2+0x2b4/0x410 [10468.453125] do_sys_open+0x73/0xa0 [10468.456239] __x64_sys_openat+0x24/0x30 [10468.461113] do_syscall_64+0xc1/0x440 [10468.463645] entry_SYSCALL_64_after_hwframe+0x49/0xae [10468.466322] RIP: 0033:0x7fa749552332 [10468.469007] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10468.492587] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10468.497881] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10468.502280] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10468.510061] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10468.515899] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10468.522124] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10472.421390] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8555s! [10472.426551] Showing busy workqueues and worker pools: [10472.428744] workqueue events: flags=0x0 [10472.430292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10472.433368] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10472.438880] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10472.441493] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10472.452868] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.458762] pending: kfree_rcu_monitor [10472.460945] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10472.464360] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10472.472497] workqueue events_long: flags=0x0 [10472.479640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.484614] pending: lu_objects_flush [obdclass] [10472.489753] workqueue events_unbound: flags=0x2 [10472.493048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10472.498378] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10472.507733] workqueue events_power_efficient: flags=0x80 [10472.510740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.514424] pending: check_lifetime [10472.515812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.521692] pending: neigh_periodic_work [10472.526609] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.528372] pending: neigh_periodic_work [10472.534703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.540035] pending: do_cache_clean [10472.543825] workqueue mm_percpu_wq: flags=0x8 [10472.548679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.552944] pending: vmstat_update [10472.556239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.558935] pending: vmstat_update [10472.565469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10472.569376] pending: vmstat_update [10472.573654] workqueue writeback: flags=0x4a [10472.576808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10472.584344] pending: wb_workfn, wb_workfn [10472.586921] workqueue xprtiod: flags=0xa [10472.589143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10472.598519] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10472.605787] workqueue ipv6_addrconf: flags=0x40008 [10472.610822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10472.612783] pending: addrconf_verify_work [10472.615435] workqueue knbd0-recv: flags=0x1a [10472.619653] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10472.625808] in-flight: 651:recv_work [10472.629258] workqueue ptlrpc_pinger: flags=0x2 [10472.631650] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10472.635377] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10472.637745] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8555s workers=4 idle: 37 31846 28897 [10472.640898] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10462s workers=2 idle: 653 [10488.057219] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10488.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10488.060280] Modules linked in: [10488.064157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10488.064254] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10488.064261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10488.064264] RIP: 0010:native_safe_halt+0x2e/0x50 [10488.064285] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10488.064289] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10488.064294] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10488.064296] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10488.064299] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10488.064301] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10488.064304] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10488.064307] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10488.064310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10488.064312] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10488.064320] Call Trace: [10488.064326] [10488.064331] ? show_regs.cold.9+0x22/0x2f [10488.064341] ? watchdog_timer_fn.cold.6+0x62/0x10a [10488.064347] ? __hrtimer_run_queues+0x1c2/0x4d0 [10488.064357] ? watchdog+0x40/0x40 [10488.064362] ? hrtimer_interrupt+0x128/0x2c0 [10488.064366] ? smp_apic_timer_interrupt+0xa6/0x270 [10488.064373] ? apic_timer_interrupt+0xf/0x20 [10488.064377] [10488.064380] ? native_safe_halt+0x2e/0x50 [10488.064383] kvm_wait+0xcd/0xe0 [10488.064391] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10488.064398] do_raw_spin_lock+0x177/0x190 [10488.064402] _raw_spin_lock+0x21/0x30 [10488.064407] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10488.064904] obd_mod_free+0x46/0x140 [ptlrpc] [10488.065218] mdc_commit_open+0x91/0xb0 [mdc] [10488.065268] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10488.065470] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10488.066047] after_reply+0xe2c/0x1780 [ptlrpc] [10488.066277] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10488.066539] ? __next_timer_interrupt+0x160/0x160 [10488.066551] ptlrpcd+0xa82/0xe40 [ptlrpc] [10488.066784] ? do_wait_intr+0xf0/0xf0 [10488.066793] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10488.067291] kthread+0x1d1/0x200 [10488.067298] ? set_kthread_struct+0x70/0x70 [10488.067302] ret_from_fork+0x1f/0x30 [10488.207280] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10488.224370] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10488.230389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10488.234688] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10488.239119] RIP: 0010:native_safe_halt+0x2e/0x50 [10488.240730] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10488.248906] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10488.251416] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10488.253574] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10488.255476] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10488.257684] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10488.259501] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10488.261706] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10488.264874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10488.267425] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10488.270772] Call Trace: [10488.272507] [10488.273366] ? show_regs.cold.9+0x22/0x2f [10488.274865] ? watchdog_timer_fn.cold.6+0x62/0x10a [10488.278729] ? __hrtimer_run_queues+0x1c2/0x4d0 [10488.280825] ? watchdog+0x40/0x40 [10488.283151] ? hrtimer_interrupt+0x128/0x2c0 [10488.286442] ? smp_apic_timer_interrupt+0xa6/0x270 [10488.289402] ? apic_timer_interrupt+0xf/0x20 [10488.291396] [10488.292408] ? native_safe_halt+0x2e/0x50 [10488.294080] kvm_wait+0xcd/0xe0 [10488.295152] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10488.297347] do_raw_spin_lock+0x177/0x190 [10488.298886] _raw_spin_lock+0x21/0x30 [10488.300103] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10488.302759] process_one_work+0x2c8/0x700 [10488.304467] worker_thread+0x296/0x6e0 [10488.306396] ? rescuer_thread+0x570/0x570 [10488.308139] kthread+0x1d1/0x200 [10488.309555] ? set_kthread_struct+0x70/0x70 [10488.311656] ret_from_fork+0x1f/0x30 [10492.059215] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10492.065501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10492.084388] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10492.087970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10492.092258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10492.095446] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10492.104030] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10492.106579] RAX: 00000000000021b8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10492.109744] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10492.113130] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10492.117163] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10492.120240] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10492.123817] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10492.127700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10492.130535] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10492.134144] Call Trace: [10492.135623] [10492.136593] ? show_regs.cold.9+0x22/0x2f [10492.137889] ? watchdog_timer_fn.cold.6+0x62/0x10a [10492.139504] ? __hrtimer_run_queues+0x1c2/0x4d0 [10492.140813] ? watchdog+0x40/0x40 [10492.142443] ? hrtimer_interrupt+0x128/0x2c0 [10492.144070] ? smp_apic_timer_interrupt+0xa6/0x270 [10492.146188] ? apic_timer_interrupt+0xf/0x20 [10492.147982] [10492.148758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10492.151028] do_raw_spin_lock+0x177/0x190 [10492.152580] _raw_spin_lock+0x21/0x30 [10492.154039] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10492.156711] ? do_raw_spin_unlock+0x75/0x190 [10492.158783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10492.161141] ? do_wait_intr+0xf0/0xf0 [10492.163118] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10492.165630] mdc_close+0x53b/0xd70 [mdc] [10492.167119] lmv_close+0x18d/0x490 [lmv] [10492.169092] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10492.172020] ll_md_real_close+0xa5/0x2a0 [lustre] [10492.174539] ll_md_close+0x274/0x800 [lustre] [10492.176562] ll_file_release+0x128/0xc50 [lustre] [10492.178930] __fput+0x10d/0x440 [10492.179985] ____fput+0x12/0x20 [10492.181398] task_work_run+0xc6/0x110 [10492.183369] exit_to_usermode_loop+0x1dd/0x1f0 [10492.185609] do_syscall_64+0x3d6/0x440 [10492.187257] entry_SYSCALL_64_after_hwframe+0x49/0xae [10492.189251] RIP: 0033:0x7fa749552cd8 [10492.191648] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10492.201014] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10492.206824] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10492.214476] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10492.220879] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10492.224787] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10492.231606] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10500.054407] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10500.060570] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10500.107982] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10500.114273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10500.117980] RIP: 0010:native_safe_halt+0x2e/0x50 [10500.119768] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10500.131930] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10500.136252] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10500.140047] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [10500.144812] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10500.148932] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10500.152680] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10500.156835] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10500.168423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10500.178454] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10500.188376] Call Trace: [10500.193111] [10500.195043] ? show_regs.cold.9+0x22/0x2f [10500.197689] ? watchdog_timer_fn.cold.6+0x62/0x10a [10500.200226] ? __hrtimer_run_queues+0x1c2/0x4d0 [10500.202284] ? watchdog+0x40/0x40 [10500.204164] ? hrtimer_interrupt+0x128/0x2c0 [10500.206258] ? smp_apic_timer_interrupt+0xa6/0x270 [10500.209060] ? apic_timer_interrupt+0xf/0x20 [10500.211572] [10500.212780] ? native_safe_halt+0x2e/0x50 [10500.214662] kvm_wait+0xcd/0xe0 [10500.216408] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10500.219084] do_raw_spin_lock+0x177/0x190 [10500.221206] _raw_spin_lock+0x21/0x30 [10500.223282] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10500.226252] ? do_raw_spin_unlock+0x75/0x190 [10500.228595] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10500.231272] ? do_wait_intr+0xf0/0xf0 [10500.233171] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10500.235953] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10500.238683] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10500.242036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10500.245834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10500.247804] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10500.250672] mdc_intent_lock+0x353/0x7e0 [mdc] [10500.253137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10500.255955] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10500.259399] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10500.262239] lmv_intent_open+0x338/0x1150 [lmv] [10500.264655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10500.267926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10500.270825] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10500.273417] ll_intent_lock+0xc5/0xaa0 [lustre] [10500.275629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10500.278482] ll_lookup_it+0xe0e/0x2270 [lustre] [10500.281031] ? do_raw_spin_unlock+0x75/0x190 [10500.283131] ? _raw_spin_unlock+0x12/0x30 [10500.285148] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10500.287047] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10500.289777] ll_atomic_open+0x36a/0xd70 [lustre] [10500.292160] lookup_open+0x356/0xb40 [10500.295102] do_last+0x313/0xfc0 [10500.296618] ? nd_jump_root+0xe5/0x160 [10500.298644] ? path_init+0x437/0x520 [10500.301031] path_openat+0xf7/0x500 [10500.302621] ? do_raw_spin_unlock+0x75/0x190 [10500.304568] do_filp_open+0x99/0x140 [10500.306610] ? getname_flags+0x6e/0x330 [10500.308464] ? __check_object_size+0xff/0x256 [10500.311399] ? do_raw_spin_unlock+0x75/0x190 [10500.314389] ? _raw_spin_unlock+0x12/0x30 [10500.316652] do_sys_openat2+0x2b4/0x410 [10500.319015] do_sys_open+0x73/0xa0 [10500.321105] __x64_sys_openat+0x24/0x30 [10500.328090] do_syscall_64+0xc1/0x440 [10500.329458] entry_SYSCALL_64_after_hwframe+0x49/0xae [10500.335489] RIP: 0033:0x7fa749552332 [10500.338220] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10500.354643] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10500.362596] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10500.372171] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10500.378842] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10500.391267] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10500.404469] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10503.137319] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8585s! [10503.152749] Showing busy workqueues and worker pools: [10503.161970] workqueue events: flags=0x0 [10503.170643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10503.182149] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10503.195243] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10503.202646] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10503.230089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.242345] pending: kfree_rcu_monitor [10503.245680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10503.254666] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10503.259283] workqueue events_long: flags=0x0 [10503.261092] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.269184] pending: lu_objects_flush [obdclass] [10503.273953] workqueue events_unbound: flags=0x2 [10503.276707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10503.280463] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10503.288590] workqueue events_power_efficient: flags=0x80 [10503.294147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.302751] pending: check_lifetime [10503.306305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.311097] pending: neigh_periodic_work [10503.313096] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.316166] pending: neigh_periodic_work [10503.318096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.325218] pending: do_cache_clean [10503.327822] workqueue mm_percpu_wq: flags=0x8 [10503.329538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.334805] pending: vmstat_update [10503.338243] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.341391] pending: vmstat_update [10503.344105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10503.348161] pending: vmstat_update [10503.350081] workqueue writeback: flags=0x4a [10503.352242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10503.358993] pending: wb_workfn, wb_workfn [10503.360534] workqueue xprtiod: flags=0xa [10503.362900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10503.365899] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10503.371252] workqueue ipv6_addrconf: flags=0x40008 [10503.374247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10503.382205] pending: addrconf_verify_work [10503.388624] workqueue knbd0-recv: flags=0x1a [10503.390910] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10503.395202] in-flight: 651:recv_work [10503.401322] workqueue ptlrpc_pinger: flags=0x2 [10503.404365] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10503.411160] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10503.414087] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8586s workers=4 idle: 37 31846 28897 [10503.418907] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10493s workers=2 idle: 653 [10516.058574] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10516.060488] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10516.065265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10516.065340] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10516.065346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10516.065352] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10516.072818] Modules linked in: lustre(O) osp(O) [10516.091304] [10516.091315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10516.091330] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10516.091334] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10516.091337] RAX: 0000000000007f2a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10516.091339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10516.091341] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10516.091342] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10516.091344] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10516.091347] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10516.091349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10516.091350] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10516.091357] Call Trace: [10516.091362] [10516.091367] ? show_regs.cold.9+0x22/0x2f [10516.091374] ? watchdog_timer_fn.cold.6+0x62/0x10a [10516.091379] ? __hrtimer_run_queues+0x1c2/0x4d0 [10516.091382] ? watchdog+0x40/0x40 [10516.091385] ? hrtimer_interrupt+0x128/0x2c0 [10516.091387] ? smp_apic_timer_interrupt+0xa6/0x270 [10516.091394] ? apic_timer_interrupt+0xf/0x20 [10516.091397] [10516.091399] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10516.091402] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10516.091409] do_raw_spin_lock+0x177/0x190 [10516.091412] _raw_spin_lock+0x21/0x30 [10516.091415] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10516.091828] process_one_work+0x2c8/0x700 [10516.091836] worker_thread+0x296/0x6e0 [10516.091840] ? rescuer_thread+0x570/0x570 [10516.091845] kthread+0x1d1/0x200 [10516.091853] ? set_kthread_struct+0x70/0x70 [10516.091857] ret_from_fork+0x1f/0x30 [10516.277938] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10516.318287] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10516.337601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10516.344942] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10516.352705] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10516.380985] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10516.383356] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10516.392853] RDX: 00000000000055f6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10516.405820] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10516.413918] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10516.423419] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10516.425475] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10516.446650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10516.452960] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10516.456500] Call Trace: [10516.459536] [10516.460725] ? show_regs.cold.9+0x22/0x2f [10516.462999] ? watchdog_timer_fn.cold.6+0x62/0x10a [10516.468538] ? __hrtimer_run_queues+0x1c2/0x4d0 [10516.470981] ? watchdog+0x40/0x40 [10516.475104] ? hrtimer_interrupt+0x128/0x2c0 [10516.477870] ? smp_apic_timer_interrupt+0xa6/0x270 [10516.480414] ? apic_timer_interrupt+0xf/0x20 [10516.484861] [10516.486042] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10516.488490] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10516.490974] do_raw_spin_lock+0x177/0x190 [10516.492197] _raw_spin_lock+0x21/0x30 [10516.495076] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10516.499754] obd_mod_free+0x46/0x140 [ptlrpc] [10516.502366] mdc_commit_open+0x91/0xb0 [mdc] [10516.504291] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10516.506530] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10516.508638] after_reply+0xe2c/0x1780 [ptlrpc] [10516.510750] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10516.512887] ? __next_timer_interrupt+0x160/0x160 [10516.514963] ptlrpcd+0xa82/0xe40 [ptlrpc] [10516.517978] ? do_wait_intr+0xf0/0xf0 [10516.519055] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10516.521047] kthread+0x1d1/0x200 [10516.522192] ? set_kthread_struct+0x70/0x70 [10516.523994] ret_from_fork+0x1f/0x30 [10520.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10520.061849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10520.085624] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10520.090897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10520.102043] RIP: 0010:native_safe_halt+0x2e/0x50 [10520.107000] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10520.117031] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10520.130173] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10520.139866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10520.147624] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10520.149776] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10520.153955] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10520.159754] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10520.165904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10520.169283] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10520.174278] Call Trace: [10520.177428] [10520.178526] ? show_regs.cold.9+0x22/0x2f [10520.183503] ? watchdog_timer_fn.cold.6+0x62/0x10a [10520.187298] ? __hrtimer_run_queues+0x1c2/0x4d0 [10520.191505] ? watchdog+0x40/0x40 [10520.197144] ? hrtimer_interrupt+0x128/0x2c0 [10520.200536] ? smp_apic_timer_interrupt+0xa6/0x270 [10520.205810] ? apic_timer_interrupt+0xf/0x20 [10520.207735] [10520.209707] ? native_safe_halt+0x2e/0x50 [10520.216073] kvm_wait+0xcd/0xe0 [10520.219577] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10520.222610] do_raw_spin_lock+0x177/0x190 [10520.226936] _raw_spin_lock+0x21/0x30 [10520.230105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10520.235215] ? do_raw_spin_unlock+0x75/0x190 [10520.237219] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10520.240872] ? do_wait_intr+0xf0/0xf0 [10520.249900] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10520.260695] mdc_close+0x53b/0xd70 [mdc] [10520.262902] lmv_close+0x18d/0x490 [lmv] [10520.268487] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10520.271859] ll_md_real_close+0xa5/0x2a0 [lustre] [10520.276969] ll_md_close+0x274/0x800 [lustre] [10520.281222] ll_file_release+0x128/0xc50 [lustre] [10520.285932] __fput+0x10d/0x440 [10520.288825] ____fput+0x12/0x20 [10520.292544] task_work_run+0xc6/0x110 [10520.296793] exit_to_usermode_loop+0x1dd/0x1f0 [10520.301662] do_syscall_64+0x3d6/0x440 [10520.304742] entry_SYSCALL_64_after_hwframe+0x49/0xae [10520.307515] RIP: 0033:0x7fa749552cd8 [10520.310067] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10520.329456] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10520.336198] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10520.342449] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10520.347479] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10520.350698] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10520.356885] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10528.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10528.057622] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10528.080631] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10528.084917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10528.088295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10528.090857] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10528.097588] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10528.100635] RAX: 0000000000006188 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10528.103008] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10528.105745] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10528.108449] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10528.111141] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10528.113880] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10528.116981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10528.119712] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10528.123566] Call Trace: [10528.124519] [10528.125295] ? show_regs.cold.9+0x22/0x2f [10528.126861] ? watchdog_timer_fn.cold.6+0x62/0x10a [10528.128743] ? __hrtimer_run_queues+0x1c2/0x4d0 [10528.130667] ? watchdog+0x40/0x40 [10528.131940] ? hrtimer_interrupt+0x128/0x2c0 [10528.133963] ? smp_apic_timer_interrupt+0xa6/0x270 [10528.135532] ? apic_timer_interrupt+0xf/0x20 [10528.137380] [10528.138135] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10528.140764] do_raw_spin_lock+0x177/0x190 [10528.142300] _raw_spin_lock+0x21/0x30 [10528.143863] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10528.145949] ? do_raw_spin_unlock+0x75/0x190 [10528.147810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10528.150130] ? do_wait_intr+0xf0/0xf0 [10528.151630] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10528.153584] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10528.155452] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10528.157973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10528.160110] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10528.161977] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10528.163388] mdc_intent_lock+0x353/0x7e0 [mdc] [10528.165257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10528.167623] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10528.169936] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10528.172281] lmv_intent_open+0x338/0x1150 [lmv] [10528.174029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10528.176412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10528.178566] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10528.179986] ll_intent_lock+0xc5/0xaa0 [lustre] [10528.181480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10528.183480] ll_lookup_it+0xe0e/0x2270 [lustre] [10528.185430] ? do_raw_spin_unlock+0x75/0x190 [10528.187090] ? _raw_spin_unlock+0x12/0x30 [10528.188994] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10528.192717] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10528.194535] ll_atomic_open+0x36a/0xd70 [lustre] [10528.196478] lookup_open+0x356/0xb40 [10528.198137] do_last+0x313/0xfc0 [10528.199866] ? nd_jump_root+0xe5/0x160 [10528.201451] ? path_init+0x437/0x520 [10528.202940] path_openat+0xf7/0x500 [10528.204115] ? do_raw_spin_unlock+0x75/0x190 [10528.206005] do_filp_open+0x99/0x140 [10528.207788] ? getname_flags+0x6e/0x330 [10528.209509] ? __check_object_size+0xff/0x256 [10528.212041] ? do_raw_spin_unlock+0x75/0x190 [10528.214350] ? _raw_spin_unlock+0x12/0x30 [10528.215857] do_sys_openat2+0x2b4/0x410 [10528.217149] do_sys_open+0x73/0xa0 [10528.218398] __x64_sys_openat+0x24/0x30 [10528.219870] do_syscall_64+0xc1/0x440 [10528.221389] entry_SYSCALL_64_after_hwframe+0x49/0xae [10528.223418] RIP: 0033:0x7fa749552332 [10528.224766] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10528.232077] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10528.234917] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10528.237805] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10528.244273] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10528.247694] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10528.250948] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10533.863026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8616s! [10533.866911] Showing busy workqueues and worker pools: [10533.869814] workqueue events: flags=0x0 [10533.872384] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10533.875192] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10533.880476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10533.883049] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10533.887541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.890563] pending: kfree_rcu_monitor [10533.892550] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10533.895074] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10533.898809] workqueue events_long: flags=0x0 [10533.901155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.904027] pending: lu_objects_flush [obdclass] [10533.906472] workqueue events_unbound: flags=0x2 [10533.908397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10533.911759] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10533.918216] workqueue events_power_efficient: flags=0x80 [10533.921046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.923950] pending: check_lifetime [10533.926027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.930256] pending: neigh_periodic_work [10533.932266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.935979] pending: neigh_periodic_work [10533.938264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.941829] pending: do_cache_clean [10533.943724] workqueue mm_percpu_wq: flags=0x8 [10533.945582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.948724] pending: vmstat_update [10533.951527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.954806] pending: vmstat_update [10533.956846] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10533.959877] pending: vmstat_update [10533.961733] workqueue writeback: flags=0x4a [10533.963652] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10533.967220] pending: wb_workfn, wb_workfn [10533.970741] workqueue xprtiod: flags=0xa [10533.972635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10533.975242] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10533.979266] workqueue ipv6_addrconf: flags=0x40008 [10533.981267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10533.984287] pending: addrconf_verify_work [10533.986664] workqueue knbd0-recv: flags=0x1a [10533.988545] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10533.991384] in-flight: 651:recv_work [10533.993058] workqueue ptlrpc_pinger: flags=0x2 [10533.994770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10533.997228] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10533.999813] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8616s workers=4 idle: 37 31846 28897 [10534.003865] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10523s workers=2 idle: 653 [10544.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10544.058205] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10544.060237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10544.060322] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10544.060330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10544.060336] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10544.060860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.060872] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10544.060876] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10544.060881] RAX: 0000000000006971 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10544.060883] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10544.060886] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10544.060888] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10544.060891] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10544.060895] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10544.060898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10544.060900] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10544.060910] Call Trace: [10544.060915] [10544.060921] ? show_regs.cold.9+0x22/0x2f [10544.060931] ? watchdog_timer_fn.cold.6+0x62/0x10a [10544.061002] ? __hrtimer_run_queues+0x1c2/0x4d0 [10544.061008] ? watchdog+0x40/0x40 [10544.061013] ? hrtimer_interrupt+0x128/0x2c0 [10544.061017] ? smp_apic_timer_interrupt+0xa6/0x270 [10544.061026] ? apic_timer_interrupt+0xf/0x20 [10544.061031] [10544.061034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.061039] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10544.061045] do_raw_spin_lock+0x177/0x190 [10544.061050] _raw_spin_lock+0x21/0x30 [10544.061055] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10544.061297] process_one_work+0x2c8/0x700 [10544.061305] worker_thread+0x296/0x6e0 [10544.061310] ? rescuer_thread+0x570/0x570 [10544.061314] kthread+0x1d1/0x200 [10544.061318] ? set_kthread_struct+0x70/0x70 [10544.061322] ret_from_fork+0x1f/0x30 [10544.169830] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10544.188584] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10544.193486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10544.197411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10544.199625] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10544.207394] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10544.210776] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10544.213871] RDX: 0000000000000761 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10544.217180] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10544.220293] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10544.222911] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10544.226470] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10544.230132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10544.232399] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10544.234798] Call Trace: [10544.236197] [10544.236864] ? show_regs.cold.9+0x22/0x2f [10544.238514] ? watchdog_timer_fn.cold.6+0x62/0x10a [10544.240297] ? __hrtimer_run_queues+0x1c2/0x4d0 [10544.241836] ? watchdog+0x40/0x40 [10544.243252] ? hrtimer_interrupt+0x128/0x2c0 [10544.244667] ? smp_apic_timer_interrupt+0xa6/0x270 [10544.246566] ? apic_timer_interrupt+0xf/0x20 [10544.248089] [10544.248796] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10544.250848] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10544.252896] do_raw_spin_lock+0x177/0x190 [10544.254577] _raw_spin_lock+0x21/0x30 [10544.255827] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10544.259084] obd_mod_free+0x46/0x140 [ptlrpc] [10544.261123] mdc_commit_open+0x91/0xb0 [mdc] [10544.262947] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10544.265269] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10544.268102] after_reply+0xe2c/0x1780 [ptlrpc] [10544.270337] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10544.272582] ? __next_timer_interrupt+0x160/0x160 [10544.274301] ptlrpcd+0xa82/0xe40 [ptlrpc] [10544.276371] ? do_wait_intr+0xf0/0xf0 [10544.277875] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10544.280345] kthread+0x1d1/0x200 [10544.281745] ? set_kthread_struct+0x70/0x70 [10544.284385] ret_from_fork+0x1f/0x30 [10548.059075] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10548.061776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10548.093957] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10548.099138] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10548.107179] RIP: 0010:native_safe_halt+0x2e/0x50 [10548.114725] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10548.131103] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10548.140988] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10548.144551] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10548.147770] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10548.155741] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10548.160025] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10548.168323] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10548.180984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10548.186437] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10548.202599] Call Trace: [10548.204163] [10548.205246] ? show_regs.cold.9+0x22/0x2f [10548.209741] ? watchdog_timer_fn.cold.6+0x62/0x10a [10548.213049] ? __hrtimer_run_queues+0x1c2/0x4d0 [10548.218815] ? watchdog+0x40/0x40 [10548.221755] ? hrtimer_interrupt+0x128/0x2c0 [10548.224407] ? smp_apic_timer_interrupt+0xa6/0x270 [10548.231590] ? apic_timer_interrupt+0xf/0x20 [10548.234217] [10548.235428] ? native_safe_halt+0x2e/0x50 [10548.238070] kvm_wait+0xcd/0xe0 [10548.240313] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10548.242797] do_raw_spin_lock+0x177/0x190 [10548.247252] _raw_spin_lock+0x21/0x30 [10548.249148] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10548.256007] ? do_raw_spin_unlock+0x75/0x190 [10548.259536] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10548.263189] ? do_wait_intr+0xf0/0xf0 [10548.271570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10548.276386] mdc_close+0x53b/0xd70 [mdc] [10548.278383] lmv_close+0x18d/0x490 [lmv] [10548.283286] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10548.286495] ll_md_real_close+0xa5/0x2a0 [lustre] [10548.296635] ll_md_close+0x274/0x800 [lustre] [10548.309053] ll_file_release+0x128/0xc50 [lustre] [10548.310916] __fput+0x10d/0x440 [10548.313826] ____fput+0x12/0x20 [10548.316243] task_work_run+0xc6/0x110 [10548.318126] exit_to_usermode_loop+0x1dd/0x1f0 [10548.329303] do_syscall_64+0x3d6/0x440 [10548.333203] entry_SYSCALL_64_after_hwframe+0x49/0xae [10548.338105] RIP: 0033:0x7fa749552cd8 [10548.345929] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10548.364870] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10548.371896] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10548.384669] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10548.392437] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10548.408399] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10548.411852] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10560.054360] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10560.058148] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10560.074330] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10560.077966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10560.081477] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10560.083563] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10560.092918] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10560.096601] RAX: 0000000000002310 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10560.099882] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10560.103455] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10560.107213] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10560.110653] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10560.114001] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10560.118289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10560.120571] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10560.123419] Call Trace: [10560.124599] [10560.125462] ? show_regs.cold.9+0x22/0x2f [10560.127418] ? watchdog_timer_fn.cold.6+0x62/0x10a [10560.129455] ? __hrtimer_run_queues+0x1c2/0x4d0 [10560.131877] ? watchdog+0x40/0x40 [10560.133782] ? hrtimer_interrupt+0x128/0x2c0 [10560.135807] ? smp_apic_timer_interrupt+0xa6/0x270 [10560.138384] ? apic_timer_interrupt+0xf/0x20 [10560.140575] [10560.142386] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10560.146282] do_raw_spin_lock+0x177/0x190 [10560.148200] _raw_spin_lock+0x21/0x30 [10560.150224] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10560.153899] ? do_raw_spin_unlock+0x75/0x190 [10560.155784] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10560.157829] ? do_wait_intr+0xf0/0xf0 [10560.161996] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10560.166271] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10560.169438] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10560.172354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10560.174954] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10560.177831] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10560.180136] mdc_intent_lock+0x353/0x7e0 [mdc] [10560.182295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10560.184515] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10560.187488] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10560.189905] lmv_intent_open+0x338/0x1150 [lmv] [10560.192212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10560.194818] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10560.197590] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10560.199871] ll_intent_lock+0xc5/0xaa0 [lustre] [10560.202637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10560.205439] ll_lookup_it+0xe0e/0x2270 [lustre] [10560.207536] ? do_raw_spin_unlock+0x75/0x190 [10560.209808] ? _raw_spin_unlock+0x12/0x30 [10560.211510] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10560.214144] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10560.216901] ll_atomic_open+0x36a/0xd70 [lustre] [10560.218861] lookup_open+0x356/0xb40 [10560.220466] do_last+0x313/0xfc0 [10560.222553] ? nd_jump_root+0xe5/0x160 [10560.224455] ? path_init+0x437/0x520 [10560.226236] path_openat+0xf7/0x500 [10560.227835] ? do_raw_spin_unlock+0x75/0x190 [10560.229798] do_filp_open+0x99/0x140 [10560.231550] ? getname_flags+0x6e/0x330 [10560.233388] ? __check_object_size+0xff/0x256 [10560.235532] ? do_raw_spin_unlock+0x75/0x190 [10560.237616] ? _raw_spin_unlock+0x12/0x30 [10560.240216] do_sys_openat2+0x2b4/0x410 [10560.242752] do_sys_open+0x73/0xa0 [10560.243997] __x64_sys_openat+0x24/0x30 [10560.245550] do_syscall_64+0xc1/0x440 [10560.246602] entry_SYSCALL_64_after_hwframe+0x49/0xae [10560.248091] RIP: 0033:0x7fa749552332 [10560.249973] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10560.259200] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10560.262463] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10560.265567] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10560.269448] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10560.272622] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10560.275863] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10564.066042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8646s! [10564.070425] Showing busy workqueues and worker pools: [10564.073236] workqueue events: flags=0x0 [10564.075806] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10564.079291] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10564.085433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10564.088829] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10564.093198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.096993] pending: kfree_rcu_monitor [10564.098830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10564.102271] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10564.107404] workqueue events_long: flags=0x0 [10564.109835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.113394] pending: lu_objects_flush [obdclass] [10564.116439] workqueue events_unbound: flags=0x2 [10564.118453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10564.121060] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10564.126083] workqueue events_power_efficient: flags=0x80 [10564.129188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.132964] pending: check_lifetime [10564.134853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.141301] pending: neigh_periodic_work [10564.146355] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.152276] pending: neigh_periodic_work [10564.154489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.159705] pending: do_cache_clean [10564.162764] workqueue mm_percpu_wq: flags=0x8 [10564.172884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.180055] pending: vmstat_update [10564.181620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.185314] pending: vmstat_update [10564.190067] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10564.194898] pending: vmstat_update [10564.200328] workqueue writeback: flags=0x4a [10564.205776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10564.210384] pending: wb_workfn, wb_workfn [10564.212250] workqueue xprtiod: flags=0xa [10564.214648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10564.217911] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10564.224246] workqueue ipv6_addrconf: flags=0x40008 [10564.229993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10564.233428] pending: addrconf_verify_work [10564.239539] workqueue knbd0-recv: flags=0x1a [10564.244961] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10564.247359] in-flight: 651:recv_work [10564.253647] workqueue ptlrpc_pinger: flags=0x2 [10564.257399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10564.263823] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10564.266746] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8647s workers=4 idle: 37 31846 28897 [10564.272284] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10554s workers=2 idle: 653 [10572.057052] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10572.059274] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10572.059674] Modules linked in: [10572.062873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10572.063121] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10572.063130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10572.063133] RIP: 0010:native_safe_halt+0x2e/0x50 [10572.067572] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10572.067645] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10572.067652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10572.067711] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10572.068158] RIP: 0010:native_safe_halt+0x2e/0x50 [10572.068170] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10572.068174] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10572.068179] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10572.068181] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10572.068184] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10572.068186] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10572.068189] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10572.068192] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10572.068195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10572.068198] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10572.068205] Call Trace: [10572.068223] [10572.068230] ? show_regs.cold.9+0x22/0x2f [10572.068239] ? watchdog_timer_fn.cold.6+0x62/0x10a [10572.068246] ? __hrtimer_run_queues+0x1c2/0x4d0 [10572.068250] ? watchdog+0x40/0x40 [10572.068255] ? hrtimer_interrupt+0x128/0x2c0 [10572.068259] ? smp_apic_timer_interrupt+0xa6/0x270 [10572.068266] ? apic_timer_interrupt+0xf/0x20 [10572.068270] [10572.068273] ? native_safe_halt+0x2e/0x50 [10572.068277] kvm_wait+0xcd/0xe0 [10572.068284] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10572.068292] do_raw_spin_lock+0x177/0x190 [10572.068297] _raw_spin_lock+0x21/0x30 [10572.068301] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10572.068562] process_one_work+0x2c8/0x700 [10572.068570] worker_thread+0x296/0x6e0 [10572.068574] ? rescuer_thread+0x570/0x570 [10572.068578] kthread+0x1d1/0x200 [10572.068582] ? set_kthread_struct+0x70/0x70 [10572.068587] ret_from_fork+0x1f/0x30 [10572.223213] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10572.234363] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10572.237850] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10572.241658] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10572.245304] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10572.249680] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10572.253929] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10572.257802] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10572.262197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10572.265687] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10572.271306] Call Trace: [10572.272821] [10572.275427] ? show_regs.cold.9+0x22/0x2f [10572.280398] ? watchdog_timer_fn.cold.6+0x62/0x10a [10572.281910] ? __hrtimer_run_queues+0x1c2/0x4d0 [10572.283888] ? watchdog+0x40/0x40 [10572.285164] ? hrtimer_interrupt+0x128/0x2c0 [10572.288031] ? smp_apic_timer_interrupt+0xa6/0x270 [10572.290038] ? apic_timer_interrupt+0xf/0x20 [10572.293212] [10572.294739] ? native_safe_halt+0x2e/0x50 [10572.296227] kvm_wait+0xcd/0xe0 [10572.297819] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10572.302948] do_raw_spin_lock+0x177/0x190 [10572.307070] _raw_spin_lock+0x21/0x30 [10572.311134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10572.315440] obd_mod_free+0x46/0x140 [ptlrpc] [10572.317960] mdc_commit_open+0x91/0xb0 [mdc] [10572.319424] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10572.327905] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10572.331215] after_reply+0xe2c/0x1780 [ptlrpc] [10572.334574] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10572.338343] ? __next_timer_interrupt+0x160/0x160 [10572.341165] ptlrpcd+0xa82/0xe40 [ptlrpc] [10572.343328] ? do_wait_intr+0xf0/0xf0 [10572.345336] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10572.348110] kthread+0x1d1/0x200 [10572.349649] ? set_kthread_struct+0x70/0x70 [10572.353127] ret_from_fork+0x1f/0x30 [10576.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10576.063250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10576.097016] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10576.103271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10576.112179] RIP: 0010:native_safe_halt+0x2e/0x50 [10576.117273] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10576.129988] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10576.132806] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10576.135761] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10576.138968] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10576.142305] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10576.147498] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10576.151127] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10576.153950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10576.156454] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10576.160200] Call Trace: [10576.161303] [10576.163942] ? show_regs.cold.9+0x22/0x2f [10576.166914] ? watchdog_timer_fn.cold.6+0x62/0x10a [10576.168858] ? __hrtimer_run_queues+0x1c2/0x4d0 [10576.170956] ? watchdog+0x40/0x40 [10576.172072] ? hrtimer_interrupt+0x128/0x2c0 [10576.173265] ? smp_apic_timer_interrupt+0xa6/0x270 [10576.174414] ? apic_timer_interrupt+0xf/0x20 [10576.175856] [10576.176607] ? native_safe_halt+0x2e/0x50 [10576.180972] kvm_wait+0xcd/0xe0 [10576.185995] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10576.187469] do_raw_spin_lock+0x177/0x190 [10576.189210] _raw_spin_lock+0x21/0x30 [10576.190842] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10576.194202] ? do_raw_spin_unlock+0x75/0x190 [10576.197453] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10576.200020] ? do_wait_intr+0xf0/0xf0 [10576.201174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10576.210040] mdc_close+0x53b/0xd70 [mdc] [10576.212002] lmv_close+0x18d/0x490 [lmv] [10576.217899] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10576.223033] ll_md_real_close+0xa5/0x2a0 [lustre] [10576.227267] ll_md_close+0x274/0x800 [lustre] [10576.229608] ll_file_release+0x128/0xc50 [lustre] [10576.232864] __fput+0x10d/0x440 [10576.233881] ____fput+0x12/0x20 [10576.235408] task_work_run+0xc6/0x110 [10576.236607] exit_to_usermode_loop+0x1dd/0x1f0 [10576.237952] do_syscall_64+0x3d6/0x440 [10576.239292] entry_SYSCALL_64_after_hwframe+0x49/0xae [10576.241211] RIP: 0033:0x7fa749552cd8 [10576.245012] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10576.259109] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10576.271672] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10576.275020] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10576.285145] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10576.293974] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10576.297151] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10592.056352] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10592.067552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10592.131271] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10592.165443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10592.178750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10592.184097] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10592.218372] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10592.227607] RAX: 000000000000573e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10592.236439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10592.251931] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10592.266101] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10592.271967] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10592.275658] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10592.281180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10592.293472] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10592.314051] Call Trace: [10592.326038] [10592.332742] ? show_regs.cold.9+0x22/0x2f [10592.351984] ? watchdog_timer_fn.cold.6+0x62/0x10a [10592.370346] ? __hrtimer_run_queues+0x1c2/0x4d0 [10592.388548] ? watchdog+0x40/0x40 [10592.398147] ? hrtimer_interrupt+0x128/0x2c0 [10592.411076] ? smp_apic_timer_interrupt+0xa6/0x270 [10592.417490] ? apic_timer_interrupt+0xf/0x20 [10592.429832] [10592.430674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10592.432312] do_raw_spin_lock+0x177/0x190 [10592.453767] _raw_spin_lock+0x21/0x30 [10592.463590] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10592.472789] ? do_raw_spin_unlock+0x75/0x190 [10592.478323] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10592.481537] ? do_wait_intr+0xf0/0xf0 [10592.484465] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10592.489846] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10592.500489] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10592.512452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.516413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10592.523825] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10592.526070] mdc_intent_lock+0x353/0x7e0 [mdc] [10592.531005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.535471] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10592.551491] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10592.564403] lmv_intent_open+0x338/0x1150 [lmv] [10592.572028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.578523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.588613] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10592.592588] ll_intent_lock+0xc5/0xaa0 [lustre] [10592.600177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.607505] ll_lookup_it+0xe0e/0x2270 [lustre] [10592.616984] ? do_raw_spin_unlock+0x75/0x190 [10592.621262] ? _raw_spin_unlock+0x12/0x30 [10592.627328] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10592.637211] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10592.641971] ll_atomic_open+0x36a/0xd70 [lustre] [10592.645411] lookup_open+0x356/0xb40 [10592.650858] do_last+0x313/0xfc0 [10592.655923] ? nd_jump_root+0xe5/0x160 [10592.665381] ? path_init+0x437/0x520 [10592.670328] path_openat+0xf7/0x500 [10592.675349] ? do_raw_spin_unlock+0x75/0x190 [10592.683135] do_filp_open+0x99/0x140 [10592.687526] ? getname_flags+0x6e/0x330 [10592.688942] ? __check_object_size+0xff/0x256 [10592.694870] ? do_raw_spin_unlock+0x75/0x190 [10592.702527] ? _raw_spin_unlock+0x12/0x30 [10592.711047] do_sys_openat2+0x2b4/0x410 [10592.720580] do_sys_open+0x73/0xa0 [10592.725059] __x64_sys_openat+0x24/0x30 [10592.729617] do_syscall_64+0xc1/0x440 [10592.737046] entry_SYSCALL_64_after_hwframe+0x49/0xae [10592.745681] RIP: 0033:0x7fa749552332 [10592.752134] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10592.779458] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10592.785956] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10592.810359] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10592.824440] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10592.830189] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10592.839843] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10594.788027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8677s! [10594.801807] Showing busy workqueues and worker pools: [10594.811505] workqueue events: flags=0x0 [10594.817565] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10594.826023] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10594.851542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10594.861372] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10594.871174] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10594.879395] pending: kfree_rcu_monitor [10594.884438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10594.888060] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10594.902100] workqueue events_long: flags=0x0 [10594.911117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10594.923222] pending: lu_objects_flush [obdclass] [10594.933835] workqueue events_unbound: flags=0x2 [10594.940079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10594.947176] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10594.958260] workqueue events_power_efficient: flags=0x80 [10594.965789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10594.987469] pending: check_lifetime [10594.994296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10595.003537] pending: neigh_periodic_work [10595.011347] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10595.016417] pending: neigh_periodic_work [10595.018714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10595.021408] pending: do_cache_clean [10595.022887] workqueue mm_percpu_wq: flags=0x8 [10595.025457] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10595.037849] pending: vmstat_update [10595.048681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10595.050979] pending: vmstat_update [10595.060303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10595.070342] pending: vmstat_update [10595.076634] workqueue writeback: flags=0x4a [10595.079791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10595.095009] pending: wb_workfn, wb_workfn [10595.101535] workqueue xprtiod: flags=0xa [10595.110428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10595.134005] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10595.143453] workqueue ipv6_addrconf: flags=0x40008 [10595.149370] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10595.161494] pending: addrconf_verify_work [10595.168468] workqueue knbd0-recv: flags=0x1a [10595.169862] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10595.189163] in-flight: 651:recv_work [10595.201478] workqueue ptlrpc_pinger: flags=0x2 [10595.205898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10595.231582] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10595.242346] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8678s workers=4 idle: 37 31846 28897 [10595.256443] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10585s workers=2 idle: 653 [10600.057171] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10600.061545] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10600.068517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10600.068680] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10600.068826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10600.068832] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10600.069599] RIP: 0010:native_safe_halt+0x2e/0x50 [10600.069612] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10600.069616] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10600.069621] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10600.069624] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10600.069626] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10600.069628] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10600.069630] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10600.069634] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10600.069637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10600.069639] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10600.069647] Call Trace: [10600.069654] [10600.069659] ? show_regs.cold.9+0x22/0x2f [10600.069668] ? watchdog_timer_fn.cold.6+0x62/0x10a [10600.069676] ? __hrtimer_run_queues+0x1c2/0x4d0 [10600.069681] ? watchdog+0x40/0x40 [10600.069755] ? hrtimer_interrupt+0x128/0x2c0 [10600.069761] ? smp_apic_timer_interrupt+0xa6/0x270 [10600.069767] ? apic_timer_interrupt+0xf/0x20 [10600.069772] [10600.069775] ? native_safe_halt+0x2e/0x50 [10600.069779] kvm_wait+0xcd/0xe0 [10600.069786] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10600.069794] do_raw_spin_lock+0x177/0x190 [10600.069798] _raw_spin_lock+0x21/0x30 [10600.069802] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10600.085858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10600.086137] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10600.086145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10600.086148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10600.086161] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10600.086165] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10600.086169] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10600.086172] RDX: 0000000000007880 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10600.086174] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10600.086176] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10600.086179] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10600.086181] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10600.086184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10600.086187] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10600.086197] Call Trace: [10600.086202] [10600.086208] ? show_regs.cold.9+0x22/0x2f [10600.086219] ? watchdog_timer_fn.cold.6+0x62/0x10a [10600.086224] ? __hrtimer_run_queues+0x1c2/0x4d0 [10600.086228] ? watchdog+0x40/0x40 [10600.086233] ? hrtimer_interrupt+0x128/0x2c0 [10600.086237] ? smp_apic_timer_interrupt+0xa6/0x270 [10600.086263] ? apic_timer_interrupt+0xf/0x20 [10600.086269] [10600.086272] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10600.086276] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10600.086281] do_raw_spin_lock+0x177/0x190 [10600.086286] _raw_spin_lock+0x21/0x30 [10600.159499] process_one_work+0x2c8/0x700 [10600.159646] worker_thread+0x296/0x6e0 [10600.159654] ? rescuer_thread+0x570/0x570 [10600.159658] kthread+0x1d1/0x200 [10600.159663] ? set_kthread_struct+0x70/0x70 [10600.159667] ret_from_fork+0x1f/0x30 [10600.674563] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10600.679017] obd_mod_free+0x46/0x140 [ptlrpc] [10600.681632] mdc_commit_open+0x91/0xb0 [mdc] [10600.683713] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10600.694342] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10600.702300] after_reply+0xe2c/0x1780 [ptlrpc] [10600.709874] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10600.712925] ? __next_timer_interrupt+0x160/0x160 [10600.728668] ptlrpcd+0xa82/0xe40 [ptlrpc] [10600.738188] ? do_wait_intr+0xf0/0xf0 [10600.742751] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10600.745799] kthread+0x1d1/0x200 [10600.747324] ? set_kthread_struct+0x70/0x70 [10600.750115] ret_from_fork+0x1f/0x30 [10604.059504] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10604.070903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10604.118603] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10604.134723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10604.142565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10604.150841] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10604.168149] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10604.171605] RAX: 0000000000006b1c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10604.175925] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10604.187752] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10604.202034] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10604.206481] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10604.210737] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10604.216303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10604.226147] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10604.237861] Call Trace: [10604.241833] [10604.243591] ? show_regs.cold.9+0x22/0x2f [10604.248268] ? watchdog_timer_fn.cold.6+0x62/0x10a [10604.257451] ? __hrtimer_run_queues+0x1c2/0x4d0 [10604.266374] ? watchdog+0x40/0x40 [10604.267846] ? hrtimer_interrupt+0x128/0x2c0 [10604.270407] ? smp_apic_timer_interrupt+0xa6/0x270 [10604.273088] ? apic_timer_interrupt+0xf/0x20 [10604.275616] [10604.276980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10604.279792] do_raw_spin_lock+0x177/0x190 [10604.281860] _raw_spin_lock+0x21/0x30 [10604.283624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10604.286731] ? do_raw_spin_unlock+0x75/0x190 [10604.291574] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10604.295128] ? do_wait_intr+0xf0/0xf0 [10604.297140] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10604.301628] mdc_close+0x53b/0xd70 [mdc] [10604.304085] lmv_close+0x18d/0x490 [lmv] [10604.306265] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10604.313017] ll_md_real_close+0xa5/0x2a0 [lustre] [10604.315705] ll_md_close+0x274/0x800 [lustre] [10604.318438] ll_file_release+0x128/0xc50 [lustre] [10604.324960] __fput+0x10d/0x440 [10604.329068] ____fput+0x12/0x20 [10604.334892] task_work_run+0xc6/0x110 [10604.339851] exit_to_usermode_loop+0x1dd/0x1f0 [10604.342269] do_syscall_64+0x3d6/0x440 [10604.344458] entry_SYSCALL_64_after_hwframe+0x49/0xae [10604.348213] RIP: 0033:0x7fa749552cd8 [10604.350623] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10604.373450] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10604.382729] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10604.385382] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10604.392971] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10604.400564] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10604.405528] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10613.874201] rcu: INFO: rcu_sched self-detected stall on CPU [10613.879127] rcu: 2-...!: (4896968 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [10613.891959] (t=8700185 jiffies g=258701 q=4059) [10613.900912] rcu: rcu_sched kthread starved for 8696980 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10613.915406] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10613.940072] rcu: RCU grace-period kthread stack dump: [10613.951075] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10613.966942] Call Trace: [10613.971182] __schedule+0x351/0xcb0 [10613.979281] schedule+0xc0/0x180 [10613.985032] schedule_timeout+0xb4/0x190 [10613.989648] ? do_raw_spin_unlock+0x75/0x190 [10613.997225] ? __next_timer_interrupt+0x160/0x160 [10614.005091] ? prepare_to_swait_event+0xf3/0x140 [10614.012672] rcu_gp_kthread+0x8dd/0x15b0 [10614.019475] ? force_qs_rnp+0x2c0/0x2c0 [10614.025703] kthread+0x1d1/0x200 [10614.027335] ? set_kthread_struct+0x70/0x70 [10614.035413] ret_from_fork+0x1f/0x30 [10614.037428] rcu: Stack dump where RCU GP kthread last ran: [10614.044465] Sending NMI from CPU 2 to CPUs 0: [10614.052388] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [10614.053346] NMI backtrace for cpu 2 [10614.063126] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10614.076436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10614.086789] Call Trace: [10614.089279] [10614.097584] ? dump_stack+0xbb/0x10e [10614.101369] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10614.114200] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10614.125309] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10614.134861] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10614.139969] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10614.153268] ? rcu_dump_cpu_stacks+0x11b/0x171 [10614.156443] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10614.167085] ? update_process_times+0x6d/0xc0 [10614.182429] ? tick_sched_handle.isra.18+0x27/0x90 [10614.190520] ? tick_sched_timer+0x53/0xb0 [10614.206971] ? __hrtimer_run_queues+0x1c2/0x4d0 [10614.216594] ? tick_sched_handle.isra.18+0x90/0x90 [10614.225097] ? hrtimer_interrupt+0x128/0x2c0 [10614.226678] ? smp_apic_timer_interrupt+0xa6/0x270 [10614.245872] ? apic_timer_interrupt+0xf/0x20 [10614.247280] [10614.247988] ? native_safe_halt+0x2e/0x50 [10614.293448] ? kvm_wait+0xcd/0xe0 [10614.294567] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10614.318258] ? do_raw_spin_lock+0x177/0x190 [10614.319844] ? _raw_spin_lock+0x21/0x30 [10614.340873] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10614.352495] ? obd_mod_free+0x46/0x140 [ptlrpc] [10614.370468] ? mdc_commit_open+0x91/0xb0 [mdc] [10614.375966] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10614.388040] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10614.402791] ? after_reply+0xe2c/0x1780 [ptlrpc] [10614.422927] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10614.447556] ? __next_timer_interrupt+0x160/0x160 [10614.450404] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10614.457982] ? do_wait_intr+0xf0/0xf0 [10614.463043] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10614.466700] ? kthread+0x1d1/0x200 [10614.469272] ? set_kthread_struct+0x70/0x70 [10614.471221] ? ret_from_fork+0x1f/0x30 [10625.506039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8708s! [10625.509488] Showing busy workqueues and worker pools: [10625.511692] workqueue events: flags=0x0 [10625.513609] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10625.516704] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10625.521961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10625.525358] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10625.528206] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.530188] pending: kfree_rcu_monitor [10625.531412] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10625.534200] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10625.539957] workqueue events_long: flags=0x0 [10625.544143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.548267] pending: lu_objects_flush [obdclass] [10625.551145] workqueue events_unbound: flags=0x2 [10625.553312] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10625.556983] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10625.561743] workqueue events_power_efficient: flags=0x80 [10625.564392] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.568146] pending: check_lifetime [10625.570187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.573637] pending: neigh_periodic_work [10625.575951] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.579268] pending: neigh_periodic_work [10625.581481] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.584288] pending: do_cache_clean [10625.585981] workqueue mm_percpu_wq: flags=0x8 [10625.588387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.591491] pending: vmstat_update [10625.593311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.596460] pending: vmstat_update [10625.598483] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10625.601663] pending: vmstat_update [10625.603591] workqueue writeback: flags=0x4a [10625.605601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10625.608199] pending: wb_workfn, wb_workfn [10625.609812] workqueue xprtiod: flags=0xa [10625.611254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10625.614002] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10625.617489] workqueue ipv6_addrconf: flags=0x40008 [10625.619825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10625.622503] pending: addrconf_verify_work [10625.624499] workqueue knbd0-recv: flags=0x1a [10625.626824] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10625.629704] in-flight: 651:recv_work [10625.631723] workqueue ptlrpc_pinger: flags=0x2 [10625.633721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10625.636612] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10625.639853] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8708s workers=4 idle: 37 31846 28897 [10625.643446] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10615s workers=2 idle: 653 [10628.061265] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10628.077760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10628.142101] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10628.155348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10628.170735] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10628.179356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10628.182355] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10628.206340] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10628.213358] RAX: 0000000000004e93 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10628.226766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10628.237471] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10628.244401] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10628.255347] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10628.268679] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10628.288834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10628.300324] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10628.306871] Call Trace: [10628.308226] [10628.309458] ? show_regs.cold.9+0x22/0x2f [10628.317230] ? watchdog_timer_fn.cold.6+0x62/0x10a [10628.325017] ? __hrtimer_run_queues+0x1c2/0x4d0 [10628.326737] ? watchdog+0x40/0x40 [10628.333212] ? hrtimer_interrupt+0x128/0x2c0 [10628.337450] ? smp_apic_timer_interrupt+0xa6/0x270 [10628.346330] ? apic_timer_interrupt+0xf/0x20 [10628.354795] [10628.356401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10628.363715] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10628.366443] do_raw_spin_lock+0x177/0x190 [10628.368656] _raw_spin_lock+0x21/0x30 [10628.370015] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10628.372887] process_one_work+0x2c8/0x700 [10628.375649] worker_thread+0x296/0x6e0 [10628.378413] ? rescuer_thread+0x570/0x570 [10628.380228] kthread+0x1d1/0x200 [10628.381759] ? set_kthread_struct+0x70/0x70 [10628.384015] ret_from_fork+0x1f/0x30 [10632.059783] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10632.067415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10632.121568] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10632.138847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10632.149084] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10632.159319] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10632.183933] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10632.192691] RAX: 0000000000006fab RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10632.199692] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10632.209547] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10632.220419] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10632.238106] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10632.247538] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10632.261605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10632.270633] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10632.278247] Call Trace: [10632.279538] [10632.285246] ? show_regs.cold.9+0x22/0x2f [10632.289107] ? watchdog_timer_fn.cold.6+0x62/0x10a [10632.303348] ? __hrtimer_run_queues+0x1c2/0x4d0 [10632.307218] ? watchdog+0x40/0x40 [10632.311710] ? hrtimer_interrupt+0x128/0x2c0 [10632.314428] ? smp_apic_timer_interrupt+0xa6/0x270 [10632.317461] ? apic_timer_interrupt+0xf/0x20 [10632.322509] [10632.326585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10632.333997] do_raw_spin_lock+0x177/0x190 [10632.338939] _raw_spin_lock+0x21/0x30 [10632.346740] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10632.351131] ? do_raw_spin_unlock+0x75/0x190 [10632.352561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10632.364219] ? do_wait_intr+0xf0/0xf0 [10632.368349] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10632.377808] mdc_close+0x53b/0xd70 [mdc] [10632.382985] lmv_close+0x18d/0x490 [lmv] [10632.387590] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10632.405309] ll_md_real_close+0xa5/0x2a0 [lustre] [10632.409985] ll_md_close+0x274/0x800 [lustre] [10632.419923] ll_file_release+0x128/0xc50 [lustre] [10632.433879] __fput+0x10d/0x440 [10632.435035] ____fput+0x12/0x20 [10632.440280] task_work_run+0xc6/0x110 [10632.443387] exit_to_usermode_loop+0x1dd/0x1f0 [10632.448132] do_syscall_64+0x3d6/0x440 [10632.456188] entry_SYSCALL_64_after_hwframe+0x49/0xae [10632.466035] RIP: 0033:0x7fa749552cd8 [10632.474109] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10632.504243] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10632.512064] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10632.527777] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10632.532495] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10632.538944] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10632.552691] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10640.059571] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10640.069425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10640.112162] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10640.119492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10640.125164] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10640.128116] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10640.151063] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10640.163161] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10640.176810] RDX: 0000000000007c29 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10640.181175] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10640.188611] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10640.196149] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10640.199171] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10640.202204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10640.207652] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10640.212596] Call Trace: [10640.214112] [10640.216550] ? show_regs.cold.9+0x22/0x2f [10640.220910] ? watchdog_timer_fn.cold.6+0x62/0x10a [10640.226014] ? __hrtimer_run_queues+0x1c2/0x4d0 [10640.228009] ? watchdog+0x40/0x40 [10640.231587] ? hrtimer_interrupt+0x128/0x2c0 [10640.233981] ? smp_apic_timer_interrupt+0xa6/0x270 [10640.238524] ? apic_timer_interrupt+0xf/0x20 [10640.244794] [10640.247532] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10640.254241] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10640.261891] do_raw_spin_lock+0x177/0x190 [10640.267824] _raw_spin_lock+0x21/0x30 [10640.274814] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10640.278002] obd_mod_free+0x46/0x140 [ptlrpc] [10640.283121] mdc_commit_open+0x91/0xb0 [mdc] [10640.285312] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10640.289139] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10640.295225] after_reply+0xe2c/0x1780 [ptlrpc] [10640.300566] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10640.304178] ? __next_timer_interrupt+0x160/0x160 [10640.307410] ptlrpcd+0xa82/0xe40 [ptlrpc] [10640.311294] ? do_wait_intr+0xf0/0xf0 [10640.315143] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10640.318412] kthread+0x1d1/0x200 [10640.321332] ? set_kthread_struct+0x70/0x70 [10640.323325] ret_from_fork+0x1f/0x30 [10652.058274] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10652.071012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10652.128759] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10652.143019] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10652.155670] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10652.163037] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10652.187722] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10652.199387] RAX: 0000000000006f12 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10652.205042] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10652.209623] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10652.215867] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10652.221260] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10652.226959] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10652.232739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10652.235945] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10652.239927] Call Trace: [10652.241146] [10652.242009] ? show_regs.cold.9+0x22/0x2f [10652.244034] ? watchdog_timer_fn.cold.6+0x62/0x10a [10652.245972] ? __hrtimer_run_queues+0x1c2/0x4d0 [10652.248550] ? watchdog+0x40/0x40 [10652.250011] ? hrtimer_interrupt+0x128/0x2c0 [10652.253169] ? smp_apic_timer_interrupt+0xa6/0x270 [10652.259736] ? apic_timer_interrupt+0xf/0x20 [10652.264010] [10652.264648] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10652.270806] do_raw_spin_lock+0x177/0x190 [10652.274805] _raw_spin_lock+0x21/0x30 [10652.278967] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10652.286219] ? do_raw_spin_unlock+0x75/0x190 [10652.290742] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10652.293212] ? do_wait_intr+0xf0/0xf0 [10652.300318] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10652.302296] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10652.307980] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10652.316399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10652.322910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10652.326588] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10652.330024] mdc_intent_lock+0x353/0x7e0 [mdc] [10652.336432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10652.338922] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10652.345495] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10652.351921] lmv_intent_open+0x338/0x1150 [lmv] [10652.359992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10652.371705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10652.385913] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10652.389084] ll_intent_lock+0xc5/0xaa0 [lustre] [10652.397425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10652.405795] ll_lookup_it+0xe0e/0x2270 [lustre] [10652.411742] ? do_raw_spin_unlock+0x75/0x190 [10652.413406] ? _raw_spin_unlock+0x12/0x30 [10652.417736] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10652.420144] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10652.428540] ll_atomic_open+0x36a/0xd70 [lustre] [10652.434777] lookup_open+0x356/0xb40 [10652.437859] do_last+0x313/0xfc0 [10652.441110] ? nd_jump_root+0xe5/0x160 [10652.443217] ? path_init+0x437/0x520 [10652.446537] path_openat+0xf7/0x500 [10652.448967] ? do_raw_spin_unlock+0x75/0x190 [10652.454290] do_filp_open+0x99/0x140 [10652.465603] ? getname_flags+0x6e/0x330 [10652.467035] ? __check_object_size+0xff/0x256 [10652.473427] ? do_raw_spin_unlock+0x75/0x190 [10652.475048] ? _raw_spin_unlock+0x12/0x30 [10652.481339] do_sys_openat2+0x2b4/0x410 [10652.485950] do_sys_open+0x73/0xa0 [10652.496014] __x64_sys_openat+0x24/0x30 [10652.501896] do_syscall_64+0xc1/0x440 [10652.506267] entry_SYSCALL_64_after_hwframe+0x49/0xae [10652.513931] RIP: 0033:0x7fa749552332 [10652.517215] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10652.537148] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10652.548572] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10652.563470] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10652.578979] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10652.596020] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10652.624758] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10655.713050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8738s! [10655.717856] Showing busy workqueues and worker pools: [10655.724072] workqueue events: flags=0x0 [10655.726432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10655.733823] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10655.749068] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10655.753563] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10655.764726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.773849] pending: kfree_rcu_monitor [10655.777583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10655.780096] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10655.788516] workqueue events_long: flags=0x0 [10655.794730] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.801114] pending: lu_objects_flush [obdclass] [10655.803039] workqueue events_unbound: flags=0x2 [10655.806798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10655.810195] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10655.821619] workqueue events_power_efficient: flags=0x80 [10655.825932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.832469] pending: check_lifetime [10655.834801] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.840726] pending: neigh_periodic_work [10655.842854] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.851906] pending: neigh_periodic_work [10655.856676] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.859850] pending: do_cache_clean [10655.861148] workqueue mm_percpu_wq: flags=0x8 [10655.864485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.873201] pending: vmstat_update [10655.875738] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.879143] pending: vmstat_update [10655.883714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.886792] pending: vmstat_update [10655.889488] workqueue writeback: flags=0x4a [10655.892496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10655.898479] pending: wb_workfn, wb_workfn [10655.903406] workqueue xprtiod: flags=0xa [10655.905585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10655.908671] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10655.912255] workqueue ipv6_addrconf: flags=0x40008 [10655.919568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10655.921667] pending: addrconf_verify_work [10655.926885] workqueue knbd0-recv: flags=0x1a [10655.930625] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10655.935670] in-flight: 651:recv_work [10655.940653] workqueue ptlrpc_pinger: flags=0x2 [10655.942774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10655.946130] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10655.948168] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8738s workers=4 idle: 37 31846 28897 [10655.957680] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10645s workers=2 idle: 653 [10656.060471] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10656.074078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10656.121148] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10656.129918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10656.136697] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10656.140569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10656.146268] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10656.162256] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10656.169763] RAX: 0000000000007607 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10656.178619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10656.183483] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10656.188699] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10656.197643] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10656.201066] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10656.205226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10656.207758] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10656.214559] Call Trace: [10656.216686] [10656.218672] ? show_regs.cold.9+0x22/0x2f [10656.223362] ? watchdog_timer_fn.cold.6+0x62/0x10a [10656.226562] ? __hrtimer_run_queues+0x1c2/0x4d0 [10656.230558] ? watchdog+0x40/0x40 [10656.232890] ? hrtimer_interrupt+0x128/0x2c0 [10656.237471] ? smp_apic_timer_interrupt+0xa6/0x270 [10656.240316] ? apic_timer_interrupt+0xf/0x20 [10656.244253] [10656.246600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10656.252700] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10656.256803] do_raw_spin_lock+0x177/0x190 [10656.259827] _raw_spin_lock+0x21/0x30 [10656.262862] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10656.273782] process_one_work+0x2c8/0x700 [10656.278454] worker_thread+0x296/0x6e0 [10656.281714] ? rescuer_thread+0x570/0x570 [10656.283932] kthread+0x1d1/0x200 [10656.285654] ? set_kthread_struct+0x70/0x70 [10656.287696] ret_from_fork+0x1f/0x30 [10660.064434] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10660.071595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10660.136133] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10660.146764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10660.163719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10660.176319] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10660.186199] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10660.188759] RAX: 0000000000007a64 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10660.193861] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10660.199292] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10660.208729] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10660.221748] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10660.233541] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10660.246490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10660.264256] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10660.270817] Call Trace: [10660.276751] [10660.277878] ? show_regs.cold.9+0x22/0x2f [10660.288693] ? watchdog_timer_fn.cold.6+0x62/0x10a [10660.290389] ? __hrtimer_run_queues+0x1c2/0x4d0 [10660.300091] ? watchdog+0x40/0x40 [10660.302953] ? hrtimer_interrupt+0x128/0x2c0 [10660.317295] ? smp_apic_timer_interrupt+0xa6/0x270 [10660.325288] ? apic_timer_interrupt+0xf/0x20 [10660.334903] [10660.336064] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10660.342514] do_raw_spin_lock+0x177/0x190 [10660.345455] _raw_spin_lock+0x21/0x30 [10660.356077] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10660.366472] ? do_raw_spin_unlock+0x75/0x190 [10660.371530] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10660.380358] ? do_wait_intr+0xf0/0xf0 [10660.382215] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10660.387148] mdc_close+0x53b/0xd70 [mdc] [10660.391673] lmv_close+0x18d/0x490 [lmv] [10660.394844] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10660.408125] ll_md_real_close+0xa5/0x2a0 [lustre] [10660.417394] ll_md_close+0x274/0x800 [lustre] [10660.429575] ll_file_release+0x128/0xc50 [lustre] [10660.441782] __fput+0x10d/0x440 [10660.443490] ____fput+0x12/0x20 [10660.445733] task_work_run+0xc6/0x110 [10660.449497] exit_to_usermode_loop+0x1dd/0x1f0 [10660.453269] do_syscall_64+0x3d6/0x440 [10660.458600] entry_SYSCALL_64_after_hwframe+0x49/0xae [10660.465511] RIP: 0033:0x7fa749552cd8 [10660.470027] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10660.492112] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10660.497047] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10660.507386] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10660.523750] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10660.539541] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10660.556171] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10668.062355] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10668.072126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10668.119067] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10668.138501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10668.149253] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10668.152904] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10668.165555] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10668.169624] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10668.173604] RDX: 00000000000044bc RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10668.177224] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10668.180117] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10668.185372] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10668.189723] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10668.195249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10668.198731] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10668.204230] Call Trace: [10668.205885] [10668.207151] ? show_regs.cold.9+0x22/0x2f [10668.209125] ? watchdog_timer_fn.cold.6+0x62/0x10a [10668.212022] ? __hrtimer_run_queues+0x1c2/0x4d0 [10668.219512] ? watchdog+0x40/0x40 [10668.221667] ? hrtimer_interrupt+0x128/0x2c0 [10668.224369] ? smp_apic_timer_interrupt+0xa6/0x270 [10668.228213] ? apic_timer_interrupt+0xf/0x20 [10668.230593] [10668.232578] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10668.236877] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10668.240369] do_raw_spin_lock+0x177/0x190 [10668.244555] _raw_spin_lock+0x21/0x30 [10668.246019] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10668.249847] obd_mod_free+0x46/0x140 [ptlrpc] [10668.254069] mdc_commit_open+0x91/0xb0 [mdc] [10668.256060] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10668.260731] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10668.264906] after_reply+0xe2c/0x1780 [ptlrpc] [10668.270909] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10668.275260] ? __next_timer_interrupt+0x160/0x160 [10668.279646] ptlrpcd+0xa82/0xe40 [ptlrpc] [10668.285987] ? do_wait_intr+0xf0/0xf0 [10668.288348] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10668.297338] kthread+0x1d1/0x200 [10668.300542] ? set_kthread_struct+0x70/0x70 [10668.303577] ret_from_fork+0x1f/0x30 [10680.054077] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10680.058127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10680.082185] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10680.087386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10680.091109] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.093438] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10680.101020] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10680.105118] RAX: 0000000000000a2c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10680.109112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10680.112473] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10680.116256] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10680.118307] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10680.122677] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10680.125460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10680.128301] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10680.131146] Call Trace: [10680.132179] [10680.133487] ? show_regs.cold.9+0x22/0x2f [10680.135223] ? watchdog_timer_fn.cold.6+0x62/0x10a [10680.137531] ? __hrtimer_run_queues+0x1c2/0x4d0 [10680.139484] ? watchdog+0x40/0x40 [10680.141327] ? hrtimer_interrupt+0x128/0x2c0 [10680.143294] ? smp_apic_timer_interrupt+0xa6/0x270 [10680.146122] ? apic_timer_interrupt+0xf/0x20 [10680.148398] [10680.149483] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.152486] do_raw_spin_lock+0x177/0x190 [10680.154804] _raw_spin_lock+0x21/0x30 [10680.157341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10680.160141] ? do_raw_spin_unlock+0x75/0x190 [10680.161470] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10680.163935] ? do_wait_intr+0xf0/0xf0 [10680.165138] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10680.168071] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10680.170733] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10680.173426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.176406] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10680.178611] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10680.180482] mdc_intent_lock+0x353/0x7e0 [mdc] [10680.182683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.185150] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10680.189215] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10680.192079] lmv_intent_open+0x338/0x1150 [lmv] [10680.195137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.197565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.200165] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10680.202194] ll_intent_lock+0xc5/0xaa0 [lustre] [10680.204471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.208378] ll_lookup_it+0xe0e/0x2270 [lustre] [10680.210134] ? do_raw_spin_unlock+0x75/0x190 [10680.213303] ? _raw_spin_unlock+0x12/0x30 [10680.215179] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10680.218335] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10680.220490] ll_atomic_open+0x36a/0xd70 [lustre] [10680.222654] lookup_open+0x356/0xb40 [10680.224120] do_last+0x313/0xfc0 [10680.225689] ? nd_jump_root+0xe5/0x160 [10680.228602] ? path_init+0x437/0x520 [10680.230183] path_openat+0xf7/0x500 [10680.231333] ? do_raw_spin_unlock+0x75/0x190 [10680.235154] do_filp_open+0x99/0x140 [10680.236685] ? getname_flags+0x6e/0x330 [10680.238683] ? __check_object_size+0xff/0x256 [10680.240390] ? do_raw_spin_unlock+0x75/0x190 [10680.243039] ? _raw_spin_unlock+0x12/0x30 [10680.244578] do_sys_openat2+0x2b4/0x410 [10680.249110] do_sys_open+0x73/0xa0 [10680.250043] __x64_sys_openat+0x24/0x30 [10680.252347] do_syscall_64+0xc1/0x440 [10680.253812] entry_SYSCALL_64_after_hwframe+0x49/0xae [10680.256333] RIP: 0033:0x7fa749552332 [10680.257545] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10680.266369] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10680.272032] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10680.276270] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10680.278785] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10680.281216] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10680.286757] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10684.063209] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10684.070571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10684.146164] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10684.165322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10684.180104] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10684.194144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10684.200454] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10684.224910] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10684.230489] RAX: 0000000000007f8a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10684.239857] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10684.243344] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10684.247012] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10684.251391] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10684.254121] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10684.257057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10684.261754] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10684.265222] Call Trace: [10684.266122] [10684.266884] ? show_regs.cold.9+0x22/0x2f [10684.268294] ? watchdog_timer_fn.cold.6+0x62/0x10a [10684.270557] ? __hrtimer_run_queues+0x1c2/0x4d0 [10684.274008] ? watchdog+0x40/0x40 [10684.276058] ? hrtimer_interrupt+0x128/0x2c0 [10684.279146] ? smp_apic_timer_interrupt+0xa6/0x270 [10684.282827] ? apic_timer_interrupt+0xf/0x20 [10684.284782] [10684.285941] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10684.289139] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10684.292331] do_raw_spin_lock+0x177/0x190 [10684.294185] _raw_spin_lock+0x21/0x30 [10684.296299] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10684.299078] process_one_work+0x2c8/0x700 [10684.300418] worker_thread+0x296/0x6e0 [10684.303458] ? rescuer_thread+0x570/0x570 [10684.306626] kthread+0x1d1/0x200 [10684.307632] ? set_kthread_struct+0x70/0x70 [10684.315902] ret_from_fork+0x1f/0x30 [10686.433155] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8769s! [10686.437237] Showing busy workqueues and worker pools: [10686.439770] workqueue events: flags=0x0 [10686.441930] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10686.445168] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10686.451460] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10686.454491] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10686.459481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.462760] pending: kfree_rcu_monitor [10686.465189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10686.468630] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10686.473132] workqueue events_long: flags=0x0 [10686.475356] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.478828] pending: lu_objects_flush [obdclass] [10686.481562] workqueue events_unbound: flags=0x2 [10686.483888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10686.487151] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10686.493350] workqueue events_power_efficient: flags=0x80 [10686.496025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.499170] pending: check_lifetime [10686.501515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.504602] pending: neigh_periodic_work [10686.506761] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.509955] pending: neigh_periodic_work [10686.511760] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.514865] pending: do_cache_clean [10686.516528] workqueue mm_percpu_wq: flags=0x8 [10686.518634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.522805] pending: vmstat_update [10686.524860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.528199] pending: vmstat_update [10686.529975] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10686.533632] pending: vmstat_update [10686.535433] workqueue writeback: flags=0x4a [10686.537361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10686.540348] pending: wb_workfn, wb_workfn [10686.542166] workqueue xprtiod: flags=0xa [10686.544065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10686.546986] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10686.550271] workqueue ipv6_addrconf: flags=0x40008 [10686.552175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10686.554441] pending: addrconf_verify_work [10686.556705] workqueue knbd0-recv: flags=0x1a [10686.559057] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10686.562375] in-flight: 651:recv_work [10686.564655] workqueue ptlrpc_pinger: flags=0x2 [10686.567051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10686.570153] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10686.573378] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8769s workers=4 idle: 37 31846 28897 [10686.577723] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10676s workers=2 idle: 653 [10688.059033] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10688.065684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10688.093237] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10688.102527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10688.109996] RIP: 0010:native_safe_halt+0x2e/0x50 [10688.114744] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10688.132468] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10688.144584] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10688.147857] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10688.156084] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10688.168615] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10688.175003] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10688.179950] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10688.186863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10688.192304] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10688.197152] Call Trace: [10688.198871] [10688.199567] ? show_regs.cold.9+0x22/0x2f [10688.205111] ? watchdog_timer_fn.cold.6+0x62/0x10a [10688.208384] ? __hrtimer_run_queues+0x1c2/0x4d0 [10688.214530] ? watchdog+0x40/0x40 [10688.215801] ? hrtimer_interrupt+0x128/0x2c0 [10688.217123] ? smp_apic_timer_interrupt+0xa6/0x270 [10688.224367] ? apic_timer_interrupt+0xf/0x20 [10688.227479] [10688.228308] ? native_safe_halt+0x2e/0x50 [10688.232740] kvm_wait+0xcd/0xe0 [10688.235548] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10688.239948] do_raw_spin_lock+0x177/0x190 [10688.244112] _raw_spin_lock+0x21/0x30 [10688.245525] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10688.251307] ? do_raw_spin_unlock+0x75/0x190 [10688.253577] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10688.255791] ? do_wait_intr+0xf0/0xf0 [10688.263382] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10688.266281] mdc_close+0x53b/0xd70 [mdc] [10688.269628] lmv_close+0x18d/0x490 [lmv] [10688.275171] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10688.281048] ll_md_real_close+0xa5/0x2a0 [lustre] [10688.290271] ll_md_close+0x274/0x800 [lustre] [10688.295282] ll_file_release+0x128/0xc50 [lustre] [10688.299414] __fput+0x10d/0x440 [10688.300738] ____fput+0x12/0x20 [10688.303108] task_work_run+0xc6/0x110 [10688.304733] exit_to_usermode_loop+0x1dd/0x1f0 [10688.307361] do_syscall_64+0x3d6/0x440 [10688.311650] entry_SYSCALL_64_after_hwframe+0x49/0xae [10688.317229] RIP: 0033:0x7fa749552cd8 [10688.319198] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10688.330067] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10688.333299] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10688.336588] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10688.345247] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10688.349039] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10688.355814] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10696.058036] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10696.061923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10696.083797] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10696.089198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10696.093004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10696.095328] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10696.104084] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10696.107489] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10696.110225] RDX: 0000000000006269 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10696.113286] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10696.116700] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10696.120577] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10696.124104] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10696.132214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10696.137921] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10696.144735] Call Trace: [10696.145832] [10696.147137] ? show_regs.cold.9+0x22/0x2f [10696.151441] ? watchdog_timer_fn.cold.6+0x62/0x10a [10696.156073] ? __hrtimer_run_queues+0x1c2/0x4d0 [10696.160113] ? watchdog+0x40/0x40 [10696.163245] ? hrtimer_interrupt+0x128/0x2c0 [10696.165973] ? smp_apic_timer_interrupt+0xa6/0x270 [10696.170719] ? apic_timer_interrupt+0xf/0x20 [10696.173217] [10696.177013] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10696.179342] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10696.181574] do_raw_spin_lock+0x177/0x190 [10696.183262] _raw_spin_lock+0x21/0x30 [10696.184541] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10696.188499] obd_mod_free+0x46/0x140 [ptlrpc] [10696.190479] mdc_commit_open+0x91/0xb0 [mdc] [10696.192973] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10696.200815] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10696.203971] after_reply+0xe2c/0x1780 [ptlrpc] [10696.208212] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10696.212679] ? __next_timer_interrupt+0x160/0x160 [10696.214687] ptlrpcd+0xa82/0xe40 [ptlrpc] [10696.220314] ? do_wait_intr+0xf0/0xf0 [10696.225238] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10696.228097] kthread+0x1d1/0x200 [10696.233017] ? set_kthread_struct+0x70/0x70 [10696.235024] ret_from_fork+0x1f/0x30 [10712.056136] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10712.065117] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10712.070977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10712.071118] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10712.071126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10712.071129] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.071142] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10712.071147] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10712.071151] RAX: 0000000000005284 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10712.071154] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10712.071156] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10712.071159] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10712.071161] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10712.071164] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10712.071167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10712.071169] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10712.072390] Call Trace: [10712.072401] [10712.072408] ? show_regs.cold.9+0x22/0x2f [10712.072423] ? watchdog_timer_fn.cold.6+0x62/0x10a [10712.072431] ? __hrtimer_run_queues+0x1c2/0x4d0 [10712.072435] ? watchdog+0x40/0x40 [10712.072439] ? hrtimer_interrupt+0x128/0x2c0 [10712.072443] ? smp_apic_timer_interrupt+0xa6/0x270 [10712.072451] ? apic_timer_interrupt+0xf/0x20 [10712.072456] [10712.072459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.072469] do_raw_spin_lock+0x177/0x190 [10712.072474] _raw_spin_lock+0x21/0x30 [10712.072478] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10712.072774] ? do_raw_spin_unlock+0x75/0x190 [10712.072780] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10712.073221] ? do_wait_intr+0xf0/0xf0 [10712.073231] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10712.077138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10712.077228] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10712.077235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10712.077243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10712.151613] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10712.151922] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10712.152151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10712.152261] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10712.152315] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10712.152344] mdc_intent_lock+0x353/0x7e0 [mdc] [10712.152369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10712.152449] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10712.152662] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10712.152690] lmv_intent_open+0x338/0x1150 [lmv] [10712.152723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10712.152806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10712.152887] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10712.152908] ll_intent_lock+0xc5/0xaa0 [lustre] [10712.152989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10712.153068] ll_lookup_it+0xe0e/0x2270 [lustre] [10712.153148] ? do_raw_spin_unlock+0x75/0x190 [10712.153157] ? _raw_spin_unlock+0x12/0x30 [10712.153163] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10712.153190] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10712.153208] ll_atomic_open+0x36a/0xd70 [lustre] [10712.153306] lookup_open+0x356/0xb40 [10712.153316] do_last+0x313/0xfc0 [10712.153321] ? nd_jump_root+0xe5/0x160 [10712.153326] ? path_init+0x437/0x520 [10712.153331] path_openat+0xf7/0x500 [10712.153335] ? do_raw_spin_unlock+0x75/0x190 [10712.153340] do_filp_open+0x99/0x140 [10712.153346] ? getname_flags+0x6e/0x330 [10712.153351] ? __check_object_size+0xff/0x256 [10712.153358] ? do_raw_spin_unlock+0x75/0x190 [10712.153362] ? _raw_spin_unlock+0x12/0x30 [10712.153367] do_sys_openat2+0x2b4/0x410 [10712.153371] do_sys_open+0x73/0xa0 [10712.153375] __x64_sys_openat+0x24/0x30 [10712.153379] do_syscall_64+0xc1/0x440 [10712.153385] entry_SYSCALL_64_after_hwframe+0x49/0xae [10712.153391] RIP: 0033:0x7fa749552332 [10712.153399] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10712.153403] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10712.153408] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10712.153410] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10712.153412] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10712.153414] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10712.153416] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10712.970959] [10712.971494] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.983402] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10713.031273] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10713.037658] RAX: 000000000000471f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10713.043073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10713.048631] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10713.056413] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10713.064292] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10713.068630] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10713.075345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10713.081320] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10713.085432] Call Trace: [10713.087274] [10713.088125] ? show_regs.cold.9+0x22/0x2f [10713.089688] ? watchdog_timer_fn.cold.6+0x62/0x10a [10713.092569] ? __hrtimer_run_queues+0x1c2/0x4d0 [10713.094111] ? watchdog+0x40/0x40 [10713.095215] ? hrtimer_interrupt+0x128/0x2c0 [10713.098404] ? smp_apic_timer_interrupt+0xa6/0x270 [10713.099926] ? apic_timer_interrupt+0xf/0x20 [10713.106310] [10713.107117] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10713.110155] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10713.112490] do_raw_spin_lock+0x177/0x190 [10713.114480] _raw_spin_lock+0x21/0x30 [10713.116476] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10713.123556] process_one_work+0x2c8/0x700 [10713.125082] worker_thread+0x296/0x6e0 [10713.126723] ? rescuer_thread+0x570/0x570 [10713.128146] kthread+0x1d1/0x200 [10713.131222] ? set_kthread_struct+0x70/0x70 [10713.132557] ret_from_fork+0x1f/0x30 [10716.063195] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10716.070630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10716.131729] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10716.140637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10716.150426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10716.158018] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10716.168696] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10716.171781] RAX: 0000000000004319 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10716.178678] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10716.185800] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10716.197848] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10716.206785] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10716.223555] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10716.230548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10716.239321] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10716.254251] Call Trace: [10716.255120] [10716.261173] ? show_regs.cold.9+0x22/0x2f [10716.267452] ? watchdog_timer_fn.cold.6+0x62/0x10a [10716.268752] ? __hrtimer_run_queues+0x1c2/0x4d0 [10716.279995] ? watchdog+0x40/0x40 [10716.282003] ? hrtimer_interrupt+0x128/0x2c0 [10716.285694] ? smp_apic_timer_interrupt+0xa6/0x270 [10716.287946] ? apic_timer_interrupt+0xf/0x20 [10716.291942] [10716.292635] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10716.294959] do_raw_spin_lock+0x177/0x190 [10716.297091] _raw_spin_lock+0x21/0x30 [10716.300354] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10716.303845] ? do_raw_spin_unlock+0x75/0x190 [10716.305451] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10716.313457] ? do_wait_intr+0xf0/0xf0 [10716.315866] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10716.326892] mdc_close+0x53b/0xd70 [mdc] [10716.329865] lmv_close+0x18d/0x490 [lmv] [10716.334548] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10716.339563] ll_md_real_close+0xa5/0x2a0 [lustre] [10716.344609] ll_md_close+0x274/0x800 [lustre] [10716.347460] ll_file_release+0x128/0xc50 [lustre] [10716.349894] __fput+0x10d/0x440 [10716.352112] ____fput+0x12/0x20 [10716.353459] task_work_run+0xc6/0x110 [10716.354874] exit_to_usermode_loop+0x1dd/0x1f0 [10716.356836] do_syscall_64+0x3d6/0x440 [10716.358874] entry_SYSCALL_64_after_hwframe+0x49/0xae [10716.361365] RIP: 0033:0x7fa749552cd8 [10716.362773] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10716.379383] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10716.386903] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10716.393578] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10716.400408] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10716.409792] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10716.414323] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10716.651718] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8799s! [10716.665475] Showing busy workqueues and worker pools: [10716.672179] workqueue events: flags=0x0 [10716.676897] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10716.686542] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10716.706806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10716.714802] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10716.722794] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.728833] pending: kfree_rcu_monitor [10716.732419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10716.741980] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10716.760815] workqueue events_long: flags=0x0 [10716.769216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.773568] pending: lu_objects_flush [obdclass] [10716.787028] workqueue events_unbound: flags=0x2 [10716.795793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10716.801501] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10716.817574] workqueue events_power_efficient: flags=0x80 [10716.826780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.836966] pending: check_lifetime [10716.845152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.853605] pending: neigh_periodic_work [10716.859219] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.878161] pending: neigh_periodic_work [10716.879568] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.890607] pending: do_cache_clean [10716.895411] workqueue mm_percpu_wq: flags=0x8 [10716.902128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.912010] pending: vmstat_update [10716.918139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.929368] pending: vmstat_update [10716.931900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.937814] pending: vmstat_update [10716.940899] workqueue writeback: flags=0x4a [10716.948748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10716.951774] pending: wb_workfn, wb_workfn [10716.956308] workqueue xprtiod: flags=0xa [10716.960797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10716.970191] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10716.977764] workqueue ipv6_addrconf: flags=0x40008 [10716.984202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10717.002273] pending: addrconf_verify_work [10717.005600] workqueue knbd0-recv: flags=0x1a [10717.011126] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10717.022709] in-flight: 651:recv_work [10717.033853] workqueue ptlrpc_pinger: flags=0x2 [10717.044306] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10717.049274] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10717.054720] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8799s workers=4 idle: 37 31846 28897 [10717.062014] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10706s workers=2 idle: 653 [10724.058025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10724.060963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10724.079938] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10724.095985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10724.102974] RIP: 0010:native_safe_halt+0x2e/0x50 [10724.106133] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10724.117382] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10724.121297] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10724.125058] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10724.128612] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10724.132283] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10724.134970] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10724.138537] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10724.142180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10724.145001] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10724.151136] Call Trace: [10724.152615] [10724.153940] ? show_regs.cold.9+0x22/0x2f [10724.156446] ? watchdog_timer_fn.cold.6+0x62/0x10a [10724.160118] ? __hrtimer_run_queues+0x1c2/0x4d0 [10724.164354] ? watchdog+0x40/0x40 [10724.167746] ? hrtimer_interrupt+0x128/0x2c0 [10724.169949] ? smp_apic_timer_interrupt+0xa6/0x270 [10724.172078] ? apic_timer_interrupt+0xf/0x20 [10724.173377] [10724.174453] ? native_safe_halt+0x2e/0x50 [10724.176953] kvm_wait+0xcd/0xe0 [10724.178254] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10724.183515] do_raw_spin_lock+0x177/0x190 [10724.186514] _raw_spin_lock+0x21/0x30 [10724.188751] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10724.190378] obd_mod_free+0x46/0x140 [ptlrpc] [10724.192561] mdc_commit_open+0x91/0xb0 [mdc] [10724.194290] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10724.198450] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10724.202843] after_reply+0xe2c/0x1780 [ptlrpc] [10724.205641] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10724.208469] ? __next_timer_interrupt+0x160/0x160 [10724.210827] ptlrpcd+0xa82/0xe40 [ptlrpc] [10724.212678] ? do_wait_intr+0xf0/0xf0 [10724.214648] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10724.218503] kthread+0x1d1/0x200 [10724.220637] ? set_kthread_struct+0x70/0x70 [10724.222985] ret_from_fork+0x1f/0x30 [10740.062302] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10740.066033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10740.111276] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10740.121011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10740.128136] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10740.131172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10740.136133] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10740.160951] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10740.175933] RAX: 0000000000006e83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10740.178647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10740.182985] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10740.187370] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10740.192042] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10740.195899] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10740.205558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10740.212208] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10740.216396] Call Trace: [10740.218417] [10740.220963] ? show_regs.cold.9+0x22/0x2f [10740.225386] ? watchdog_timer_fn.cold.6+0x62/0x10a [10740.228923] ? __hrtimer_run_queues+0x1c2/0x4d0 [10740.233221] ? watchdog+0x40/0x40 [10740.237589] ? hrtimer_interrupt+0x128/0x2c0 [10740.239955] ? smp_apic_timer_interrupt+0xa6/0x270 [10740.243722] ? apic_timer_interrupt+0xf/0x20 [10740.248397] [10740.250128] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10740.256433] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10740.259478] do_raw_spin_lock+0x177/0x190 [10740.264974] _raw_spin_lock+0x21/0x30 [10740.270350] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10740.276886] process_one_work+0x2c8/0x700 [10740.279086] worker_thread+0x296/0x6e0 [10740.280408] ? rescuer_thread+0x570/0x570 [10740.282456] kthread+0x1d1/0x200 [10740.285265] ? set_kthread_struct+0x70/0x70 [10740.289243] ret_from_fork+0x1f/0x30 [10744.054012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10744.058611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10744.061824] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10744.094377] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10744.094395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10744.094398] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10744.094414] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10744.094419] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10744.094425] RAX: 0000000000003ddf RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10744.094428] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10744.098268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10744.098345] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10744.098352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10744.098356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10744.098369] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10744.098373] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10744.098378] RAX: 0000000000002a85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10744.098380] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10744.098383] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10744.098385] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10744.098388] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10744.098391] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10744.098394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10744.098396] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10744.098405] Call Trace: [10744.098410] [10744.098416] ? show_regs.cold.9+0x22/0x2f [10744.098425] ? watchdog_timer_fn.cold.6+0x62/0x10a [10744.098431] ? __hrtimer_run_queues+0x1c2/0x4d0 [10744.098435] ? watchdog+0x40/0x40 [10744.098439] ? hrtimer_interrupt+0x128/0x2c0 [10744.098443] ? smp_apic_timer_interrupt+0xa6/0x270 [10744.098451] ? apic_timer_interrupt+0xf/0x20 [10744.098456] [10744.098459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10744.098465] do_raw_spin_lock+0x177/0x190 [10744.098470] _raw_spin_lock+0x21/0x30 [10744.098475] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10744.098762] ? do_raw_spin_unlock+0x75/0x190 [10744.098769] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10744.098965] ? do_wait_intr+0xf0/0xf0 [10744.098974] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10744.099175] mdc_close+0x53b/0xd70 [mdc] [10744.099214] lmv_close+0x18d/0x490 [lmv] [10744.099243] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10744.099352] ll_md_real_close+0xa5/0x2a0 [lustre] [10744.099433] ll_md_close+0x274/0x800 [lustre] [10744.099516] ll_file_release+0x128/0xc50 [lustre] [10744.099596] __fput+0x10d/0x440 [10744.099605] ____fput+0x12/0x20 [10744.099610] task_work_run+0xc6/0x110 [10744.099617] exit_to_usermode_loop+0x1dd/0x1f0 [10744.099624] do_syscall_64+0x3d6/0x440 [10744.099629] entry_SYSCALL_64_after_hwframe+0x49/0xae [10744.099635] RIP: 0033:0x7fa749552cd8 [10744.099642] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10744.099646] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10744.099650] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10744.099652] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10744.099655] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10744.099657] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10744.099659] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10744.415856] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10744.421858] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10744.427119] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10744.429118] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10744.440310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10744.447116] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10744.450837] Call Trace: [10744.455752] [10744.457191] ? show_regs.cold.9+0x22/0x2f [10744.460678] ? watchdog_timer_fn.cold.6+0x62/0x10a [10744.463201] ? __hrtimer_run_queues+0x1c2/0x4d0 [10744.465051] ? watchdog+0x40/0x40 [10744.466961] ? hrtimer_interrupt+0x128/0x2c0 [10744.468982] ? smp_apic_timer_interrupt+0xa6/0x270 [10744.470889] ? apic_timer_interrupt+0xf/0x20 [10744.475414] [10744.476296] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10744.478807] do_raw_spin_lock+0x177/0x190 [10744.483352] _raw_spin_lock+0x21/0x30 [10744.485222] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10744.488092] ? do_raw_spin_unlock+0x75/0x190 [10744.490413] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10744.492659] ? do_wait_intr+0xf0/0xf0 [10744.493786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10744.499279] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10744.500990] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10744.509749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10744.523665] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10744.527829] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10744.529835] mdc_intent_lock+0x353/0x7e0 [mdc] [10744.531612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10744.536640] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10744.540095] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10744.544819] lmv_intent_open+0x338/0x1150 [lmv] [10744.547452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10744.555254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10744.559949] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10744.562992] ll_intent_lock+0xc5/0xaa0 [lustre] [10744.565112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10744.567862] ll_lookup_it+0xe0e/0x2270 [lustre] [10744.570685] ? do_raw_spin_unlock+0x75/0x190 [10744.572365] ? _raw_spin_unlock+0x12/0x30 [10744.573556] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10744.579113] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10744.582784] ll_atomic_open+0x36a/0xd70 [lustre] [10744.587094] lookup_open+0x356/0xb40 [10744.588964] do_last+0x313/0xfc0 [10744.592186] ? nd_jump_root+0xe5/0x160 [10744.597072] ? path_init+0x437/0x520 [10744.599735] path_openat+0xf7/0x500 [10744.602760] ? do_raw_spin_unlock+0x75/0x190 [10744.605889] do_filp_open+0x99/0x140 [10744.609170] ? getname_flags+0x6e/0x330 [10744.615917] ? __check_object_size+0xff/0x256 [10744.618208] ? do_raw_spin_unlock+0x75/0x190 [10744.624233] ? _raw_spin_unlock+0x12/0x30 [10744.629236] do_sys_openat2+0x2b4/0x410 [10744.631331] do_sys_open+0x73/0xa0 [10744.633024] __x64_sys_openat+0x24/0x30 [10744.638890] do_syscall_64+0xc1/0x440 [10744.643460] entry_SYSCALL_64_after_hwframe+0x49/0xae [10744.648587] RIP: 0033:0x7fa749552332 [10744.651483] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10744.667083] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10744.669501] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10744.672979] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10744.676647] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10744.685039] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10744.688196] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10747.360640] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8830s! [10747.365355] Showing busy workqueues and worker pools: [10747.366851] workqueue events: flags=0x0 [10747.367924] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10747.371828] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10747.378104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10747.380801] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10747.384189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.386740] pending: kfree_rcu_monitor [10747.388106] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10747.390117] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10747.393096] workqueue events_long: flags=0x0 [10747.400798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.403152] pending: lu_objects_flush [obdclass] [10747.405214] workqueue events_unbound: flags=0x2 [10747.406585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10747.408517] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10747.411751] workqueue events_power_efficient: flags=0x80 [10747.413871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.417160] pending: check_lifetime [10747.418927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.423146] pending: neigh_periodic_work [10747.425064] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.428197] pending: neigh_periodic_work [10747.431044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.434266] pending: do_cache_clean [10747.436323] workqueue mm_percpu_wq: flags=0x8 [10747.439014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.442129] pending: vmstat_update [10747.443978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.446207] pending: vmstat_update [10747.452188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10747.455918] pending: vmstat_update [10747.459651] workqueue writeback: flags=0x4a [10747.461761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10747.466874] pending: wb_workfn, wb_workfn [10747.469578] workqueue xprtiod: flags=0xa [10747.472089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10747.474431] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10747.481716] workqueue ipv6_addrconf: flags=0x40008 [10747.488521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10747.491239] pending: addrconf_verify_work [10747.492596] workqueue knbd0-recv: flags=0x1a [10747.495219] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10747.499204] in-flight: 651:recv_work [10747.503446] workqueue ptlrpc_pinger: flags=0x2 [10747.506760] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10747.510592] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10747.515242] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8830s workers=4 idle: 37 31846 28897 [10747.521017] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10737s workers=2 idle: 653 [10752.059695] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10752.065562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10752.111897] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10752.126256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10752.132659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10752.135525] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10752.146482] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10752.151833] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10752.161091] RDX: 0000000000001818 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10752.166647] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10752.170428] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10752.176850] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10752.181353] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10752.185001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10752.192067] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10752.196640] Call Trace: [10752.198161] [10752.200023] ? show_regs.cold.9+0x22/0x2f [10752.203111] ? watchdog_timer_fn.cold.6+0x62/0x10a [10752.205526] ? __hrtimer_run_queues+0x1c2/0x4d0 [10752.209849] ? watchdog+0x40/0x40 [10752.212763] ? hrtimer_interrupt+0x128/0x2c0 [10752.215540] ? smp_apic_timer_interrupt+0xa6/0x270 [10752.220200] ? apic_timer_interrupt+0xf/0x20 [10752.222234] [10752.222970] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10752.226862] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10752.231757] do_raw_spin_lock+0x177/0x190 [10752.234746] _raw_spin_lock+0x21/0x30 [10752.236808] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10752.240559] obd_mod_free+0x46/0x140 [ptlrpc] [10752.244470] mdc_commit_open+0x91/0xb0 [mdc] [10752.248192] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10752.252556] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10752.267540] after_reply+0xe2c/0x1780 [ptlrpc] [10752.271452] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10752.276250] ? __next_timer_interrupt+0x160/0x160 [10752.279539] ptlrpcd+0xa82/0xe40 [ptlrpc] [10752.295017] ? do_wait_intr+0xf0/0xf0 [10752.299480] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10752.301565] kthread+0x1d1/0x200 [10752.307195] ? set_kthread_struct+0x70/0x70 [10752.311538] ret_from_fork+0x1f/0x30 [10768.057174] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10768.065667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10768.126306] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10768.138851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10768.151700] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10768.160568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10768.166612] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10768.203188] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10768.210958] RAX: 0000000000003d0f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10768.218283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10768.222268] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10768.230112] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10768.240280] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10768.244881] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10768.254244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10768.262557] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10768.270762] Call Trace: [10768.273404] [10768.274673] ? show_regs.cold.9+0x22/0x2f [10768.277244] ? watchdog_timer_fn.cold.6+0x62/0x10a [10768.280615] ? __hrtimer_run_queues+0x1c2/0x4d0 [10768.285917] ? watchdog+0x40/0x40 [10768.287086] ? hrtimer_interrupt+0x128/0x2c0 [10768.296909] ? smp_apic_timer_interrupt+0xa6/0x270 [10768.303406] ? apic_timer_interrupt+0xf/0x20 [10768.308993] [10768.311305] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10768.315111] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10768.318755] do_raw_spin_lock+0x177/0x190 [10768.324799] _raw_spin_lock+0x21/0x30 [10768.328801] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10768.337370] process_one_work+0x2c8/0x700 [10768.341412] worker_thread+0x296/0x6e0 [10768.346354] ? rescuer_thread+0x570/0x570 [10768.347851] kthread+0x1d1/0x200 [10768.349005] ? set_kthread_struct+0x70/0x70 [10768.359861] ret_from_fork+0x1f/0x30 [10772.054023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10772.056573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10772.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10772.092282] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10772.096162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10772.096251] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10772.096259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10772.096262] RIP: 0010:native_safe_halt+0x2e/0x50 [10772.096275] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10772.096279] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10772.096284] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10772.096286] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10772.096288] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10772.096290] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10772.096293] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10772.096296] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10772.096299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10772.096301] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10772.096309] Call Trace: [10772.096314] [10772.096319] ? show_regs.cold.9+0x22/0x2f [10772.096330] ? watchdog_timer_fn.cold.6+0x62/0x10a [10772.096337] ? __hrtimer_run_queues+0x1c2/0x4d0 [10772.096341] ? watchdog+0x40/0x40 [10772.096346] ? hrtimer_interrupt+0x128/0x2c0 [10772.096350] ? smp_apic_timer_interrupt+0xa6/0x270 [10772.096357] ? apic_timer_interrupt+0xf/0x20 [10772.096362] [10772.096365] ? native_safe_halt+0x2e/0x50 [10772.096370] kvm_wait+0xcd/0xe0 [10772.096380] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10772.096389] do_raw_spin_lock+0x177/0x190 [10772.096394] _raw_spin_lock+0x21/0x30 [10772.096399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10772.096687] ? do_raw_spin_unlock+0x75/0x190 [10772.096694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10772.107818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10772.107829] RIP: 0010:native_safe_halt+0x2e/0x50 [10772.107843] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10772.107847] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10772.107852] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10772.107855] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [10772.107857] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10772.107859] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10772.107861] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10772.107864] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10772.107867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10772.107869] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10772.107878] Call Trace: [10772.107883] [10772.107888] ? show_regs.cold.9+0x22/0x2f [10772.107897] ? watchdog_timer_fn.cold.6+0x62/0x10a [10772.107903] ? __hrtimer_run_queues+0x1c2/0x4d0 [10772.107908] ? watchdog+0x40/0x40 [10772.107912] ? hrtimer_interrupt+0x128/0x2c0 [10772.107916] ? smp_apic_timer_interrupt+0xa6/0x270 [10772.107923] ? apic_timer_interrupt+0xf/0x20 [10772.107928] [10772.107931] ? native_safe_halt+0x2e/0x50 [10772.107935] kvm_wait+0xcd/0xe0 [10772.107942] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10772.107951] do_raw_spin_lock+0x177/0x190 [10772.107955] _raw_spin_lock+0x21/0x30 [10772.107960] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10772.132516] ? do_wait_intr+0xf0/0xf0 [10772.132537] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10772.141533] ? do_raw_spin_unlock+0x75/0x190 [10772.147308] mdc_close+0x53b/0xd70 [mdc] [10772.147359] lmv_close+0x18d/0x490 [lmv] [10772.147390] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10772.147502] ll_md_real_close+0xa5/0x2a0 [lustre] [10772.147585] ll_md_close+0x274/0x800 [lustre] [10772.147667] ll_file_release+0x128/0xc50 [lustre] [10772.147915] __fput+0x10d/0x440 [10772.147930] ____fput+0x12/0x20 [10772.147936] task_work_run+0xc6/0x110 [10772.148011] exit_to_usermode_loop+0x1dd/0x1f0 [10772.148019] do_syscall_64+0x3d6/0x440 [10772.148024] entry_SYSCALL_64_after_hwframe+0x49/0xae [10772.148037] RIP: 0033:0x7fa749552cd8 [10772.148085] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10772.148090] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10772.148096] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10772.148099] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10772.148101] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10772.148103] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10772.148105] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10772.603410] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10772.611415] ? do_wait_intr+0xf0/0xf0 [10772.613646] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10772.618521] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10772.624729] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10772.630921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10772.636413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10772.638560] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10772.646022] mdc_intent_lock+0x353/0x7e0 [mdc] [10772.651390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10772.656314] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10772.662101] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10772.671394] lmv_intent_open+0x338/0x1150 [lmv] [10772.679120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10772.688976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10772.692897] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10772.695906] ll_intent_lock+0xc5/0xaa0 [lustre] [10772.701193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10772.704106] ll_lookup_it+0xe0e/0x2270 [lustre] [10772.708913] ? do_raw_spin_unlock+0x75/0x190 [10772.710114] ? _raw_spin_unlock+0x12/0x30 [10772.712367] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10772.716991] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10772.719796] ll_atomic_open+0x36a/0xd70 [lustre] [10772.722424] lookup_open+0x356/0xb40 [10772.727386] do_last+0x313/0xfc0 [10772.729471] ? nd_jump_root+0xe5/0x160 [10772.730770] ? path_init+0x437/0x520 [10772.731697] path_openat+0xf7/0x500 [10772.738210] ? do_raw_spin_unlock+0x75/0x190 [10772.740204] do_filp_open+0x99/0x140 [10772.741525] ? getname_flags+0x6e/0x330 [10772.742972] ? __check_object_size+0xff/0x256 [10772.746720] ? do_raw_spin_unlock+0x75/0x190 [10772.752408] ? _raw_spin_unlock+0x12/0x30 [10772.753508] do_sys_openat2+0x2b4/0x410 [10772.755273] do_sys_open+0x73/0xa0 [10772.756700] __x64_sys_openat+0x24/0x30 [10772.758651] do_syscall_64+0xc1/0x440 [10772.760127] entry_SYSCALL_64_after_hwframe+0x49/0xae [10772.762659] RIP: 0033:0x7fa749552332 [10772.764444] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10772.773869] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10772.776499] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10772.779291] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10772.782150] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10772.784916] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10772.788135] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10777.572605] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8860s! [10777.583336] Showing busy workqueues and worker pools: [10777.588368] workqueue events: flags=0x0 [10777.593059] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10777.600758] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10777.611829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10777.618988] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10777.630725] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.640077] pending: kfree_rcu_monitor [10777.644229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10777.650967] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10777.658680] workqueue events_long: flags=0x0 [10777.661782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.669958] pending: lu_objects_flush [obdclass] [10777.675139] workqueue events_unbound: flags=0x2 [10777.676783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10777.684335] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10777.688037] workqueue events_power_efficient: flags=0x80 [10777.690218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.693073] pending: check_lifetime [10777.694652] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.697332] pending: neigh_periodic_work [10777.699508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.702271] pending: neigh_periodic_work [10777.704103] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.706963] pending: do_cache_clean [10777.708594] workqueue mm_percpu_wq: flags=0x8 [10777.710466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.713292] pending: vmstat_update [10777.714944] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.717378] pending: vmstat_update [10777.718807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.721734] pending: vmstat_update [10777.723328] workqueue writeback: flags=0x4a [10777.725115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10777.727991] pending: wb_workfn, wb_workfn [10777.729899] workqueue xprtiod: flags=0xa [10777.731513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10777.734056] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10777.737968] workqueue ipv6_addrconf: flags=0x40008 [10777.740170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10777.742836] pending: addrconf_verify_work [10777.744786] workqueue knbd0-recv: flags=0x1a [10777.746678] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10777.749991] in-flight: 651:recv_work [10777.751815] workqueue ptlrpc_pinger: flags=0x2 [10777.753909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10777.756746] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10777.759626] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8860s workers=4 idle: 37 31846 28897 [10777.763259] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10767s workers=2 idle: 653 [10780.058046] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10780.061350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10780.080462] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10780.085190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10780.090574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10780.094291] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10780.102633] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10780.106149] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10780.109195] RDX: 000000000000025f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10780.113194] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10780.116524] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10780.119604] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10780.122705] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10780.126508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10780.128727] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10780.131874] Call Trace: [10780.132981] [10780.134125] ? show_regs.cold.9+0x22/0x2f [10780.135817] ? watchdog_timer_fn.cold.6+0x62/0x10a [10780.137862] ? __hrtimer_run_queues+0x1c2/0x4d0 [10780.140046] ? watchdog+0x40/0x40 [10780.141334] ? hrtimer_interrupt+0x128/0x2c0 [10780.143435] ? smp_apic_timer_interrupt+0xa6/0x270 [10780.145394] ? apic_timer_interrupt+0xf/0x20 [10780.147466] [10780.148435] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10780.150519] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10780.152686] do_raw_spin_lock+0x177/0x190 [10780.154659] _raw_spin_lock+0x21/0x30 [10780.156247] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10780.158591] obd_mod_free+0x46/0x140 [ptlrpc] [10780.165394] mdc_commit_open+0x91/0xb0 [mdc] [10780.166992] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10780.174242] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10780.182216] after_reply+0xe2c/0x1780 [ptlrpc] [10780.186692] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10780.190066] ? __next_timer_interrupt+0x160/0x160 [10780.193558] ptlrpcd+0xa82/0xe40 [ptlrpc] [10780.196043] ? do_wait_intr+0xf0/0xf0 [10780.198814] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10780.202753] kthread+0x1d1/0x200 [10780.207932] ? set_kthread_struct+0x70/0x70 [10780.214105] ret_from_fork+0x1f/0x30 [10793.878423] rcu: INFO: rcu_sched self-detected stall on CPU [10793.886122] rcu: 2-...!: (4987427 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [10793.908976] (t=8880189 jiffies g=258701 q=4118) [10793.915284] rcu: rcu_sched kthread starved for 8876984 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10793.936071] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10793.945511] rcu: RCU grace-period kthread stack dump: [10793.951102] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10793.964947] Call Trace: [10793.967714] __schedule+0x351/0xcb0 [10793.971420] schedule+0xc0/0x180 [10793.977606] schedule_timeout+0xb4/0x190 [10793.981120] ? do_raw_spin_unlock+0x75/0x190 [10793.989958] ? __next_timer_interrupt+0x160/0x160 [10793.995782] ? prepare_to_swait_event+0xf3/0x140 [10794.002436] rcu_gp_kthread+0x8dd/0x15b0 [10794.006320] ? force_qs_rnp+0x2c0/0x2c0 [10794.010417] kthread+0x1d1/0x200 [10794.015951] ? set_kthread_struct+0x70/0x70 [10794.023058] ret_from_fork+0x1f/0x30 [10794.026004] rcu: Stack dump where RCU GP kthread last ran: [10794.038410] Sending NMI from CPU 2 to CPUs 0: [10794.045867] NMI backtrace for cpu 0 [10794.045874] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10794.045876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10794.045878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10794.045879] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10794.045881] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [10794.045884] RAX: 0000000000006f89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10794.045886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10794.045887] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10794.045888] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10794.045889] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10794.045890] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10794.045892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10794.045893] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10794.045894] Call Trace: [10794.045895] [10794.045900] ? show_regs.cold.9+0x22/0x2f [10794.045901] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10794.045902] ? nmi_cpu_backtrace_handler+0x15/0x40 [10794.045903] ? nmi_handle+0xbd/0x210 [10794.045904] ? default_do_nmi+0x4b/0x1b0 [10794.045905] ? do_nmi+0x280/0x380 [10794.045906] ? end_repeat_nmi+0x16/0x1f [10794.045907] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10794.045908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10794.045910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10794.045911] [10794.045912] do_raw_spin_lock+0x177/0x190 [10794.045913] _raw_spin_lock+0x21/0x30 [10794.045914] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10794.045915] ? do_raw_spin_unlock+0x75/0x190 [10794.045917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10794.045918] ? do_wait_intr+0xf0/0xf0 [10794.045919] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10794.045920] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10794.045921] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10794.045922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.045924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10794.045925] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10794.045926] mdc_intent_lock+0x353/0x7e0 [mdc] [10794.045927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.045929] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10794.045930] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10794.045931] lmv_intent_open+0x338/0x1150 [lmv] [10794.045933] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.045934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.045935] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10794.045936] ll_intent_lock+0xc5/0xaa0 [lustre] [10794.045937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.045944] ll_lookup_it+0xe0e/0x2270 [lustre] [10794.045945] ? do_raw_spin_unlock+0x75/0x190 [10794.045946] ? _raw_spin_unlock+0x12/0x30 [10794.045948] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10794.045949] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10794.045951] ll_atomic_open+0x36a/0xd70 [lustre] [10794.045952] lookup_open+0x356/0xb40 [10794.045953] do_last+0x313/0xfc0 [10794.045954] ? nd_jump_root+0xe5/0x160 [10794.045955] ? path_init+0x437/0x520 [10794.045957] path_openat+0xf7/0x500 [10794.045958] ? do_raw_spin_unlock+0x75/0x190 [10794.045959] do_filp_open+0x99/0x140 [10794.045960] ? getname_flags+0x6e/0x330 [10794.045961] ? __check_object_size+0xff/0x256 [10794.045962] ? do_raw_spin_unlock+0x75/0x190 [10794.045963] ? _raw_spin_unlock+0x12/0x30 [10794.045964] do_sys_openat2+0x2b4/0x410 [10794.045965] do_sys_open+0x73/0xa0 [10794.045966] __x64_sys_openat+0x24/0x30 [10794.045966] do_syscall_64+0xc1/0x440 [10794.045967] entry_SYSCALL_64_after_hwframe+0x49/0xae [10794.045968] RIP: 0033:0x7fa749552332 [10794.045969] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10794.045972] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10794.045974] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10794.045975] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10794.045976] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10794.045977] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10794.045978] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10794.050262] NMI backtrace for cpu 2 [10794.565445] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10794.574659] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10794.586444] Call Trace: [10794.597823] [10794.598622] ? dump_stack+0xbb/0x10e [10794.599926] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10794.623398] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10794.627299] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10794.633683] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10794.644562] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10794.646253] ? rcu_dump_cpu_stacks+0x11b/0x171 [10794.654671] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10794.659925] ? update_process_times+0x6d/0xc0 [10794.662600] ? tick_sched_handle.isra.18+0x27/0x90 [10794.667728] ? tick_sched_timer+0x53/0xb0 [10794.669478] ? __hrtimer_run_queues+0x1c2/0x4d0 [10794.675889] ? tick_sched_handle.isra.18+0x90/0x90 [10794.677660] ? hrtimer_interrupt+0x128/0x2c0 [10794.687896] ? smp_apic_timer_interrupt+0xa6/0x270 [10794.692525] ? apic_timer_interrupt+0xf/0x20 [10794.695324] [10794.699398] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10794.703435] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10794.709513] ? do_raw_spin_lock+0x177/0x190 [10794.712878] ? _raw_spin_lock+0x21/0x30 [10794.715373] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10794.721415] ? obd_mod_free+0x46/0x140 [ptlrpc] [10794.727460] ? mdc_commit_open+0x91/0xb0 [mdc] [10794.732334] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10794.735718] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10794.740723] ? after_reply+0xe2c/0x1780 [ptlrpc] [10794.747113] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10794.760353] ? __next_timer_interrupt+0x160/0x160 [10794.765275] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10794.769302] ? do_wait_intr+0xf0/0xf0 [10794.772100] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10794.778268] ? kthread+0x1d1/0x200 [10794.779500] ? set_kthread_struct+0x70/0x70 [10794.792577] ? ret_from_fork+0x1f/0x30 [10796.057027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10796.059765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10796.078709] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10796.083169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10796.086895] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10796.089631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10796.092191] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10796.099499] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10796.102850] RAX: 0000000000002398 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10796.106513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10796.110147] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10796.113588] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10796.116978] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10796.119917] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10796.123127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10796.125467] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10796.128477] Call Trace: [10796.129719] [10796.130571] ? show_regs.cold.9+0x22/0x2f [10796.132401] ? watchdog_timer_fn.cold.6+0x62/0x10a [10796.134424] ? __hrtimer_run_queues+0x1c2/0x4d0 [10796.136404] ? watchdog+0x40/0x40 [10796.137691] ? hrtimer_interrupt+0x128/0x2c0 [10796.139167] ? smp_apic_timer_interrupt+0xa6/0x270 [10796.141176] ? apic_timer_interrupt+0xf/0x20 [10796.142765] [10796.143687] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10796.145843] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10796.148262] do_raw_spin_lock+0x177/0x190 [10796.149933] _raw_spin_lock+0x21/0x30 [10796.151443] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10796.153940] process_one_work+0x2c8/0x700 [10796.155485] worker_thread+0x296/0x6e0 [10796.157078] ? rescuer_thread+0x570/0x570 [10796.158898] kthread+0x1d1/0x200 [10796.160216] ? set_kthread_struct+0x70/0x70 [10796.161973] ret_from_fork+0x1f/0x30 [10800.059422] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [10800.069400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10800.111466] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10800.142706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10800.152799] RIP: 0010:native_safe_halt+0x2e/0x50 [10800.161290] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10800.187416] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10800.202258] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10800.212846] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10800.222146] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10800.234506] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10800.244748] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10800.258826] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10800.274365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10800.283567] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10800.292793] Call Trace: [10800.294660] [10800.296540] ? show_regs.cold.9+0x22/0x2f [10800.306755] ? watchdog_timer_fn.cold.6+0x62/0x10a [10800.314852] ? __hrtimer_run_queues+0x1c2/0x4d0 [10800.316462] ? watchdog+0x40/0x40 [10800.318900] ? hrtimer_interrupt+0x128/0x2c0 [10800.325020] ? smp_apic_timer_interrupt+0xa6/0x270 [10800.331257] ? apic_timer_interrupt+0xf/0x20 [10800.337127] [10800.341700] ? native_safe_halt+0x2e/0x50 [10800.346145] kvm_wait+0xcd/0xe0 [10800.348615] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10800.351768] do_raw_spin_lock+0x177/0x190 [10800.358584] _raw_spin_lock+0x21/0x30 [10800.364655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10800.370107] ? do_raw_spin_unlock+0x75/0x190 [10800.371117] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10800.375495] ? do_wait_intr+0xf0/0xf0 [10800.376559] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10800.380509] mdc_close+0x53b/0xd70 [mdc] [10800.386121] lmv_close+0x18d/0x490 [lmv] [10800.390851] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10800.398452] ll_md_real_close+0xa5/0x2a0 [lustre] [10800.401754] ll_md_close+0x274/0x800 [lustre] [10800.406319] ll_file_release+0x128/0xc50 [lustre] [10800.409380] __fput+0x10d/0x440 [10800.411212] ____fput+0x12/0x20 [10800.412437] task_work_run+0xc6/0x110 [10800.414206] exit_to_usermode_loop+0x1dd/0x1f0 [10800.416379] do_syscall_64+0x3d6/0x440 [10800.418659] entry_SYSCALL_64_after_hwframe+0x49/0xae [10800.420646] RIP: 0033:0x7fa749552cd8 [10800.422976] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10800.435827] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10800.440953] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10800.444707] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10800.451249] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10800.455344] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10800.458791] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10807.777050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8890s! [10807.781712] Showing busy workqueues and worker pools: [10807.784110] workqueue events: flags=0x0 [10807.786629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10807.794038] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10807.803916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10807.809935] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10807.820579] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.826451] pending: kfree_rcu_monitor [10807.832810] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10807.838209] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10807.848152] workqueue events_long: flags=0x0 [10807.853177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.858128] pending: lu_objects_flush [obdclass] [10807.863991] workqueue events_unbound: flags=0x2 [10807.868865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10807.874829] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10807.885616] workqueue events_power_efficient: flags=0x80 [10807.887570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.902152] pending: check_lifetime [10807.904743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.908672] pending: neigh_periodic_work [10807.913058] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.921951] pending: neigh_periodic_work [10807.925802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.931039] pending: do_cache_clean [10807.934216] workqueue mm_percpu_wq: flags=0x8 [10807.940709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.949049] pending: vmstat_update [10807.954827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.967543] pending: vmstat_update [10807.971267] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.978064] pending: vmstat_update [10807.982839] workqueue writeback: flags=0x4a [10807.990721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10807.998400] pending: wb_workfn, wb_workfn [10808.004246] workqueue xprtiod: flags=0xa [10808.007499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10808.010821] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10808.014788] workqueue ipv6_addrconf: flags=0x40008 [10808.017035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10808.021451] pending: addrconf_verify_work [10808.023831] workqueue knbd0-recv: flags=0x1a [10808.026461] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10808.028919] in-flight: 651:recv_work [10808.030704] workqueue ptlrpc_pinger: flags=0x2 [10808.032676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10808.035759] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10808.038419] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8890s workers=4 idle: 37 31846 28897 [10808.041858] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10797s workers=2 idle: 653 [10820.058416] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [10820.061956] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10820.104665] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10820.121635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10820.134703] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10820.136705] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10820.156290] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10820.164477] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10820.173568] RDX: 000000000000671c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10820.181324] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10820.189187] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10820.195463] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10820.205256] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10820.217279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10820.225805] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10820.234462] Call Trace: [10820.237900] [10820.241846] ? show_regs.cold.9+0x22/0x2f [10820.249878] ? watchdog_timer_fn.cold.6+0x62/0x10a [10820.252972] ? __hrtimer_run_queues+0x1c2/0x4d0 [10820.256889] ? watchdog+0x40/0x40 [10820.260352] ? hrtimer_interrupt+0x128/0x2c0 [10820.267972] ? smp_apic_timer_interrupt+0xa6/0x270 [10820.274538] ? apic_timer_interrupt+0xf/0x20 [10820.279355] [10820.281356] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10820.286827] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10820.290855] do_raw_spin_lock+0x177/0x190 [10820.294974] _raw_spin_lock+0x21/0x30 [10820.300964] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10820.306396] obd_mod_free+0x46/0x140 [ptlrpc] [10820.313029] mdc_commit_open+0x91/0xb0 [mdc] [10820.316530] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10820.324221] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10820.333071] after_reply+0xe2c/0x1780 [ptlrpc] [10820.344079] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10820.350108] ? __next_timer_interrupt+0x160/0x160 [10820.354249] ptlrpcd+0xa82/0xe40 [ptlrpc] [10820.358027] ? do_wait_intr+0xf0/0xf0 [10820.362755] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10820.372010] kthread+0x1d1/0x200 [10820.373850] ? set_kthread_struct+0x70/0x70 [10820.377929] ret_from_fork+0x1f/0x30 [10824.058300] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10824.063873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10824.087238] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10824.091247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10824.095691] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10824.099247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10824.102583] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10824.112229] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10824.116304] RAX: 0000000000007d9d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10824.119858] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10824.123478] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10824.126950] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10824.130982] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10824.134155] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10824.137898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10824.140716] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10824.143977] Call Trace: [10824.145169] [10824.146523] ? show_regs.cold.9+0x22/0x2f [10824.148778] ? watchdog_timer_fn.cold.6+0x62/0x10a [10824.151255] ? __hrtimer_run_queues+0x1c2/0x4d0 [10824.153081] ? watchdog+0x40/0x40 [10824.154825] ? hrtimer_interrupt+0x128/0x2c0 [10824.156576] ? smp_apic_timer_interrupt+0xa6/0x270 [10824.159333] ? apic_timer_interrupt+0xf/0x20 [10824.161232] [10824.162496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10824.165238] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10824.167902] do_raw_spin_lock+0x177/0x190 [10824.169884] _raw_spin_lock+0x21/0x30 [10824.171829] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10824.174940] process_one_work+0x2c8/0x700 [10824.176761] worker_thread+0x296/0x6e0 [10824.180866] ? rescuer_thread+0x570/0x570 [10824.184017] kthread+0x1d1/0x200 [10824.185846] ? set_kthread_struct+0x70/0x70 [10824.187835] ret_from_fork+0x1f/0x30 [10828.059023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10828.062385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10828.111743] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10828.121227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10828.135432] RIP: 0010:native_safe_halt+0x2e/0x50 [10828.142646] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10828.170230] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10828.179503] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10828.186106] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10828.191304] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10828.198015] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10828.209757] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10828.217407] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10828.224335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10828.231259] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10828.240801] Call Trace: [10828.243946] [10828.248892] ? show_regs.cold.9+0x22/0x2f [10828.252715] ? watchdog_timer_fn.cold.6+0x62/0x10a [10828.256606] ? __hrtimer_run_queues+0x1c2/0x4d0 [10828.262330] ? watchdog+0x40/0x40 [10828.264724] ? hrtimer_interrupt+0x128/0x2c0 [10828.270986] ? smp_apic_timer_interrupt+0xa6/0x270 [10828.276849] ? apic_timer_interrupt+0xf/0x20 [10828.282191] [10828.283089] ? native_safe_halt+0x2e/0x50 [10828.290211] kvm_wait+0xcd/0xe0 [10828.293928] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10828.300529] do_raw_spin_lock+0x177/0x190 [10828.306728] _raw_spin_lock+0x21/0x30 [10828.311762] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10828.322303] ? do_raw_spin_unlock+0x75/0x190 [10828.326867] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10828.330818] ? do_wait_intr+0xf0/0xf0 [10828.332600] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10828.336272] mdc_close+0x53b/0xd70 [mdc] [10828.342951] lmv_close+0x18d/0x490 [lmv] [10828.344721] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10828.352198] ll_md_real_close+0xa5/0x2a0 [lustre] [10828.353972] ll_md_close+0x274/0x800 [lustre] [10828.357639] ll_file_release+0x128/0xc50 [lustre] [10828.360772] __fput+0x10d/0x440 [10828.362324] ____fput+0x12/0x20 [10828.364320] task_work_run+0xc6/0x110 [10828.367690] exit_to_usermode_loop+0x1dd/0x1f0 [10828.371362] do_syscall_64+0x3d6/0x440 [10828.373583] entry_SYSCALL_64_after_hwframe+0x49/0xae [10828.378331] RIP: 0033:0x7fa749552cd8 [10828.381625] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10828.398844] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10828.417007] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10828.425356] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10828.436032] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10828.446475] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10828.453839] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10832.054097] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [10832.058267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10832.095107] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10832.109473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10832.118187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10832.123673] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10832.138378] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10832.143780] RAX: 00000000000025a7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10832.151156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10832.157499] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10832.166259] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10832.172600] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10832.181069] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10832.190137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10832.194541] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10832.202581] Call Trace: [10832.203458] [10832.204202] ? show_regs.cold.9+0x22/0x2f [10832.210072] ? watchdog_timer_fn.cold.6+0x62/0x10a [10832.216153] ? __hrtimer_run_queues+0x1c2/0x4d0 [10832.222187] ? watchdog+0x40/0x40 [10832.223378] ? hrtimer_interrupt+0x128/0x2c0 [10832.228778] ? smp_apic_timer_interrupt+0xa6/0x270 [10832.234197] ? apic_timer_interrupt+0xf/0x20 [10832.244115] [10832.244919] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10832.251148] do_raw_spin_lock+0x177/0x190 [10832.255693] _raw_spin_lock+0x21/0x30 [10832.261259] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10832.263397] ? do_raw_spin_unlock+0x75/0x190 [10832.265131] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10832.267190] ? do_wait_intr+0xf0/0xf0 [10832.268950] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10832.271051] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10832.273094] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10832.275377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10832.277471] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10832.279603] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10832.281383] mdc_intent_lock+0x353/0x7e0 [mdc] [10832.283085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10832.285150] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10832.287478] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10832.289089] lmv_intent_open+0x338/0x1150 [lmv] [10832.290304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10832.291994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10832.294388] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10832.296365] ll_intent_lock+0xc5/0xaa0 [lustre] [10832.298193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10832.300048] ll_lookup_it+0xe0e/0x2270 [lustre] [10832.301760] ? do_raw_spin_unlock+0x75/0x190 [10832.303342] ? _raw_spin_unlock+0x12/0x30 [10832.304868] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10832.306791] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10832.308752] ll_atomic_open+0x36a/0xd70 [lustre] [10832.310548] lookup_open+0x356/0xb40 [10832.311874] do_last+0x313/0xfc0 [10832.313102] ? nd_jump_root+0xe5/0x160 [10832.314496] ? path_init+0x437/0x520 [10832.316127] path_openat+0xf7/0x500 [10832.317454] ? do_raw_spin_unlock+0x75/0x190 [10832.319319] do_filp_open+0x99/0x140 [10832.320530] ? getname_flags+0x6e/0x330 [10832.321507] ? __check_object_size+0xff/0x256 [10832.322630] ? do_raw_spin_unlock+0x75/0x190 [10832.323875] ? _raw_spin_unlock+0x12/0x30 [10832.326945] do_sys_openat2+0x2b4/0x410 [10832.329296] do_sys_open+0x73/0xa0 [10832.330743] __x64_sys_openat+0x24/0x30 [10832.332326] do_syscall_64+0xc1/0x440 [10832.333797] entry_SYSCALL_64_after_hwframe+0x49/0xae [10832.336157] RIP: 0033:0x7fa749552332 [10832.337663] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10832.347690] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10832.350864] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10832.357308] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10832.361060] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10832.366576] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10832.370548] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10838.498101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8921s! [10838.504647] Showing busy workqueues and worker pools: [10838.508076] workqueue events: flags=0x0 [10838.510476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10838.519180] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10838.530391] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10838.539687] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10838.545018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.548995] pending: kfree_rcu_monitor [10838.550566] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10838.553538] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10838.558040] workqueue events_long: flags=0x0 [10838.559956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.562620] pending: lu_objects_flush [obdclass] [10838.565218] workqueue events_unbound: flags=0x2 [10838.567629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10838.570418] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10838.574656] workqueue events_power_efficient: flags=0x80 [10838.577426] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.585223] pending: check_lifetime [10838.589681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.595344] pending: neigh_periodic_work [10838.600255] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.606074] pending: neigh_periodic_work [10838.608413] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.612285] pending: do_cache_clean [10838.614730] workqueue mm_percpu_wq: flags=0x8 [10838.618296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.622451] pending: vmstat_update [10838.624490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.627979] pending: vmstat_update [10838.629969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.634030] pending: vmstat_update [10838.636145] workqueue writeback: flags=0x4a [10838.638345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10838.641790] pending: wb_workfn, wb_workfn [10838.649183] workqueue xprtiod: flags=0xa [10838.651016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10838.661591] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10838.670095] workqueue ipv6_addrconf: flags=0x40008 [10838.678016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10838.684639] pending: addrconf_verify_work [10838.687348] workqueue knbd0-recv: flags=0x1a [10838.692082] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10838.696272] in-flight: 651:recv_work [10838.703586] workqueue ptlrpc_pinger: flags=0x2 [10838.705160] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10838.708376] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10838.716098] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8921s workers=4 idle: 37 31846 28897 [10838.723150] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10828s workers=2 idle: 653 [10848.058967] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10848.064481] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10848.100405] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10848.107269] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10848.112978] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10848.118000] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10848.132811] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10848.142483] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10848.149146] RDX: 0000000000002cad RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10848.151605] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10848.180859] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10848.185667] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10848.198813] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10848.207242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10848.212241] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10848.219070] Call Trace: [10848.220280] [10848.221349] ? show_regs.cold.9+0x22/0x2f [10848.226856] ? watchdog_timer_fn.cold.6+0x62/0x10a [10848.231546] ? __hrtimer_run_queues+0x1c2/0x4d0 [10848.236349] ? watchdog+0x40/0x40 [10848.238351] ? hrtimer_interrupt+0x128/0x2c0 [10848.241324] ? smp_apic_timer_interrupt+0xa6/0x270 [10848.243408] ? apic_timer_interrupt+0xf/0x20 [10848.245918] [10848.246595] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10848.249026] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10848.251194] do_raw_spin_lock+0x177/0x190 [10848.252832] _raw_spin_lock+0x21/0x30 [10848.254222] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10848.256344] obd_mod_free+0x46/0x140 [ptlrpc] [10848.258383] mdc_commit_open+0x91/0xb0 [mdc] [10848.260329] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10848.262956] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10848.265579] after_reply+0xe2c/0x1780 [ptlrpc] [10848.267958] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10848.270589] ? __next_timer_interrupt+0x160/0x160 [10848.272283] ptlrpcd+0xa82/0xe40 [ptlrpc] [10848.274276] ? do_wait_intr+0xf0/0xf0 [10848.275823] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10848.278147] kthread+0x1d1/0x200 [10848.279429] ? set_kthread_struct+0x70/0x70 [10848.281144] ret_from_fork+0x1f/0x30 [10852.057025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10852.060075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10852.109996] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10852.122415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10852.132245] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10852.137342] RIP: 0010:native_safe_halt+0x2e/0x50 [10852.141002] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10852.158940] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10852.164147] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10852.166447] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10852.174527] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10852.178987] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10852.185523] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10852.194837] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10852.200199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10852.203490] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10852.211370] Call Trace: [10852.214165] [10852.217086] ? show_regs.cold.9+0x22/0x2f [10852.222665] ? watchdog_timer_fn.cold.6+0x62/0x10a [10852.227447] ? __hrtimer_run_queues+0x1c2/0x4d0 [10852.231608] ? watchdog+0x40/0x40 [10852.236206] ? hrtimer_interrupt+0x128/0x2c0 [10852.241096] ? smp_apic_timer_interrupt+0xa6/0x270 [10852.245380] ? apic_timer_interrupt+0xf/0x20 [10852.259442] [10852.260225] ? native_safe_halt+0x2e/0x50 [10852.262099] kvm_wait+0xcd/0xe0 [10852.263251] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10852.265463] do_raw_spin_lock+0x177/0x190 [10852.267361] _raw_spin_lock+0x21/0x30 [10852.269281] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10852.273086] process_one_work+0x2c8/0x700 [10852.275775] worker_thread+0x296/0x6e0 [10852.278934] ? rescuer_thread+0x570/0x570 [10852.281035] kthread+0x1d1/0x200 [10852.286432] ? set_kthread_struct+0x70/0x70 [10852.288339] ret_from_fork+0x1f/0x30 [10856.063045] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10856.068789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10856.122626] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10856.128814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10856.136698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10856.142821] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10856.166251] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10856.174046] RAX: 0000000000000342 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10856.187143] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10856.194734] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10856.200941] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10856.205785] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10856.222656] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10856.230578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10856.246121] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10856.258500] Call Trace: [10856.259322] [10856.259990] ? show_regs.cold.9+0x22/0x2f [10856.272410] ? watchdog_timer_fn.cold.6+0x62/0x10a [10856.278220] ? __hrtimer_run_queues+0x1c2/0x4d0 [10856.285674] ? watchdog+0x40/0x40 [10856.294198] ? hrtimer_interrupt+0x128/0x2c0 [10856.300275] ? smp_apic_timer_interrupt+0xa6/0x270 [10856.303207] ? apic_timer_interrupt+0xf/0x20 [10856.306887] [10856.308194] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10856.315034] do_raw_spin_lock+0x177/0x190 [10856.317311] _raw_spin_lock+0x21/0x30 [10856.319351] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10856.325862] ? do_raw_spin_unlock+0x75/0x190 [10856.329762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10856.335725] ? do_wait_intr+0xf0/0xf0 [10856.338294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10856.341723] mdc_close+0x53b/0xd70 [mdc] [10856.350061] lmv_close+0x18d/0x490 [lmv] [10856.354905] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10856.365461] ll_md_real_close+0xa5/0x2a0 [lustre] [10856.368853] ll_md_close+0x274/0x800 [lustre] [10856.378482] ll_file_release+0x128/0xc50 [lustre] [10856.383222] __fput+0x10d/0x440 [10856.387920] ____fput+0x12/0x20 [10856.391373] task_work_run+0xc6/0x110 [10856.393543] exit_to_usermode_loop+0x1dd/0x1f0 [10856.395935] do_syscall_64+0x3d6/0x440 [10856.398259] entry_SYSCALL_64_after_hwframe+0x49/0xae [10856.401109] RIP: 0033:0x7fa749552cd8 [10856.402653] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10856.411843] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10856.415439] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10856.418281] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10856.420802] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10856.423957] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10856.426845] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10864.059854] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10864.067754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10864.108901] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10864.119871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10864.128789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10864.135530] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10864.148186] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10864.152454] RAX: 0000000000001084 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10864.156301] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10864.159759] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10864.162513] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10864.165559] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10864.169535] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10864.173442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10864.175956] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10864.179727] Call Trace: [10864.181558] [10864.182476] ? show_regs.cold.9+0x22/0x2f [10864.183906] ? watchdog_timer_fn.cold.6+0x62/0x10a [10864.187544] ? __hrtimer_run_queues+0x1c2/0x4d0 [10864.195148] ? watchdog+0x40/0x40 [10864.196768] ? hrtimer_interrupt+0x128/0x2c0 [10864.202729] ? smp_apic_timer_interrupt+0xa6/0x270 [10864.204949] ? apic_timer_interrupt+0xf/0x20 [10864.207271] [10864.208210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10864.211003] do_raw_spin_lock+0x177/0x190 [10864.213445] _raw_spin_lock+0x21/0x30 [10864.216316] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10864.222888] ? do_raw_spin_unlock+0x75/0x190 [10864.225810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10864.231776] ? do_wait_intr+0xf0/0xf0 [10864.237704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10864.244285] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10864.250726] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10864.255392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10864.263802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10864.270419] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10864.273742] mdc_intent_lock+0x353/0x7e0 [mdc] [10864.278723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10864.283569] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10864.287986] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10864.293904] lmv_intent_open+0x338/0x1150 [lmv] [10864.297486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10864.303151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10864.306218] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10864.308852] ll_intent_lock+0xc5/0xaa0 [lustre] [10864.311602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10864.318210] ll_lookup_it+0xe0e/0x2270 [lustre] [10864.321316] ? do_raw_spin_unlock+0x75/0x190 [10864.325314] ? _raw_spin_unlock+0x12/0x30 [10864.334360] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10864.337977] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10864.346440] ll_atomic_open+0x36a/0xd70 [lustre] [10864.348712] lookup_open+0x356/0xb40 [10864.350999] do_last+0x313/0xfc0 [10864.354234] ? nd_jump_root+0xe5/0x160 [10864.363392] ? path_init+0x437/0x520 [10864.368400] path_openat+0xf7/0x500 [10864.369582] ? do_raw_spin_unlock+0x75/0x190 [10864.373740] do_filp_open+0x99/0x140 [10864.375414] ? getname_flags+0x6e/0x330 [10864.383053] ? __check_object_size+0xff/0x256 [10864.387629] ? do_raw_spin_unlock+0x75/0x190 [10864.392309] ? _raw_spin_unlock+0x12/0x30 [10864.397154] do_sys_openat2+0x2b4/0x410 [10864.399284] do_sys_open+0x73/0xa0 [10864.403496] __x64_sys_openat+0x24/0x30 [10864.405058] do_syscall_64+0xc1/0x440 [10864.406606] entry_SYSCALL_64_after_hwframe+0x49/0xae [10864.412538] RIP: 0033:0x7fa749552332 [10864.418684] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10864.439582] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10864.451642] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10864.457499] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10864.470291] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10864.475077] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10864.483271] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10869.223029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8952s! [10869.225849] Showing busy workqueues and worker pools: [10869.227739] workqueue events: flags=0x0 [10869.229169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10869.231833] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10869.235965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10869.238129] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10869.241413] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.243751] pending: kfree_rcu_monitor [10869.245387] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10869.247579] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10869.251016] workqueue events_long: flags=0x0 [10869.252543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.255194] pending: lu_objects_flush [obdclass] [10869.257504] workqueue events_unbound: flags=0x2 [10869.259180] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10869.261725] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10869.266510] workqueue events_power_efficient: flags=0x80 [10869.268987] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.272345] pending: check_lifetime [10869.274230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.277556] pending: neigh_periodic_work [10869.279967] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.282944] pending: neigh_periodic_work [10869.285010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.289013] pending: do_cache_clean [10869.291074] workqueue mm_percpu_wq: flags=0x8 [10869.293041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.300945] pending: vmstat_update [10869.305929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.309038] pending: vmstat_update [10869.310755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10869.318851] pending: vmstat_update [10869.320125] workqueue writeback: flags=0x4a [10869.324378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10869.331057] pending: wb_workfn, wb_workfn [10869.334732] workqueue xprtiod: flags=0xa [10869.339047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10869.342163] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10869.356408] workqueue ipv6_addrconf: flags=0x40008 [10869.360140] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10869.368061] pending: addrconf_verify_work [10869.369790] workqueue knbd0-recv: flags=0x1a [10869.371247] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10869.378511] in-flight: 651:recv_work [10869.382594] workqueue ptlrpc_pinger: flags=0x2 [10869.387395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10869.392353] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10869.398439] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8952s workers=4 idle: 37 31846 28897 [10869.407573] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10859s workers=2 idle: 653 [10876.060854] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10876.071592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10876.108074] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10876.117319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10876.123317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10876.127493] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10876.145012] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10876.153287] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10876.161085] RDX: 0000000000003fe0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10876.166555] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10876.171850] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10876.176094] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10876.181216] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10876.189262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10876.192989] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10876.197424] Call Trace: [10876.198552] [10876.199671] ? show_regs.cold.9+0x22/0x2f [10876.201750] ? watchdog_timer_fn.cold.6+0x62/0x10a [10876.204242] ? __hrtimer_run_queues+0x1c2/0x4d0 [10876.206536] ? watchdog+0x40/0x40 [10876.208267] ? hrtimer_interrupt+0x128/0x2c0 [10876.214122] ? smp_apic_timer_interrupt+0xa6/0x270 [10876.217575] ? apic_timer_interrupt+0xf/0x20 [10876.221680] [10876.223026] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10876.226650] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10876.230341] do_raw_spin_lock+0x177/0x190 [10876.232465] _raw_spin_lock+0x21/0x30 [10876.234045] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10876.236739] obd_mod_free+0x46/0x140 [ptlrpc] [10876.241425] mdc_commit_open+0x91/0xb0 [mdc] [10876.246050] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10876.252964] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10876.255694] after_reply+0xe2c/0x1780 [ptlrpc] [10876.259357] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10876.266439] ? __next_timer_interrupt+0x160/0x160 [10876.269658] ptlrpcd+0xa82/0xe40 [ptlrpc] [10876.272972] ? do_wait_intr+0xf0/0xf0 [10876.275277] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10876.278850] kthread+0x1d1/0x200 [10876.280391] ? set_kthread_struct+0x70/0x70 [10876.282530] ret_from_fork+0x1f/0x30 [10880.061187] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10880.068458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10880.112961] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10880.125723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10880.136297] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10880.143873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10880.147027] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10880.157585] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10880.160432] RAX: 0000000000007f62 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10880.163002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10880.165469] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10880.167468] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10880.169323] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10880.171449] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10880.173979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10880.175946] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10880.177723] Call Trace: [10880.178319] [10880.178841] ? show_regs.cold.9+0x22/0x2f [10880.179812] ? watchdog_timer_fn.cold.6+0x62/0x10a [10880.181004] ? __hrtimer_run_queues+0x1c2/0x4d0 [10880.182160] ? watchdog+0x40/0x40 [10880.183100] ? hrtimer_interrupt+0x128/0x2c0 [10880.184667] ? smp_apic_timer_interrupt+0xa6/0x270 [10880.186599] ? apic_timer_interrupt+0xf/0x20 [10880.187745] [10880.188791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10880.191894] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10880.198436] do_raw_spin_lock+0x177/0x190 [10880.200999] _raw_spin_lock+0x21/0x30 [10880.204718] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10880.212939] process_one_work+0x2c8/0x700 [10880.214952] worker_thread+0x296/0x6e0 [10880.218895] ? rescuer_thread+0x570/0x570 [10880.222893] kthread+0x1d1/0x200 [10880.223975] ? set_kthread_struct+0x70/0x70 [10880.225796] ret_from_fork+0x1f/0x30 [10884.059171] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10884.062213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10884.100434] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10884.107389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10884.112637] RIP: 0010:native_safe_halt+0x2e/0x50 [10884.117533] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10884.138319] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10884.146972] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10884.154527] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10884.162557] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10884.168929] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10884.176808] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10884.183193] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10884.190520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10884.194822] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10884.203022] Call Trace: [10884.204939] [10884.207094] ? show_regs.cold.9+0x22/0x2f [10884.211153] ? watchdog_timer_fn.cold.6+0x62/0x10a [10884.217087] ? __hrtimer_run_queues+0x1c2/0x4d0 [10884.221487] ? watchdog+0x40/0x40 [10884.223161] ? hrtimer_interrupt+0x128/0x2c0 [10884.229617] ? smp_apic_timer_interrupt+0xa6/0x270 [10884.232223] ? apic_timer_interrupt+0xf/0x20 [10884.234593] [10884.235560] ? native_safe_halt+0x2e/0x50 [10884.237129] kvm_wait+0xcd/0xe0 [10884.238260] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10884.240800] do_raw_spin_lock+0x177/0x190 [10884.243093] _raw_spin_lock+0x21/0x30 [10884.245003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10884.248053] ? do_raw_spin_unlock+0x75/0x190 [10884.250421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10884.253567] ? do_wait_intr+0xf0/0xf0 [10884.255566] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10884.258501] mdc_close+0x53b/0xd70 [mdc] [10884.260549] lmv_close+0x18d/0x490 [lmv] [10884.262556] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10884.265509] ll_md_real_close+0xa5/0x2a0 [lustre] [10884.268220] ll_md_close+0x274/0x800 [lustre] [10884.270561] ll_file_release+0x128/0xc50 [lustre] [10884.273211] __fput+0x10d/0x440 [10884.274789] ____fput+0x12/0x20 [10884.276432] task_work_run+0xc6/0x110 [10884.278253] exit_to_usermode_loop+0x1dd/0x1f0 [10884.280518] do_syscall_64+0x3d6/0x440 [10884.282401] entry_SYSCALL_64_after_hwframe+0x49/0xae [10884.285006] RIP: 0033:0x7fa749552cd8 [10884.286807] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10884.298075] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10884.303199] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10884.306820] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10884.310693] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10884.314421] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10884.318154] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10896.054540] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10896.058214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10896.084254] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10896.089067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10896.093301] RIP: 0010:native_safe_halt+0x2e/0x50 [10896.095279] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10896.103706] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10896.107087] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10896.111070] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [10896.115867] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10896.119712] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10896.123666] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10896.127200] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10896.131077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10896.133522] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10896.135909] Call Trace: [10896.137223] [10896.138395] ? show_regs.cold.9+0x22/0x2f [10896.140190] ? watchdog_timer_fn.cold.6+0x62/0x10a [10896.142655] ? __hrtimer_run_queues+0x1c2/0x4d0 [10896.144781] ? watchdog+0x40/0x40 [10896.145913] ? hrtimer_interrupt+0x128/0x2c0 [10896.147477] ? smp_apic_timer_interrupt+0xa6/0x270 [10896.149954] ? apic_timer_interrupt+0xf/0x20 [10896.151445] [10896.152289] ? native_safe_halt+0x2e/0x50 [10896.153314] kvm_wait+0xcd/0xe0 [10896.154260] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10896.157375] do_raw_spin_lock+0x177/0x190 [10896.159554] _raw_spin_lock+0x21/0x30 [10896.161130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10896.163430] ? do_raw_spin_unlock+0x75/0x190 [10896.165393] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10896.167741] ? do_wait_intr+0xf0/0xf0 [10896.169273] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10896.172416] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10896.175234] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10896.178397] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10896.181040] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10896.183430] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10896.186144] mdc_intent_lock+0x353/0x7e0 [mdc] [10896.188103] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10896.190970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10896.193907] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10896.195928] lmv_intent_open+0x338/0x1150 [lmv] [10896.197872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10896.200849] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10896.203428] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10896.205020] ll_intent_lock+0xc5/0xaa0 [lustre] [10896.206829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10896.209660] ll_lookup_it+0xe0e/0x2270 [lustre] [10896.211829] ? do_raw_spin_unlock+0x75/0x190 [10896.213804] ? _raw_spin_unlock+0x12/0x30 [10896.215878] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10896.218356] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10896.220761] ll_atomic_open+0x36a/0xd70 [lustre] [10896.222960] lookup_open+0x356/0xb40 [10896.224375] do_last+0x313/0xfc0 [10896.225721] ? nd_jump_root+0xe5/0x160 [10896.227994] ? path_init+0x437/0x520 [10896.229644] path_openat+0xf7/0x500 [10896.231299] ? do_raw_spin_unlock+0x75/0x190 [10896.233478] do_filp_open+0x99/0x140 [10896.234932] ? getname_flags+0x6e/0x330 [10896.236665] ? __check_object_size+0xff/0x256 [10896.238559] ? do_raw_spin_unlock+0x75/0x190 [10896.240296] ? _raw_spin_unlock+0x12/0x30 [10896.241916] do_sys_openat2+0x2b4/0x410 [10896.243436] do_sys_open+0x73/0xa0 [10896.245027] __x64_sys_openat+0x24/0x30 [10896.246707] do_syscall_64+0xc1/0x440 [10896.248247] entry_SYSCALL_64_after_hwframe+0x49/0xae [10896.251841] RIP: 0033:0x7fa749552332 [10896.253157] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10896.259925] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10896.262731] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10896.266038] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10896.269009] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10896.272276] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10896.275535] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10899.425047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8982s! [10899.430863] Showing busy workqueues and worker pools: [10899.436510] workqueue events: flags=0x0 [10899.441359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10899.444627] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10899.453877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10899.457390] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10899.468534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.476419] pending: kfree_rcu_monitor [10899.479566] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10899.489410] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10899.501878] workqueue events_long: flags=0x0 [10899.505638] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.509194] pending: lu_objects_flush [obdclass] [10899.513437] workqueue events_unbound: flags=0x2 [10899.516385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10899.519688] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10899.529575] workqueue events_power_efficient: flags=0x80 [10899.531423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.533542] pending: check_lifetime [10899.534696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.536871] pending: neigh_periodic_work [10899.540622] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.548426] pending: neigh_periodic_work [10899.551357] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.557981] pending: do_cache_clean [10899.565254] workqueue mm_percpu_wq: flags=0x8 [10899.568410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.574078] pending: vmstat_update [10899.576618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.581325] pending: vmstat_update [10899.583454] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10899.588470] pending: vmstat_update [10899.592103] workqueue writeback: flags=0x4a [10899.594827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10899.599455] pending: wb_workfn, wb_workfn [10899.604011] workqueue xprtiod: flags=0xa [10899.605349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10899.616026] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10899.622194] workqueue ipv6_addrconf: flags=0x40008 [10899.626214] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10899.630403] pending: addrconf_verify_work [10899.635662] workqueue knbd0-recv: flags=0x1a [10899.642398] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10899.651374] in-flight: 651:recv_work [10899.656482] workqueue ptlrpc_pinger: flags=0x2 [10899.661201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10899.666827] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10899.670052] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8982s workers=4 idle: 37 31846 28897 [10899.673959] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10889s workers=2 idle: 653 [10904.058020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10904.060323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10904.073911] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10904.077242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10904.080109] RIP: 0010:native_safe_halt+0x2e/0x50 [10904.081767] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10904.088041] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10904.090792] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10904.092562] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10904.094679] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10904.096598] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10904.099129] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10904.101495] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10904.103973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10904.105797] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10904.108094] Call Trace: [10904.108957] [10904.109714] ? show_regs.cold.9+0x22/0x2f [10904.111468] ? watchdog_timer_fn.cold.6+0x62/0x10a [10904.113233] ? __hrtimer_run_queues+0x1c2/0x4d0 [10904.115066] ? watchdog+0x40/0x40 [10904.116126] ? hrtimer_interrupt+0x128/0x2c0 [10904.118311] ? smp_apic_timer_interrupt+0xa6/0x270 [10904.120027] ? apic_timer_interrupt+0xf/0x20 [10904.121741] [10904.122506] ? native_safe_halt+0x2e/0x50 [10904.124126] kvm_wait+0xcd/0xe0 [10904.125335] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10904.127302] do_raw_spin_lock+0x177/0x190 [10904.128837] _raw_spin_lock+0x21/0x30 [10904.130184] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10904.132026] obd_mod_free+0x46/0x140 [ptlrpc] [10904.133993] mdc_commit_open+0x91/0xb0 [mdc] [10904.135582] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10904.137820] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10904.139793] after_reply+0xe2c/0x1780 [ptlrpc] [10904.142095] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10904.143950] ? __next_timer_interrupt+0x160/0x160 [10904.145695] ptlrpcd+0xa82/0xe40 [ptlrpc] [10904.147106] ? do_wait_intr+0xf0/0xf0 [10904.148473] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10904.150359] kthread+0x1d1/0x200 [10904.151421] ? set_kthread_struct+0x70/0x70 [10904.152970] ret_from_fork+0x1f/0x30 [10908.057543] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10908.063366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10908.102682] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10908.113796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10908.119519] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10908.123408] RIP: 0010:native_safe_halt+0x2e/0x50 [10908.126223] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10908.136102] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10908.142295] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10908.146305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10908.150840] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10908.155280] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10908.159228] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10908.161724] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10908.166800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10908.169786] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10908.173400] Call Trace: [10908.174618] [10908.175516] ? show_regs.cold.9+0x22/0x2f [10908.178272] ? watchdog_timer_fn.cold.6+0x62/0x10a [10908.180592] ? __hrtimer_run_queues+0x1c2/0x4d0 [10908.183219] ? watchdog+0x40/0x40 [10908.184619] ? hrtimer_interrupt+0x128/0x2c0 [10908.186437] ? smp_apic_timer_interrupt+0xa6/0x270 [10908.188981] ? apic_timer_interrupt+0xf/0x20 [10908.191481] [10908.192368] ? native_safe_halt+0x2e/0x50 [10908.194920] kvm_wait+0xcd/0xe0 [10908.196507] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10908.199170] do_raw_spin_lock+0x177/0x190 [10908.200913] _raw_spin_lock+0x21/0x30 [10908.202695] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10908.205245] process_one_work+0x2c8/0x700 [10908.207012] worker_thread+0x296/0x6e0 [10908.209016] ? rescuer_thread+0x570/0x570 [10908.210656] kthread+0x1d1/0x200 [10908.212399] ? set_kthread_struct+0x70/0x70 [10908.214246] ret_from_fork+0x1f/0x30 [10912.059071] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10912.061661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10912.084786] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10912.090407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10912.095292] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10912.098881] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10912.110712] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10912.116590] RAX: 0000000000007592 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10912.121785] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10912.124860] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10912.127745] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10912.130427] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10912.135052] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10912.138479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10912.142313] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10912.146718] Call Trace: [10912.148243] [10912.149388] ? show_regs.cold.9+0x22/0x2f [10912.151358] ? watchdog_timer_fn.cold.6+0x62/0x10a [10912.153341] ? __hrtimer_run_queues+0x1c2/0x4d0 [10912.155002] ? watchdog+0x40/0x40 [10912.156071] ? hrtimer_interrupt+0x128/0x2c0 [10912.158310] ? smp_apic_timer_interrupt+0xa6/0x270 [10912.160402] ? apic_timer_interrupt+0xf/0x20 [10912.162229] [10912.163062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10912.165245] do_raw_spin_lock+0x177/0x190 [10912.166916] _raw_spin_lock+0x21/0x30 [10912.168841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10912.171594] ? do_raw_spin_unlock+0x75/0x190 [10912.173266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10912.176145] ? do_wait_intr+0xf0/0xf0 [10912.177821] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10912.180321] mdc_close+0x53b/0xd70 [mdc] [10912.182244] lmv_close+0x18d/0x490 [lmv] [10912.184081] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10912.186852] ll_md_real_close+0xa5/0x2a0 [lustre] [10912.189076] ll_md_close+0x274/0x800 [lustre] [10912.191209] ll_file_release+0x128/0xc50 [lustre] [10912.193449] __fput+0x10d/0x440 [10912.195191] ____fput+0x12/0x20 [10912.196852] task_work_run+0xc6/0x110 [10912.198365] exit_to_usermode_loop+0x1dd/0x1f0 [10912.201338] do_syscall_64+0x3d6/0x440 [10912.202893] entry_SYSCALL_64_after_hwframe+0x49/0xae [10912.205402] RIP: 0033:0x7fa749552cd8 [10912.207013] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10912.215856] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10912.218838] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10912.222141] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10912.225195] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10912.228199] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10912.231413] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10924.058892] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10924.080624] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10924.159051] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10924.177008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10924.195176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10924.197441] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10924.213574] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10924.218967] RAX: 0000000000003256 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10924.231994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10924.236060] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10924.239399] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10924.245198] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10924.249651] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10924.256999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10924.260160] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10924.265574] Call Trace: [10924.266544] [10924.269329] ? show_regs.cold.9+0x22/0x2f [10924.271326] ? watchdog_timer_fn.cold.6+0x62/0x10a [10924.273306] ? __hrtimer_run_queues+0x1c2/0x4d0 [10924.275533] ? watchdog+0x40/0x40 [10924.276937] ? hrtimer_interrupt+0x128/0x2c0 [10924.279901] ? smp_apic_timer_interrupt+0xa6/0x270 [10924.282486] ? apic_timer_interrupt+0xf/0x20 [10924.284042] [10924.289453] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10924.291999] do_raw_spin_lock+0x177/0x190 [10924.302337] _raw_spin_lock+0x21/0x30 [10924.303823] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10924.307999] ? do_raw_spin_unlock+0x75/0x190 [10924.311296] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10924.314239] ? do_wait_intr+0xf0/0xf0 [10924.316028] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10924.320912] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10924.324400] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10924.328066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10924.331754] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10924.334817] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10924.336986] mdc_intent_lock+0x353/0x7e0 [mdc] [10924.338533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10924.341386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10924.343990] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10924.346692] lmv_intent_open+0x338/0x1150 [lmv] [10924.348889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10924.352942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10924.369337] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10924.372761] ll_intent_lock+0xc5/0xaa0 [lustre] [10924.374086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10924.376069] ll_lookup_it+0xe0e/0x2270 [lustre] [10924.378511] ? do_raw_spin_unlock+0x75/0x190 [10924.379905] ? _raw_spin_unlock+0x12/0x30 [10924.381196] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10924.382528] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10924.384449] ll_atomic_open+0x36a/0xd70 [lustre] [10924.394770] lookup_open+0x356/0xb40 [10924.402442] do_last+0x313/0xfc0 [10924.403666] ? nd_jump_root+0xe5/0x160 [10924.412921] ? path_init+0x437/0x520 [10924.415711] path_openat+0xf7/0x500 [10924.430012] ? do_raw_spin_unlock+0x75/0x190 [10924.430949] do_filp_open+0x99/0x140 [10924.432173] ? getname_flags+0x6e/0x330 [10924.433334] ? __check_object_size+0xff/0x256 [10924.434956] ? do_raw_spin_unlock+0x75/0x190 [10924.436157] ? _raw_spin_unlock+0x12/0x30 [10924.437547] do_sys_openat2+0x2b4/0x410 [10924.439012] do_sys_open+0x73/0xa0 [10924.440006] __x64_sys_openat+0x24/0x30 [10924.441179] do_syscall_64+0xc1/0x440 [10924.442542] entry_SYSCALL_64_after_hwframe+0x49/0xae [10924.444223] RIP: 0033:0x7fa749552332 [10924.445365] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10924.451514] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10924.453934] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10924.456934] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10924.462532] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10924.471300] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10924.475521] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10930.151063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9012s! [10930.157129] Showing busy workqueues and worker pools: [10930.162714] workqueue events: flags=0x0 [10930.166248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10930.179718] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10930.197643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10930.203796] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10930.212894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.217753] pending: kfree_rcu_monitor [10930.219965] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10930.225025] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10930.235789] workqueue events_long: flags=0x0 [10930.238247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.245024] pending: lu_objects_flush [obdclass] [10930.248944] workqueue events_unbound: flags=0x2 [10930.254043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10930.261427] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10930.272519] workqueue events_power_efficient: flags=0x80 [10930.274888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.278302] pending: check_lifetime [10930.280088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.283435] pending: neigh_periodic_work [10930.285480] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.288697] pending: neigh_periodic_work [10930.290820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.293873] pending: do_cache_clean [10930.295442] workqueue mm_percpu_wq: flags=0x8 [10930.297649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.300782] pending: vmstat_update [10930.301941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.306877] pending: vmstat_update [10930.309126] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10930.312593] pending: vmstat_update [10930.314575] workqueue writeback: flags=0x4a [10930.316903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10930.319774] pending: wb_workfn, wb_workfn [10930.322097] workqueue xprtiod: flags=0xa [10930.323865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10930.326625] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10930.330530] workqueue ipv6_addrconf: flags=0x40008 [10930.332578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10930.335586] pending: addrconf_verify_work [10930.337908] workqueue knbd0-recv: flags=0x1a [10930.339790] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10930.342484] in-flight: 651:recv_work [10930.344112] workqueue ptlrpc_pinger: flags=0x2 [10930.346055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10930.348753] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10930.352046] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9013s workers=4 idle: 37 31846 28897 [10930.355822] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10920s workers=2 idle: 653 [10932.058035] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10932.069541] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10932.090930] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10932.094835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10932.097832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10932.101399] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10932.122548] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10932.126177] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10932.130190] RDX: 0000000000007632 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10932.133226] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10932.135192] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10932.138174] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10932.140344] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10932.142873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10932.145761] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10932.148209] Call Trace: [10932.148857] [10932.149751] ? show_regs.cold.9+0x22/0x2f [10932.151017] ? watchdog_timer_fn.cold.6+0x62/0x10a [10932.153070] ? __hrtimer_run_queues+0x1c2/0x4d0 [10932.154628] ? watchdog+0x40/0x40 [10932.155839] ? hrtimer_interrupt+0x128/0x2c0 [10932.157242] ? smp_apic_timer_interrupt+0xa6/0x270 [10932.158786] ? apic_timer_interrupt+0xf/0x20 [10932.160049] [10932.160548] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10932.169585] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10932.171682] do_raw_spin_lock+0x177/0x190 [10932.175847] _raw_spin_lock+0x21/0x30 [10932.177106] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10932.182653] obd_mod_free+0x46/0x140 [ptlrpc] [10932.187159] mdc_commit_open+0x91/0xb0 [mdc] [10932.191111] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10932.193461] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10932.200001] after_reply+0xe2c/0x1780 [ptlrpc] [10932.209522] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10932.212304] ? __next_timer_interrupt+0x160/0x160 [10932.217724] ptlrpcd+0xa82/0xe40 [ptlrpc] [10932.222929] ? do_wait_intr+0xf0/0xf0 [10932.224348] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10932.228447] kthread+0x1d1/0x200 [10932.230668] ? set_kthread_struct+0x70/0x70 [10932.232057] ret_from_fork+0x1f/0x30 [10936.057022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [10936.064410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10936.099384] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10936.115266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10936.127021] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10936.131166] RIP: 0010:native_safe_halt+0x2e/0x50 [10936.134430] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10936.144703] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10936.152371] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10936.159193] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10936.167183] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10936.172637] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10936.177017] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10936.186553] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10936.192253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10936.199455] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10936.216725] Call Trace: [10936.219562] [10936.224513] ? show_regs.cold.9+0x22/0x2f [10936.232007] ? watchdog_timer_fn.cold.6+0x62/0x10a [10936.235622] ? __hrtimer_run_queues+0x1c2/0x4d0 [10936.241952] ? watchdog+0x40/0x40 [10936.245206] ? hrtimer_interrupt+0x128/0x2c0 [10936.249370] ? smp_apic_timer_interrupt+0xa6/0x270 [10936.256053] ? apic_timer_interrupt+0xf/0x20 [10936.259120] [10936.261181] ? native_safe_halt+0x2e/0x50 [10936.263554] kvm_wait+0xcd/0xe0 [10936.269720] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10936.274395] do_raw_spin_lock+0x177/0x190 [10936.276192] _raw_spin_lock+0x21/0x30 [10936.278310] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10936.282419] process_one_work+0x2c8/0x700 [10936.285207] worker_thread+0x296/0x6e0 [10936.287537] ? rescuer_thread+0x570/0x570 [10936.295986] kthread+0x1d1/0x200 [10936.300262] ? set_kthread_struct+0x70/0x70 [10936.302448] ret_from_fork+0x1f/0x30 [10940.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10940.065061] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10940.090778] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10940.098222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10940.106839] RIP: 0010:native_safe_halt+0x2e/0x50 [10940.109917] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10940.123251] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10940.126227] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10940.128232] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10940.132939] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10940.136243] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10940.139105] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10940.141479] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10940.144330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10940.145962] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10940.148450] Call Trace: [10940.149115] [10940.149766] ? show_regs.cold.9+0x22/0x2f [10940.157497] ? watchdog_timer_fn.cold.6+0x62/0x10a [10940.161759] ? __hrtimer_run_queues+0x1c2/0x4d0 [10940.164144] ? watchdog+0x40/0x40 [10940.165711] ? hrtimer_interrupt+0x128/0x2c0 [10940.167088] ? smp_apic_timer_interrupt+0xa6/0x270 [10940.168347] ? apic_timer_interrupt+0xf/0x20 [10940.171865] [10940.173422] ? native_safe_halt+0x2e/0x50 [10940.174867] kvm_wait+0xcd/0xe0 [10940.175757] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10940.177837] do_raw_spin_lock+0x177/0x190 [10940.179526] _raw_spin_lock+0x21/0x30 [10940.181153] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10940.183940] ? do_raw_spin_unlock+0x75/0x190 [10940.186991] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10940.189563] ? do_wait_intr+0xf0/0xf0 [10940.190984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10940.194723] mdc_close+0x53b/0xd70 [mdc] [10940.195796] lmv_close+0x18d/0x490 [lmv] [10940.197004] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10940.202360] ll_md_real_close+0xa5/0x2a0 [lustre] [10940.206530] ll_md_close+0x274/0x800 [lustre] [10940.212348] ll_file_release+0x128/0xc50 [lustre] [10940.216769] __fput+0x10d/0x440 [10940.218512] ____fput+0x12/0x20 [10940.220836] task_work_run+0xc6/0x110 [10940.222311] exit_to_usermode_loop+0x1dd/0x1f0 [10940.224792] do_syscall_64+0x3d6/0x440 [10940.226668] entry_SYSCALL_64_after_hwframe+0x49/0xae [10940.228692] RIP: 0033:0x7fa749552cd8 [10940.234183] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10940.242296] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10940.248771] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10940.252196] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10940.256990] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10940.259716] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10940.262709] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10956.054061] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10956.067127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10956.108437] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10956.112788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10956.119659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10956.124097] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10956.142177] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10956.144820] RAX: 00000000000049d4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10956.148880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10956.155048] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10956.160848] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10956.169510] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10956.175542] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10956.181084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10956.184650] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10956.189047] Call Trace: [10956.189861] [10956.193408] ? show_regs.cold.9+0x22/0x2f [10956.195667] ? watchdog_timer_fn.cold.6+0x62/0x10a [10956.201499] ? __hrtimer_run_queues+0x1c2/0x4d0 [10956.204346] ? watchdog+0x40/0x40 [10956.206717] ? hrtimer_interrupt+0x128/0x2c0 [10956.208106] ? smp_apic_timer_interrupt+0xa6/0x270 [10956.211397] ? apic_timer_interrupt+0xf/0x20 [10956.213543] [10956.218336] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10956.223468] do_raw_spin_lock+0x177/0x190 [10956.232540] _raw_spin_lock+0x21/0x30 [10956.235146] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10956.238831] ? do_raw_spin_unlock+0x75/0x190 [10956.242663] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10956.244638] ? do_wait_intr+0xf0/0xf0 [10956.245950] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10956.256595] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10956.262552] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10956.271532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10956.273342] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10956.283894] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10956.287805] mdc_intent_lock+0x353/0x7e0 [mdc] [10956.290662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10956.295185] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10956.299378] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10956.302887] lmv_intent_open+0x338/0x1150 [lmv] [10956.307779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10956.314947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10956.317533] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10956.323359] ll_intent_lock+0xc5/0xaa0 [lustre] [10956.327530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10956.336840] ll_lookup_it+0xe0e/0x2270 [lustre] [10956.339648] ? do_raw_spin_unlock+0x75/0x190 [10956.342439] ? _raw_spin_unlock+0x12/0x30 [10956.346503] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10956.349752] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10956.354056] ll_atomic_open+0x36a/0xd70 [lustre] [10956.360054] lookup_open+0x356/0xb40 [10956.363458] do_last+0x313/0xfc0 [10956.369155] ? nd_jump_root+0xe5/0x160 [10956.376436] ? path_init+0x437/0x520 [10956.379126] path_openat+0xf7/0x500 [10956.381773] ? do_raw_spin_unlock+0x75/0x190 [10956.384625] do_filp_open+0x99/0x140 [10956.386619] ? getname_flags+0x6e/0x330 [10956.389193] ? __check_object_size+0xff/0x256 [10956.391089] ? do_raw_spin_unlock+0x75/0x190 [10956.393482] ? _raw_spin_unlock+0x12/0x30 [10956.395973] do_sys_openat2+0x2b4/0x410 [10956.398576] do_sys_open+0x73/0xa0 [10956.402208] __x64_sys_openat+0x24/0x30 [10956.403539] do_syscall_64+0xc1/0x440 [10956.411360] entry_SYSCALL_64_after_hwframe+0x49/0xae [10956.413638] RIP: 0033:0x7fa749552332 [10956.416478] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10956.440595] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10956.453612] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10956.458416] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10956.465335] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10956.471258] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10956.478258] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10960.061419] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [10960.076614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10960.122746] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10960.130868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10960.138653] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10960.140798] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10960.161402] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10960.166473] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [10960.174557] RDX: 0000000000006235 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [10960.178623] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [10960.187322] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [10960.194313] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10960.201653] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [10960.210724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10960.215509] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [10960.219836] Call Trace: [10960.222465] [10960.223385] ? show_regs.cold.9+0x22/0x2f [10960.230529] ? watchdog_timer_fn.cold.6+0x62/0x10a [10960.234539] ? __hrtimer_run_queues+0x1c2/0x4d0 [10960.243806] ? watchdog+0x40/0x40 [10960.245490] ? hrtimer_interrupt+0x128/0x2c0 [10960.250751] ? smp_apic_timer_interrupt+0xa6/0x270 [10960.258113] ? apic_timer_interrupt+0xf/0x20 [10960.262307] [10960.263530] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10960.266227] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10960.275943] do_raw_spin_lock+0x177/0x190 [10960.282088] _raw_spin_lock+0x21/0x30 [10960.289868] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10960.296440] obd_mod_free+0x46/0x140 [ptlrpc] [10960.303039] mdc_commit_open+0x91/0xb0 [mdc] [10960.307210] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10960.308908] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10960.321348] after_reply+0xe2c/0x1780 [ptlrpc] [10960.328439] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10960.331434] ? __next_timer_interrupt+0x160/0x160 [10960.333942] ptlrpcd+0xa82/0xe40 [ptlrpc] [10960.335889] ? do_wait_intr+0xf0/0xf0 [10960.337260] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10960.339704] kthread+0x1d1/0x200 [10960.340962] ? set_kthread_struct+0x70/0x70 [10960.342599] ret_from_fork+0x1f/0x30 [10960.870250] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9043s! [10960.875936] Showing busy workqueues and worker pools: [10960.878224] workqueue events: flags=0x0 [10960.880979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10960.884895] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10960.894862] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10960.896528] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10960.906610] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10960.911399] pending: kfree_rcu_monitor [10960.924848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10960.928512] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10960.941909] workqueue events_long: flags=0x0 [10960.946882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10960.950910] pending: lu_objects_flush [obdclass] [10960.955219] workqueue events_unbound: flags=0x2 [10960.960721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10960.964441] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10960.975489] workqueue events_power_efficient: flags=0x80 [10960.987769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10960.994196] pending: check_lifetime [10960.998077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10961.003128] pending: neigh_periodic_work [10961.006360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10961.010072] pending: neigh_periodic_work [10961.015577] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10961.019837] pending: do_cache_clean [10961.020823] workqueue mm_percpu_wq: flags=0x8 [10961.022335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10961.035095] pending: vmstat_update [10961.039140] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10961.043848] pending: vmstat_update [10961.045515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10961.050970] pending: vmstat_update [10961.055695] workqueue writeback: flags=0x4a [10961.058506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10961.069790] pending: wb_workfn, wb_workfn [10961.071327] workqueue xprtiod: flags=0xa [10961.073365] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10961.077185] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10961.085740] workqueue ipv6_addrconf: flags=0x40008 [10961.089421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10961.093905] pending: addrconf_verify_work [10961.096603] workqueue knbd0-recv: flags=0x1a [10961.101838] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10961.105461] in-flight: 651:recv_work [10961.108029] workqueue ptlrpc_pinger: flags=0x2 [10961.109971] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10961.113043] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10961.116185] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9043s workers=4 idle: 37 31846 28897 [10961.119537] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10951s workers=2 idle: 653 [10964.057027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10964.059188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10964.094357] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10964.099919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10964.104096] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10964.108482] RIP: 0010:native_safe_halt+0x2e/0x50 [10964.110129] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10964.126164] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10964.133488] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10964.137201] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10964.139309] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10964.141927] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10964.147134] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10964.152971] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10964.156473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10964.162055] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10964.168508] Call Trace: [10964.170172] [10964.171457] ? show_regs.cold.9+0x22/0x2f [10964.174407] ? watchdog_timer_fn.cold.6+0x62/0x10a [10964.177792] ? __hrtimer_run_queues+0x1c2/0x4d0 [10964.181399] ? watchdog+0x40/0x40 [10964.182419] ? hrtimer_interrupt+0x128/0x2c0 [10964.187372] ? smp_apic_timer_interrupt+0xa6/0x270 [10964.190193] ? apic_timer_interrupt+0xf/0x20 [10964.194181] [10964.194751] ? native_safe_halt+0x2e/0x50 [10964.195836] kvm_wait+0xcd/0xe0 [10964.196785] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10964.199924] do_raw_spin_lock+0x177/0x190 [10964.201233] _raw_spin_lock+0x21/0x30 [10964.204710] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10964.207187] process_one_work+0x2c8/0x700 [10964.208813] worker_thread+0x296/0x6e0 [10964.211393] ? rescuer_thread+0x570/0x570 [10964.212456] kthread+0x1d1/0x200 [10964.213403] ? set_kthread_struct+0x70/0x70 [10964.216892] ret_from_fork+0x1f/0x30 [10968.063927] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10968.080805] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10968.126352] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10968.139181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10968.146230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10968.153209] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10968.175078] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10968.179056] RAX: 0000000000007f0f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10968.189810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10968.196801] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10968.200460] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10968.205194] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10968.209836] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10968.215948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10968.218974] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10968.222170] Call Trace: [10968.223321] [10968.224246] ? show_regs.cold.9+0x22/0x2f [10968.226233] ? watchdog_timer_fn.cold.6+0x62/0x10a [10968.228355] ? __hrtimer_run_queues+0x1c2/0x4d0 [10968.230747] ? watchdog+0x40/0x40 [10968.233848] ? hrtimer_interrupt+0x128/0x2c0 [10968.237042] ? smp_apic_timer_interrupt+0xa6/0x270 [10968.240749] ? apic_timer_interrupt+0xf/0x20 [10968.246476] [10968.248455] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10968.251262] do_raw_spin_lock+0x177/0x190 [10968.253687] _raw_spin_lock+0x21/0x30 [10968.258669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10968.263699] ? do_raw_spin_unlock+0x75/0x190 [10968.268294] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10968.273222] ? do_wait_intr+0xf0/0xf0 [10968.276566] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10968.279814] mdc_close+0x53b/0xd70 [mdc] [10968.283249] lmv_close+0x18d/0x490 [lmv] [10968.286986] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10968.293288] ll_md_real_close+0xa5/0x2a0 [lustre] [10968.296689] ll_md_close+0x274/0x800 [lustre] [10968.303809] ll_file_release+0x128/0xc50 [lustre] [10968.309118] __fput+0x10d/0x440 [10968.311763] ____fput+0x12/0x20 [10968.317927] task_work_run+0xc6/0x110 [10968.319736] exit_to_usermode_loop+0x1dd/0x1f0 [10968.323888] do_syscall_64+0x3d6/0x440 [10968.326966] entry_SYSCALL_64_after_hwframe+0x49/0xae [10968.332012] RIP: 0033:0x7fa749552cd8 [10968.334036] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10968.354182] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10968.360257] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10968.366812] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10968.371784] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10968.380889] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10968.387679] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10973.881211] rcu: INFO: rcu_sched self-detected stall on CPU [10973.885886] rcu: 2-...!: (5102053 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [10973.902667] (t=9060192 jiffies g=258701 q=4140) [10973.905770] rcu: rcu_sched kthread starved for 9056987 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [10973.916244] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10973.925652] rcu: RCU grace-period kthread stack dump: [10973.927214] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10973.944469] Call Trace: [10973.945677] __schedule+0x351/0xcb0 [10973.949976] schedule+0xc0/0x180 [10973.953947] schedule_timeout+0xb4/0x190 [10973.959569] ? do_raw_spin_unlock+0x75/0x190 [10973.962197] ? __next_timer_interrupt+0x160/0x160 [10973.967871] ? prepare_to_swait_event+0xf3/0x140 [10973.969946] rcu_gp_kthread+0x8dd/0x15b0 [10973.971268] ? force_qs_rnp+0x2c0/0x2c0 [10973.972823] kthread+0x1d1/0x200 [10973.974897] ? set_kthread_struct+0x70/0x70 [10973.982605] ret_from_fork+0x1f/0x30 [10973.985267] rcu: Stack dump where RCU GP kthread last ran: [10973.994211] NMI backtrace for cpu 2 [10973.998021] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10974.011132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10974.020034] Call Trace: [10974.021178] [10974.021967] ? dump_stack+0xbb/0x10e [10974.030603] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10974.032981] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10974.038856] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10974.040984] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10974.050913] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10974.057958] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [10974.060566] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [10974.062923] ? update_process_times+0x6d/0xc0 [10974.065942] ? tick_sched_handle.isra.18+0x27/0x90 [10974.067515] ? tick_sched_timer+0x53/0xb0 [10974.072505] ? __hrtimer_run_queues+0x1c2/0x4d0 [10974.074010] ? tick_sched_handle.isra.18+0x90/0x90 [10974.081076] ? hrtimer_interrupt+0x128/0x2c0 [10974.083534] ? smp_apic_timer_interrupt+0xa6/0x270 [10974.089408] ? apic_timer_interrupt+0xf/0x20 [10974.090787] [10974.091418] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10974.100003] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10974.102782] ? do_raw_spin_lock+0x177/0x190 [10974.104179] ? _raw_spin_lock+0x21/0x30 [10974.110839] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10974.114883] ? obd_mod_free+0x46/0x140 [ptlrpc] [10974.116545] ? mdc_commit_open+0x91/0xb0 [mdc] [10974.123104] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10974.125613] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10974.129838] ? after_reply+0xe2c/0x1780 [ptlrpc] [10974.135220] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10974.147011] ? __next_timer_interrupt+0x160/0x160 [10974.150261] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10974.153472] ? do_wait_intr+0xf0/0xf0 [10974.159284] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10974.166229] ? kthread+0x1d1/0x200 [10974.170640] ? set_kthread_struct+0x70/0x70 [10974.173891] ? ret_from_fork+0x1f/0x30 [10974.175726] NMI backtrace for cpu 2 [10974.176893] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10974.183637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10974.196443] Call Trace: [10974.197229] [10974.197854] ? dump_stack+0xbb/0x10e [10974.198921] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10974.204584] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10974.206074] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10974.210976] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10974.216220] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10974.222635] ? rcu_dump_cpu_stacks+0x11b/0x171 [10974.225373] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10974.230301] ? update_process_times+0x6d/0xc0 [10974.234347] ? tick_sched_handle.isra.18+0x27/0x90 [10974.238286] ? tick_sched_timer+0x53/0xb0 [10974.240163] ? __hrtimer_run_queues+0x1c2/0x4d0 [10974.241818] ? tick_sched_handle.isra.18+0x90/0x90 [10974.245568] ? hrtimer_interrupt+0x128/0x2c0 [10974.248379] ? smp_apic_timer_interrupt+0xa6/0x270 [10974.252632] ? apic_timer_interrupt+0xf/0x20 [10974.255796] [10974.258516] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10974.262843] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10974.270115] ? do_raw_spin_lock+0x177/0x190 [10974.272813] ? _raw_spin_lock+0x21/0x30 [10974.274708] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10974.282453] ? obd_mod_free+0x46/0x140 [ptlrpc] [10974.286650] ? mdc_commit_open+0x91/0xb0 [mdc] [10974.290494] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10974.294224] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10974.299556] ? after_reply+0xe2c/0x1780 [ptlrpc] [10974.306922] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10974.310214] ? __next_timer_interrupt+0x160/0x160 [10974.314318] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10974.318810] ? do_wait_intr+0xf0/0xf0 [10974.324432] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10974.336932] ? kthread+0x1d1/0x200 [10974.343286] ? set_kthread_struct+0x70/0x70 [10974.351598] ? ret_from_fork+0x1f/0x30 [10988.054120] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [10988.063925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10988.108415] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10988.115975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10988.129969] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10988.135042] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10988.146936] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10988.151359] RAX: 0000000000005280 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10988.155240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [10988.159758] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [10988.163126] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [10988.166506] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [10988.170283] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [10988.173342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10988.175846] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [10988.179262] Call Trace: [10988.180541] [10988.181821] ? show_regs.cold.9+0x22/0x2f [10988.185378] ? watchdog_timer_fn.cold.6+0x62/0x10a [10988.188814] ? __hrtimer_run_queues+0x1c2/0x4d0 [10988.192115] ? watchdog+0x40/0x40 [10988.193525] ? hrtimer_interrupt+0x128/0x2c0 [10988.196077] ? smp_apic_timer_interrupt+0xa6/0x270 [10988.198976] ? apic_timer_interrupt+0xf/0x20 [10988.201840] [10988.202935] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10988.205952] do_raw_spin_lock+0x177/0x190 [10988.208298] _raw_spin_lock+0x21/0x30 [10988.210009] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10988.213076] ? do_raw_spin_unlock+0x75/0x190 [10988.215268] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10988.218542] ? do_wait_intr+0xf0/0xf0 [10988.220422] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10988.223208] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10988.225956] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10988.229219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.231855] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10988.234917] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10988.237365] mdc_intent_lock+0x353/0x7e0 [mdc] [10988.239816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.243039] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10988.246259] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10988.249889] lmv_intent_open+0x338/0x1150 [lmv] [10988.251864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.254388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.257904] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10988.260400] ll_intent_lock+0xc5/0xaa0 [lustre] [10988.262877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.266310] ll_lookup_it+0xe0e/0x2270 [lustre] [10988.268982] ? do_raw_spin_unlock+0x75/0x190 [10988.271074] ? _raw_spin_unlock+0x12/0x30 [10988.273554] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10988.276075] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10988.278533] ll_atomic_open+0x36a/0xd70 [lustre] [10988.280825] lookup_open+0x356/0xb40 [10988.282555] do_last+0x313/0xfc0 [10988.284009] ? nd_jump_root+0xe5/0x160 [10988.285877] ? path_init+0x437/0x520 [10988.287686] path_openat+0xf7/0x500 [10988.289831] ? do_raw_spin_unlock+0x75/0x190 [10988.292512] do_filp_open+0x99/0x140 [10988.294321] ? getname_flags+0x6e/0x330 [10988.296099] ? __check_object_size+0xff/0x256 [10988.298700] ? do_raw_spin_unlock+0x75/0x190 [10988.301048] ? _raw_spin_unlock+0x12/0x30 [10988.302895] do_sys_openat2+0x2b4/0x410 [10988.305013] do_sys_open+0x73/0xa0 [10988.307212] __x64_sys_openat+0x24/0x30 [10988.309261] do_syscall_64+0xc1/0x440 [10988.311068] entry_SYSCALL_64_after_hwframe+0x49/0xae [10988.313930] RIP: 0033:0x7fa749552332 [10988.315708] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10988.334605] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10988.346648] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [10988.355549] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [10988.358610] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [10988.361547] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [10988.369664] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [10991.585047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9074s! [10991.591546] Showing busy workqueues and worker pools: [10991.593660] workqueue events: flags=0x0 [10991.595259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10991.597915] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [10991.605316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10991.612424] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10991.622526] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.628876] pending: kfree_rcu_monitor [10991.634438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10991.641980] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [10991.659820] workqueue events_long: flags=0x0 [10991.665001] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.668143] pending: lu_objects_flush [obdclass] [10991.669963] workqueue events_unbound: flags=0x2 [10991.679532] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [10991.683492] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10991.691660] workqueue events_power_efficient: flags=0x80 [10991.694073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.696610] pending: check_lifetime [10991.698059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.700953] pending: neigh_periodic_work [10991.704595] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.708801] pending: neigh_periodic_work [10991.710651] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.713370] pending: do_cache_clean [10991.715193] workqueue mm_percpu_wq: flags=0x8 [10991.717431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.720632] pending: vmstat_update [10991.723028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.727294] pending: vmstat_update [10991.729068] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10991.731501] pending: vmstat_update [10991.736529] workqueue writeback: flags=0x4a [10991.739260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10991.743909] pending: wb_workfn, wb_workfn [10991.745565] workqueue xprtiod: flags=0xa [10991.749352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10991.752012] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10991.758319] workqueue ipv6_addrconf: flags=0x40008 [10991.761330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10991.764450] pending: addrconf_verify_work [10991.767677] workqueue knbd0-recv: flags=0x1a [10991.770059] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10991.774234] in-flight: 651:recv_work [10991.775275] workqueue ptlrpc_pinger: flags=0x2 [10991.776549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10991.779972] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10991.784180] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9074s workers=4 idle: 37 31846 28897 [10991.789304] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10981s workers=2 idle: 653 [10992.057025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [10992.060020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10992.096883] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10992.105614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10992.115031] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10992.117742] RIP: 0010:native_safe_halt+0x2e/0x50 [10992.119356] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10992.135191] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10992.139726] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10992.156163] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [10992.168354] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [10992.176300] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [10992.183418] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [10992.192937] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [10992.196889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10992.205211] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [10992.213619] Call Trace: [10992.216328] [10992.217608] ? show_regs.cold.9+0x22/0x2f [10992.219674] ? watchdog_timer_fn.cold.6+0x62/0x10a [10992.223277] ? __hrtimer_run_queues+0x1c2/0x4d0 [10992.227548] ? watchdog+0x40/0x40 [10992.233703] ? hrtimer_interrupt+0x128/0x2c0 [10992.236235] ? smp_apic_timer_interrupt+0xa6/0x270 [10992.238754] ? apic_timer_interrupt+0xf/0x20 [10992.242237] [10992.244071] ? native_safe_halt+0x2e/0x50 [10992.246140] kvm_wait+0xcd/0xe0 [10992.248204] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10992.254381] do_raw_spin_lock+0x177/0x190 [10992.256963] _raw_spin_lock+0x21/0x30 [10992.262881] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10992.265352] process_one_work+0x2c8/0x700 [10992.266912] worker_thread+0x296/0x6e0 [10992.271078] ? rescuer_thread+0x570/0x570 [10992.274519] kthread+0x1d1/0x200 [10992.275785] ? set_kthread_struct+0x70/0x70 [10992.279398] ret_from_fork+0x1f/0x30 [10996.059458] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [10996.068832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10996.086516] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10996.091133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10996.094653] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10996.097153] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10996.103944] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10996.107221] RAX: 00000000000016b9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [10996.111613] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [10996.117995] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [10996.121944] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [10996.125877] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [10996.131807] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [10996.136805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10996.141506] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [10996.145203] Call Trace: [10996.147907] [10996.148870] ? show_regs.cold.9+0x22/0x2f [10996.150779] ? watchdog_timer_fn.cold.6+0x62/0x10a [10996.153179] ? __hrtimer_run_queues+0x1c2/0x4d0 [10996.154980] ? watchdog+0x40/0x40 [10996.158469] ? hrtimer_interrupt+0x128/0x2c0 [10996.160241] ? smp_apic_timer_interrupt+0xa6/0x270 [10996.163875] ? apic_timer_interrupt+0xf/0x20 [10996.166256] [10996.167288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10996.169735] do_raw_spin_lock+0x177/0x190 [10996.172468] _raw_spin_lock+0x21/0x30 [10996.173485] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10996.176394] ? do_raw_spin_unlock+0x75/0x190 [10996.179774] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10996.183128] ? do_wait_intr+0xf0/0xf0 [10996.186740] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10996.192410] mdc_close+0x53b/0xd70 [mdc] [10996.194667] lmv_close+0x18d/0x490 [lmv] [10996.196496] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10996.199345] ll_md_real_close+0xa5/0x2a0 [lustre] [10996.201542] ll_md_close+0x274/0x800 [lustre] [10996.203530] ll_file_release+0x128/0xc50 [lustre] [10996.205366] __fput+0x10d/0x440 [10996.207999] ____fput+0x12/0x20 [10996.209373] task_work_run+0xc6/0x110 [10996.212040] exit_to_usermode_loop+0x1dd/0x1f0 [10996.214536] do_syscall_64+0x3d6/0x440 [10996.215870] entry_SYSCALL_64_after_hwframe+0x49/0xae [10996.221090] RIP: 0033:0x7fa749552cd8 [10996.223225] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10996.232406] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10996.239064] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [10996.241675] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [10996.245369] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [10996.251357] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [10996.257763] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11000.058054] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11000.060704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11000.094381] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11000.101515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11000.113932] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11000.120101] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11000.141320] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11000.150430] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11000.159006] RDX: 0000000000005b3e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11000.170292] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11000.176669] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11000.184888] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11000.196449] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11000.210097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11000.227666] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11000.239470] Call Trace: [11000.246585] [11000.247629] ? show_regs.cold.9+0x22/0x2f [11000.252656] ? watchdog_timer_fn.cold.6+0x62/0x10a [11000.258572] ? __hrtimer_run_queues+0x1c2/0x4d0 [11000.266293] ? watchdog+0x40/0x40 [11000.268936] ? hrtimer_interrupt+0x128/0x2c0 [11000.273486] ? smp_apic_timer_interrupt+0xa6/0x270 [11000.275690] ? apic_timer_interrupt+0xf/0x20 [11000.277779] [11000.281224] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11000.284756] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11000.289783] do_raw_spin_lock+0x177/0x190 [11000.292777] _raw_spin_lock+0x21/0x30 [11000.294676] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11000.299678] obd_mod_free+0x46/0x140 [ptlrpc] [11000.301739] mdc_commit_open+0x91/0xb0 [mdc] [11000.303536] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11000.311537] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11000.316262] after_reply+0xe2c/0x1780 [ptlrpc] [11000.320048] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11000.323577] ? __next_timer_interrupt+0x160/0x160 [11000.326007] ptlrpcd+0xa82/0xe40 [ptlrpc] [11000.328414] ? do_wait_intr+0xf0/0xf0 [11000.329978] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11000.338537] kthread+0x1d1/0x200 [11000.340612] ? set_kthread_struct+0x70/0x70 [11000.343582] ret_from_fork+0x1f/0x30 [11016.065544] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11016.067944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11016.102050] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11016.112555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11016.128133] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11016.142733] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11016.161979] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11016.175030] RAX: 0000000000007e86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11016.184243] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11016.200690] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11016.210492] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11016.220513] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11016.222895] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11016.233772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11016.247111] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11016.254797] Call Trace: [11016.258469] [11016.263937] ? show_regs.cold.9+0x22/0x2f [11016.270831] ? watchdog_timer_fn.cold.6+0x62/0x10a [11016.274385] ? __hrtimer_run_queues+0x1c2/0x4d0 [11016.280892] ? watchdog+0x40/0x40 [11016.284816] ? hrtimer_interrupt+0x128/0x2c0 [11016.291553] ? smp_apic_timer_interrupt+0xa6/0x270 [11016.296205] ? apic_timer_interrupt+0xf/0x20 [11016.300057] [11016.300823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11016.307762] do_raw_spin_lock+0x177/0x190 [11016.313399] _raw_spin_lock+0x21/0x30 [11016.319378] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11016.334657] ? do_raw_spin_unlock+0x75/0x190 [11016.348858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11016.363299] ? do_wait_intr+0xf0/0xf0 [11016.367013] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11016.372826] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11016.374847] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11016.385931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.399514] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11016.413016] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11016.433444] mdc_intent_lock+0x353/0x7e0 [mdc] [11016.435720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.444525] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11016.450016] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11016.453078] lmv_intent_open+0x338/0x1150 [lmv] [11016.457554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.461584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.466836] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11016.469039] ll_intent_lock+0xc5/0xaa0 [lustre] [11016.471287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.473765] ll_lookup_it+0xe0e/0x2270 [lustre] [11016.475853] ? do_raw_spin_unlock+0x75/0x190 [11016.477838] ? _raw_spin_unlock+0x12/0x30 [11016.479910] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11016.482304] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11016.487865] ll_atomic_open+0x36a/0xd70 [lustre] [11016.493354] lookup_open+0x356/0xb40 [11016.495466] do_last+0x313/0xfc0 [11016.501068] ? nd_jump_root+0xe5/0x160 [11016.502925] ? path_init+0x437/0x520 [11016.504217] path_openat+0xf7/0x500 [11016.509862] ? do_raw_spin_unlock+0x75/0x190 [11016.514296] do_filp_open+0x99/0x140 [11016.516288] ? getname_flags+0x6e/0x330 [11016.518195] ? __check_object_size+0xff/0x256 [11016.523471] ? do_raw_spin_unlock+0x75/0x190 [11016.525958] ? _raw_spin_unlock+0x12/0x30 [11016.527765] do_sys_openat2+0x2b4/0x410 [11016.529644] do_sys_open+0x73/0xa0 [11016.531235] __x64_sys_openat+0x24/0x30 [11016.533216] do_syscall_64+0xc1/0x440 [11016.534886] entry_SYSCALL_64_after_hwframe+0x49/0xae [11016.537549] RIP: 0033:0x7fa749552332 [11016.539345] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11016.547876] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11016.552798] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11016.557570] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11016.560149] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11016.562839] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11016.566176] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11020.070134] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11020.079291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11020.130092] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11020.144054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11020.151722] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11020.155947] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11020.159067] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11020.174644] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11020.184694] RAX: 0000000000004d87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11020.207504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11020.215914] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11020.226105] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11020.229759] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11020.238240] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11020.249946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11020.251855] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11020.264133] Call Trace: [11020.265080] [11020.265856] ? show_regs.cold.9+0x22/0x2f [11020.272829] ? watchdog_timer_fn.cold.6+0x62/0x10a [11020.275697] ? __hrtimer_run_queues+0x1c2/0x4d0 [11020.282236] ? watchdog+0x40/0x40 [11020.286069] ? hrtimer_interrupt+0x128/0x2c0 [11020.289360] ? smp_apic_timer_interrupt+0xa6/0x270 [11020.294905] ? apic_timer_interrupt+0xf/0x20 [11020.297191] [11020.302823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11020.312542] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11020.314530] do_raw_spin_lock+0x177/0x190 [11020.323771] _raw_spin_lock+0x21/0x30 [11020.325799] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11020.328897] process_one_work+0x2c8/0x700 [11020.331117] worker_thread+0x296/0x6e0 [11020.333065] ? rescuer_thread+0x570/0x570 [11020.335669] kthread+0x1d1/0x200 [11020.337462] ? set_kthread_struct+0x70/0x70 [11020.339906] ret_from_fork+0x1f/0x30 [11022.307141] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9105s! [11022.312900] Showing busy workqueues and worker pools: [11022.316840] workqueue events: flags=0x0 [11022.320972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11022.323371] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11022.340495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11022.345580] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11022.353369] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.356293] pending: kfree_rcu_monitor [11022.361649] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11022.367608] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11022.380069] workqueue events_long: flags=0x0 [11022.384334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.390951] pending: lu_objects_flush [obdclass] [11022.393899] workqueue events_unbound: flags=0x2 [11022.396263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11022.399301] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11022.403716] workqueue events_power_efficient: flags=0x80 [11022.406234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.411163] pending: check_lifetime [11022.417354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.422059] pending: neigh_periodic_work [11022.425186] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.432943] pending: neigh_periodic_work [11022.440471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.443952] pending: do_cache_clean [11022.449533] workqueue mm_percpu_wq: flags=0x8 [11022.452837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.457732] pending: vmstat_update [11022.461776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.467876] pending: vmstat_update [11022.469364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11022.476344] pending: vmstat_update [11022.477715] workqueue writeback: flags=0x4a [11022.480885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11022.485374] pending: wb_workfn, wb_workfn [11022.490373] workqueue xprtiod: flags=0xa [11022.492050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11022.499119] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11022.502268] workqueue ipv6_addrconf: flags=0x40008 [11022.506854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11022.513277] pending: addrconf_verify_work [11022.518144] workqueue knbd0-recv: flags=0x1a [11022.523812] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11022.526732] in-flight: 651:recv_work [11022.530324] workqueue ptlrpc_pinger: flags=0x2 [11022.536881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11022.547317] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11022.559018] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9105s workers=4 idle: 37 31846 28897 [11022.567523] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11012s workers=2 idle: 653 [11024.059181] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11024.063252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11024.110122] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11024.124213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11024.136549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11024.140681] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11024.167785] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11024.171719] RAX: 0000000000007f14 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11024.175216] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11024.178758] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11024.182365] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11024.185877] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11024.201682] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11024.210355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11024.215972] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11024.226440] Call Trace: [11024.227320] [11024.227933] ? show_regs.cold.9+0x22/0x2f [11024.231161] ? watchdog_timer_fn.cold.6+0x62/0x10a [11024.242570] ? __hrtimer_run_queues+0x1c2/0x4d0 [11024.247377] ? watchdog+0x40/0x40 [11024.250988] ? hrtimer_interrupt+0x128/0x2c0 [11024.255239] ? smp_apic_timer_interrupt+0xa6/0x270 [11024.260499] ? apic_timer_interrupt+0xf/0x20 [11024.264095] [11024.267155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11024.276223] do_raw_spin_lock+0x177/0x190 [11024.281107] _raw_spin_lock+0x21/0x30 [11024.284664] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11024.290717] ? do_raw_spin_unlock+0x75/0x190 [11024.295708] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11024.298712] ? do_wait_intr+0xf0/0xf0 [11024.302674] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11024.307739] mdc_close+0x53b/0xd70 [mdc] [11024.311739] lmv_close+0x18d/0x490 [lmv] [11024.316804] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11024.318867] ll_md_real_close+0xa5/0x2a0 [lustre] [11024.328610] ll_md_close+0x274/0x800 [lustre] [11024.332006] ll_file_release+0x128/0xc50 [lustre] [11024.336825] __fput+0x10d/0x440 [11024.339222] ____fput+0x12/0x20 [11024.342669] task_work_run+0xc6/0x110 [11024.350664] exit_to_usermode_loop+0x1dd/0x1f0 [11024.352355] do_syscall_64+0x3d6/0x440 [11024.353828] entry_SYSCALL_64_after_hwframe+0x49/0xae [11024.355964] RIP: 0033:0x7fa749552cd8 [11024.357397] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11024.364597] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11024.367502] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11024.369959] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11024.376380] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11024.381833] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11024.384513] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11028.058012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11028.064825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11028.114461] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11028.118813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11028.130407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11028.137771] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11028.152856] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11028.159751] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11028.164670] RDX: 000000000000127b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11028.171227] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11028.178016] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11028.184967] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11028.196939] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11028.200767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11028.204084] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11028.208610] Call Trace: [11028.209806] [11028.211056] ? show_regs.cold.9+0x22/0x2f [11028.214097] ? watchdog_timer_fn.cold.6+0x62/0x10a [11028.222153] ? __hrtimer_run_queues+0x1c2/0x4d0 [11028.227799] ? watchdog+0x40/0x40 [11028.229386] ? hrtimer_interrupt+0x128/0x2c0 [11028.239679] ? smp_apic_timer_interrupt+0xa6/0x270 [11028.243634] ? apic_timer_interrupt+0xf/0x20 [11028.252100] [11028.256714] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11028.261920] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11028.272167] do_raw_spin_lock+0x177/0x190 [11028.273480] _raw_spin_lock+0x21/0x30 [11028.274549] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11028.281973] obd_mod_free+0x46/0x140 [ptlrpc] [11028.285443] mdc_commit_open+0x91/0xb0 [mdc] [11028.294801] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11028.302514] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11028.309118] after_reply+0xe2c/0x1780 [ptlrpc] [11028.315604] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11028.321284] ? __next_timer_interrupt+0x160/0x160 [11028.326226] ptlrpcd+0xa82/0xe40 [ptlrpc] [11028.331721] ? do_wait_intr+0xf0/0xf0 [11028.336562] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11028.343976] kthread+0x1d1/0x200 [11028.346578] ? set_kthread_struct+0x70/0x70 [11028.350071] ret_from_fork+0x1f/0x30 [11048.054063] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11048.057100] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11048.058491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 [11048.062075] Modules linked in: [11048.062077] pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [11048.075727] lustre(O) [11048.078253] dm_mod [11048.091848] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11048.091922] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11048.091929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11048.091935] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11048.092282] RIP: 0010:native_safe_halt+0x2e/0x50 [11048.092291] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11048.092296] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11048.092300] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11048.092303] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11048.092305] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11048.092307] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11048.092309] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11048.092312] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11048.092315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11048.092317] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11048.092327] Call Trace: [11048.092332] [11048.092336] ? show_regs.cold.9+0x22/0x2f [11048.095517] sha512_ssse3 sha512_generic [11048.095532] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11048.095539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11048.095541] RIP: 0010:native_safe_halt+0x2e/0x50 [11048.095556] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11048.095560] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11048.095565] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11048.095567] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11048.095569] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11048.095571] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11048.095573] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11048.095577] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11048.095580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11048.095582] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11048.095591] Call Trace: [11048.095596] [11048.095601] ? show_regs.cold.9+0x22/0x2f [11048.095611] ? watchdog_timer_fn.cold.6+0x62/0x10a [11048.095617] ? __hrtimer_run_queues+0x1c2/0x4d0 [11048.095621] ? watchdog+0x40/0x40 [11048.095625] ? hrtimer_interrupt+0x128/0x2c0 [11048.095629] ? smp_apic_timer_interrupt+0xa6/0x270 [11048.095637] ? apic_timer_interrupt+0xf/0x20 [11048.095642] [11048.095645] ? native_safe_halt+0x2e/0x50 [11048.095649] kvm_wait+0xcd/0xe0 [11048.095657] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11048.095667] do_raw_spin_lock+0x177/0x190 [11048.095673] _raw_spin_lock+0x21/0x30 [11048.095678] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11048.097441] ? watchdog_timer_fn.cold.6+0x62/0x10a [11048.125504] ? do_raw_spin_unlock+0x75/0x190 [11048.130628] ? __hrtimer_run_queues+0x1c2/0x4d0 [11048.135373] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11048.137691] ? watchdog+0x40/0x40 [11048.140496] ? do_wait_intr+0xf0/0xf0 [11048.140520] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11048.140938] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11048.154345] ? hrtimer_interrupt+0x128/0x2c0 [11048.154364] ? smp_apic_timer_interrupt+0xa6/0x270 [11048.154373] ? apic_timer_interrupt+0xf/0x20 [11048.154378] [11048.154383] ? native_safe_halt+0x2e/0x50 [11048.154387] kvm_wait+0xcd/0xe0 [11048.154410] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11048.154420] do_raw_spin_lock+0x177/0x190 [11048.154424] _raw_spin_lock+0x21/0x30 [11048.154429] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11048.160716] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11048.162914] process_one_work+0x2c8/0x700 [11048.169099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11048.169545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11048.169586] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11048.169619] mdc_intent_lock+0x353/0x7e0 [mdc] [11048.169650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11048.169736] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11048.170264] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11048.178178] worker_thread+0x296/0x6e0 [11048.181272] lmv_intent_open+0x338/0x1150 [lmv] [11048.192569] ? rescuer_thread+0x570/0x570 [11048.196008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11048.196133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11048.196218] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11048.196248] ll_intent_lock+0xc5/0xaa0 [lustre] [11048.196474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11048.196555] ll_lookup_it+0xe0e/0x2270 [lustre] [11048.196635] ? do_raw_spin_unlock+0x75/0x190 [11048.196644] ? _raw_spin_unlock+0x12/0x30 [11048.196651] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11048.196677] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11048.196694] ll_atomic_open+0x36a/0xd70 [lustre] [11048.196774] lookup_open+0x356/0xb40 [11048.196783] do_last+0x313/0xfc0 [11048.196788] ? nd_jump_root+0xe5/0x160 [11048.196792] ? path_init+0x437/0x520 [11048.196796] path_openat+0xf7/0x500 [11048.196801] ? do_raw_spin_unlock+0x75/0x190 [11048.196805] do_filp_open+0x99/0x140 [11048.196811] ? getname_flags+0x6e/0x330 [11048.196815] ? __check_object_size+0xff/0x256 [11048.196822] ? do_raw_spin_unlock+0x75/0x190 [11048.196825] ? _raw_spin_unlock+0x12/0x30 [11048.196829] do_sys_openat2+0x2b4/0x410 [11048.196833] do_sys_open+0x73/0xa0 [11048.196836] __x64_sys_openat+0x24/0x30 [11048.196840] do_syscall_64+0xc1/0x440 [11048.196846] entry_SYSCALL_64_after_hwframe+0x49/0xae [11048.196853] RIP: 0033:0x7fa749552332 [11048.196860] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11048.196864] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11048.196869] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11048.196871] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11048.196873] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11048.196875] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11048.196877] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11048.661756] kthread+0x1d1/0x200 [11048.662998] ? set_kthread_struct+0x70/0x70 [11048.664479] ret_from_fork+0x1f/0x30 [11052.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11052.069656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11052.111498] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11052.117579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11052.122662] RIP: 0010:native_safe_halt+0x2e/0x50 [11052.124732] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11052.135791] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11052.139505] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11052.142801] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11052.146056] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11052.149271] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11052.152774] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11052.155687] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11052.159578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11052.162130] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11052.165831] Call Trace: [11052.167041] [11052.168279] ? show_regs.cold.9+0x22/0x2f [11052.171018] ? watchdog_timer_fn.cold.6+0x62/0x10a [11052.173776] ? __hrtimer_run_queues+0x1c2/0x4d0 [11052.175263] ? watchdog+0x40/0x40 [11052.176297] ? hrtimer_interrupt+0x128/0x2c0 [11052.178147] ? smp_apic_timer_interrupt+0xa6/0x270 [11052.180040] ? apic_timer_interrupt+0xf/0x20 [11052.183770] [11052.184955] ? native_safe_halt+0x2e/0x50 [11052.186773] kvm_wait+0xcd/0xe0 [11052.188469] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11052.191506] do_raw_spin_lock+0x177/0x190 [11052.193136] _raw_spin_lock+0x21/0x30 [11052.194912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11052.197363] ? do_raw_spin_unlock+0x75/0x190 [11052.199979] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11052.203007] ? do_wait_intr+0xf0/0xf0 [11052.204727] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11052.207286] mdc_close+0x53b/0xd70 [mdc] [11052.208799] lmv_close+0x18d/0x490 [lmv] [11052.210514] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11052.213559] ll_md_real_close+0xa5/0x2a0 [lustre] [11052.216901] ll_md_close+0x274/0x800 [lustre] [11052.218890] ll_file_release+0x128/0xc50 [lustre] [11052.220894] __fput+0x10d/0x440 [11052.222346] ____fput+0x12/0x20 [11052.228514] task_work_run+0xc6/0x110 [11052.229874] exit_to_usermode_loop+0x1dd/0x1f0 [11052.247607] do_syscall_64+0x3d6/0x440 [11052.250940] entry_SYSCALL_64_after_hwframe+0x49/0xae [11052.263990] RIP: 0033:0x7fa749552cd8 [11052.272181] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11052.282080] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11052.289280] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11052.297206] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11052.300601] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11052.305495] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11052.309834] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11053.028800] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9135s! [11053.038575] Showing busy workqueues and worker pools: [11053.041651] workqueue events: flags=0x0 [11053.047757] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11053.057887] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11053.071625] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11053.076803] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11053.082919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.087679] pending: kfree_rcu_monitor [11053.090112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11053.093797] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11053.099120] workqueue events_long: flags=0x0 [11053.101589] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.105122] pending: lu_objects_flush [obdclass] [11053.108569] workqueue events_unbound: flags=0x2 [11053.111286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11053.114594] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11053.122991] workqueue events_power_efficient: flags=0x80 [11053.126821] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.130472] pending: check_lifetime [11053.131991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.134993] pending: neigh_periodic_work [11053.136559] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.138707] pending: neigh_periodic_work [11053.141737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.144416] pending: do_cache_clean [11053.146494] workqueue mm_percpu_wq: flags=0x8 [11053.147700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.150423] pending: vmstat_update [11053.151557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.154136] pending: vmstat_update [11053.156044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11053.159238] pending: vmstat_update [11053.161333] workqueue writeback: flags=0x4a [11053.163322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11053.166426] pending: wb_workfn, wb_workfn [11053.168871] workqueue xprtiod: flags=0xa [11053.170747] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11053.173652] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11053.176768] workqueue ipv6_addrconf: flags=0x40008 [11053.178333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11053.180410] pending: addrconf_verify_work [11053.181887] workqueue knbd0-recv: flags=0x1a [11053.183250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11053.185617] in-flight: 651:recv_work [11053.186915] workqueue ptlrpc_pinger: flags=0x2 [11053.188487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11053.190556] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11053.193366] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9136s workers=4 idle: 37 31846 28897 [11053.196621] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11043s workers=2 idle: 653 [11056.058292] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11056.064048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11056.101519] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11056.110128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11056.115833] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11056.119303] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11056.127811] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11056.131465] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11056.138382] RDX: 0000000000007342 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11056.141416] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11056.145897] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11056.149490] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11056.153317] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11056.157529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11056.167488] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11056.176010] Call Trace: [11056.177207] [11056.179578] ? show_regs.cold.9+0x22/0x2f [11056.185174] ? watchdog_timer_fn.cold.6+0x62/0x10a [11056.193675] ? __hrtimer_run_queues+0x1c2/0x4d0 [11056.199614] ? watchdog+0x40/0x40 [11056.202295] ? hrtimer_interrupt+0x128/0x2c0 [11056.209007] ? smp_apic_timer_interrupt+0xa6/0x270 [11056.213874] ? apic_timer_interrupt+0xf/0x20 [11056.219884] [11056.220848] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11056.225235] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11056.234227] do_raw_spin_lock+0x177/0x190 [11056.244440] _raw_spin_lock+0x21/0x30 [11056.246952] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11056.252646] obd_mod_free+0x46/0x140 [ptlrpc] [11056.257242] mdc_commit_open+0x91/0xb0 [mdc] [11056.261798] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11056.264637] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11056.268615] after_reply+0xe2c/0x1780 [ptlrpc] [11056.274007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11056.282794] ? __next_timer_interrupt+0x160/0x160 [11056.284562] ptlrpcd+0xa82/0xe40 [ptlrpc] [11056.287030] ? do_wait_intr+0xf0/0xf0 [11056.288775] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11056.291270] kthread+0x1d1/0x200 [11056.292697] ? set_kthread_struct+0x70/0x70 [11056.295141] ret_from_fork+0x1f/0x30 [11076.058641] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11076.067932] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11076.114283] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11076.130266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11076.139849] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11076.146562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11076.152922] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11076.163750] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11076.167492] RAX: 0000000000006c8c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11076.171404] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11076.174934] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11076.178867] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11076.183003] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11076.187475] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11076.192071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11076.194822] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11076.198205] Call Trace: [11076.199363] [11076.200531] ? show_regs.cold.9+0x22/0x2f [11076.203052] ? watchdog_timer_fn.cold.6+0x62/0x10a [11076.205346] ? __hrtimer_run_queues+0x1c2/0x4d0 [11076.207955] ? watchdog+0x40/0x40 [11076.209550] ? hrtimer_interrupt+0x128/0x2c0 [11076.211928] ? smp_apic_timer_interrupt+0xa6/0x270 [11076.214288] ? apic_timer_interrupt+0xf/0x20 [11076.216553] [11076.217838] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11076.220846] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11076.223513] do_raw_spin_lock+0x177/0x190 [11076.225444] _raw_spin_lock+0x21/0x30 [11076.227344] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11076.230743] process_one_work+0x2c8/0x700 [11076.232959] worker_thread+0x296/0x6e0 [11076.234883] ? rescuer_thread+0x570/0x570 [11076.236863] kthread+0x1d1/0x200 [11076.238921] ? set_kthread_struct+0x70/0x70 [11076.241016] ret_from_fork+0x1f/0x30 [11080.061372] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11080.063628] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11080.064259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11080.064330] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11080.064337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11080.064340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11080.064353] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11080.064358] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11080.064362] RAX: 0000000000006f85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11080.064365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11080.064367] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11080.064369] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11080.064371] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11080.064374] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11080.064377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11080.064379] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11080.064388] Call Trace: [11080.064395] [11080.064401] ? show_regs.cold.9+0x22/0x2f [11080.064412] ? watchdog_timer_fn.cold.6+0x62/0x10a [11080.064419] ? __hrtimer_run_queues+0x1c2/0x4d0 [11080.064423] ? watchdog+0x40/0x40 [11080.064427] ? hrtimer_interrupt+0x128/0x2c0 [11080.064431] ? smp_apic_timer_interrupt+0xa6/0x270 [11080.064440] ? apic_timer_interrupt+0xf/0x20 [11080.064445] [11080.064448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11080.064454] do_raw_spin_lock+0x177/0x190 [11080.064459] _raw_spin_lock+0x21/0x30 [11080.064464] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11080.064740] ? do_raw_spin_unlock+0x75/0x190 [11080.064746] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11080.071112] ? do_wait_intr+0xf0/0xf0 [11080.071125] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11080.071367] mdc_close+0x53b/0xd70 [mdc] [11080.071404] lmv_close+0x18d/0x490 [lmv] [11080.071439] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11080.071566] ll_md_real_close+0xa5/0x2a0 [lustre] [11080.071667] ll_md_close+0x274/0x800 [lustre] [11080.071753] ll_file_release+0x128/0xc50 [lustre] [11080.071861] __fput+0x10d/0x440 [11080.071870] ____fput+0x12/0x20 [11080.071874] task_work_run+0xc6/0x110 [11080.071881] exit_to_usermode_loop+0x1dd/0x1f0 [11080.071887] do_syscall_64+0x3d6/0x440 [11080.071891] entry_SYSCALL_64_after_hwframe+0x49/0xae [11080.071900] RIP: 0033:0x7fa749552cd8 [11080.071908] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11080.071912] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11080.071918] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11080.071920] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11080.071922] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11080.071925] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11080.071927] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11080.416364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11080.434840] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11080.439153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11080.442777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11080.445107] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11080.451913] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11080.454517] RAX: 000000000000320e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11080.457073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11080.459832] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11080.462418] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11080.465397] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11080.468262] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11080.471384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11080.477007] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11080.479487] Call Trace: [11080.482014] [11080.482981] ? show_regs.cold.9+0x22/0x2f [11080.485323] ? watchdog_timer_fn.cold.6+0x62/0x10a [11080.487197] ? __hrtimer_run_queues+0x1c2/0x4d0 [11080.502906] ? watchdog+0x40/0x40 [11080.504779] ? hrtimer_interrupt+0x128/0x2c0 [11080.508347] ? smp_apic_timer_interrupt+0xa6/0x270 [11080.515016] ? apic_timer_interrupt+0xf/0x20 [11080.520462] [11080.522190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11080.525966] do_raw_spin_lock+0x177/0x190 [11080.528110] _raw_spin_lock+0x21/0x30 [11080.530064] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11080.534932] ? do_raw_spin_unlock+0x75/0x190 [11080.539024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11080.544029] ? do_wait_intr+0xf0/0xf0 [11080.545393] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11080.561934] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11080.568851] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11080.576977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11080.581708] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11080.586859] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11080.593696] mdc_intent_lock+0x353/0x7e0 [mdc] [11080.596310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11080.599868] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11080.603765] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11080.608926] lmv_intent_open+0x338/0x1150 [lmv] [11080.611299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11080.614360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11080.618035] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11080.620758] ll_intent_lock+0xc5/0xaa0 [lustre] [11080.623528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11080.626560] ll_lookup_it+0xe0e/0x2270 [lustre] [11080.629114] ? do_raw_spin_unlock+0x75/0x190 [11080.634840] ? _raw_spin_unlock+0x12/0x30 [11080.637006] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11080.645001] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11080.649303] ll_atomic_open+0x36a/0xd70 [lustre] [11080.652426] lookup_open+0x356/0xb40 [11080.660465] do_last+0x313/0xfc0 [11080.662436] ? nd_jump_root+0xe5/0x160 [11080.664388] ? path_init+0x437/0x520 [11080.666817] path_openat+0xf7/0x500 [11080.669405] ? do_raw_spin_unlock+0x75/0x190 [11080.671915] do_filp_open+0x99/0x140 [11080.673208] ? getname_flags+0x6e/0x330 [11080.677591] ? __check_object_size+0xff/0x256 [11080.681132] ? do_raw_spin_unlock+0x75/0x190 [11080.682935] ? _raw_spin_unlock+0x12/0x30 [11080.684413] do_sys_openat2+0x2b4/0x410 [11080.686137] do_sys_open+0x73/0xa0 [11080.691583] __x64_sys_openat+0x24/0x30 [11080.693004] do_syscall_64+0xc1/0x440 [11080.699179] entry_SYSCALL_64_after_hwframe+0x49/0xae [11080.704284] RIP: 0033:0x7fa749552332 [11080.707056] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11080.723008] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11080.730792] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11080.734455] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11080.741924] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11080.750243] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11080.755242] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11083.238122] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9166s! [11083.245206] Showing busy workqueues and worker pools: [11083.246846] workqueue events: flags=0x0 [11083.249269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11083.255980] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11083.264886] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11083.272694] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11083.285360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.289556] pending: kfree_rcu_monitor [11083.291433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11083.294815] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11083.299020] workqueue events_long: flags=0x0 [11083.306597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.311561] pending: lu_objects_flush [obdclass] [11083.314421] workqueue events_unbound: flags=0x2 [11083.317909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11083.320586] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11083.328129] workqueue events_power_efficient: flags=0x80 [11083.331458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.334170] pending: check_lifetime [11083.336051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.338796] pending: neigh_periodic_work [11083.340829] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.345275] pending: neigh_periodic_work [11083.347279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.350411] pending: do_cache_clean [11083.353193] workqueue mm_percpu_wq: flags=0x8 [11083.359355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.361585] pending: vmstat_update [11083.366500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.375426] pending: vmstat_update [11083.378013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11083.385101] pending: vmstat_update [11083.388111] workqueue writeback: flags=0x4a [11083.389538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11083.393193] pending: wb_workfn, wb_workfn [11083.395993] workqueue xprtiod: flags=0xa [11083.398133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11083.400576] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11083.403480] workqueue ipv6_addrconf: flags=0x40008 [11083.405518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11083.408677] pending: addrconf_verify_work [11083.410024] workqueue knbd0-recv: flags=0x1a [11083.413843] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11083.417132] in-flight: 651:recv_work [11083.418889] workqueue ptlrpc_pinger: flags=0x2 [11083.422075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11083.424630] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11083.431651] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9166s workers=4 idle: 37 31846 28897 [11083.435628] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11073s workers=2 idle: 653 [11084.058018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11084.061953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11084.088403] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11084.094096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11084.099262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11084.103128] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11084.115279] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11084.122149] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11084.126971] RDX: 0000000000007044 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11084.129501] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11084.134754] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11084.139680] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11084.144256] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11084.149696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11084.152531] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11084.160772] Call Trace: [11084.162018] [11084.163173] ? show_regs.cold.9+0x22/0x2f [11084.165102] ? watchdog_timer_fn.cold.6+0x62/0x10a [11084.167179] ? __hrtimer_run_queues+0x1c2/0x4d0 [11084.170053] ? watchdog+0x40/0x40 [11084.171632] ? hrtimer_interrupt+0x128/0x2c0 [11084.174012] ? smp_apic_timer_interrupt+0xa6/0x270 [11084.176130] ? apic_timer_interrupt+0xf/0x20 [11084.178008] [11084.179020] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11084.181019] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11084.183337] do_raw_spin_lock+0x177/0x190 [11084.185198] _raw_spin_lock+0x21/0x30 [11084.187112] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11084.189465] obd_mod_free+0x46/0x140 [ptlrpc] [11084.193966] mdc_commit_open+0x91/0xb0 [mdc] [11084.196746] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11084.202844] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11084.208012] after_reply+0xe2c/0x1780 [ptlrpc] [11084.211995] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11084.215807] ? __next_timer_interrupt+0x160/0x160 [11084.219852] ptlrpcd+0xa82/0xe40 [ptlrpc] [11084.221684] ? do_wait_intr+0xf0/0xf0 [11084.225855] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11084.230425] kthread+0x1d1/0x200 [11084.232222] ? set_kthread_struct+0x70/0x70 [11084.236112] ret_from_fork+0x1f/0x30 [11104.057552] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11104.062829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11104.091420] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11104.098810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11104.104939] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11104.109878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11104.113815] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11104.127715] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11104.132479] RAX: 0000000000004fd4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11104.135982] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11104.139703] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11104.142894] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11104.147219] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11104.150440] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11104.155320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11104.158299] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11104.161623] Call Trace: [11104.163380] [11104.164219] ? show_regs.cold.9+0x22/0x2f [11104.166459] ? watchdog_timer_fn.cold.6+0x62/0x10a [11104.168499] ? __hrtimer_run_queues+0x1c2/0x4d0 [11104.170921] ? watchdog+0x40/0x40 [11104.172577] ? hrtimer_interrupt+0x128/0x2c0 [11104.174905] ? smp_apic_timer_interrupt+0xa6/0x270 [11104.177693] ? apic_timer_interrupt+0xf/0x20 [11104.180187] [11104.181399] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11104.184891] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11104.188137] do_raw_spin_lock+0x177/0x190 [11104.190235] _raw_spin_lock+0x21/0x30 [11104.192139] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11104.194685] process_one_work+0x2c8/0x700 [11104.196964] worker_thread+0x296/0x6e0 [11104.199568] ? rescuer_thread+0x570/0x570 [11104.200809] kthread+0x1d1/0x200 [11104.203631] ? set_kthread_struct+0x70/0x70 [11104.206432] ret_from_fork+0x1f/0x30 [11108.054031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11108.057206] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11108.059029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11108.076184] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11108.076200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11108.076203] RIP: 0010:native_safe_halt+0x2e/0x50 [11108.076217] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11108.076221] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11108.076227] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11108.076229] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11108.076232] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11108.076234] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11108.076236] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11108.076240] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11108.076243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11108.076245] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11108.076254] Call Trace: [11108.076259] [11108.076265] ? show_regs.cold.9+0x22/0x2f [11108.076273] ? watchdog_timer_fn.cold.6+0x62/0x10a [11108.076279] ? __hrtimer_run_queues+0x1c2/0x4d0 [11108.076283] ? watchdog+0x40/0x40 [11108.076287] ? hrtimer_interrupt+0x128/0x2c0 [11108.076290] ? smp_apic_timer_interrupt+0xa6/0x270 [11108.076297] ? apic_timer_interrupt+0xf/0x20 [11108.076301] [11108.076303] ? native_safe_halt+0x2e/0x50 [11108.076307] kvm_wait+0xcd/0xe0 [11108.076315] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11108.084512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11108.084587] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11108.084595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11108.084598] RIP: 0010:native_safe_halt+0x2e/0x50 [11108.084610] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11108.085137] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 [11108.090074] do_raw_spin_lock+0x177/0x190 [11108.090105] _raw_spin_lock+0x21/0x30 [11108.090113] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11108.090640] ? do_raw_spin_unlock+0x75/0x190 [11108.090647] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11108.090932] ? do_wait_intr+0xf0/0xf0 [11108.090942] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11108.101403] ORIG_RAX: ffffffffffffff13 [11108.103310] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11108.121485] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11108.121494] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11108.121497] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11108.121499] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11108.121501] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11108.121505] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11108.121508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11108.121511] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11108.121520] Call Trace: [11108.121527] [11108.121533] ? show_regs.cold.9+0x22/0x2f [11108.121545] ? watchdog_timer_fn.cold.6+0x62/0x10a [11108.121552] ? __hrtimer_run_queues+0x1c2/0x4d0 [11108.121556] ? watchdog+0x40/0x40 [11108.121561] ? hrtimer_interrupt+0x128/0x2c0 [11108.121565] ? smp_apic_timer_interrupt+0xa6/0x270 [11108.121574] ? apic_timer_interrupt+0xf/0x20 [11108.121578] [11108.121581] ? native_safe_halt+0x2e/0x50 [11108.121586] kvm_wait+0xcd/0xe0 [11108.121593] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11108.121603] do_raw_spin_lock+0x177/0x190 [11108.121607] _raw_spin_lock+0x21/0x30 [11108.121611] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11108.124983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11108.134672] ? do_raw_spin_unlock+0x75/0x190 [11108.134694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11108.137473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11108.144790] ? do_wait_intr+0xf0/0xf0 [11108.144812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11108.149581] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11108.154808] mdc_close+0x53b/0xd70 [mdc] [11108.158435] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11108.158492] mdc_intent_lock+0x353/0x7e0 [mdc] [11108.166792] lmv_close+0x18d/0x490 [lmv] [11108.166836] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11108.166952] ll_md_real_close+0xa5/0x2a0 [lustre] [11108.167042] ll_md_close+0x274/0x800 [lustre] [11108.172537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11108.172753] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11108.174429] ll_file_release+0x128/0xc50 [lustre] [11108.174587] __fput+0x10d/0x440 [11108.174604] ____fput+0x12/0x20 [11108.174609] task_work_run+0xc6/0x110 [11108.174618] exit_to_usermode_loop+0x1dd/0x1f0 [11108.174626] do_syscall_64+0x3d6/0x440 [11108.174631] entry_SYSCALL_64_after_hwframe+0x49/0xae [11108.174640] RIP: 0033:0x7fa749552cd8 [11108.174649] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11108.174653] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11108.176953] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11108.177015] lmv_intent_open+0x338/0x1150 [lmv] [11108.177078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11108.178872] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11108.180874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11108.184469] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11108.184478] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11108.184481] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11108.184483] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11108.396465] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11108.402013] ll_intent_lock+0xc5/0xaa0 [lustre] [11108.404479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11108.409021] ll_lookup_it+0xe0e/0x2270 [lustre] [11108.411642] ? do_raw_spin_unlock+0x75/0x190 [11108.413942] ? _raw_spin_unlock+0x12/0x30 [11108.415853] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11108.418133] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11108.420961] ll_atomic_open+0x36a/0xd70 [lustre] [11108.423636] lookup_open+0x356/0xb40 [11108.425135] do_last+0x313/0xfc0 [11108.427586] ? nd_jump_root+0xe5/0x160 [11108.429744] ? path_init+0x437/0x520 [11108.431869] path_openat+0xf7/0x500 [11108.433489] ? do_raw_spin_unlock+0x75/0x190 [11108.436227] do_filp_open+0x99/0x140 [11108.438145] ? getname_flags+0x6e/0x330 [11108.440396] ? __check_object_size+0xff/0x256 [11108.442465] ? do_raw_spin_unlock+0x75/0x190 [11108.444736] ? _raw_spin_unlock+0x12/0x30 [11108.446534] do_sys_openat2+0x2b4/0x410 [11108.448944] do_sys_open+0x73/0xa0 [11108.450686] __x64_sys_openat+0x24/0x30 [11108.452453] do_syscall_64+0xc1/0x440 [11108.454263] entry_SYSCALL_64_after_hwframe+0x49/0xae [11108.456569] RIP: 0033:0x7fa749552332 [11108.458099] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11108.466512] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11108.469582] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11108.472929] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11108.476595] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11108.480195] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11108.483730] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11112.058644] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11112.063743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11112.090934] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11112.095284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11112.098980] RIP: 0010:native_safe_halt+0x2e/0x50 [11112.100950] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11112.108428] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11112.111451] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11112.114653] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11112.117535] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11112.120352] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11112.123447] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11112.126249] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11112.129633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11112.132091] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11112.135016] Call Trace: [11112.136083] [11112.136862] ? show_regs.cold.9+0x22/0x2f [11112.138447] ? watchdog_timer_fn.cold.6+0x62/0x10a [11112.140364] ? __hrtimer_run_queues+0x1c2/0x4d0 [11112.142597] ? watchdog+0x40/0x40 [11112.144068] ? hrtimer_interrupt+0x128/0x2c0 [11112.145736] ? smp_apic_timer_interrupt+0xa6/0x270 [11112.147813] ? apic_timer_interrupt+0xf/0x20 [11112.149575] [11112.150561] ? native_safe_halt+0x2e/0x50 [11112.152254] kvm_wait+0xcd/0xe0 [11112.153663] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11112.155635] do_raw_spin_lock+0x177/0x190 [11112.157471] _raw_spin_lock+0x21/0x30 [11112.158946] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11112.161341] obd_mod_free+0x46/0x140 [ptlrpc] [11112.163512] mdc_commit_open+0x91/0xb0 [mdc] [11112.165320] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11112.167805] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11112.170427] after_reply+0xe2c/0x1780 [ptlrpc] [11112.172625] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11112.174975] ? __next_timer_interrupt+0x160/0x160 [11112.177091] ptlrpcd+0xa82/0xe40 [ptlrpc] [11112.179360] ? do_wait_intr+0xf0/0xf0 [11112.180895] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11112.184571] kthread+0x1d1/0x200 [11112.186058] ? set_kthread_struct+0x70/0x70 [11112.187785] ret_from_fork+0x1f/0x30 [11113.442438] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9196s! [11113.447890] Showing busy workqueues and worker pools: [11113.450514] workqueue events: flags=0x0 [11113.454682] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11113.460040] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11113.483049] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11113.487153] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11113.506458] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.519925] pending: kfree_rcu_monitor [11113.532386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11113.541523] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11113.554122] workqueue events_long: flags=0x0 [11113.564446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.574732] pending: lu_objects_flush [obdclass] [11113.578107] workqueue events_unbound: flags=0x2 [11113.579716] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11113.587466] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11113.601204] workqueue events_power_efficient: flags=0x80 [11113.605722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.608744] pending: check_lifetime [11113.610099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.624567] pending: neigh_periodic_work [11113.627086] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.635697] pending: neigh_periodic_work [11113.637770] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.651614] pending: do_cache_clean [11113.654350] workqueue mm_percpu_wq: flags=0x8 [11113.657376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.660810] pending: vmstat_update [11113.663436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.665803] pending: vmstat_update [11113.667635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11113.670611] pending: vmstat_update [11113.672293] workqueue writeback: flags=0x4a [11113.674927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11113.677645] pending: wb_workfn, wb_workfn [11113.679782] workqueue xprtiod: flags=0xa [11113.681802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11113.685048] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11113.688925] workqueue ipv6_addrconf: flags=0x40008 [11113.691307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11113.694325] pending: addrconf_verify_work [11113.696148] workqueue knbd0-recv: flags=0x1a [11113.698344] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11113.701253] in-flight: 651:recv_work [11113.703015] workqueue ptlrpc_pinger: flags=0x2 [11113.704810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11113.707531] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11113.710261] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9196s workers=4 idle: 37 31846 28897 [11113.712874] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11103s workers=2 idle: 653 [11132.057214] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11132.060403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11132.107255] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11132.121808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11132.128911] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11132.134712] RIP: 0010:native_safe_halt+0x2e/0x50 [11132.143773] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11132.159977] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11132.165008] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11132.172075] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11132.177169] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11132.180870] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11132.184760] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11132.189201] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11132.195290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11132.200757] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11132.204622] Call Trace: [11132.206277] [11132.207736] ? show_regs.cold.9+0x22/0x2f [11132.209684] ? watchdog_timer_fn.cold.6+0x62/0x10a [11132.212065] ? __hrtimer_run_queues+0x1c2/0x4d0 [11132.214716] ? watchdog+0x40/0x40 [11132.216217] ? hrtimer_interrupt+0x128/0x2c0 [11132.219879] ? smp_apic_timer_interrupt+0xa6/0x270 [11132.225071] ? apic_timer_interrupt+0xf/0x20 [11132.229908] [11132.231310] ? native_safe_halt+0x2e/0x50 [11132.236905] kvm_wait+0xcd/0xe0 [11132.239217] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11132.243016] do_raw_spin_lock+0x177/0x190 [11132.244557] _raw_spin_lock+0x21/0x30 [11132.246711] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11132.250977] process_one_work+0x2c8/0x700 [11132.254690] worker_thread+0x296/0x6e0 [11132.258235] ? rescuer_thread+0x570/0x570 [11132.262290] kthread+0x1d1/0x200 [11132.266199] ? set_kthread_struct+0x70/0x70 [11132.270038] ret_from_fork+0x1f/0x30 [11136.059052] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11136.063638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11136.100549] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11136.107682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11136.112955] RIP: 0010:native_safe_halt+0x2e/0x50 [11136.116297] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11136.128533] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11136.133644] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11136.137997] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11136.146689] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11136.151207] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11136.159689] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11136.165186] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11136.169682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11136.178855] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11136.187298] Call Trace: [11136.188540] [11136.189831] ? show_regs.cold.9+0x22/0x2f [11136.194932] ? watchdog_timer_fn.cold.6+0x62/0x10a [11136.199328] ? __hrtimer_run_queues+0x1c2/0x4d0 [11136.208988] ? watchdog+0x40/0x40 [11136.213496] ? hrtimer_interrupt+0x128/0x2c0 [11136.218554] ? smp_apic_timer_interrupt+0xa6/0x270 [11136.221639] ? apic_timer_interrupt+0xf/0x20 [11136.223946] [11136.225829] ? native_safe_halt+0x2e/0x50 [11136.227626] kvm_wait+0xcd/0xe0 [11136.229516] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11136.232209] do_raw_spin_lock+0x177/0x190 [11136.234789] _raw_spin_lock+0x21/0x30 [11136.239264] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11136.241825] ? do_raw_spin_unlock+0x75/0x190 [11136.245113] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11136.254061] ? do_wait_intr+0xf0/0xf0 [11136.261089] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11136.269873] mdc_close+0x53b/0xd70 [mdc] [11136.274740] lmv_close+0x18d/0x490 [lmv] [11136.278942] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11136.288893] ll_md_real_close+0xa5/0x2a0 [lustre] [11136.296073] ll_md_close+0x274/0x800 [lustre] [11136.301149] ll_file_release+0x128/0xc50 [lustre] [11136.308743] __fput+0x10d/0x440 [11136.310429] ____fput+0x12/0x20 [11136.318500] task_work_run+0xc6/0x110 [11136.320128] exit_to_usermode_loop+0x1dd/0x1f0 [11136.328227] do_syscall_64+0x3d6/0x440 [11136.330329] entry_SYSCALL_64_after_hwframe+0x49/0xae [11136.337459] RIP: 0033:0x7fa749552cd8 [11136.340084] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11136.349546] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11136.353670] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11136.362718] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11136.370548] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11136.375426] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11136.379916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11140.055533] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11140.064805] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11140.066211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11140.067619] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11140.067632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11140.067637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11140.067661] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11140.067668] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11140.067674] RAX: 0000000000003685 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11140.067677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11140.067680] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11140.067683] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11140.067685] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11140.067689] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11140.067693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11140.067696] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11140.067708] Call Trace: [11140.067715] [11140.067721] ? show_regs.cold.9+0x22/0x2f [11140.067732] ? watchdog_timer_fn.cold.6+0x62/0x10a [11140.067738] ? __hrtimer_run_queues+0x1c2/0x4d0 [11140.067743] ? watchdog+0x40/0x40 [11140.067747] ? hrtimer_interrupt+0x128/0x2c0 [11140.067751] ? smp_apic_timer_interrupt+0xa6/0x270 [11140.067769] ? apic_timer_interrupt+0xf/0x20 [11140.067798] [11140.067804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11140.067823] do_raw_spin_lock+0x177/0x190 [11140.067838] _raw_spin_lock+0x21/0x30 [11140.067850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11140.077381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11140.077455] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11140.077463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11140.077465] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11140.077480] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11140.077484] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11140.077488] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11140.077490] RDX: 00000000000023d5 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11140.077492] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11140.077494] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11140.077497] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11140.077500] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11140.077503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11140.077505] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11140.077514] Call Trace: [11140.077520] [11140.077526] ? show_regs.cold.9+0x22/0x2f [11140.077536] ? watchdog_timer_fn.cold.6+0x62/0x10a [11140.077543] ? __hrtimer_run_queues+0x1c2/0x4d0 [11140.077547] ? watchdog+0x40/0x40 [11140.077553] ? hrtimer_interrupt+0x128/0x2c0 [11140.077556] ? smp_apic_timer_interrupt+0xa6/0x270 [11140.077564] ? apic_timer_interrupt+0xf/0x20 [11140.077569] [11140.077572] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11140.077577] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11140.077581] do_raw_spin_lock+0x177/0x190 [11140.077586] _raw_spin_lock+0x21/0x30 [11140.077592] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11140.079368] obd_mod_free+0x46/0x140 [ptlrpc] [11140.164124] ? do_raw_spin_unlock+0x75/0x190 [11140.164155] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11140.177562] mdc_commit_open+0x91/0xb0 [mdc] [11140.177616] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11140.178853] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11140.180164] after_reply+0xe2c/0x1780 [ptlrpc] [11140.181087] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11140.202791] ? do_wait_intr+0xf0/0xf0 [11140.202813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11140.203282] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11140.204055] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11140.212449] ? __next_timer_interrupt+0x160/0x160 [11140.212471] ptlrpcd+0xa82/0xe40 [ptlrpc] [11140.213025] ? do_wait_intr+0xf0/0xf0 [11140.213034] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11140.213423] kthread+0x1d1/0x200 [11140.213430] ? set_kthread_struct+0x70/0x70 [11140.213434] ret_from_fork+0x1f/0x30 [11140.969238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11140.971532] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11140.985998] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11140.996428] mdc_intent_lock+0x353/0x7e0 [mdc] [11141.001077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11141.003397] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11141.022463] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11141.025364] lmv_intent_open+0x338/0x1150 [lmv] [11141.034174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11141.046955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11141.051514] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11141.053254] ll_intent_lock+0xc5/0xaa0 [lustre] [11141.054582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11141.056674] ll_lookup_it+0xe0e/0x2270 [lustre] [11141.063011] ? do_raw_spin_unlock+0x75/0x190 [11141.064225] ? _raw_spin_unlock+0x12/0x30 [11141.065979] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11141.068654] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11141.071004] ll_atomic_open+0x36a/0xd70 [lustre] [11141.073078] lookup_open+0x356/0xb40 [11141.074637] do_last+0x313/0xfc0 [11141.076058] ? nd_jump_root+0xe5/0x160 [11141.077694] ? path_init+0x437/0x520 [11141.079620] path_openat+0xf7/0x500 [11141.080946] ? do_raw_spin_unlock+0x75/0x190 [11141.082345] do_filp_open+0x99/0x140 [11141.083926] ? getname_flags+0x6e/0x330 [11141.085254] ? __check_object_size+0xff/0x256 [11141.086888] ? do_raw_spin_unlock+0x75/0x190 [11141.088849] ? _raw_spin_unlock+0x12/0x30 [11141.092744] do_sys_openat2+0x2b4/0x410 [11141.095664] do_sys_open+0x73/0xa0 [11141.101007] __x64_sys_openat+0x24/0x30 [11141.104862] do_syscall_64+0xc1/0x440 [11141.106564] entry_SYSCALL_64_after_hwframe+0x49/0xae [11141.111898] RIP: 0033:0x7fa749552332 [11141.114953] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11141.126089] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11141.128889] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11141.132057] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11141.135245] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11141.138194] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11141.141236] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11144.166540] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9227s! [11144.169648] Showing busy workqueues and worker pools: [11144.173110] workqueue events: flags=0x0 [11144.175028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11144.181247] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11144.186645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11144.193021] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11144.196773] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.200055] pending: kfree_rcu_monitor [11144.205296] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11144.209973] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11144.213311] workqueue events_long: flags=0x0 [11144.214953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.217684] pending: lu_objects_flush [obdclass] [11144.220225] workqueue events_unbound: flags=0x2 [11144.221960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11144.223976] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11144.227914] workqueue events_power_efficient: flags=0x80 [11144.229392] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.233679] pending: check_lifetime [11144.235210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.237277] pending: neigh_periodic_work [11144.239109] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.241222] pending: neigh_periodic_work [11144.245830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.252099] pending: do_cache_clean [11144.255410] workqueue mm_percpu_wq: flags=0x8 [11144.261200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.268651] pending: vmstat_update [11144.273568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.282018] pending: vmstat_update [11144.285670] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11144.290925] pending: vmstat_update [11144.295254] workqueue writeback: flags=0x4a [11144.297645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11144.300125] pending: wb_workfn, wb_workfn [11144.304981] workqueue xprtiod: flags=0xa [11144.308658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11144.310482] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11144.315310] workqueue ipv6_addrconf: flags=0x40008 [11144.321588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11144.325784] pending: addrconf_verify_work [11144.327646] workqueue knbd0-recv: flags=0x1a [11144.329206] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11144.332127] in-flight: 651:recv_work [11144.336131] workqueue ptlrpc_pinger: flags=0x2 [11144.340287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11144.343221] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11144.345797] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9227s workers=4 idle: 37 31846 28897 [11144.350125] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11134s workers=2 idle: 653 [11153.885509] rcu: INFO: rcu_sched self-detected stall on CPU [11153.888846] rcu: 2-...!: (5210129 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [11153.894624] (t=9240196 jiffies g=258701 q=4156) [11153.897884] rcu: rcu_sched kthread starved for 9236991 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [11153.904411] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11153.909732] rcu: RCU grace-period kthread stack dump: [11153.912411] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11153.917078] Call Trace: [11153.918098] __schedule+0x351/0xcb0 [11153.919649] schedule+0xc0/0x180 [11153.920866] schedule_timeout+0xb4/0x190 [11153.922646] ? do_raw_spin_unlock+0x75/0x190 [11153.923878] ? __next_timer_interrupt+0x160/0x160 [11153.925448] ? prepare_to_swait_event+0xf3/0x140 [11153.926849] rcu_gp_kthread+0x8dd/0x15b0 [11153.932224] ? force_qs_rnp+0x2c0/0x2c0 [11153.933687] kthread+0x1d1/0x200 [11153.935381] ? set_kthread_struct+0x70/0x70 [11153.940228] ret_from_fork+0x1f/0x30 [11153.941245] rcu: Stack dump where RCU GP kthread last ran: [11153.946532] Sending NMI from CPU 2 to CPUs 1: [11153.954866] NMI backtrace for cpu 1 [11153.954875] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11153.954877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11153.954879] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11153.954882] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11153.954884] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11153.954886] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 [11153.954888] RAX: 0000000000007d89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11153.954890] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11153.954892] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11153.954893] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11153.954894] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11153.954896] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11153.954897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11153.954899] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11153.954900] Call Trace: [11153.954901] [11153.954903] ? show_regs.cold.9+0x22/0x2f [11153.954904] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11153.954905] ? nmi_cpu_backtrace_handler+0x15/0x40 [11153.954906] ? nmi_handle+0xbd/0x210 [11153.954907] ? default_do_nmi+0x4b/0x1b0 [11153.954908] ? do_nmi+0x280/0x380 [11153.954909] ? end_repeat_nmi+0x16/0x1f [11153.954910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11153.954911] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11153.954912] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11153.954913] [11153.954914] do_raw_spin_lock+0x177/0x190 [11153.954915] _raw_spin_lock+0x21/0x30 [11153.954916] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11153.954917] process_one_work+0x2c8/0x700 [11153.954918] worker_thread+0x296/0x6e0 [11153.954919] ? rescuer_thread+0x570/0x570 [11153.954921] kthread+0x1d1/0x200 [11153.954921] ? set_kthread_struct+0x70/0x70 [11153.954923] ret_from_fork+0x1f/0x30 [11153.955453] NMI backtrace for cpu 2 [11154.091825] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11154.097391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11154.106088] Call Trace: [11154.106914] [11154.110722] ? dump_stack+0xbb/0x10e [11154.114835] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11154.117084] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11154.121994] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11154.125760] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11154.128870] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11154.131484] ? rcu_dump_cpu_stacks+0x11b/0x171 [11154.136503] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11154.139415] ? update_process_times+0x6d/0xc0 [11154.141555] ? tick_sched_handle.isra.18+0x27/0x90 [11154.143666] ? tick_sched_timer+0x53/0xb0 [11154.148751] ? __hrtimer_run_queues+0x1c2/0x4d0 [11154.152277] ? tick_sched_handle.isra.18+0x90/0x90 [11154.157579] ? hrtimer_interrupt+0x128/0x2c0 [11154.163383] ? smp_apic_timer_interrupt+0xa6/0x270 [11154.166596] ? apic_timer_interrupt+0xf/0x20 [11154.168411] [11154.169266] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11154.171763] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11154.175764] ? do_raw_spin_lock+0x177/0x190 [11154.177494] ? _raw_spin_lock+0x21/0x30 [11154.178871] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11154.181376] ? obd_mod_free+0x46/0x140 [ptlrpc] [11154.187191] ? mdc_commit_open+0x91/0xb0 [mdc] [11154.191796] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11154.194260] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11154.196748] ? after_reply+0xe2c/0x1780 [ptlrpc] [11154.198880] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11154.204018] ? __next_timer_interrupt+0x160/0x160 [11154.205774] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11154.209831] ? do_wait_intr+0xf0/0xf0 [11154.211431] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11154.214040] ? kthread+0x1d1/0x200 [11154.215430] ? set_kthread_struct+0x70/0x70 [11154.217510] ? ret_from_fork+0x1f/0x30 [11164.059724] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11164.068807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11164.098468] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11164.110864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11164.113172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11164.129058] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11164.138227] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11164.142463] RAX: 0000000000005804 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11164.149716] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11164.154169] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11164.157682] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11164.160986] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11164.168307] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11164.174058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11164.180314] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11164.186042] Call Trace: [11164.186820] [11164.187351] ? show_regs.cold.9+0x22/0x2f [11164.188929] ? watchdog_timer_fn.cold.6+0x62/0x10a [11164.195665] ? __hrtimer_run_queues+0x1c2/0x4d0 [11164.197853] ? watchdog+0x40/0x40 [11164.199041] ? hrtimer_interrupt+0x128/0x2c0 [11164.207305] ? smp_apic_timer_interrupt+0xa6/0x270 [11164.218468] ? apic_timer_interrupt+0xf/0x20 [11164.221331] [11164.222193] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11164.227889] do_raw_spin_lock+0x177/0x190 [11164.230046] _raw_spin_lock+0x21/0x30 [11164.231093] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11164.235183] ? do_raw_spin_unlock+0x75/0x190 [11164.237514] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11164.239798] ? do_wait_intr+0xf0/0xf0 [11164.241697] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11164.244405] mdc_close+0x53b/0xd70 [mdc] [11164.246495] lmv_close+0x18d/0x490 [lmv] [11164.248796] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11164.250691] ll_md_real_close+0xa5/0x2a0 [lustre] [11164.252147] ll_md_close+0x274/0x800 [lustre] [11164.257449] ll_file_release+0x128/0xc50 [lustre] [11164.259296] __fput+0x10d/0x440 [11164.261792] ____fput+0x12/0x20 [11164.265463] task_work_run+0xc6/0x110 [11164.266647] exit_to_usermode_loop+0x1dd/0x1f0 [11164.270174] do_syscall_64+0x3d6/0x440 [11164.272573] entry_SYSCALL_64_after_hwframe+0x49/0xae [11164.274519] RIP: 0033:0x7fa749552cd8 [11164.275727] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11164.288782] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11164.292773] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11164.296191] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11164.301242] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11164.302924] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11164.305443] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11172.054109] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11172.068751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11172.103571] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11172.112894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11172.120419] RIP: 0010:native_safe_halt+0x2e/0x50 [11172.124331] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11172.137396] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11172.140483] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11172.143677] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11172.146864] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11172.149880] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11172.153298] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11172.156505] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11172.160384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11172.164077] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11172.168776] Call Trace: [11172.170205] [11172.171556] ? show_regs.cold.9+0x22/0x2f [11172.174298] ? watchdog_timer_fn.cold.6+0x62/0x10a [11172.177395] ? __hrtimer_run_queues+0x1c2/0x4d0 [11172.179800] ? watchdog+0x40/0x40 [11172.183184] ? hrtimer_interrupt+0x128/0x2c0 [11172.185533] ? smp_apic_timer_interrupt+0xa6/0x270 [11172.188502] ? apic_timer_interrupt+0xf/0x20 [11172.190542] [11172.191422] ? native_safe_halt+0x2e/0x50 [11172.193551] kvm_wait+0xcd/0xe0 [11172.198524] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11172.205790] do_raw_spin_lock+0x177/0x190 [11172.209097] _raw_spin_lock+0x21/0x30 [11172.211607] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11172.215318] ? do_raw_spin_unlock+0x75/0x190 [11172.217927] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11172.221320] ? do_wait_intr+0xf0/0xf0 [11172.222940] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11172.225967] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11172.228424] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11172.235378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11172.237689] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11172.240367] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11172.243946] mdc_intent_lock+0x353/0x7e0 [mdc] [11172.245925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11172.248762] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11172.251427] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11172.254152] lmv_intent_open+0x338/0x1150 [lmv] [11172.256357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11172.258451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11172.261242] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11172.263334] ll_intent_lock+0xc5/0xaa0 [lustre] [11172.265611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11172.268039] ll_lookup_it+0xe0e/0x2270 [lustre] [11172.270310] ? do_raw_spin_unlock+0x75/0x190 [11172.271969] ? _raw_spin_unlock+0x12/0x30 [11172.273736] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11172.276236] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11172.278461] ll_atomic_open+0x36a/0xd70 [lustre] [11172.280794] lookup_open+0x356/0xb40 [11172.282518] do_last+0x313/0xfc0 [11172.283896] ? nd_jump_root+0xe5/0x160 [11172.285544] ? path_init+0x437/0x520 [11172.287276] path_openat+0xf7/0x500 [11172.288980] ? do_raw_spin_unlock+0x75/0x190 [11172.290985] do_filp_open+0x99/0x140 [11172.293183] ? getname_flags+0x6e/0x330 [11172.294861] ? __check_object_size+0xff/0x256 [11172.296925] ? do_raw_spin_unlock+0x75/0x190 [11172.298993] ? _raw_spin_unlock+0x12/0x30 [11172.300763] do_sys_openat2+0x2b4/0x410 [11172.302419] do_sys_open+0x73/0xa0 [11172.303762] __x64_sys_openat+0x24/0x30 [11172.305311] do_syscall_64+0xc1/0x440 [11172.306906] entry_SYSCALL_64_after_hwframe+0x49/0xae [11172.308667] RIP: 0033:0x7fa749552332 [11172.310328] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11172.319050] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11172.322289] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11172.324971] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11172.328237] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11172.331420] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11172.334520] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11174.378296] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9257s! [11174.382920] Showing busy workqueues and worker pools: [11174.385621] workqueue events: flags=0x0 [11174.387124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11174.389906] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11174.395233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11174.397972] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11174.402132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.404751] pending: kfree_rcu_monitor [11174.406433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11174.409816] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11174.414190] workqueue events_long: flags=0x0 [11174.416952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.419713] pending: lu_objects_flush [obdclass] [11174.422073] workqueue events_unbound: flags=0x2 [11174.424689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11174.427610] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11174.431869] workqueue events_power_efficient: flags=0x80 [11174.434524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.437741] pending: check_lifetime [11174.439294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.442289] pending: neigh_periodic_work [11174.444739] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.447643] pending: neigh_periodic_work [11174.449325] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.452795] pending: do_cache_clean [11174.454783] workqueue mm_percpu_wq: flags=0x8 [11174.456995] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.460600] pending: vmstat_update [11174.462737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.466724] pending: vmstat_update [11174.468683] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11174.471820] pending: vmstat_update [11174.473959] workqueue writeback: flags=0x4a [11174.476149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11174.479673] pending: wb_workfn, wb_workfn [11174.482135] workqueue xprtiod: flags=0xa [11174.484018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11174.487796] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11174.491253] workqueue ipv6_addrconf: flags=0x40008 [11174.493671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11174.496948] pending: addrconf_verify_work [11174.499128] workqueue knbd0-recv: flags=0x1a [11174.501340] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11174.504256] in-flight: 651:recv_work [11174.505985] workqueue ptlrpc_pinger: flags=0x2 [11174.508107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11174.510770] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11174.513283] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9257s workers=4 idle: 37 31846 28897 [11174.516496] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11164s workers=2 idle: 653 [11180.060378] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11180.062538] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11180.064564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11180.064655] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11180.064662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11180.064666] RIP: 0010:native_safe_halt+0x2e/0x50 [11180.064679] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11180.064683] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11180.064688] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11180.064691] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11180.064693] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11180.064695] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11180.064697] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11180.064701] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11180.064704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11180.064706] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11180.064715] Call Trace: [11180.064720] [11180.064724] ? show_regs.cold.9+0x22/0x2f [11180.064734] ? watchdog_timer_fn.cold.6+0x62/0x10a [11180.064740] ? __hrtimer_run_queues+0x1c2/0x4d0 [11180.064745] ? watchdog+0x40/0x40 [11180.064749] ? hrtimer_interrupt+0x128/0x2c0 [11180.064753] ? smp_apic_timer_interrupt+0xa6/0x270 [11180.064760] ? apic_timer_interrupt+0xf/0x20 [11180.064765] [11180.064767] ? native_safe_halt+0x2e/0x50 [11180.064771] kvm_wait+0xcd/0xe0 [11180.064779] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11180.064786] do_raw_spin_lock+0x177/0x190 [11180.064791] _raw_spin_lock+0x21/0x30 [11180.064796] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11180.074367] Modules linked in: lustre(O) [11180.105647] obd_mod_free+0x46/0x140 [ptlrpc] [11180.122242] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11180.122316] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11180.122322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11180.122328] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11180.128622] mdc_commit_open+0x91/0xb0 [mdc] [11180.128671] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11180.132007] [11180.132012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11180.132027] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11180.132032] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11180.132038] RAX: 0000000000007e13 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11180.132040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11180.132043] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11180.132045] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11180.132047] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11180.132050] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11180.132053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11180.132055] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11180.132064] Call Trace: [11180.132069] [11180.132074] ? show_regs.cold.9+0x22/0x2f [11180.132096] ? watchdog_timer_fn.cold.6+0x62/0x10a [11180.132104] ? __hrtimer_run_queues+0x1c2/0x4d0 [11180.132108] ? watchdog+0x40/0x40 [11180.132113] ? hrtimer_interrupt+0x128/0x2c0 [11180.132117] ? smp_apic_timer_interrupt+0xa6/0x270 [11180.132124] ? apic_timer_interrupt+0xf/0x20 [11180.132129] [11180.132132] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11180.132137] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11180.132143] do_raw_spin_lock+0x177/0x190 [11180.132148] _raw_spin_lock+0x21/0x30 [11180.132153] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11180.153578] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11180.162100] process_one_work+0x2c8/0x700 [11180.169609] after_reply+0xe2c/0x1780 [ptlrpc] [11180.169903] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11180.178145] worker_thread+0x296/0x6e0 [11180.182690] ? __next_timer_interrupt+0x160/0x160 [11180.182711] ptlrpcd+0xa82/0xe40 [ptlrpc] [11180.183276] ? do_wait_intr+0xf0/0xf0 [11180.183287] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11180.183489] kthread+0x1d1/0x200 [11180.183495] ? set_kthread_struct+0x70/0x70 [11180.183499] ret_from_fork+0x1f/0x30 [11180.507320] ? rescuer_thread+0x570/0x570 [11180.508585] kthread+0x1d1/0x200 [11180.511646] ? set_kthread_struct+0x70/0x70 [11180.516136] ret_from_fork+0x1f/0x30 [11192.059244] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11192.069845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11192.096486] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11192.101576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11192.105062] RIP: 0010:native_safe_halt+0x2e/0x50 [11192.107525] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11192.116428] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11192.120821] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11192.124110] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11192.126668] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11192.129274] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11192.132344] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11192.135241] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11192.138241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11192.140250] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11192.143057] Call Trace: [11192.143856] [11192.144435] ? show_regs.cold.9+0x22/0x2f [11192.145834] ? watchdog_timer_fn.cold.6+0x62/0x10a [11192.147267] ? __hrtimer_run_queues+0x1c2/0x4d0 [11192.148881] ? watchdog+0x40/0x40 [11192.150072] ? hrtimer_interrupt+0x128/0x2c0 [11192.151435] ? smp_apic_timer_interrupt+0xa6/0x270 [11192.153060] ? apic_timer_interrupt+0xf/0x20 [11192.154356] [11192.154983] ? native_safe_halt+0x2e/0x50 [11192.156366] kvm_wait+0xcd/0xe0 [11192.157242] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11192.158515] do_raw_spin_lock+0x177/0x190 [11192.159823] _raw_spin_lock+0x21/0x30 [11192.160752] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11192.162480] ? do_raw_spin_unlock+0x75/0x190 [11192.163525] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11192.165447] ? do_wait_intr+0xf0/0xf0 [11192.166383] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11192.168926] mdc_close+0x53b/0xd70 [mdc] [11192.170358] lmv_close+0x18d/0x490 [lmv] [11192.171517] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11192.173254] ll_md_real_close+0xa5/0x2a0 [lustre] [11192.174561] ll_md_close+0x274/0x800 [lustre] [11192.176109] ll_file_release+0x128/0xc50 [lustre] [11192.177854] __fput+0x10d/0x440 [11192.178973] ____fput+0x12/0x20 [11192.180086] task_work_run+0xc6/0x110 [11192.181156] exit_to_usermode_loop+0x1dd/0x1f0 [11192.182188] do_syscall_64+0x3d6/0x440 [11192.183061] entry_SYSCALL_64_after_hwframe+0x49/0xae [11192.184430] RIP: 0033:0x7fa749552cd8 [11192.185320] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11192.191381] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11192.193483] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11192.195383] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11192.197626] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11192.200169] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11192.202217] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11200.054023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11200.056789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11200.104708] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11200.116287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11200.125980] RIP: 0010:native_safe_halt+0x2e/0x50 [11200.130295] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11200.147932] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11200.150609] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11200.161163] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11200.170204] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11200.176286] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11200.182195] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11200.192860] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11200.205270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11200.209189] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11200.223609] Call Trace: [11200.228111] [11200.230392] ? show_regs.cold.9+0x22/0x2f [11200.237097] ? watchdog_timer_fn.cold.6+0x62/0x10a [11200.244719] ? __hrtimer_run_queues+0x1c2/0x4d0 [11200.251217] ? watchdog+0x40/0x40 [11200.255259] ? hrtimer_interrupt+0x128/0x2c0 [11200.258266] ? smp_apic_timer_interrupt+0xa6/0x270 [11200.261634] ? apic_timer_interrupt+0xf/0x20 [11200.263764] [11200.265252] ? native_safe_halt+0x2e/0x50 [11200.272591] kvm_wait+0xcd/0xe0 [11200.273740] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11200.284259] do_raw_spin_lock+0x177/0x190 [11200.296919] _raw_spin_lock+0x21/0x30 [11200.304085] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11200.315045] ? do_raw_spin_unlock+0x75/0x190 [11200.319320] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11200.328235] ? do_wait_intr+0xf0/0xf0 [11200.332374] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11200.334439] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11200.338648] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11200.351168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11200.354407] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11200.361050] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11200.365859] mdc_intent_lock+0x353/0x7e0 [mdc] [11200.367415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11200.373345] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11200.385100] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11200.394171] lmv_intent_open+0x338/0x1150 [lmv] [11200.398250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11200.403971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11200.408203] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11200.413260] ll_intent_lock+0xc5/0xaa0 [lustre] [11200.415817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11200.422216] ll_lookup_it+0xe0e/0x2270 [lustre] [11200.425552] ? do_raw_spin_unlock+0x75/0x190 [11200.427986] ? _raw_spin_unlock+0x12/0x30 [11200.430015] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11200.432223] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11200.435608] ll_atomic_open+0x36a/0xd70 [lustre] [11200.438807] lookup_open+0x356/0xb40 [11200.442758] do_last+0x313/0xfc0 [11200.444594] ? nd_jump_root+0xe5/0x160 [11200.447291] ? path_init+0x437/0x520 [11200.448598] path_openat+0xf7/0x500 [11200.450299] ? do_raw_spin_unlock+0x75/0x190 [11200.455044] do_filp_open+0x99/0x140 [11200.460291] ? getname_flags+0x6e/0x330 [11200.463531] ? __check_object_size+0xff/0x256 [11200.472380] ? do_raw_spin_unlock+0x75/0x190 [11200.474145] ? _raw_spin_unlock+0x12/0x30 [11200.486862] do_sys_openat2+0x2b4/0x410 [11200.488706] do_sys_open+0x73/0xa0 [11200.494669] __x64_sys_openat+0x24/0x30 [11200.499068] do_syscall_64+0xc1/0x440 [11200.506607] entry_SYSCALL_64_after_hwframe+0x49/0xae [11200.509474] RIP: 0033:0x7fa749552332 [11200.510708] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11200.525913] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11200.536838] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11200.545921] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11200.554017] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11200.561901] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11200.567417] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11204.586866] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9287s! [11204.590017] Showing busy workqueues and worker pools: [11204.592221] workqueue events: flags=0x0 [11204.594639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11204.599242] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11204.616730] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11204.626929] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11204.637323] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.647553] pending: kfree_rcu_monitor [11204.651592] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11204.659376] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11204.668538] workqueue events_long: flags=0x0 [11204.673609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.677702] pending: lu_objects_flush [obdclass] [11204.680965] workqueue events_unbound: flags=0x2 [11204.683354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11204.687236] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11204.697128] workqueue events_power_efficient: flags=0x80 [11204.700845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.703333] pending: check_lifetime [11204.705518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.709004] pending: neigh_periodic_work [11204.714438] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.719506] pending: neigh_periodic_work [11204.721678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.729639] pending: do_cache_clean [11204.735980] workqueue mm_percpu_wq: flags=0x8 [11204.739335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.743052] pending: vmstat_update [11204.744918] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.752509] pending: vmstat_update [11204.756048] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11204.760627] pending: vmstat_update [11204.762668] workqueue writeback: flags=0x4a [11204.765852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11204.772664] pending: wb_workfn, wb_workfn [11204.775114] workqueue xprtiod: flags=0xa [11204.777873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11204.782124] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11204.788574] workqueue ipv6_addrconf: flags=0x40008 [11204.796498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11204.800544] pending: addrconf_verify_work [11204.803244] workqueue knbd0-recv: flags=0x1a [11204.808789] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11204.817635] in-flight: 651:recv_work [11204.820204] workqueue ptlrpc_pinger: flags=0x2 [11204.823085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11204.826912] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11204.830121] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9287s workers=4 idle: 37 31846 28897 [11204.837820] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11194s workers=2 idle: 653 [11208.057719] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11208.058462] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11208.063825] Modules linked in: [11208.069427] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11208.069501] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11208.069509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11208.069512] RIP: 0010:native_safe_halt+0x2e/0x50 [11208.069525] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11208.069529] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11208.069534] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11208.069536] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11208.069538] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11208.069541] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11208.069543] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11208.069546] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11208.069549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11208.069552] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11208.069561] Call Trace: [11208.069567] [11208.069572] ? show_regs.cold.9+0x22/0x2f [11208.069583] ? watchdog_timer_fn.cold.6+0x62/0x10a [11208.069662] ? __hrtimer_run_queues+0x1c2/0x4d0 [11208.069666] ? watchdog+0x40/0x40 [11208.069670] ? hrtimer_interrupt+0x128/0x2c0 [11208.069674] ? smp_apic_timer_interrupt+0xa6/0x270 [11208.069681] ? apic_timer_interrupt+0xf/0x20 [11208.069686] [11208.069690] ? native_safe_halt+0x2e/0x50 [11208.069694] kvm_wait+0xcd/0xe0 [11208.069703] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11208.069710] do_raw_spin_lock+0x177/0x190 [11208.069714] _raw_spin_lock+0x21/0x30 [11208.069719] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11208.070458] obd_mod_free+0x46/0x140 [ptlrpc] [11208.071506] mdc_commit_open+0x91/0xb0 [mdc] [11208.071543] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11208.071766] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11208.073001] lustre(O) [11208.111184] after_reply+0xe2c/0x1780 [ptlrpc] [11208.111453] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11208.112536] ? __next_timer_interrupt+0x160/0x160 [11208.112548] ptlrpcd+0xa82/0xe40 [ptlrpc] [11208.112832] ? do_wait_intr+0xf0/0xf0 [11208.112842] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11208.126528] osp(O) [11208.136839] kthread+0x1d1/0x200 [11208.141816] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11208.141890] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11208.141898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11208.141907] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11208.146041] RIP: 0010:native_safe_halt+0x2e/0x50 [11208.146057] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11208.146062] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11208.146079] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11208.146082] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11208.146085] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11208.146087] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11208.146090] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11208.146093] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11208.146097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11208.146099] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11208.146107] Call Trace: [11208.146113] [11208.146120] ? show_regs.cold.9+0x22/0x2f [11208.146132] ? watchdog_timer_fn.cold.6+0x62/0x10a [11208.146139] ? __hrtimer_run_queues+0x1c2/0x4d0 [11208.146143] ? watchdog+0x40/0x40 [11208.146147] ? hrtimer_interrupt+0x128/0x2c0 [11208.146152] ? smp_apic_timer_interrupt+0xa6/0x270 [11208.146159] ? apic_timer_interrupt+0xf/0x20 [11208.146164] [11208.146167] ? native_safe_halt+0x2e/0x50 [11208.146182] kvm_wait+0xcd/0xe0 [11208.146193] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11208.146203] do_raw_spin_lock+0x177/0x190 [11208.146208] _raw_spin_lock+0x21/0x30 [11208.146212] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11208.146483] process_one_work+0x2c8/0x700 [11208.146490] worker_thread+0x296/0x6e0 [11208.146499] ? rescuer_thread+0x570/0x570 [11208.146503] kthread+0x1d1/0x200 [11208.146509] ? set_kthread_struct+0x70/0x70 [11208.146512] ret_from_fork+0x1f/0x30 [11208.576811] ? set_kthread_struct+0x70/0x70 [11208.581288] ret_from_fork+0x1f/0x30 [11220.059048] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11220.066645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11220.111364] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11220.120079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11220.130465] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11220.132689] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11220.140783] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11220.144065] RAX: 0000000000001d0f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11220.148374] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11220.151294] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11220.154923] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11220.157924] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11220.160573] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11220.164162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11220.166466] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11220.169754] Call Trace: [11220.170928] [11220.171939] ? show_regs.cold.9+0x22/0x2f [11220.173623] ? watchdog_timer_fn.cold.6+0x62/0x10a [11220.175769] ? __hrtimer_run_queues+0x1c2/0x4d0 [11220.178026] ? watchdog+0x40/0x40 [11220.179444] ? hrtimer_interrupt+0x128/0x2c0 [11220.181416] ? smp_apic_timer_interrupt+0xa6/0x270 [11220.183496] ? apic_timer_interrupt+0xf/0x20 [11220.185437] [11220.186285] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11220.188336] do_raw_spin_lock+0x177/0x190 [11220.189889] _raw_spin_lock+0x21/0x30 [11220.191531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11220.193968] ? do_raw_spin_unlock+0x75/0x190 [11220.195791] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11220.198282] ? do_wait_intr+0xf0/0xf0 [11220.199900] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11220.202472] mdc_close+0x53b/0xd70 [mdc] [11220.204433] lmv_close+0x18d/0x490 [lmv] [11220.206362] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11220.208593] ll_md_real_close+0xa5/0x2a0 [lustre] [11220.210956] ll_md_close+0x274/0x800 [lustre] [11220.212733] ll_file_release+0x128/0xc50 [lustre] [11220.215126] __fput+0x10d/0x440 [11220.216464] ____fput+0x12/0x20 [11220.217775] task_work_run+0xc6/0x110 [11220.219843] exit_to_usermode_loop+0x1dd/0x1f0 [11220.221666] do_syscall_64+0x3d6/0x440 [11220.223704] entry_SYSCALL_64_after_hwframe+0x49/0xae [11220.226204] RIP: 0033:0x7fa749552cd8 [11220.228402] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11220.237145] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11220.240467] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11220.243685] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11220.246889] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11220.250490] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11220.253591] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11232.054182] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11232.057897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11232.075691] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11232.080150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11232.084200] RIP: 0010:native_safe_halt+0x2e/0x50 [11232.086627] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11232.096024] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11232.098841] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11232.102491] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11232.106264] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11232.110147] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11232.114426] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11232.116871] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11232.120379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11232.123615] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11232.128540] Call Trace: [11232.129725] [11232.130778] ? show_regs.cold.9+0x22/0x2f [11232.132933] ? watchdog_timer_fn.cold.6+0x62/0x10a [11232.135500] ? __hrtimer_run_queues+0x1c2/0x4d0 [11232.137624] ? watchdog+0x40/0x40 [11232.139767] ? hrtimer_interrupt+0x128/0x2c0 [11232.141918] ? smp_apic_timer_interrupt+0xa6/0x270 [11232.144763] ? apic_timer_interrupt+0xf/0x20 [11232.147094] [11232.148082] ? native_safe_halt+0x2e/0x50 [11232.149938] kvm_wait+0xcd/0xe0 [11232.151300] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11232.153800] do_raw_spin_lock+0x177/0x190 [11232.155403] _raw_spin_lock+0x21/0x30 [11232.157422] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11232.160230] ? do_raw_spin_unlock+0x75/0x190 [11232.162660] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11232.165531] ? do_wait_intr+0xf0/0xf0 [11232.166678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11232.169565] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11232.173483] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11232.177522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11232.179662] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11232.184294] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11232.186570] mdc_intent_lock+0x353/0x7e0 [mdc] [11232.189304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11232.192027] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11232.197337] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11232.200830] lmv_intent_open+0x338/0x1150 [lmv] [11232.203468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11232.206733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11232.211003] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11232.213639] ll_intent_lock+0xc5/0xaa0 [lustre] [11232.218272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11232.221720] ll_lookup_it+0xe0e/0x2270 [lustre] [11232.224986] ? do_raw_spin_unlock+0x75/0x190 [11232.226840] ? _raw_spin_unlock+0x12/0x30 [11232.229849] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11232.233834] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11232.237221] ll_atomic_open+0x36a/0xd70 [lustre] [11232.239062] lookup_open+0x356/0xb40 [11232.240998] do_last+0x313/0xfc0 [11232.242261] ? nd_jump_root+0xe5/0x160 [11232.243814] ? path_init+0x437/0x520 [11232.245455] path_openat+0xf7/0x500 [11232.246806] ? do_raw_spin_unlock+0x75/0x190 [11232.248553] do_filp_open+0x99/0x140 [11232.250201] ? getname_flags+0x6e/0x330 [11232.251949] ? __check_object_size+0xff/0x256 [11232.254009] ? do_raw_spin_unlock+0x75/0x190 [11232.255717] ? _raw_spin_unlock+0x12/0x30 [11232.257511] do_sys_openat2+0x2b4/0x410 [11232.259149] do_sys_open+0x73/0xa0 [11232.260413] __x64_sys_openat+0x24/0x30 [11232.262338] do_syscall_64+0xc1/0x440 [11232.264034] entry_SYSCALL_64_after_hwframe+0x49/0xae [11232.266086] RIP: 0033:0x7fa749552332 [11232.267622] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11232.276266] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11232.280413] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11232.283849] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11232.287336] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11232.290855] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11232.294620] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11235.299649] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9318s! [11235.312677] Showing busy workqueues and worker pools: [11235.318686] workqueue events: flags=0x0 [11235.320887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11235.329876] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11235.339402] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11235.352977] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11235.359269] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.365691] pending: kfree_rcu_monitor [11235.369658] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11235.376066] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11235.383669] workqueue events_long: flags=0x0 [11235.387649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.390374] pending: lu_objects_flush [obdclass] [11235.394469] workqueue events_unbound: flags=0x2 [11235.399260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11235.403479] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11235.410985] workqueue events_power_efficient: flags=0x80 [11235.414654] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.418396] pending: check_lifetime [11235.423597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.429958] pending: neigh_periodic_work [11235.434455] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.448142] pending: neigh_periodic_work [11235.453551] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.463429] pending: do_cache_clean [11235.469959] workqueue mm_percpu_wq: flags=0x8 [11235.474484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.482749] pending: vmstat_update [11235.484190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.501828] pending: vmstat_update [11235.503540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11235.510691] pending: vmstat_update [11235.517291] workqueue writeback: flags=0x4a [11235.523281] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11235.531699] pending: wb_workfn, wb_workfn [11235.542790] workqueue xprtiod: flags=0xa [11235.544505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11235.548573] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11235.565212] workqueue ipv6_addrconf: flags=0x40008 [11235.570394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11235.576752] pending: addrconf_verify_work [11235.586022] workqueue knbd0-recv: flags=0x1a [11235.591770] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11235.602179] in-flight: 651:recv_work [11235.605403] workqueue ptlrpc_pinger: flags=0x2 [11235.614713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11235.623671] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11235.628582] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9318s workers=4 idle: 37 31846 28897 [11235.632727] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11225s workers=2 idle: 653 [11236.061279] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11236.064179] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11236.071625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11236.071716] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11236.071723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11236.071729] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11236.072525] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11236.072537] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11236.072542] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11236.072546] RAX: 0000000000007d0a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11236.072549] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11236.072551] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11236.072553] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11236.072556] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11236.072559] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11236.072562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11236.072564] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11236.072573] Call Trace: [11236.072578] [11236.072584] ? show_regs.cold.9+0x22/0x2f [11236.072593] ? watchdog_timer_fn.cold.6+0x62/0x10a [11236.072599] ? __hrtimer_run_queues+0x1c2/0x4d0 [11236.072604] ? watchdog+0x40/0x40 [11236.072608] ? hrtimer_interrupt+0x128/0x2c0 [11236.072612] ? smp_apic_timer_interrupt+0xa6/0x270 [11236.072620] ? apic_timer_interrupt+0xf/0x20 [11236.072625] [11236.072638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11236.072643] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11236.072648] do_raw_spin_lock+0x177/0x190 [11236.072712] _raw_spin_lock+0x21/0x30 [11236.072717] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11236.072935] process_one_work+0x2c8/0x700 [11236.072942] worker_thread+0x296/0x6e0 [11236.072946] ? rescuer_thread+0x570/0x570 [11236.072950] kthread+0x1d1/0x200 [11236.072954] ? set_kthread_struct+0x70/0x70 [11236.072958] ret_from_fork+0x1f/0x30 [11236.293606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11236.318656] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11236.325350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11236.330672] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11236.333477] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11236.353460] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11236.358350] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11236.363803] RDX: 000000000000716a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11236.368178] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11236.372995] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11236.376373] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11236.380667] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11236.384786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11236.389453] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11236.393733] Call Trace: [11236.394351] [11236.394806] ? show_regs.cold.9+0x22/0x2f [11236.396059] ? watchdog_timer_fn.cold.6+0x62/0x10a [11236.401325] ? __hrtimer_run_queues+0x1c2/0x4d0 [11236.403339] ? watchdog+0x40/0x40 [11236.407294] ? hrtimer_interrupt+0x128/0x2c0 [11236.411024] ? smp_apic_timer_interrupt+0xa6/0x270 [11236.413176] ? apic_timer_interrupt+0xf/0x20 [11236.417362] [11236.419429] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11236.421831] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11236.426075] do_raw_spin_lock+0x177/0x190 [11236.430202] _raw_spin_lock+0x21/0x30 [11236.432355] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11236.439489] obd_mod_free+0x46/0x140 [ptlrpc] [11236.441396] mdc_commit_open+0x91/0xb0 [mdc] [11236.444605] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11236.447588] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11236.450331] after_reply+0xe2c/0x1780 [ptlrpc] [11236.452122] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11236.461012] ? __next_timer_interrupt+0x160/0x160 [11236.463526] ptlrpcd+0xa82/0xe40 [ptlrpc] [11236.467789] ? do_wait_intr+0xf0/0xf0 [11236.470556] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11236.474619] kthread+0x1d1/0x200 [11236.475879] ? set_kthread_struct+0x70/0x70 [11236.478735] ret_from_fork+0x1f/0x30 [11248.059022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11248.064703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11248.101428] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11248.115363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11248.128148] RIP: 0010:native_safe_halt+0x2e/0x50 [11248.131973] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11248.154988] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11248.158521] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11248.161554] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11248.164577] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11248.167744] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11248.170589] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11248.173970] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11248.177407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11248.180012] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11248.183043] Call Trace: [11248.184067] [11248.185367] ? show_regs.cold.9+0x22/0x2f [11248.188856] ? watchdog_timer_fn.cold.6+0x62/0x10a [11248.193569] ? __hrtimer_run_queues+0x1c2/0x4d0 [11248.197072] ? watchdog+0x40/0x40 [11248.201979] ? hrtimer_interrupt+0x128/0x2c0 [11248.203680] ? smp_apic_timer_interrupt+0xa6/0x270 [11248.211950] ? apic_timer_interrupt+0xf/0x20 [11248.213517] [11248.214281] ? native_safe_halt+0x2e/0x50 [11248.217819] kvm_wait+0xcd/0xe0 [11248.219206] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11248.223988] do_raw_spin_lock+0x177/0x190 [11248.226079] _raw_spin_lock+0x21/0x30 [11248.227762] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11248.233075] ? do_raw_spin_unlock+0x75/0x190 [11248.236061] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11248.241370] ? do_wait_intr+0xf0/0xf0 [11248.245244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11248.250305] mdc_close+0x53b/0xd70 [mdc] [11248.252745] lmv_close+0x18d/0x490 [lmv] [11248.254694] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11248.258383] ll_md_real_close+0xa5/0x2a0 [lustre] [11248.260224] ll_md_close+0x274/0x800 [lustre] [11248.262062] ll_file_release+0x128/0xc50 [lustre] [11248.264026] __fput+0x10d/0x440 [11248.265125] ____fput+0x12/0x20 [11248.266532] task_work_run+0xc6/0x110 [11248.268026] exit_to_usermode_loop+0x1dd/0x1f0 [11248.269804] do_syscall_64+0x3d6/0x440 [11248.271267] entry_SYSCALL_64_after_hwframe+0x49/0xae [11248.273674] RIP: 0033:0x7fa749552cd8 [11248.275120] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11248.282311] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11248.285720] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11248.288746] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11248.292414] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11248.296070] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11248.299045] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11260.069468] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11260.074488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11260.111965] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11260.117627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11260.121797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11260.123886] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11260.133965] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11260.137350] RAX: 000000000000330b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11260.139961] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11260.142834] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11260.145875] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11260.159184] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11260.163773] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11260.167094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11260.169395] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11260.172828] Call Trace: [11260.173724] [11260.175841] ? show_regs.cold.9+0x22/0x2f [11260.179942] ? watchdog_timer_fn.cold.6+0x62/0x10a [11260.185545] ? __hrtimer_run_queues+0x1c2/0x4d0 [11260.188194] ? watchdog+0x40/0x40 [11260.189416] ? hrtimer_interrupt+0x128/0x2c0 [11260.194619] ? smp_apic_timer_interrupt+0xa6/0x270 [11260.197618] ? apic_timer_interrupt+0xf/0x20 [11260.202349] [11260.203557] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11260.208959] do_raw_spin_lock+0x177/0x190 [11260.212764] _raw_spin_lock+0x21/0x30 [11260.218955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11260.229863] ? do_raw_spin_unlock+0x75/0x190 [11260.236054] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11260.240738] ? do_wait_intr+0xf0/0xf0 [11260.243056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11260.250598] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11260.258856] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11260.268152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11260.276733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11260.282193] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11260.285875] mdc_intent_lock+0x353/0x7e0 [mdc] [11260.293200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11260.297930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11260.301864] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11260.304839] lmv_intent_open+0x338/0x1150 [lmv] [11260.313302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11260.319572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11260.325243] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11260.329689] ll_intent_lock+0xc5/0xaa0 [lustre] [11260.331907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11260.335836] ll_lookup_it+0xe0e/0x2270 [lustre] [11260.337940] ? do_raw_spin_unlock+0x75/0x190 [11260.342810] ? _raw_spin_unlock+0x12/0x30 [11260.346558] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11260.350623] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11260.353720] ll_atomic_open+0x36a/0xd70 [lustre] [11260.356000] lookup_open+0x356/0xb40 [11260.358333] do_last+0x313/0xfc0 [11260.362901] ? nd_jump_root+0xe5/0x160 [11260.370184] ? path_init+0x437/0x520 [11260.372119] path_openat+0xf7/0x500 [11260.376736] ? do_raw_spin_unlock+0x75/0x190 [11260.380114] do_filp_open+0x99/0x140 [11260.381183] ? getname_flags+0x6e/0x330 [11260.382476] ? __check_object_size+0xff/0x256 [11260.389895] ? do_raw_spin_unlock+0x75/0x190 [11260.392782] ? _raw_spin_unlock+0x12/0x30 [11260.399485] do_sys_openat2+0x2b4/0x410 [11260.405205] do_sys_open+0x73/0xa0 [11260.407048] __x64_sys_openat+0x24/0x30 [11260.408385] do_syscall_64+0xc1/0x440 [11260.413418] entry_SYSCALL_64_after_hwframe+0x49/0xae [11260.415888] RIP: 0033:0x7fa749552332 [11260.419557] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11260.437887] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11260.448937] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11260.458473] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11260.460319] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11260.466180] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11260.472961] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11264.058755] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11264.059227] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11264.062606] Modules linked in: [11264.066238] Modules linked in: [11264.066241] lustre(O) osp(O) ofd(O) [11264.067940] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [11264.069541] lod(O) [11264.071529] fld(O) ptlrpc_gss(O) [11264.078275] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11264.078353] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11264.080503] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [11264.082215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11264.102973] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11264.103016] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11264.103022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11264.103026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11264.112290] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11264.115130] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11264.115147] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11264.115154] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11264.115157] RDX: 0000000000000b60 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11264.115159] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11264.115162] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11264.115164] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11264.115168] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11264.115171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11264.115173] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11264.115182] Call Trace: [11264.115188] [11264.115193] ? show_regs.cold.9+0x22/0x2f [11264.115207] ? watchdog_timer_fn.cold.6+0x62/0x10a [11264.115215] ? __hrtimer_run_queues+0x1c2/0x4d0 [11264.115220] ? watchdog+0x40/0x40 [11264.115224] ? hrtimer_interrupt+0x128/0x2c0 [11264.115228] ? smp_apic_timer_interrupt+0xa6/0x270 [11264.115237] ? apic_timer_interrupt+0xf/0x20 [11264.115242] [11264.115245] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11264.115252] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11264.115257] do_raw_spin_lock+0x177/0x190 [11264.115262] _raw_spin_lock+0x21/0x30 [11264.115269] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11264.120855] [11264.120869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11264.134790] obd_mod_free+0x46/0x140 [ptlrpc] [11264.142208] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11264.142227] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11264.142234] RAX: 0000000000007e83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11264.142236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11264.142239] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11264.142241] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11264.142247] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11264.142253] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11264.142256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11264.142259] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11264.142266] Call Trace: [11264.142277] [11264.142286] ? show_regs.cold.9+0x22/0x2f [11264.142298] ? watchdog_timer_fn.cold.6+0x62/0x10a [11264.142310] ? __hrtimer_run_queues+0x1c2/0x4d0 [11264.142315] ? watchdog+0x40/0x40 [11264.142320] ? hrtimer_interrupt+0x128/0x2c0 [11264.142327] ? smp_apic_timer_interrupt+0xa6/0x270 [11264.142339] ? apic_timer_interrupt+0xf/0x20 [11264.142344] [11264.142347] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11264.142360] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11264.142367] do_raw_spin_lock+0x177/0x190 [11264.142372] _raw_spin_lock+0x21/0x30 [11264.142383] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11264.142723] process_one_work+0x2c8/0x700 [11264.142731] worker_thread+0x296/0x6e0 [11264.142735] ? rescuer_thread+0x570/0x570 [11264.142739] kthread+0x1d1/0x200 [11264.142743] ? set_kthread_struct+0x70/0x70 [11264.142747] ret_from_fork+0x1f/0x30 [11264.434468] mdc_commit_open+0x91/0xb0 [mdc] [11264.435927] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11264.438267] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11264.440739] after_reply+0xe2c/0x1780 [ptlrpc] [11264.443149] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11264.447636] ? __next_timer_interrupt+0x160/0x160 [11264.450951] ptlrpcd+0xa82/0xe40 [ptlrpc] [11264.453693] ? do_wait_intr+0xf0/0xf0 [11264.455154] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11264.457330] kthread+0x1d1/0x200 [11264.458660] ? set_kthread_struct+0x70/0x70 [11264.460259] ret_from_fork+0x1f/0x30 [11266.018036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9348s! [11266.025084] Showing busy workqueues and worker pools: [11266.030892] workqueue events: flags=0x0 [11266.036709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11266.040717] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11266.046449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11266.052966] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11266.061125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.065349] pending: kfree_rcu_monitor [11266.066555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11266.068986] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11266.074349] workqueue events_long: flags=0x0 [11266.077283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.081901] pending: lu_objects_flush [obdclass] [11266.085131] workqueue events_unbound: flags=0x2 [11266.087187] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11266.093184] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11266.101029] workqueue events_power_efficient: flags=0x80 [11266.105851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.121799] pending: check_lifetime [11266.124068] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.127122] pending: neigh_periodic_work [11266.128530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.137721] pending: neigh_periodic_work [11266.140834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.147233] pending: do_cache_clean [11266.150515] workqueue mm_percpu_wq: flags=0x8 [11266.152747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.156268] pending: vmstat_update [11266.160022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.166148] pending: vmstat_update [11266.170124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11266.172435] pending: vmstat_update [11266.176145] workqueue writeback: flags=0x4a [11266.178551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11266.181772] pending: wb_workfn, wb_workfn [11266.187415] workqueue xprtiod: flags=0xa [11266.191211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11266.197256] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11266.203415] workqueue ipv6_addrconf: flags=0x40008 [11266.207703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11266.211024] pending: addrconf_verify_work [11266.213744] workqueue knbd0-recv: flags=0x1a [11266.216986] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11266.220890] in-flight: 651:recv_work [11266.223162] workqueue ptlrpc_pinger: flags=0x2 [11266.227985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11266.231922] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11266.237818] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9349s workers=4 idle: 37 31846 28897 [11266.242975] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11256s workers=2 idle: 653 [11276.060076] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11276.064467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11276.090374] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11276.096551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11276.101572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11276.104609] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11276.114014] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11276.118953] RAX: 0000000000002de1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11276.122859] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11276.126560] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11276.129960] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11276.133851] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11276.137377] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11276.141231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11276.144617] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11276.148171] Call Trace: [11276.149081] [11276.150052] ? show_regs.cold.9+0x22/0x2f [11276.151926] ? watchdog_timer_fn.cold.6+0x62/0x10a [11276.153957] ? __hrtimer_run_queues+0x1c2/0x4d0 [11276.156024] ? watchdog+0x40/0x40 [11276.157451] ? hrtimer_interrupt+0x128/0x2c0 [11276.158976] ? smp_apic_timer_interrupt+0xa6/0x270 [11276.161999] ? apic_timer_interrupt+0xf/0x20 [11276.164505] [11276.165863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11276.168442] do_raw_spin_lock+0x177/0x190 [11276.170264] _raw_spin_lock+0x21/0x30 [11276.172854] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11276.175628] ? do_raw_spin_unlock+0x75/0x190 [11276.177887] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11276.180891] ? do_wait_intr+0xf0/0xf0 [11276.182512] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11276.185871] mdc_close+0x53b/0xd70 [mdc] [11276.188026] lmv_close+0x18d/0x490 [lmv] [11276.190283] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11276.194464] ll_md_real_close+0xa5/0x2a0 [lustre] [11276.197824] ll_md_close+0x274/0x800 [lustre] [11276.200101] ll_file_release+0x128/0xc50 [lustre] [11276.202641] __fput+0x10d/0x440 [11276.203891] ____fput+0x12/0x20 [11276.205089] task_work_run+0xc6/0x110 [11276.206933] exit_to_usermode_loop+0x1dd/0x1f0 [11276.209093] do_syscall_64+0x3d6/0x440 [11276.211539] entry_SYSCALL_64_after_hwframe+0x49/0xae [11276.214443] RIP: 0033:0x7fa749552cd8 [11276.216901] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11276.225642] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11276.228787] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11276.232899] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11276.236156] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11276.239612] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11276.242990] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11292.054011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11292.057025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11292.058034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11292.058047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11292.058137] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11292.058143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11292.058146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11292.058159] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11292.058163] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11292.058168] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11292.058171] RDX: 0000000000006c09 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11292.058173] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11292.058176] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11292.058179] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11292.058182] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11292.058185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11292.058188] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11292.058197] Call Trace: [11292.058200] Modules linked in: lustre(O) [11292.058202] [11292.058209] ? show_regs.cold.9+0x22/0x2f [11292.058224] ? watchdog_timer_fn.cold.6+0x62/0x10a [11292.058232] ? __hrtimer_run_queues+0x1c2/0x4d0 [11292.058236] ? watchdog+0x40/0x40 [11292.058240] ? hrtimer_interrupt+0x128/0x2c0 [11292.058244] ? smp_apic_timer_interrupt+0xa6/0x270 [11292.058252] ? apic_timer_interrupt+0xf/0x20 [11292.058256] [11292.058259] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11292.058264] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11292.058268] do_raw_spin_lock+0x177/0x190 [11292.058273] _raw_spin_lock+0x21/0x30 [11292.058279] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11292.061888] obd_mod_free+0x46/0x140 [ptlrpc] [11292.062094] mdc_commit_open+0x91/0xb0 [mdc] [11292.062141] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11292.062407] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11292.066559] Modules linked in: lustre(O) [11292.066253] after_reply+0xe2c/0x1780 [ptlrpc] [11292.067693] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11292.068279] ? __next_timer_interrupt+0x160/0x160 [11292.068290] ptlrpcd+0xa82/0xe40 [ptlrpc] [11292.068585] ? do_wait_intr+0xf0/0xf0 [11292.068594] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11292.068788] kthread+0x1d1/0x200 [11292.068793] ? set_kthread_struct+0x70/0x70 [11292.068796] ret_from_fork+0x1f/0x30 [11292.073370] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11292.073459] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11292.073467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11292.073469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11292.073535] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11292.073540] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11292.073545] RAX: 0000000000000d84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11292.073547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11292.073549] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11292.073551] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11292.073554] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11292.073556] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11292.073559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11292.073561] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11292.073569] Call Trace: [11292.073574] [11292.073579] ? show_regs.cold.9+0x22/0x2f [11292.073588] ? watchdog_timer_fn.cold.6+0x62/0x10a [11292.073594] ? __hrtimer_run_queues+0x1c2/0x4d0 [11292.073597] ? watchdog+0x40/0x40 [11292.073602] ? hrtimer_interrupt+0x128/0x2c0 [11292.073607] ? smp_apic_timer_interrupt+0xa6/0x270 [11292.073614] ? apic_timer_interrupt+0xf/0x20 [11292.073618] [11292.073621] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11292.073627] do_raw_spin_lock+0x177/0x190 [11292.073631] _raw_spin_lock+0x21/0x30 [11292.073636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11292.127328] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11292.128600] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11292.128608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11292.128614] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11292.129971] RIP: 0010:native_safe_halt+0x2e/0x50 [11292.129984] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11292.129988] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11292.129991] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11292.129994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11292.129996] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11292.129997] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11292.129999] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11292.130002] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11292.130005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11292.130007] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11292.130015] Call Trace: [11292.130020] [11292.130025] ? show_regs.cold.9+0x22/0x2f [11292.130033] ? watchdog_timer_fn.cold.6+0x62/0x10a [11292.130039] ? __hrtimer_run_queues+0x1c2/0x4d0 [11292.130043] ? watchdog+0x40/0x40 [11292.130047] ? hrtimer_interrupt+0x128/0x2c0 [11292.130051] ? smp_apic_timer_interrupt+0xa6/0x270 [11292.130057] ? apic_timer_interrupt+0xf/0x20 [11292.130061] [11292.130065] ? native_safe_halt+0x2e/0x50 [11292.130069] kvm_wait+0xcd/0xe0 [11292.130076] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11292.130083] do_raw_spin_lock+0x177/0x190 [11292.130087] _raw_spin_lock+0x21/0x30 [11292.130091] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11292.130320] process_one_work+0x2c8/0x700 [11292.130327] worker_thread+0x296/0x6e0 [11292.130332] ? rescuer_thread+0x570/0x570 [11292.130336] kthread+0x1d1/0x200 [11292.130340] ? set_kthread_struct+0x70/0x70 [11292.130344] ret_from_fork+0x1f/0x30 [11292.887209] ? do_raw_spin_unlock+0x75/0x190 [11292.896417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11292.907592] ? do_wait_intr+0xf0/0xf0 [11292.910424] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11292.923347] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11292.930522] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11292.942220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11292.952563] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11292.962268] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11292.969358] mdc_intent_lock+0x353/0x7e0 [mdc] [11292.973677] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11292.986385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11292.999450] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11293.010613] lmv_intent_open+0x338/0x1150 [lmv] [11293.014646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11293.024994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11293.031871] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11293.048679] ll_intent_lock+0xc5/0xaa0 [lustre] [11293.054071] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11293.064426] ll_lookup_it+0xe0e/0x2270 [lustre] [11293.073666] ? do_raw_spin_unlock+0x75/0x190 [11293.085037] ? _raw_spin_unlock+0x12/0x30 [11293.087643] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11293.094876] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11293.099510] ll_atomic_open+0x36a/0xd70 [lustre] [11293.103672] lookup_open+0x356/0xb40 [11293.105525] do_last+0x313/0xfc0 [11293.108955] ? nd_jump_root+0xe5/0x160 [11293.114762] ? path_init+0x437/0x520 [11293.117788] path_openat+0xf7/0x500 [11293.119812] ? do_raw_spin_unlock+0x75/0x190 [11293.125722] do_filp_open+0x99/0x140 [11293.128529] ? getname_flags+0x6e/0x330 [11293.133727] ? __check_object_size+0xff/0x256 [11293.140580] ? do_raw_spin_unlock+0x75/0x190 [11293.142686] ? _raw_spin_unlock+0x12/0x30 [11293.149367] do_sys_openat2+0x2b4/0x410 [11293.152354] do_sys_open+0x73/0xa0 [11293.157315] __x64_sys_openat+0x24/0x30 [11293.158822] do_syscall_64+0xc1/0x440 [11293.163638] entry_SYSCALL_64_after_hwframe+0x49/0xae [11293.168452] RIP: 0033:0x7fa749552332 [11293.172478] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11293.192973] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11293.206560] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11293.227293] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11293.247881] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11293.261423] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11293.278571] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11296.737048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9379s! [11296.740125] Showing busy workqueues and worker pools: [11296.743198] workqueue events: flags=0x0 [11296.745395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11296.747840] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11296.758309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11296.764669] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11296.767989] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.783839] pending: kfree_rcu_monitor [11296.791833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11296.797740] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11296.806684] workqueue events_long: flags=0x0 [11296.812179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.825731] pending: lu_objects_flush [obdclass] [11296.832272] workqueue events_unbound: flags=0x2 [11296.834738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11296.839509] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11296.849695] workqueue events_power_efficient: flags=0x80 [11296.858836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.862263] pending: check_lifetime [11296.864144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.869935] pending: neigh_periodic_work [11296.872611] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.876696] pending: neigh_periodic_work [11296.879213] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.883254] pending: do_cache_clean [11296.886155] workqueue mm_percpu_wq: flags=0x8 [11296.888370] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.891232] pending: vmstat_update [11296.893250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.906472] pending: vmstat_update [11296.910831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11296.915329] pending: vmstat_update [11296.921983] workqueue writeback: flags=0x4a [11296.924569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11296.927681] pending: wb_workfn, wb_workfn [11296.929477] workqueue xprtiod: flags=0xa [11296.931741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11296.939982] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11296.952754] workqueue ipv6_addrconf: flags=0x40008 [11296.956470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11296.964243] pending: addrconf_verify_work [11296.970837] workqueue knbd0-recv: flags=0x1a [11296.978393] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11296.986648] in-flight: 651:recv_work [11296.991683] workqueue ptlrpc_pinger: flags=0x2 [11296.995589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11296.999982] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11297.008274] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9379s workers=4 idle: 37 31846 28897 [11297.018232] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11286s workers=2 idle: 653 [11304.060110] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11304.074974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11304.106270] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11304.113289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11304.124962] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11304.139595] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11304.177740] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11304.197595] RAX: 0000000000002408 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11304.210257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11304.220323] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11304.225540] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11304.232876] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11304.249792] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11304.270261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11304.281919] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11304.288860] Call Trace: [11304.291892] [11304.296522] ? show_regs.cold.9+0x22/0x2f [11304.307174] ? watchdog_timer_fn.cold.6+0x62/0x10a [11304.312424] ? __hrtimer_run_queues+0x1c2/0x4d0 [11304.324453] ? watchdog+0x40/0x40 [11304.332193] ? hrtimer_interrupt+0x128/0x2c0 [11304.340234] ? smp_apic_timer_interrupt+0xa6/0x270 [11304.345498] ? apic_timer_interrupt+0xf/0x20 [11304.348902] [11304.349969] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11304.362474] do_raw_spin_lock+0x177/0x190 [11304.366379] _raw_spin_lock+0x21/0x30 [11304.371658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11304.385123] ? do_raw_spin_unlock+0x75/0x190 [11304.394811] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11304.416709] ? do_wait_intr+0xf0/0xf0 [11304.425407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11304.443614] mdc_close+0x53b/0xd70 [mdc] [11304.448315] lmv_close+0x18d/0x490 [lmv] [11304.457428] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11304.471590] ll_md_real_close+0xa5/0x2a0 [lustre] [11304.484561] ll_md_close+0x274/0x800 [lustre] [11304.504782] ll_file_release+0x128/0xc50 [lustre] [11304.524182] __fput+0x10d/0x440 [11304.530479] ____fput+0x12/0x20 [11304.535273] task_work_run+0xc6/0x110 [11304.539696] exit_to_usermode_loop+0x1dd/0x1f0 [11304.550265] do_syscall_64+0x3d6/0x440 [11304.552998] entry_SYSCALL_64_after_hwframe+0x49/0xae [11304.557594] RIP: 0033:0x7fa749552cd8 [11304.564238] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11304.583871] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11304.592557] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11304.599139] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11304.606424] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11304.612883] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11304.626139] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11320.059190] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11320.062230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11320.064363] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11320.083483] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11320.083497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11320.083505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11320.083518] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11320.083522] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11320.083528] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11320.083531] RDX: 000000000000390e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11320.083533] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11320.083535] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11320.083538] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11320.083541] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11320.083544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11320.083547] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11320.083555] Call Trace: [11320.083560] [11320.083566] ? show_regs.cold.9+0x22/0x2f [11320.083575] ? watchdog_timer_fn.cold.6+0x62/0x10a [11320.083587] ? __hrtimer_run_queues+0x1c2/0x4d0 [11320.083591] ? watchdog+0x40/0x40 [11320.083596] ? hrtimer_interrupt+0x128/0x2c0 [11320.083600] ? smp_apic_timer_interrupt+0xa6/0x270 [11320.083608] ? apic_timer_interrupt+0xf/0x20 [11320.083805] [11320.083809] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11320.083814] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11320.083819] do_raw_spin_lock+0x177/0x190 [11320.083824] _raw_spin_lock+0x21/0x30 [11320.083831] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11320.084096] obd_mod_free+0x46/0x140 [ptlrpc] [11320.090210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [11320.096219] mdc_commit_open+0x91/0xb0 [mdc] [11320.096267] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11320.096531] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11320.096863] after_reply+0xe2c/0x1780 [ptlrpc] [11320.097075] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11320.097286] ? __next_timer_interrupt+0x160/0x160 [11320.097297] ptlrpcd+0xa82/0xe40 [ptlrpc] [11320.102644] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11320.102701] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11320.102708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11320.102714] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11320.102992] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11320.103004] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11320.103008] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11320.103012] RAX: 0000000000005223 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11320.103015] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11320.103018] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11320.103225] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11320.103228] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11320.103231] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11320.103234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11320.103236] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11320.103245] Call Trace: [11320.103250] [11320.103255] ? show_regs.cold.9+0x22/0x2f [11320.103319] ? watchdog_timer_fn.cold.6+0x62/0x10a [11320.103326] ? __hrtimer_run_queues+0x1c2/0x4d0 [11320.103331] ? watchdog+0x40/0x40 [11320.103335] ? hrtimer_interrupt+0x128/0x2c0 [11320.103338] ? smp_apic_timer_interrupt+0xa6/0x270 [11320.103348] ? apic_timer_interrupt+0xf/0x20 [11320.103352] [11320.103355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11320.103360] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11320.103365] do_raw_spin_lock+0x177/0x190 [11320.103370] _raw_spin_lock+0x21/0x30 [11320.103375] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11320.103605] process_one_work+0x2c8/0x700 [11320.103612] worker_thread+0x296/0x6e0 [11320.103617] ? rescuer_thread+0x570/0x570 [11320.103621] kthread+0x1d1/0x200 [11320.103625] ? set_kthread_struct+0x70/0x70 [11320.103641] ret_from_fork+0x1f/0x30 [11320.313944] ? do_wait_intr+0xf0/0xf0 [11320.315479] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11320.317947] kthread+0x1d1/0x200 [11320.319535] ? set_kthread_struct+0x70/0x70 [11320.321351] ret_from_fork+0x1f/0x30 [11324.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11324.057282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11324.169980] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11324.182949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11324.189855] RIP: 0010:native_safe_halt+0x2e/0x50 [11324.194839] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11324.241012] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11324.252443] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11324.268258] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11324.283715] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11324.294458] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11324.309960] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11324.318614] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11324.330206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11324.345086] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11324.357793] Call Trace: [11324.362078] [11324.364156] ? show_regs.cold.9+0x22/0x2f [11324.367839] ? watchdog_timer_fn.cold.6+0x62/0x10a [11324.373130] ? __hrtimer_run_queues+0x1c2/0x4d0 [11324.378080] ? watchdog+0x40/0x40 [11324.379895] ? hrtimer_interrupt+0x128/0x2c0 [11324.384407] ? smp_apic_timer_interrupt+0xa6/0x270 [11324.388626] ? apic_timer_interrupt+0xf/0x20 [11324.390948] [11324.391933] ? native_safe_halt+0x2e/0x50 [11324.400968] kvm_wait+0xcd/0xe0 [11324.407709] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11324.416510] do_raw_spin_lock+0x177/0x190 [11324.421738] _raw_spin_lock+0x21/0x30 [11324.424819] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11324.438247] ? do_raw_spin_unlock+0x75/0x190 [11324.454400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11324.464017] ? do_wait_intr+0xf0/0xf0 [11324.465455] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11324.471009] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11324.479533] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11324.485107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.491809] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11324.502613] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11324.514869] mdc_intent_lock+0x353/0x7e0 [mdc] [11324.522252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.530251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11324.544194] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11324.556651] lmv_intent_open+0x338/0x1150 [lmv] [11324.567871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.578389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.587358] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11324.592603] ll_intent_lock+0xc5/0xaa0 [lustre] [11324.602618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.614446] ll_lookup_it+0xe0e/0x2270 [lustre] [11324.620602] ? do_raw_spin_unlock+0x75/0x190 [11324.626900] ? _raw_spin_unlock+0x12/0x30 [11324.628320] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11324.644398] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11324.646234] ll_atomic_open+0x36a/0xd70 [lustre] [11324.657106] lookup_open+0x356/0xb40 [11324.662113] do_last+0x313/0xfc0 [11324.673173] ? nd_jump_root+0xe5/0x160 [11324.675758] ? path_init+0x437/0x520 [11324.680137] path_openat+0xf7/0x500 [11324.683594] ? do_raw_spin_unlock+0x75/0x190 [11324.686798] do_filp_open+0x99/0x140 [11324.691993] ? getname_flags+0x6e/0x330 [11324.696372] ? __check_object_size+0xff/0x256 [11324.698005] ? do_raw_spin_unlock+0x75/0x190 [11324.711986] ? _raw_spin_unlock+0x12/0x30 [11324.724049] do_sys_openat2+0x2b4/0x410 [11324.726154] do_sys_open+0x73/0xa0 [11324.731171] __x64_sys_openat+0x24/0x30 [11324.736756] do_syscall_64+0xc1/0x440 [11324.740467] entry_SYSCALL_64_after_hwframe+0x49/0xae [11324.747650] RIP: 0033:0x7fa749552332 [11324.752998] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11324.793442] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11324.810507] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11324.819818] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11324.829720] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11324.841822] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11324.856340] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11327.460661] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9410s! [11327.483415] Showing busy workqueues and worker pools: [11327.493896] workqueue events: flags=0x0 [11327.497453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11327.503623] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11327.516630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11327.521923] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11327.543817] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.550571] pending: kfree_rcu_monitor [11327.554619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11327.560994] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11327.564811] workqueue events_long: flags=0x0 [11327.568649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.576023] pending: lu_objects_flush [obdclass] [11327.579710] workqueue events_unbound: flags=0x2 [11327.581366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11327.592613] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11327.607726] workqueue events_power_efficient: flags=0x80 [11327.616589] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.630097] pending: check_lifetime [11327.635604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.645675] pending: neigh_periodic_work [11327.651692] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.657870] pending: neigh_periodic_work [11327.669148] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.679699] pending: do_cache_clean [11327.681708] workqueue mm_percpu_wq: flags=0x8 [11327.690843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.699551] pending: vmstat_update [11327.703525] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.712133] pending: vmstat_update [11327.715511] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11327.725262] pending: vmstat_update [11327.727109] workqueue writeback: flags=0x4a [11327.728466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11327.739028] pending: wb_workfn, wb_workfn [11327.741113] workqueue xprtiod: flags=0xa [11327.746715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11327.753813] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11327.766919] workqueue ipv6_addrconf: flags=0x40008 [11327.776847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11327.786149] pending: addrconf_verify_work [11327.789819] workqueue knbd0-recv: flags=0x1a [11327.792048] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11327.800664] in-flight: 651:recv_work [11327.813029] workqueue ptlrpc_pinger: flags=0x2 [11327.815980] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11327.826193] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11327.841117] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9410s workers=4 idle: 37 31846 28897 [11327.851366] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11317s workers=2 idle: 653 [11332.059633] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11332.074517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11332.119526] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11332.134067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11332.157716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11332.172527] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11332.209160] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11332.218056] RAX: 000000000000781f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11332.230440] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11332.246584] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11332.248886] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11332.255530] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11332.268490] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11332.274970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11332.282536] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11332.292457] Call Trace: [11332.294402] [11332.295555] ? show_regs.cold.9+0x22/0x2f [11332.298914] ? watchdog_timer_fn.cold.6+0x62/0x10a [11332.301044] ? __hrtimer_run_queues+0x1c2/0x4d0 [11332.302345] ? watchdog+0x40/0x40 [11332.303444] ? hrtimer_interrupt+0x128/0x2c0 [11332.311114] ? smp_apic_timer_interrupt+0xa6/0x270 [11332.312263] ? apic_timer_interrupt+0xf/0x20 [11332.313394] [11332.315017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11332.318494] do_raw_spin_lock+0x177/0x190 [11332.319675] _raw_spin_lock+0x21/0x30 [11332.321244] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11332.324647] ? do_raw_spin_unlock+0x75/0x190 [11332.326381] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11332.328487] ? do_wait_intr+0xf0/0xf0 [11332.330390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11332.332266] mdc_close+0x53b/0xd70 [mdc] [11332.336671] lmv_close+0x18d/0x490 [lmv] [11332.338376] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11332.341236] ll_md_real_close+0xa5/0x2a0 [lustre] [11332.343350] ll_md_close+0x274/0x800 [lustre] [11332.345512] ll_file_release+0x128/0xc50 [lustre] [11332.346761] __fput+0x10d/0x440 [11332.347703] ____fput+0x12/0x20 [11332.348668] task_work_run+0xc6/0x110 [11332.350820] exit_to_usermode_loop+0x1dd/0x1f0 [11332.353067] do_syscall_64+0x3d6/0x440 [11332.354982] entry_SYSCALL_64_after_hwframe+0x49/0xae [11332.357542] RIP: 0033:0x7fa749552cd8 [11332.359481] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11332.369410] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11332.373313] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11332.376511] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11332.379754] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11332.382711] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11332.385715] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11333.888226] rcu: INFO: rcu_sched self-detected stall on CPU [11333.890782] rcu: 2-...!: (5316475 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [11333.926282] (t=9420199 jiffies g=258701 q=4216) [11333.988878] rcu: rcu_sched kthread starved for 9416994 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [11334.002461] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11334.022617] rcu: RCU grace-period kthread stack dump: [11334.042615] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11334.053342] Call Trace: [11334.054185] __schedule+0x351/0xcb0 [11334.075665] schedule+0xc0/0x180 [11334.076774] schedule_timeout+0xb4/0x190 [11334.077941] ? do_raw_spin_unlock+0x75/0x190 [11334.089546] ? __next_timer_interrupt+0x160/0x160 [11334.093058] ? prepare_to_swait_event+0xf3/0x140 [11334.096719] rcu_gp_kthread+0x8dd/0x15b0 [11334.099822] ? force_qs_rnp+0x2c0/0x2c0 [11334.101843] kthread+0x1d1/0x200 [11334.103610] ? set_kthread_struct+0x70/0x70 [11334.105710] ret_from_fork+0x1f/0x30 [11334.107224] rcu: Stack dump where RCU GP kthread last ran: [11334.109694] Sending NMI from CPU 2 to CPUs 0: [11334.111849] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [11334.112189] NMI backtrace for cpu 2 [11334.122636] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11334.130838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11334.136199] Call Trace: [11334.141708] [11334.149354] ? dump_stack+0xbb/0x10e [11334.153686] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11334.158758] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11334.160767] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11334.162758] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11334.167346] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11334.169858] ? rcu_dump_cpu_stacks+0x11b/0x171 [11334.173500] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11334.179892] ? update_process_times+0x6d/0xc0 [11334.182251] ? tick_sched_handle.isra.18+0x27/0x90 [11334.184436] ? tick_sched_timer+0x53/0xb0 [11334.187884] ? __hrtimer_run_queues+0x1c2/0x4d0 [11334.194695] ? tick_sched_handle.isra.18+0x90/0x90 [11334.197052] ? hrtimer_interrupt+0x128/0x2c0 [11334.198884] ? smp_apic_timer_interrupt+0xa6/0x270 [11334.200945] ? apic_timer_interrupt+0xf/0x20 [11334.203486] [11334.204471] ? native_safe_halt+0x2e/0x50 [11334.206438] ? kvm_wait+0xcd/0xe0 [11334.207934] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11334.210542] ? do_raw_spin_lock+0x177/0x190 [11334.214169] ? _raw_spin_lock+0x21/0x30 [11334.217239] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11334.220533] ? obd_mod_free+0x46/0x140 [ptlrpc] [11334.222782] ? mdc_commit_open+0x91/0xb0 [mdc] [11334.224356] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11334.226567] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11334.228510] ? after_reply+0xe2c/0x1780 [ptlrpc] [11334.230545] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11334.232750] ? __next_timer_interrupt+0x160/0x160 [11334.234551] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11334.236438] ? do_wait_intr+0xf0/0xf0 [11334.237808] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11334.239893] ? kthread+0x1d1/0x200 [11334.241553] ? set_kthread_struct+0x70/0x70 [11334.243225] ? ret_from_fork+0x1f/0x30 [11348.057185] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11348.061457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11348.090284] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11348.097472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11348.103252] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11348.106407] RIP: 0010:native_safe_halt+0x2e/0x50 [11348.108359] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11348.120526] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11348.124440] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11348.127519] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11348.130910] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11348.137265] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11348.141737] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11348.146747] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11348.157665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11348.161990] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11348.169065] Call Trace: [11348.170094] [11348.170800] ? show_regs.cold.9+0x22/0x2f [11348.172732] ? watchdog_timer_fn.cold.6+0x62/0x10a [11348.174665] ? __hrtimer_run_queues+0x1c2/0x4d0 [11348.176945] ? watchdog+0x40/0x40 [11348.178205] ? hrtimer_interrupt+0x128/0x2c0 [11348.179710] ? smp_apic_timer_interrupt+0xa6/0x270 [11348.181864] ? apic_timer_interrupt+0xf/0x20 [11348.186754] [11348.190427] ? native_safe_halt+0x2e/0x50 [11348.194767] kvm_wait+0xcd/0xe0 [11348.196491] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11348.198618] do_raw_spin_lock+0x177/0x190 [11348.200235] _raw_spin_lock+0x21/0x30 [11348.201556] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11348.203966] process_one_work+0x2c8/0x700 [11348.205448] worker_thread+0x296/0x6e0 [11348.207327] ? rescuer_thread+0x570/0x570 [11348.208961] kthread+0x1d1/0x200 [11348.210232] ? set_kthread_struct+0x70/0x70 [11348.213768] ret_from_fork+0x1f/0x30 [11358.178041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9441s! [11358.181391] Showing busy workqueues and worker pools: [11358.183289] workqueue events: flags=0x0 [11358.187158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11358.191890] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11358.200495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11358.202631] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11358.208044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.212027] pending: kfree_rcu_monitor [11358.216277] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11358.219519] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11358.226276] workqueue events_long: flags=0x0 [11358.231497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.235301] pending: lu_objects_flush [obdclass] [11358.239649] workqueue events_unbound: flags=0x2 [11358.241669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11358.244566] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11358.251112] workqueue events_power_efficient: flags=0x80 [11358.255627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.260109] pending: check_lifetime [11358.264488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.267204] pending: neigh_periodic_work [11358.271245] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.273544] pending: neigh_periodic_work [11358.274902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.278392] pending: do_cache_clean [11358.279829] workqueue mm_percpu_wq: flags=0x8 [11358.285159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.288801] pending: vmstat_update [11358.290841] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.292802] pending: vmstat_update [11358.294031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11358.297067] pending: vmstat_update [11358.299530] workqueue writeback: flags=0x4a [11358.302375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11358.307815] pending: wb_workfn, wb_workfn [11358.313792] workqueue xprtiod: flags=0xa [11358.315157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11358.320555] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11358.326409] workqueue ipv6_addrconf: flags=0x40008 [11358.329116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11358.336628] pending: addrconf_verify_work [11358.343083] workqueue knbd0-recv: flags=0x1a [11358.344839] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11358.351298] in-flight: 651:recv_work [11358.353560] workqueue ptlrpc_pinger: flags=0x2 [11358.364162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11358.365794] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11358.372757] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9441s workers=4 idle: 37 31846 28897 [11358.382924] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11348s workers=2 idle: 653 [11360.058986] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11360.061492] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11360.065526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11360.065607] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11360.065614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11360.065618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11360.065631] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11360.065636] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11360.067002] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11360.067006] RDX: 0000000000007242 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11360.067009] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11360.067011] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11360.067014] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11360.072036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11360.072119] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11360.072127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11360.072130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11360.072143] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11360.072148] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11360.072152] RAX: 000000000000707a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11360.072155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11360.072157] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11360.072160] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11360.072163] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11360.072166] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11360.072169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11360.072172] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11360.072181] Call Trace: [11360.072187] [11360.072194] ? show_regs.cold.9+0x22/0x2f [11360.072205] ? watchdog_timer_fn.cold.6+0x62/0x10a [11360.072275] ? __hrtimer_run_queues+0x1c2/0x4d0 [11360.072279] ? watchdog+0x40/0x40 [11360.072283] ? hrtimer_interrupt+0x128/0x2c0 [11360.072287] ? smp_apic_timer_interrupt+0xa6/0x270 [11360.072295] ? apic_timer_interrupt+0xf/0x20 [11360.072300] [11360.072303] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11360.073071] do_raw_spin_lock+0x177/0x190 [11360.073077] _raw_spin_lock+0x21/0x30 [11360.073082] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11360.073663] ? do_raw_spin_unlock+0x75/0x190 [11360.073669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11360.132425] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11360.132436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11360.132439] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11360.132447] Call Trace: [11360.132453] [11360.132459] ? show_regs.cold.9+0x22/0x2f [11360.132470] ? watchdog_timer_fn.cold.6+0x62/0x10a [11360.132478] ? __hrtimer_run_queues+0x1c2/0x4d0 [11360.132481] ? watchdog+0x40/0x40 [11360.132485] ? hrtimer_interrupt+0x128/0x2c0 [11360.132489] ? smp_apic_timer_interrupt+0xa6/0x270 [11360.132497] ? apic_timer_interrupt+0xf/0x20 [11360.132501] [11360.132503] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11360.132510] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11360.132515] do_raw_spin_lock+0x177/0x190 [11360.132519] _raw_spin_lock+0x21/0x30 [11360.132527] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11360.133999] obd_mod_free+0x46/0x140 [ptlrpc] [11360.134419] mdc_commit_open+0x91/0xb0 [mdc] [11360.134457] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11360.134678] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11360.149180] ? do_wait_intr+0xf0/0xf0 [11360.149202] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11360.150463] mdc_close+0x53b/0xd70 [mdc] [11360.150505] lmv_close+0x18d/0x490 [lmv] [11360.150544] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11360.159886] after_reply+0xe2c/0x1780 [ptlrpc] [11360.161257] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11360.161795] ? __next_timer_interrupt+0x160/0x160 [11360.161806] ptlrpcd+0xa82/0xe40 [ptlrpc] [11360.167331] ll_md_real_close+0xa5/0x2a0 [lustre] [11360.167455] ll_md_close+0x274/0x800 [lustre] [11360.187689] ? do_wait_intr+0xf0/0xf0 [11360.187710] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11360.189186] kthread+0x1d1/0x200 [11360.189196] ? set_kthread_struct+0x70/0x70 [11360.189201] ret_from_fork+0x1f/0x30 [11360.582629] ll_file_release+0x128/0xc50 [lustre] [11360.586851] __fput+0x10d/0x440 [11360.590422] ____fput+0x12/0x20 [11360.593036] task_work_run+0xc6/0x110 [11360.595566] exit_to_usermode_loop+0x1dd/0x1f0 [11360.598657] do_syscall_64+0x3d6/0x440 [11360.601390] entry_SYSCALL_64_after_hwframe+0x49/0xae [11360.604407] RIP: 0033:0x7fa749552cd8 [11360.605723] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11360.618496] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11360.624049] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11360.632795] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11360.639501] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11360.646503] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11360.652448] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11376.057198] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11376.059934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11376.099045] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11376.109917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11376.117491] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11376.120187] RIP: 0010:native_safe_halt+0x2e/0x50 [11376.125210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11376.150261] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11376.160030] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11376.166313] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11376.171396] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11376.175070] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11376.184496] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11376.186340] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11376.190863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11376.192251] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11376.196347] Call Trace: [11376.197824] [11376.198311] ? show_regs.cold.9+0x22/0x2f [11376.205772] ? watchdog_timer_fn.cold.6+0x62/0x10a [11376.207252] ? __hrtimer_run_queues+0x1c2/0x4d0 [11376.210332] ? watchdog+0x40/0x40 [11376.214165] ? hrtimer_interrupt+0x128/0x2c0 [11376.216335] ? smp_apic_timer_interrupt+0xa6/0x270 [11376.219909] ? apic_timer_interrupt+0xf/0x20 [11376.224814] [11376.227723] ? native_safe_halt+0x2e/0x50 [11376.229279] kvm_wait+0xcd/0xe0 [11376.231128] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11376.237144] do_raw_spin_lock+0x177/0x190 [11376.240609] _raw_spin_lock+0x21/0x30 [11376.242102] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11376.246721] process_one_work+0x2c8/0x700 [11376.249134] worker_thread+0x296/0x6e0 [11376.251305] ? rescuer_thread+0x570/0x570 [11376.252299] kthread+0x1d1/0x200 [11376.253133] ? set_kthread_struct+0x70/0x70 [11376.256176] ret_from_fork+0x1f/0x30 [11384.056709] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11384.065424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11384.101448] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11384.109153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11384.116384] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11384.118964] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11384.136438] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11384.142962] RAX: 0000000000007c53 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11384.148829] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11384.152570] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11384.160019] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11384.165418] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11384.171398] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11384.175555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11384.184136] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11384.188921] Call Trace: [11384.192484] [11384.194828] ? show_regs.cold.9+0x22/0x2f [11384.196809] ? watchdog_timer_fn.cold.6+0x62/0x10a [11384.199120] ? __hrtimer_run_queues+0x1c2/0x4d0 [11384.202109] ? watchdog+0x40/0x40 [11384.203883] ? hrtimer_interrupt+0x128/0x2c0 [11384.207704] ? smp_apic_timer_interrupt+0xa6/0x270 [11384.209948] ? apic_timer_interrupt+0xf/0x20 [11384.212920] [11384.214842] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11384.217957] do_raw_spin_lock+0x177/0x190 [11384.219377] _raw_spin_lock+0x21/0x30 [11384.221114] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11384.223886] ? do_raw_spin_unlock+0x75/0x190 [11384.225836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11384.228208] ? do_wait_intr+0xf0/0xf0 [11384.230011] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11384.233300] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11384.235895] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11384.238884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11384.241515] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11384.244042] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11384.246335] mdc_intent_lock+0x353/0x7e0 [mdc] [11384.248587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11384.251702] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11384.254869] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11384.259185] lmv_intent_open+0x338/0x1150 [lmv] [11384.262066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11384.267598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11384.276567] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11384.281098] ll_intent_lock+0xc5/0xaa0 [lustre] [11384.285964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11384.290148] ll_lookup_it+0xe0e/0x2270 [lustre] [11384.293764] ? do_raw_spin_unlock+0x75/0x190 [11384.297642] ? _raw_spin_unlock+0x12/0x30 [11384.300026] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11384.302383] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11384.305602] ll_atomic_open+0x36a/0xd70 [lustre] [11384.307874] lookup_open+0x356/0xb40 [11384.309610] do_last+0x313/0xfc0 [11384.310827] ? nd_jump_root+0xe5/0x160 [11384.315700] ? path_init+0x437/0x520 [11384.317977] path_openat+0xf7/0x500 [11384.325152] ? do_raw_spin_unlock+0x75/0x190 [11384.326698] do_filp_open+0x99/0x140 [11384.330832] ? getname_flags+0x6e/0x330 [11384.334807] ? __check_object_size+0xff/0x256 [11384.337957] ? do_raw_spin_unlock+0x75/0x190 [11384.345465] ? _raw_spin_unlock+0x12/0x30 [11384.348495] do_sys_openat2+0x2b4/0x410 [11384.350831] do_sys_open+0x73/0xa0 [11384.352414] __x64_sys_openat+0x24/0x30 [11384.356105] do_syscall_64+0xc1/0x440 [11384.358771] entry_SYSCALL_64_after_hwframe+0x49/0xae [11384.368178] RIP: 0033:0x7fa749552332 [11384.369446] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11384.378480] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11384.387464] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11384.396783] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11384.405758] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11384.410765] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11384.419447] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11388.058880] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11388.060853] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11388.068819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11388.068906] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11388.068916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11388.068920] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11388.068939] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11388.068943] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11388.068948] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11388.068953] RDX: 00000000000002b6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11388.068956] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11388.068958] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11388.068960] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11388.068966] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11388.068970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11388.068972] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11388.068980] Call Trace: [11388.068985] [11388.068994] ? show_regs.cold.9+0x22/0x2f [11388.069004] ? watchdog_timer_fn.cold.6+0x62/0x10a [11388.069014] ? __hrtimer_run_queues+0x1c2/0x4d0 [11388.069019] ? watchdog+0x40/0x40 [11388.069024] ? hrtimer_interrupt+0x128/0x2c0 [11388.069028] ? smp_apic_timer_interrupt+0xa6/0x270 [11388.069114] ? apic_timer_interrupt+0xf/0x20 [11388.069120] [11388.069123] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11388.069131] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11388.069137] do_raw_spin_lock+0x177/0x190 [11388.069141] _raw_spin_lock+0x21/0x30 [11388.069147] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11388.069502] obd_mod_free+0x46/0x140 [ptlrpc] [11388.069858] mdc_commit_open+0x91/0xb0 [mdc] [11388.069894] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11388.070122] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11388.070605] after_reply+0xe2c/0x1780 [ptlrpc] [11388.070857] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11388.071131] ? __next_timer_interrupt+0x160/0x160 [11388.071140] ptlrpcd+0xa82/0xe40 [ptlrpc] [11388.071629] ? do_wait_intr+0xf0/0xf0 [11388.071638] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11388.076011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11388.076145] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11388.076152] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11388.076155] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11388.076169] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11388.076173] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11388.076177] RAX: 0000000000006190 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11388.076180] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11388.076182] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11388.076184] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11388.076186] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11388.076190] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11388.076193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11388.076195] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11388.076203] Call Trace: [11388.076209] [11388.076216] ? show_regs.cold.9+0x22/0x2f [11388.076224] ? watchdog_timer_fn.cold.6+0x62/0x10a [11388.076232] ? __hrtimer_run_queues+0x1c2/0x4d0 [11388.076236] ? watchdog+0x40/0x40 [11388.076241] ? hrtimer_interrupt+0x128/0x2c0 [11388.076245] ? smp_apic_timer_interrupt+0xa6/0x270 [11388.076253] ? apic_timer_interrupt+0xf/0x20 [11388.076258] [11388.076261] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11388.076268] do_raw_spin_lock+0x177/0x190 [11388.076273] _raw_spin_lock+0x21/0x30 [11388.076278] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11388.111096] kthread+0x1d1/0x200 [11388.111116] ? set_kthread_struct+0x70/0x70 [11388.111120] ret_from_fork+0x1f/0x30 [11388.519688] ? do_raw_spin_unlock+0x75/0x190 [11388.525195] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11388.528204] ? do_wait_intr+0xf0/0xf0 [11388.532704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11388.540015] mdc_close+0x53b/0xd70 [mdc] [11388.544671] lmv_close+0x18d/0x490 [lmv] [11388.547792] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11388.552463] ll_md_real_close+0xa5/0x2a0 [lustre] [11388.557065] ll_md_close+0x274/0x800 [lustre] [11388.558663] ll_file_release+0x128/0xc50 [lustre] [11388.563519] __fput+0x10d/0x440 [11388.568841] ____fput+0x12/0x20 [11388.571898] task_work_run+0xc6/0x110 [11388.573902] exit_to_usermode_loop+0x1dd/0x1f0 [11388.576999] do_syscall_64+0x3d6/0x440 [11388.580980] entry_SYSCALL_64_after_hwframe+0x49/0xae [11388.585119] RIP: 0033:0x7fa749552cd8 [11388.589602] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11388.607775] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11388.615733] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11388.630741] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11388.638820] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11388.643700] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11388.649870] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11388.897633] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9471s! [11388.901288] Showing busy workqueues and worker pools: [11388.904888] workqueue events: flags=0x0 [11388.906926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11388.910586] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11388.917291] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11388.920291] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11388.925433] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.929205] pending: kfree_rcu_monitor [11388.931382] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11388.933810] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11388.937204] workqueue events_long: flags=0x0 [11388.938672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.941785] pending: lu_objects_flush [obdclass] [11388.943710] workqueue events_unbound: flags=0x2 [11388.944938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11388.946945] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11388.950558] workqueue events_power_efficient: flags=0x80 [11388.952712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.955603] pending: check_lifetime [11388.957287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.959573] pending: neigh_periodic_work [11388.961078] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.963475] pending: neigh_periodic_work [11388.965060] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.967956] pending: do_cache_clean [11388.969723] workqueue mm_percpu_wq: flags=0x8 [11388.971659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.974110] pending: vmstat_update [11388.975726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.978450] pending: vmstat_update [11388.980395] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11388.983511] pending: vmstat_update [11388.985106] workqueue writeback: flags=0x4a [11388.986604] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11388.989411] pending: wb_workfn, wb_workfn [11388.990903] workqueue xprtiod: flags=0xa [11388.992660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11388.995869] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11388.999603] workqueue ipv6_addrconf: flags=0x40008 [11389.001379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11389.004419] pending: addrconf_verify_work [11389.006402] workqueue knbd0-recv: flags=0x1a [11389.008438] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11389.011182] in-flight: 651:recv_work [11389.012879] workqueue ptlrpc_pinger: flags=0x2 [11389.015564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11389.018488] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11389.021343] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9471s workers=4 idle: 37 31846 28897 [11389.024875] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11378s workers=2 idle: 653 [11404.057084] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11404.059796] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11404.102210] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11404.109565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11404.112526] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11404.115509] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11404.117647] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11404.129836] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11404.136906] RAX: 0000000000004691 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11404.146176] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11404.154509] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11404.157776] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11404.163200] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11404.169537] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11404.176620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11404.183015] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11404.192576] Call Trace: [11404.194575] [11404.197222] ? show_regs.cold.9+0x22/0x2f [11404.199611] ? watchdog_timer_fn.cold.6+0x62/0x10a [11404.206772] ? __hrtimer_run_queues+0x1c2/0x4d0 [11404.211030] ? watchdog+0x40/0x40 [11404.213330] ? hrtimer_interrupt+0x128/0x2c0 [11404.216800] ? smp_apic_timer_interrupt+0xa6/0x270 [11404.219253] ? apic_timer_interrupt+0xf/0x20 [11404.221500] [11404.223447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11404.226287] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11404.230811] do_raw_spin_lock+0x177/0x190 [11404.233489] _raw_spin_lock+0x21/0x30 [11404.235242] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11404.238621] process_one_work+0x2c8/0x700 [11404.239973] worker_thread+0x296/0x6e0 [11404.245752] ? rescuer_thread+0x570/0x570 [11404.249365] kthread+0x1d1/0x200 [11404.252831] ? set_kthread_struct+0x70/0x70 [11404.255208] ret_from_fork+0x1f/0x30 [11416.054834] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11416.058059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11416.058855] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11416.059072] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11416.059077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11416.059148] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11416.059158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11416.059161] RIP: 0010:native_safe_halt+0x2e/0x50 [11416.059205] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11416.059210] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11416.059214] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11416.059217] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11416.059219] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11416.059221] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11416.059227] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11416.059231] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11416.059234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11416.059237] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11416.059245] Call Trace: [11416.059262] [11416.059268] ? show_regs.cold.9+0x22/0x2f [11416.059277] ? watchdog_timer_fn.cold.6+0x62/0x10a [11416.059284] ? __hrtimer_run_queues+0x1c2/0x4d0 [11416.059289] ? watchdog+0x40/0x40 [11416.059293] ? hrtimer_interrupt+0x128/0x2c0 [11416.059298] ? smp_apic_timer_interrupt+0xa6/0x270 [11416.059305] ? apic_timer_interrupt+0xf/0x20 [11416.059310] [11416.059313] ? native_safe_halt+0x2e/0x50 [11416.059317] kvm_wait+0xcd/0xe0 [11416.059326] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11416.059335] do_raw_spin_lock+0x177/0x190 [11416.059340] _raw_spin_lock+0x21/0x30 [11416.059345] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11416.059679] ? do_raw_spin_unlock+0x75/0x190 [11416.059687] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11416.059890] ? do_wait_intr+0xf0/0xf0 [11416.059898] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11416.060482] mdc_close+0x53b/0xd70 [mdc] [11416.060522] lmv_close+0x18d/0x490 [lmv] [11416.060552] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11416.060660] ll_md_real_close+0xa5/0x2a0 [lustre] [11416.060973] ll_md_close+0x274/0x800 [lustre] [11416.061057] ll_file_release+0x128/0xc50 [lustre] [11416.061150] __fput+0x10d/0x440 [11416.061160] ____fput+0x12/0x20 [11416.061165] task_work_run+0xc6/0x110 [11416.061172] exit_to_usermode_loop+0x1dd/0x1f0 [11416.061179] do_syscall_64+0x3d6/0x440 [11416.061184] entry_SYSCALL_64_after_hwframe+0x49/0xae [11416.061206] RIP: 0033:0x7fa749552cd8 [11416.061238] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11416.061244] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11416.061248] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11416.061251] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11416.061253] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11416.061255] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11416.061257] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11416.082914] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11416.082938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11416.082941] RIP: 0010:native_safe_halt+0x2e/0x50 [11416.082975] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11416.082980] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11416.082989] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11416.082992] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11416.083005] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11416.083007] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11416.083010] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11416.083014] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11416.083017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11416.083020] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11416.083032] Call Trace: [11416.083042] [11416.083050] ? show_regs.cold.9+0x22/0x2f [11416.083064] ? watchdog_timer_fn.cold.6+0x62/0x10a [11416.083071] ? __hrtimer_run_queues+0x1c2/0x4d0 [11416.083075] ? watchdog+0x40/0x40 [11416.083080] ? hrtimer_interrupt+0x128/0x2c0 [11416.083084] ? smp_apic_timer_interrupt+0xa6/0x270 [11416.083091] ? apic_timer_interrupt+0xf/0x20 [11416.083095] [11416.083099] ? native_safe_halt+0x2e/0x50 [11416.083103] kvm_wait+0xcd/0xe0 [11416.083111] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11416.083125] do_raw_spin_lock+0x177/0x190 [11416.083129] _raw_spin_lock+0x21/0x30 [11416.083136] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11416.083452] ? do_raw_spin_unlock+0x75/0x190 [11416.083459] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11416.083798] ? do_wait_intr+0xf0/0xf0 [11416.083807] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11416.084162] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11416.084402] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11416.084620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.084740] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11416.084776] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11416.084804] mdc_intent_lock+0x353/0x7e0 [mdc] [11416.084832] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.085106] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11416.085340] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11416.085371] lmv_intent_open+0x338/0x1150 [lmv] [11416.085402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.085488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.085572] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11416.085602] ll_intent_lock+0xc5/0xaa0 [lustre] [11416.085710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.085796] ll_lookup_it+0xe0e/0x2270 [lustre] [11416.085880] ? do_raw_spin_unlock+0x75/0x190 [11416.085887] ? _raw_spin_unlock+0x12/0x30 [11416.085893] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11416.086112] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11416.086133] ll_atomic_open+0x36a/0xd70 [lustre] [11416.086218] lookup_open+0x356/0xb40 [11416.086227] do_last+0x313/0xfc0 [11416.086232] ? nd_jump_root+0xe5/0x160 [11416.086237] ? path_init+0x437/0x520 [11416.086243] path_openat+0xf7/0x500 [11416.086248] ? do_raw_spin_unlock+0x75/0x190 [11416.086252] do_filp_open+0x99/0x140 [11416.086259] ? getname_flags+0x6e/0x330 [11416.086264] ? __check_object_size+0xff/0x256 [11416.086270] ? do_raw_spin_unlock+0x75/0x190 [11416.086273] ? _raw_spin_unlock+0x12/0x30 [11416.086277] do_sys_openat2+0x2b4/0x410 [11416.086282] do_sys_open+0x73/0xa0 [11416.086285] __x64_sys_openat+0x24/0x30 [11416.086289] do_syscall_64+0xc1/0x440 [11416.086295] entry_SYSCALL_64_after_hwframe+0x49/0xae [11416.086302] RIP: 0033:0x7fa749552332 [11416.086328] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11416.086333] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11416.086337] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11416.086339] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11416.086342] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11416.086344] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11416.086347] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11416.626913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11416.647807] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11416.653423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11416.657354] RIP: 0010:native_safe_halt+0x2e/0x50 [11416.659804] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11416.671419] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11416.680352] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11416.690054] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11416.693906] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11416.697632] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11416.702778] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11416.707140] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11416.712038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11416.715654] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11416.719124] Call Trace: [11416.720098] [11416.720803] ? show_regs.cold.9+0x22/0x2f [11416.722918] ? watchdog_timer_fn.cold.6+0x62/0x10a [11416.725064] ? __hrtimer_run_queues+0x1c2/0x4d0 [11416.726367] ? watchdog+0x40/0x40 [11416.727264] ? hrtimer_interrupt+0x128/0x2c0 [11416.728387] ? smp_apic_timer_interrupt+0xa6/0x270 [11416.730031] ? apic_timer_interrupt+0xf/0x20 [11416.732102] [11416.733019] ? native_safe_halt+0x2e/0x50 [11416.734984] kvm_wait+0xcd/0xe0 [11416.736832] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11416.739602] do_raw_spin_lock+0x177/0x190 [11416.741288] _raw_spin_lock+0x21/0x30 [11416.743313] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11416.747890] obd_mod_free+0x46/0x140 [ptlrpc] [11416.751003] mdc_commit_open+0x91/0xb0 [mdc] [11416.753315] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11416.755860] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11416.758151] after_reply+0xe2c/0x1780 [ptlrpc] [11416.762324] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11416.766057] ? __next_timer_interrupt+0x160/0x160 [11416.768112] ptlrpcd+0xa82/0xe40 [ptlrpc] [11416.770341] ? do_wait_intr+0xf0/0xf0 [11416.772151] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11416.774497] kthread+0x1d1/0x200 [11416.775599] ? set_kthread_struct+0x70/0x70 [11416.777177] ret_from_fork+0x1f/0x30 [11419.105081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9501s! [11419.107593] Showing busy workqueues and worker pools: [11419.109544] workqueue events: flags=0x0 [11419.110873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11419.113160] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11419.117316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11419.119790] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11419.122590] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.124578] pending: kfree_rcu_monitor [11419.125956] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11419.128103] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11419.131202] workqueue events_long: flags=0x0 [11419.132378] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.134224] pending: lu_objects_flush [obdclass] [11419.135777] workqueue events_unbound: flags=0x2 [11419.137226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11419.139274] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11419.142523] workqueue events_power_efficient: flags=0x80 [11419.144015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.145994] pending: check_lifetime [11419.147052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.149240] pending: neigh_periodic_work [11419.150801] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.153040] pending: neigh_periodic_work [11419.154555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.156952] pending: do_cache_clean [11419.158055] workqueue mm_percpu_wq: flags=0x8 [11419.159263] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.161230] pending: vmstat_update [11419.162340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.164304] pending: vmstat_update [11419.165880] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11419.168578] pending: vmstat_update [11419.170078] workqueue writeback: flags=0x4a [11419.171585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11419.173930] pending: wb_workfn, wb_workfn [11419.175696] workqueue xprtiod: flags=0xa [11419.177136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11419.179377] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11419.182377] workqueue ipv6_addrconf: flags=0x40008 [11419.184395] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11419.187023] pending: addrconf_verify_work [11419.188875] workqueue knbd0-recv: flags=0x1a [11419.193341] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11419.201252] in-flight: 651:recv_work [11419.203605] workqueue ptlrpc_pinger: flags=0x2 [11419.206034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11419.210217] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11419.216155] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9502s workers=4 idle: 37 31846 28897 [11419.220602] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11409s workers=2 idle: 653 [11432.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11432.059571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11432.090175] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11432.100042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11432.108260] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11432.114398] RIP: 0010:native_safe_halt+0x2e/0x50 [11432.116111] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11432.125825] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11432.130087] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11432.133868] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11432.140313] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11432.145147] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11432.147585] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11432.154108] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11432.158030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11432.161097] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11432.169659] Call Trace: [11432.170625] [11432.171395] ? show_regs.cold.9+0x22/0x2f [11432.172853] ? watchdog_timer_fn.cold.6+0x62/0x10a [11432.175225] ? __hrtimer_run_queues+0x1c2/0x4d0 [11432.177192] ? watchdog+0x40/0x40 [11432.178856] ? hrtimer_interrupt+0x128/0x2c0 [11432.181150] ? smp_apic_timer_interrupt+0xa6/0x270 [11432.183773] ? apic_timer_interrupt+0xf/0x20 [11432.185813] [11432.187471] ? native_safe_halt+0x2e/0x50 [11432.190439] kvm_wait+0xcd/0xe0 [11432.192716] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11432.199051] do_raw_spin_lock+0x177/0x190 [11432.200524] _raw_spin_lock+0x21/0x30 [11432.201924] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11432.206819] process_one_work+0x2c8/0x700 [11432.209190] worker_thread+0x296/0x6e0 [11432.212276] ? rescuer_thread+0x570/0x570 [11432.213942] kthread+0x1d1/0x200 [11432.215035] ? set_kthread_struct+0x70/0x70 [11432.216445] ret_from_fork+0x1f/0x30 [11444.056534] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11444.058171] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11444.059355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11444.059510] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11444.059517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11444.059520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11444.059536] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11444.059541] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11444.059545] RAX: 0000000000007885 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11444.059548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11444.059550] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11444.059553] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11444.059555] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11444.059558] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11444.059562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11444.059564] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11444.059573] Call Trace: [11444.059580] [11444.059581] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11444.059588] Modules linked in: lustre(O) [11444.059587] ? show_regs.cold.9+0x22/0x2f [11444.059593] osp(O) ofd(O) lod(O) mdt(O) mdd(O) [11444.059601] ? watchdog_timer_fn.cold.6+0x62/0x10a [11444.059607] mgs(O) lquota(O) lfsck(O) [11444.059612] ? __hrtimer_run_queues+0x1c2/0x4d0 [11444.059617] obdecho(O) mgc(O) [11444.059622] ? watchdog+0x40/0x40 [11444.059628] mdc(O) lov(O) [11444.059631] ? hrtimer_interrupt+0x128/0x2c0 [11444.059636] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [11444.059640] ? smp_apic_timer_interrupt+0xa6/0x270 [11444.059649] obdclass(O) ksocklnd(O) lnet(O) [11444.059655] ? apic_timer_interrupt+0xf/0x20 [11444.059659] libcfs(O) rpcsec_gss_krb5 auth_rpcgss [11444.059666] [11444.059670] nfsv4 dns_resolver [11444.059673] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11444.059676] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr [11444.059687] do_raw_spin_lock+0x177/0x190 [11444.059692] squashfs ata_generic [11444.059698] _raw_spin_lock+0x21/0x30 [11444.059700] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11444.059723] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11444.059728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11444.059730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11444.059738] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11444.059742] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11444.059747] RAX: 0000000000007a83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11444.059749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11444.059752] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11444.059753] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11444.059756] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11444.059759] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11444.059761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11444.059764] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11444.059770] Call Trace: [11444.059774] [11444.059790] ? show_regs.cold.9+0x22/0x2f [11444.059797] ? watchdog_timer_fn.cold.6+0x62/0x10a [11444.059803] ? __hrtimer_run_queues+0x1c2/0x4d0 [11444.059807] ? watchdog+0x40/0x40 [11444.059812] ? hrtimer_interrupt+0x128/0x2c0 [11444.059815] ? smp_apic_timer_interrupt+0xa6/0x270 [11444.059822] ? apic_timer_interrupt+0xf/0x20 [11444.059826] [11444.059829] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11444.059836] do_raw_spin_lock+0x177/0x190 [11444.059842] _raw_spin_lock+0x21/0x30 [11444.059846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11444.059709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11444.060347] ? do_raw_spin_unlock+0x75/0x190 [11444.060354] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11444.060792] ? do_wait_intr+0xf0/0xf0 [11444.060800] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11444.061045] mdc_close+0x53b/0xd70 [mdc] [11444.061210] lmv_close+0x18d/0x490 [lmv] [11444.061241] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11444.061349] ll_md_real_close+0xa5/0x2a0 [lustre] [11444.061594] ? do_raw_spin_unlock+0x75/0x190 [11444.061601] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11444.061805] ? do_wait_intr+0xf0/0xf0 [11444.061440] ll_md_close+0x274/0x800 [lustre] [11444.061812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11444.062036] ll_file_release+0x128/0xc50 [lustre] [11444.062767] __fput+0x10d/0x440 [11444.062303] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11444.062917] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11444.063057] ____fput+0x12/0x20 [11444.063066] task_work_run+0xc6/0x110 [11444.063077] exit_to_usermode_loop+0x1dd/0x1f0 [11444.063085] do_syscall_64+0x3d6/0x440 [11444.063089] entry_SYSCALL_64_after_hwframe+0x49/0xae [11444.063098] RIP: 0033:0x7fa749552cd8 [11444.063106] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11444.063109] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11444.063114] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11444.063116] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11444.063119] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11444.063121] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11444.063123] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11444.063075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.063539] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11444.063585] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11444.063616] mdc_intent_lock+0x353/0x7e0 [mdc] [11444.063644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.063732] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11444.066615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11444.066698] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11444.066706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11444.066709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11444.066722] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11444.066726] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11444.066731] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11444.066733] RDX: 0000000000002fc9 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11444.066736] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11444.066739] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11444.066742] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11444.066745] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11444.066748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11444.066751] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11444.066760] Call Trace: [11444.066765] [11444.066770] ? show_regs.cold.9+0x22/0x2f [11444.066780] ? watchdog_timer_fn.cold.6+0x62/0x10a [11444.066786] ? __hrtimer_run_queues+0x1c2/0x4d0 [11444.066790] ? watchdog+0x40/0x40 [11444.066795] ? hrtimer_interrupt+0x128/0x2c0 [11444.066799] ? smp_apic_timer_interrupt+0xa6/0x270 [11444.066808] ? apic_timer_interrupt+0xf/0x20 [11444.066813] [11444.066816] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11444.066820] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11444.066825] do_raw_spin_lock+0x177/0x190 [11444.066830] _raw_spin_lock+0x21/0x30 [11444.066836] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11444.067106] obd_mod_free+0x46/0x140 [ptlrpc] [11444.068556] mdc_commit_open+0x91/0xb0 [mdc] [11444.068597] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11444.068809] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11444.069022] after_reply+0xe2c/0x1780 [ptlrpc] [11444.070417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11444.070643] ? __next_timer_interrupt+0x160/0x160 [11444.101332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11444.101385] lmv_intent_open+0x338/0x1150 [lmv] [11444.101422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.101569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.101655] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11444.101680] ll_intent_lock+0xc5/0xaa0 [lustre] [11444.101781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.101869] ll_lookup_it+0xe0e/0x2270 [lustre] [11444.101960] ? do_raw_spin_unlock+0x75/0x190 [11444.101968] ? _raw_spin_unlock+0x12/0x30 [11444.101975] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11444.102002] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11444.102023] ll_atomic_open+0x36a/0xd70 [lustre] [11444.102269] lookup_open+0x356/0xb40 [11444.102277] do_last+0x313/0xfc0 [11444.102281] ? nd_jump_root+0xe5/0x160 [11444.102286] ? path_init+0x437/0x520 [11444.102290] path_openat+0xf7/0x500 [11444.102294] ? do_raw_spin_unlock+0x75/0x190 [11444.102298] do_filp_open+0x99/0x140 [11444.102304] ? getname_flags+0x6e/0x330 [11444.102308] ? __check_object_size+0xff/0x256 [11444.102313] ? do_raw_spin_unlock+0x75/0x190 [11444.102316] ? _raw_spin_unlock+0x12/0x30 [11444.102320] do_sys_openat2+0x2b4/0x410 [11444.102323] do_sys_open+0x73/0xa0 [11444.102326] __x64_sys_openat+0x24/0x30 [11444.102329] do_syscall_64+0xc1/0x440 [11444.102335] entry_SYSCALL_64_after_hwframe+0x49/0xae [11444.102340] RIP: 0033:0x7fa749552332 [11444.102346] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11444.102350] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11444.102354] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11444.102356] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11444.102358] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11444.102360] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11444.102362] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11444.551415] ptlrpcd+0xa82/0xe40 [ptlrpc] [11444.553234] ? do_wait_intr+0xf0/0xf0 [11444.554846] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11444.557526] kthread+0x1d1/0x200 [11444.559086] ? set_kthread_struct+0x70/0x70 [11444.561068] ret_from_fork+0x1f/0x30 [11449.313041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9532s! [11449.317572] Showing busy workqueues and worker pools: [11449.320853] workqueue events: flags=0x0 [11449.323738] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11449.327669] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11449.335228] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11449.340170] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11449.344901] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.349781] pending: kfree_rcu_monitor [11449.352165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11449.355727] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11449.360713] workqueue events_long: flags=0x0 [11449.362915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.366603] pending: lu_objects_flush [obdclass] [11449.370206] workqueue events_unbound: flags=0x2 [11449.372623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11449.375937] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11449.380171] workqueue events_power_efficient: flags=0x80 [11449.384147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.389983] pending: check_lifetime [11449.391987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.395175] pending: neigh_periodic_work [11449.399628] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.404623] pending: neigh_periodic_work [11449.406934] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.409704] pending: do_cache_clean [11449.412706] workqueue mm_percpu_wq: flags=0x8 [11449.415014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.417678] pending: vmstat_update [11449.420443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.427827] pending: vmstat_update [11449.431298] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11449.433158] pending: vmstat_update [11449.434123] workqueue writeback: flags=0x4a [11449.439792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11449.442881] pending: wb_workfn, wb_workfn [11449.445232] workqueue xprtiod: flags=0xa [11449.446800] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11449.450059] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11449.454321] workqueue ipv6_addrconf: flags=0x40008 [11449.456603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11449.459862] pending: addrconf_verify_work [11449.462274] workqueue knbd0-recv: flags=0x1a [11449.464414] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11449.467616] in-flight: 651:recv_work [11449.469336] workqueue ptlrpc_pinger: flags=0x2 [11449.474286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11449.477700] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11449.482552] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9532s workers=4 idle: 37 31846 28897 [11449.486712] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11439s workers=2 idle: 653 [11460.057017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11460.060656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11460.115045] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11460.130767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11460.141860] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11460.149163] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11460.152128] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11460.173846] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11460.182261] RAX: 00000000000009f4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11460.188467] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11460.201061] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11460.203250] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11460.213606] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11460.222532] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11460.236582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11460.244518] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11460.256441] Call Trace: [11460.259444] [11460.262008] ? show_regs.cold.9+0x22/0x2f [11460.267472] ? watchdog_timer_fn.cold.6+0x62/0x10a [11460.274494] ? __hrtimer_run_queues+0x1c2/0x4d0 [11460.276729] ? watchdog+0x40/0x40 [11460.278622] ? hrtimer_interrupt+0x128/0x2c0 [11460.282839] ? smp_apic_timer_interrupt+0xa6/0x270 [11460.286159] ? apic_timer_interrupt+0xf/0x20 [11460.289712] [11460.291466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11460.295238] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11460.298082] do_raw_spin_lock+0x177/0x190 [11460.302444] _raw_spin_lock+0x21/0x30 [11460.304631] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11460.311569] process_one_work+0x2c8/0x700 [11460.312553] worker_thread+0x296/0x6e0 [11460.314014] ? rescuer_thread+0x570/0x570 [11460.315170] kthread+0x1d1/0x200 [11460.316328] ? set_kthread_struct+0x70/0x70 [11460.318854] ret_from_fork+0x1f/0x30 [11472.066216] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11472.072407] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11472.072719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11472.072788] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11472.072794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11472.072797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11472.072811] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11472.072815] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11472.072819] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11472.072821] RDX: 0000000000006d03 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11472.072823] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11472.072825] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11472.072827] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11472.072829] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11472.072833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11472.072834] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11472.072842] Call Trace: [11472.072848] [11472.072853] ? show_regs.cold.9+0x22/0x2f [11472.072862] ? watchdog_timer_fn.cold.6+0x62/0x10a [11472.072869] ? __hrtimer_run_queues+0x1c2/0x4d0 [11472.072873] ? watchdog+0x40/0x40 [11472.072878] ? hrtimer_interrupt+0x128/0x2c0 [11472.072881] ? smp_apic_timer_interrupt+0xa6/0x270 [11472.072889] ? apic_timer_interrupt+0xf/0x20 [11472.072894] [11472.072897] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11472.072902] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11472.072906] do_raw_spin_lock+0x177/0x190 [11472.072911] _raw_spin_lock+0x21/0x30 [11472.072917] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11472.073222] obd_mod_free+0x46/0x140 [ptlrpc] [11472.073419] mdc_commit_open+0x91/0xb0 [mdc] [11472.073456] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11472.094464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11472.094543] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11472.094552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11472.094555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11472.094570] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11472.094575] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11472.094579] RAX: 0000000000007d0f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11472.094581] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11472.094584] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11472.094585] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11472.094588] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11472.094591] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11472.094594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11472.094597] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11472.094605] Call Trace: [11472.094610] [11472.094615] ? show_regs.cold.9+0x22/0x2f [11472.094624] ? watchdog_timer_fn.cold.6+0x62/0x10a [11472.094630] ? __hrtimer_run_queues+0x1c2/0x4d0 [11472.094634] ? watchdog+0x40/0x40 [11472.094638] ? hrtimer_interrupt+0x128/0x2c0 [11472.094642] ? smp_apic_timer_interrupt+0xa6/0x270 [11472.094651] ? apic_timer_interrupt+0xf/0x20 [11472.094655] [11472.094658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11472.094665] do_raw_spin_lock+0x177/0x190 [11472.094669] _raw_spin_lock+0x21/0x30 [11472.094674] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11472.094956] ? do_raw_spin_unlock+0x75/0x190 [11472.094962] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11472.095169] ? do_wait_intr+0xf0/0xf0 [11472.095177] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11472.095370] mdc_close+0x53b/0xd70 [mdc] [11472.095408] lmv_close+0x18d/0x490 [lmv] [11472.095436] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11472.095545] ll_md_real_close+0xa5/0x2a0 [lustre] [11472.095628] ll_md_close+0x274/0x800 [lustre] [11472.095711] ll_file_release+0x128/0xc50 [lustre] [11472.095791] __fput+0x10d/0x440 [11472.095800] ____fput+0x12/0x20 [11472.095803] task_work_run+0xc6/0x110 [11472.095809] exit_to_usermode_loop+0x1dd/0x1f0 [11472.095816] do_syscall_64+0x3d6/0x440 [11472.095820] entry_SYSCALL_64_after_hwframe+0x49/0xae [11472.095826] RIP: 0033:0x7fa749552cd8 [11472.095832] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11472.095836] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11472.095840] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11472.095842] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11472.095844] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11472.095846] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11472.095848] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11472.596658] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11472.605244] after_reply+0xe2c/0x1780 [ptlrpc] [11472.610728] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11472.615952] ? __next_timer_interrupt+0x160/0x160 [11472.621003] ptlrpcd+0xa82/0xe40 [ptlrpc] [11472.624549] ? do_wait_intr+0xf0/0xf0 [11472.628420] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11472.638217] kthread+0x1d1/0x200 [11472.644660] ? set_kthread_struct+0x70/0x70 [11472.647945] ret_from_fork+0x1f/0x30 [11476.058234] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11476.072760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11476.146908] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11476.160645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11476.165713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11476.178430] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11476.209371] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11476.219888] RAX: 0000000000007f07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11476.226509] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11476.233586] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11476.243142] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11476.257465] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11476.269461] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11476.279205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11476.285380] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11476.296197] Call Trace: [11476.296871] [11476.297435] ? show_regs.cold.9+0x22/0x2f [11476.298588] ? watchdog_timer_fn.cold.6+0x62/0x10a [11476.309603] ? __hrtimer_run_queues+0x1c2/0x4d0 [11476.315163] ? watchdog+0x40/0x40 [11476.319117] ? hrtimer_interrupt+0x128/0x2c0 [11476.323595] ? smp_apic_timer_interrupt+0xa6/0x270 [11476.327492] ? apic_timer_interrupt+0xf/0x20 [11476.333871] [11476.335018] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11476.341394] do_raw_spin_lock+0x177/0x190 [11476.346615] _raw_spin_lock+0x21/0x30 [11476.352712] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11476.360328] ? do_raw_spin_unlock+0x75/0x190 [11476.363520] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11476.375439] ? do_wait_intr+0xf0/0xf0 [11476.383359] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11476.391016] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11476.397690] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11476.411429] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11476.414674] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11476.427880] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11476.431558] mdc_intent_lock+0x353/0x7e0 [mdc] [11476.438661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11476.445073] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11476.453952] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11476.460796] lmv_intent_open+0x338/0x1150 [lmv] [11476.466398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11476.475610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11476.484490] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11476.489261] ll_intent_lock+0xc5/0xaa0 [lustre] [11476.493255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11476.502784] ll_lookup_it+0xe0e/0x2270 [lustre] [11476.504736] ? do_raw_spin_unlock+0x75/0x190 [11476.513887] ? _raw_spin_unlock+0x12/0x30 [11476.520500] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11476.528128] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11476.535005] ll_atomic_open+0x36a/0xd70 [lustre] [11476.537841] lookup_open+0x356/0xb40 [11476.542918] do_last+0x313/0xfc0 [11476.549470] ? nd_jump_root+0xe5/0x160 [11476.562334] ? path_init+0x437/0x520 [11476.563598] path_openat+0xf7/0x500 [11476.574492] ? do_raw_spin_unlock+0x75/0x190 [11476.581785] do_filp_open+0x99/0x140 [11476.587419] ? getname_flags+0x6e/0x330 [11476.588849] ? __check_object_size+0xff/0x256 [11476.598698] ? do_raw_spin_unlock+0x75/0x190 [11476.603720] ? _raw_spin_unlock+0x12/0x30 [11476.609813] do_sys_openat2+0x2b4/0x410 [11476.615459] do_sys_open+0x73/0xa0 [11476.621582] __x64_sys_openat+0x24/0x30 [11476.628988] do_syscall_64+0xc1/0x440 [11476.631475] entry_SYSCALL_64_after_hwframe+0x49/0xae [11476.648741] RIP: 0033:0x7fa749552332 [11476.650109] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11476.678116] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11476.694776] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11476.704985] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11476.717257] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11476.730760] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11476.741569] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11479.522308] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9562s! [11479.533019] Showing busy workqueues and worker pools: [11479.536572] workqueue events: flags=0x0 [11479.543562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11479.547028] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11479.553726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11479.556782] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11479.561089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.564833] pending: kfree_rcu_monitor [11479.566962] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11479.570996] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11479.576107] workqueue events_long: flags=0x0 [11479.578825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.582987] pending: lu_objects_flush [obdclass] [11479.586190] workqueue events_unbound: flags=0x2 [11479.588966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11479.592545] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11479.598109] workqueue events_power_efficient: flags=0x80 [11479.601916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.605310] pending: check_lifetime [11479.607194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.610571] pending: neigh_periodic_work [11479.613128] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.616696] pending: neigh_periodic_work [11479.618242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.620925] pending: do_cache_clean [11479.622788] workqueue mm_percpu_wq: flags=0x8 [11479.625453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.629043] pending: vmstat_update [11479.631075] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.634300] pending: vmstat_update [11479.636768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11479.640118] pending: vmstat_update [11479.641347] workqueue writeback: flags=0x4a [11479.643165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11479.646098] pending: wb_workfn, wb_workfn [11479.648516] workqueue xprtiod: flags=0xa [11479.652214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11479.656127] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11479.661196] workqueue ipv6_addrconf: flags=0x40008 [11479.664721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11479.668797] pending: addrconf_verify_work [11479.671355] workqueue knbd0-recv: flags=0x1a [11479.673988] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11479.678418] in-flight: 651:recv_work [11479.680693] workqueue ptlrpc_pinger: flags=0x2 [11479.683288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11479.687033] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11479.691203] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9562s workers=4 idle: 37 31846 28897 [11479.695780] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11469s workers=2 idle: 653 [11488.065646] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11488.070542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11488.157197] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11488.181303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11488.198913] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11488.207410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11488.211027] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11488.221765] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11488.225264] RAX: 0000000000007d89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11488.227864] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11488.231284] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11488.235247] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11488.241251] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11488.249944] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11488.257846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11488.259876] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11488.268880] Call Trace: [11488.270857] [11488.272220] ? show_regs.cold.9+0x22/0x2f [11488.274932] ? watchdog_timer_fn.cold.6+0x62/0x10a [11488.276768] ? __hrtimer_run_queues+0x1c2/0x4d0 [11488.278991] ? watchdog+0x40/0x40 [11488.280748] ? hrtimer_interrupt+0x128/0x2c0 [11488.282602] ? smp_apic_timer_interrupt+0xa6/0x270 [11488.288156] ? apic_timer_interrupt+0xf/0x20 [11488.294493] [11488.295192] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11488.297824] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11488.300132] do_raw_spin_lock+0x177/0x190 [11488.305129] _raw_spin_lock+0x21/0x30 [11488.307033] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11488.318246] process_one_work+0x2c8/0x700 [11488.320935] worker_thread+0x296/0x6e0 [11488.327807] ? rescuer_thread+0x570/0x570 [11488.329338] kthread+0x1d1/0x200 [11488.340969] ? set_kthread_struct+0x70/0x70 [11488.346615] ret_from_fork+0x1f/0x30 [11500.061867] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11500.063544] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11500.074155] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11500.074454] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11500.074462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11500.074464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11500.074504] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11500.074508] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11500.074513] RAX: 0000000000007e92 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11500.074515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11500.074518] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11500.074520] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11500.074522] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11500.074526] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11500.074529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11500.074532] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11500.074539] Call Trace: [11500.074544] [11500.074549] ? show_regs.cold.9+0x22/0x2f [11500.074559] ? watchdog_timer_fn.cold.6+0x62/0x10a [11500.074920] ? __hrtimer_run_queues+0x1c2/0x4d0 [11500.074923] ? watchdog+0x40/0x40 [11500.074926] ? hrtimer_interrupt+0x128/0x2c0 [11500.074929] ? smp_apic_timer_interrupt+0xa6/0x270 [11500.074935] ? apic_timer_interrupt+0xf/0x20 [11500.074938] [11500.074940] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11500.074945] do_raw_spin_lock+0x177/0x190 [11500.074948] _raw_spin_lock+0x21/0x30 [11500.074952] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11500.075233] ? do_raw_spin_unlock+0x75/0x190 [11500.075239] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11500.076032] ? do_wait_intr+0xf0/0xf0 [11500.076040] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11500.076545] mdc_close+0x53b/0xd70 [mdc] [11500.076728] lmv_close+0x18d/0x490 [lmv] [11500.076757] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11500.088261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11500.088336] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11500.088343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11500.088345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11500.088357] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11500.088360] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11500.088364] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11500.088366] RDX: 000000000000715a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11500.088368] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11500.088370] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11500.088372] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11500.088375] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11500.088378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11500.088380] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11500.088388] Call Trace: [11500.088394] [11500.088412] ? show_regs.cold.9+0x22/0x2f [11500.088422] ? watchdog_timer_fn.cold.6+0x62/0x10a [11500.088428] ? __hrtimer_run_queues+0x1c2/0x4d0 [11500.088432] ? watchdog+0x40/0x40 [11500.088436] ? hrtimer_interrupt+0x128/0x2c0 [11500.088441] ? smp_apic_timer_interrupt+0xa6/0x270 [11500.088448] ? apic_timer_interrupt+0xf/0x20 [11500.088453] [11500.088456] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11500.088460] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11500.088465] do_raw_spin_lock+0x177/0x190 [11500.088469] _raw_spin_lock+0x21/0x30 [11500.088486] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11500.164982] ll_md_real_close+0xa5/0x2a0 [lustre] [11500.165119] ll_md_close+0x274/0x800 [lustre] [11500.165205] ll_file_release+0x128/0xc50 [lustre] [11500.165301] __fput+0x10d/0x440 [11500.165313] ____fput+0x12/0x20 [11500.165318] task_work_run+0xc6/0x110 [11500.165325] exit_to_usermode_loop+0x1dd/0x1f0 [11500.165333] do_syscall_64+0x3d6/0x440 [11500.165337] entry_SYSCALL_64_after_hwframe+0x49/0xae [11500.165345] RIP: 0033:0x7fa749552cd8 [11500.165353] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11500.165356] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11500.165362] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11500.165364] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11500.165366] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11500.165368] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11500.165370] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11500.767130] obd_mod_free+0x46/0x140 [ptlrpc] [11500.770068] mdc_commit_open+0x91/0xb0 [mdc] [11500.772314] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11500.777712] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11500.781876] after_reply+0xe2c/0x1780 [ptlrpc] [11500.786668] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11500.791222] ? __next_timer_interrupt+0x160/0x160 [11500.796921] ptlrpcd+0xa82/0xe40 [ptlrpc] [11500.803212] ? do_wait_intr+0xf0/0xf0 [11500.807391] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11500.811988] kthread+0x1d1/0x200 [11500.824039] ? set_kthread_struct+0x70/0x70 [11500.826839] ret_from_fork+0x1f/0x30 [11504.054422] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11504.058380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11504.087524] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11504.098868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11504.105971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11504.111612] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11504.122636] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11504.128618] RAX: 0000000000003182 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11504.134888] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11504.138474] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11504.142048] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11504.144826] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11504.148063] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11504.152080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11504.155369] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11504.158272] Call Trace: [11504.159402] [11504.160371] ? show_regs.cold.9+0x22/0x2f [11504.162081] ? watchdog_timer_fn.cold.6+0x62/0x10a [11504.164520] ? __hrtimer_run_queues+0x1c2/0x4d0 [11504.166608] ? watchdog+0x40/0x40 [11504.168364] ? hrtimer_interrupt+0x128/0x2c0 [11504.170260] ? smp_apic_timer_interrupt+0xa6/0x270 [11504.173004] ? apic_timer_interrupt+0xf/0x20 [11504.174860] [11504.175925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11504.177746] do_raw_spin_lock+0x177/0x190 [11504.178904] _raw_spin_lock+0x21/0x30 [11504.180230] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11504.183791] ? do_raw_spin_unlock+0x75/0x190 [11504.185741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11504.188413] ? do_wait_intr+0xf0/0xf0 [11504.190094] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11504.192706] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11504.195194] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11504.197918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11504.202039] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11504.204936] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11504.207382] mdc_intent_lock+0x353/0x7e0 [mdc] [11504.209882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11504.212277] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11504.215938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11504.218989] lmv_intent_open+0x338/0x1150 [lmv] [11504.221556] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11504.224647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11504.231323] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11504.235916] ll_intent_lock+0xc5/0xaa0 [lustre] [11504.239219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11504.241851] ll_lookup_it+0xe0e/0x2270 [lustre] [11504.244461] ? do_raw_spin_unlock+0x75/0x190 [11504.246989] ? _raw_spin_unlock+0x12/0x30 [11504.250252] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11504.252712] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11504.255428] ll_atomic_open+0x36a/0xd70 [lustre] [11504.262473] lookup_open+0x356/0xb40 [11504.264970] do_last+0x313/0xfc0 [11504.266554] ? nd_jump_root+0xe5/0x160 [11504.269734] ? path_init+0x437/0x520 [11504.271810] path_openat+0xf7/0x500 [11504.273591] ? do_raw_spin_unlock+0x75/0x190 [11504.276537] do_filp_open+0x99/0x140 [11504.278182] ? getname_flags+0x6e/0x330 [11504.280995] ? __check_object_size+0xff/0x256 [11504.286633] ? do_raw_spin_unlock+0x75/0x190 [11504.289103] ? _raw_spin_unlock+0x12/0x30 [11504.291976] do_sys_openat2+0x2b4/0x410 [11504.293983] do_sys_open+0x73/0xa0 [11504.295867] __x64_sys_openat+0x24/0x30 [11504.297723] do_syscall_64+0xc1/0x440 [11504.299892] entry_SYSCALL_64_after_hwframe+0x49/0xae [11504.302968] RIP: 0033:0x7fa749552332 [11504.305093] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11504.318218] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11504.326045] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11504.333387] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11504.339303] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11504.347383] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11504.353781] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11509.734876] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9592s! [11509.742871] Showing busy workqueues and worker pools: [11509.750949] workqueue events: flags=0x0 [11509.753844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11509.762695] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11509.777301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11509.783213] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11509.793938] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.800979] pending: kfree_rcu_monitor [11509.802606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11509.807167] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11509.817852] workqueue events_long: flags=0x0 [11509.820146] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.827330] pending: lu_objects_flush [obdclass] [11509.830289] workqueue events_unbound: flags=0x2 [11509.834728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11509.839085] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11509.844345] workqueue events_power_efficient: flags=0x80 [11509.849732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.855688] pending: check_lifetime [11509.859977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.864910] pending: neigh_periodic_work [11509.867353] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.875231] pending: neigh_periodic_work [11509.877116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.881221] pending: do_cache_clean [11509.883420] workqueue mm_percpu_wq: flags=0x8 [11509.884811] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.888979] pending: vmstat_update [11509.892640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.898371] pending: vmstat_update [11509.900330] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11509.904825] pending: vmstat_update [11509.907881] workqueue writeback: flags=0x4a [11509.910232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11509.914257] pending: wb_workfn, wb_workfn [11509.919028] workqueue xprtiod: flags=0xa [11509.921250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11509.925588] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11509.934102] workqueue ipv6_addrconf: flags=0x40008 [11509.936694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11509.940857] pending: addrconf_verify_work [11509.944205] workqueue knbd0-recv: flags=0x1a [11509.953592] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11509.958288] in-flight: 651:recv_work [11509.964419] workqueue ptlrpc_pinger: flags=0x2 [11509.967354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11509.970344] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11509.974506] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9592s workers=4 idle: 37 31846 28897 [11509.979394] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11499s workers=2 idle: 653 [11513.891271] rcu: INFO: rcu_sched self-detected stall on CPU [11513.893852] rcu: 2-...!: (5414166 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [11513.897988] (t=9600202 jiffies g=258701 q=4288) [11513.901293] rcu: rcu_sched kthread starved for 9596997 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [11513.907850] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11513.915947] rcu: RCU grace-period kthread stack dump: [11513.918279] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11513.926656] Call Trace: [11513.928535] __schedule+0x351/0xcb0 [11513.931473] schedule+0xc0/0x180 [11513.934545] schedule_timeout+0xb4/0x190 [11513.936539] ? do_raw_spin_unlock+0x75/0x190 [11513.939710] ? __next_timer_interrupt+0x160/0x160 [11513.942304] ? prepare_to_swait_event+0xf3/0x140 [11513.945517] rcu_gp_kthread+0x8dd/0x15b0 [11513.948193] ? force_qs_rnp+0x2c0/0x2c0 [11513.950163] kthread+0x1d1/0x200 [11513.952042] ? set_kthread_struct+0x70/0x70 [11513.954674] ret_from_fork+0x1f/0x30 [11513.956859] rcu: Stack dump where RCU GP kthread last ran: [11513.959553] Sending NMI from CPU 2 to CPUs 0: [11513.962611] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [11513.963576] NMI backtrace for cpu 2 [11513.967673] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11513.972630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11513.977168] Call Trace: [11513.978376] [11513.979370] ? dump_stack+0xbb/0x10e [11513.981131] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11513.983587] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11513.985821] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11513.988340] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11513.991073] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11513.993772] ? rcu_dump_cpu_stacks+0x11b/0x171 [11513.995787] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11513.998334] ? update_process_times+0x6d/0xc0 [11514.000774] ? tick_sched_handle.isra.18+0x27/0x90 [11514.003162] ? tick_sched_timer+0x53/0xb0 [11514.005796] ? __hrtimer_run_queues+0x1c2/0x4d0 [11514.008192] ? tick_sched_handle.isra.18+0x90/0x90 [11514.010384] ? hrtimer_interrupt+0x128/0x2c0 [11514.012526] ? smp_apic_timer_interrupt+0xa6/0x270 [11514.015536] ? apic_timer_interrupt+0xf/0x20 [11514.017193] [11514.018295] ? native_safe_halt+0x2e/0x50 [11514.019578] ? kvm_wait+0xcd/0xe0 [11514.020940] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11514.023395] ? do_raw_spin_lock+0x177/0x190 [11514.025418] ? _raw_spin_lock+0x21/0x30 [11514.027530] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11514.030282] ? obd_mod_free+0x46/0x140 [ptlrpc] [11514.032521] ? mdc_commit_open+0x91/0xb0 [mdc] [11514.034702] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11514.036777] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11514.039194] ? after_reply+0xe2c/0x1780 [ptlrpc] [11514.041104] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11514.043533] ? __next_timer_interrupt+0x160/0x160 [11514.045473] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11514.047448] ? do_wait_intr+0xf0/0xf0 [11514.048926] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11514.051798] ? kthread+0x1d1/0x200 [11514.053304] ? set_kthread_struct+0x70/0x70 [11514.054968] ? ret_from_fork+0x1f/0x30 [11516.057143] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11516.062287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11516.102176] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11516.107229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11516.114453] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11516.121065] RIP: 0010:native_safe_halt+0x2e/0x50 [11516.123558] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11516.130429] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11516.132840] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11516.134850] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11516.136701] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11516.143129] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11516.145454] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11516.148071] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11516.150753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11516.152269] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11516.154514] Call Trace: [11516.155403] [11516.156082] ? show_regs.cold.9+0x22/0x2f [11516.157157] ? watchdog_timer_fn.cold.6+0x62/0x10a [11516.158600] ? __hrtimer_run_queues+0x1c2/0x4d0 [11516.159884] ? watchdog+0x40/0x40 [11516.161675] ? hrtimer_interrupt+0x128/0x2c0 [11516.163359] ? smp_apic_timer_interrupt+0xa6/0x270 [11516.165887] ? apic_timer_interrupt+0xf/0x20 [11516.168268] [11516.171147] ? native_safe_halt+0x2e/0x50 [11516.173388] kvm_wait+0xcd/0xe0 [11516.176897] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11516.183144] do_raw_spin_lock+0x177/0x190 [11516.189178] _raw_spin_lock+0x21/0x30 [11516.190782] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11516.198175] process_one_work+0x2c8/0x700 [11516.202163] worker_thread+0x296/0x6e0 [11516.206432] ? rescuer_thread+0x570/0x570 [11516.208388] kthread+0x1d1/0x200 [11516.209918] ? set_kthread_struct+0x70/0x70 [11516.214317] ret_from_fork+0x1f/0x30 [11528.061532] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11528.067928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11528.102614] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11528.116692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11528.121379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11528.125820] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11528.138941] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11528.144555] RAX: 0000000000006932 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11528.150235] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11528.155226] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11528.162278] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11528.169604] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11528.176979] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11528.183188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11528.190536] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11528.194682] Call Trace: [11528.196153] [11528.197007] ? show_regs.cold.9+0x22/0x2f [11528.202600] ? watchdog_timer_fn.cold.6+0x62/0x10a [11528.205506] ? __hrtimer_run_queues+0x1c2/0x4d0 [11528.211187] ? watchdog+0x40/0x40 [11528.212637] ? hrtimer_interrupt+0x128/0x2c0 [11528.223001] ? smp_apic_timer_interrupt+0xa6/0x270 [11528.225178] ? apic_timer_interrupt+0xf/0x20 [11528.227764] [11528.228591] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11528.233421] do_raw_spin_lock+0x177/0x190 [11528.236960] _raw_spin_lock+0x21/0x30 [11528.239061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11528.243962] ? do_raw_spin_unlock+0x75/0x190 [11528.249419] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11528.251773] ? do_wait_intr+0xf0/0xf0 [11528.252973] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11528.255979] mdc_close+0x53b/0xd70 [mdc] [11528.257817] lmv_close+0x18d/0x490 [lmv] [11528.261994] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11528.264379] ll_md_real_close+0xa5/0x2a0 [lustre] [11528.267408] ll_md_close+0x274/0x800 [lustre] [11528.272866] ll_file_release+0x128/0xc50 [lustre] [11528.275595] __fput+0x10d/0x440 [11528.281203] ____fput+0x12/0x20 [11528.282541] task_work_run+0xc6/0x110 [11528.284980] exit_to_usermode_loop+0x1dd/0x1f0 [11528.287052] do_syscall_64+0x3d6/0x440 [11528.291927] entry_SYSCALL_64_after_hwframe+0x49/0xae [11528.297625] RIP: 0033:0x7fa749552cd8 [11528.300685] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11528.313409] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11528.321208] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11528.324940] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11528.329688] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11528.335179] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11528.339150] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11540.054021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11540.056684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11540.060118] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11540.084771] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11540.092236] Modules linked in: [11540.097302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11540.097314] RIP: 0010:native_safe_halt+0x2e/0x50 [11540.097331] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11540.097336] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11540.097342] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11540.097345] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11540.097347] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11540.097349] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11540.097352] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11540.097355] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11540.097358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11540.097361] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11540.097370] Call Trace: [11540.097378] [11540.097384] ? show_regs.cold.9+0x22/0x2f [11540.097395] ? watchdog_timer_fn.cold.6+0x62/0x10a [11540.097403] ? __hrtimer_run_queues+0x1c2/0x4d0 [11540.097407] ? watchdog+0x40/0x40 [11540.097412] ? hrtimer_interrupt+0x128/0x2c0 [11540.097417] ? smp_apic_timer_interrupt+0xa6/0x270 [11540.097423] ? apic_timer_interrupt+0xf/0x20 [11540.097428] [11540.097431] ? native_safe_halt+0x2e/0x50 [11540.097435] kvm_wait+0xcd/0xe0 [11540.097443] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11540.097453] do_raw_spin_lock+0x177/0x190 [11540.097458] _raw_spin_lock+0x21/0x30 [11540.097462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11540.097847] ? do_raw_spin_unlock+0x75/0x190 [11540.097853] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11540.098072] ? do_wait_intr+0xf0/0xf0 [11540.098080] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11540.099698] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11540.099772] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11540.099791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11540.099794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11540.099807] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11540.099811] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11540.099816] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11540.099818] RDX: 0000000000000c5c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11540.099821] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11540.099823] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11540.099826] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11540.099829] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11540.099832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11540.099834] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11540.099843] Call Trace: [11540.099848] [11540.099853] ? show_regs.cold.9+0x22/0x2f [11540.099865] ? watchdog_timer_fn.cold.6+0x62/0x10a [11540.099871] ? __hrtimer_run_queues+0x1c2/0x4d0 [11540.099875] ? watchdog+0x40/0x40 [11540.099879] ? hrtimer_interrupt+0x128/0x2c0 [11540.099883] ? smp_apic_timer_interrupt+0xa6/0x270 [11540.099893] ? apic_timer_interrupt+0xf/0x20 [11540.099898] [11540.099901] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11540.099906] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11540.099910] do_raw_spin_lock+0x177/0x190 [11540.099915] _raw_spin_lock+0x21/0x30 [11540.099921] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11540.100214] obd_mod_free+0x46/0x140 [ptlrpc] [11540.100409] mdc_commit_open+0x91/0xb0 [mdc] [11540.100446] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11540.100713] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11540.100921] after_reply+0xe2c/0x1780 [ptlrpc] [11540.101755] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11540.102039] ? __next_timer_interrupt+0x160/0x160 [11540.102051] ptlrpcd+0xa82/0xe40 [ptlrpc] [11540.102256] ? do_wait_intr+0xf0/0xf0 [11540.102265] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11540.102477] kthread+0x1d1/0x200 [11540.102482] ? set_kthread_struct+0x70/0x70 [11540.102485] ret_from_fork+0x1f/0x30 [11540.110494] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11540.393945] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11540.400982] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.404807] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11540.407173] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11540.409422] mdc_intent_lock+0x353/0x7e0 [mdc] [11540.416314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.418786] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11540.421614] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11540.423595] lmv_intent_open+0x338/0x1150 [lmv] [11540.430053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.433160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.436443] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11540.438618] ll_intent_lock+0xc5/0xaa0 [lustre] [11540.440945] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.444359] ll_lookup_it+0xe0e/0x2270 [lustre] [11540.446112] ? do_raw_spin_unlock+0x75/0x190 [11540.447907] ? _raw_spin_unlock+0x12/0x30 [11540.449430] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11540.451571] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11540.453169] ll_atomic_open+0x36a/0xd70 [lustre] [11540.457332] lookup_open+0x356/0xb40 [11540.458893] do_last+0x313/0xfc0 [11540.460286] ? nd_jump_root+0xe5/0x160 [11540.463127] ? path_init+0x437/0x520 [11540.465001] path_openat+0xf7/0x500 [11540.466681] ? do_raw_spin_unlock+0x75/0x190 [11540.467672] do_filp_open+0x99/0x140 [11540.468417] ? getname_flags+0x6e/0x330 [11540.471072] ? __check_object_size+0xff/0x256 [11540.472754] ? do_raw_spin_unlock+0x75/0x190 [11540.476435] ? _raw_spin_unlock+0x12/0x30 [11540.478410] do_sys_openat2+0x2b4/0x410 [11540.480351] do_sys_open+0x73/0xa0 [11540.482857] __x64_sys_openat+0x24/0x30 [11540.486358] do_syscall_64+0xc1/0x440 [11540.489392] entry_SYSCALL_64_after_hwframe+0x49/0xae [11540.492401] RIP: 0033:0x7fa749552332 [11540.494146] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11540.505157] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11540.508927] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11540.512960] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11540.516963] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11540.527109] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11540.531925] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11540.535164] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9623s! [11540.538227] Showing busy workqueues and worker pools: [11540.539990] workqueue events: flags=0x0 [11540.543117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11540.547219] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11540.559043] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11540.565673] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11540.573121] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.577490] pending: kfree_rcu_monitor [11540.580262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11540.583240] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11540.588391] workqueue events_long: flags=0x0 [11540.594939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.598058] pending: lu_objects_flush [obdclass] [11540.599953] workqueue events_unbound: flags=0x2 [11540.602238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11540.605542] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11540.615064] workqueue events_power_efficient: flags=0x80 [11540.618104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.620368] pending: check_lifetime [11540.623292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.630236] pending: neigh_periodic_work [11540.632723] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.638420] pending: neigh_periodic_work [11540.641639] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.646003] pending: do_cache_clean [11540.648535] workqueue mm_percpu_wq: flags=0x8 [11540.651117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.653672] pending: vmstat_update [11540.656090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.659194] pending: vmstat_update [11540.660814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11540.663317] pending: vmstat_update [11540.665543] workqueue writeback: flags=0x4a [11540.667117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11540.670817] pending: wb_workfn, wb_workfn [11540.672728] workqueue xprtiod: flags=0xa [11540.674338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11540.677440] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11540.682249] workqueue ipv6_addrconf: flags=0x40008 [11540.686026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11540.689033] pending: addrconf_verify_work [11540.690924] workqueue knbd0-recv: flags=0x1a [11540.692803] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11540.700277] in-flight: 651:recv_work [11540.703866] workqueue ptlrpc_pinger: flags=0x2 [11540.705419] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11540.711010] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11540.714476] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9623s workers=4 idle: 37 31846 28897 [11540.718817] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11530s workers=2 idle: 653 [11544.057063] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11544.059374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11544.074340] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11544.077997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11544.080960] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11544.083292] RIP: 0010:native_safe_halt+0x2e/0x50 [11544.084815] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11544.090791] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11544.093278] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11544.095548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11544.097900] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11544.100321] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11544.102724] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11544.104904] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11544.107665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11544.109693] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11544.111897] Call Trace: [11544.112747] [11544.113403] ? show_regs.cold.9+0x22/0x2f [11544.114947] ? watchdog_timer_fn.cold.6+0x62/0x10a [11544.116650] ? __hrtimer_run_queues+0x1c2/0x4d0 [11544.118341] ? watchdog+0x40/0x40 [11544.119521] ? hrtimer_interrupt+0x128/0x2c0 [11544.121103] ? smp_apic_timer_interrupt+0xa6/0x270 [11544.122702] ? apic_timer_interrupt+0xf/0x20 [11544.124381] [11544.125190] ? native_safe_halt+0x2e/0x50 [11544.126432] kvm_wait+0xcd/0xe0 [11544.127716] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11544.129576] do_raw_spin_lock+0x177/0x190 [11544.130905] _raw_spin_lock+0x21/0x30 [11544.132295] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11544.134324] process_one_work+0x2c8/0x700 [11544.135837] worker_thread+0x296/0x6e0 [11544.137178] ? rescuer_thread+0x570/0x570 [11544.138719] kthread+0x1d1/0x200 [11544.139815] ? set_kthread_struct+0x70/0x70 [11544.141156] ret_from_fork+0x1f/0x30 [11556.061423] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11556.068574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11556.114191] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11556.126897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11556.137525] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11556.142725] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11556.161284] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11556.166839] RAX: 000000000000778f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11556.169943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11556.176058] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11556.181443] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11556.187081] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11556.201150] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11556.203837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11556.210796] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11556.214853] Call Trace: [11556.215876] [11556.217187] ? show_regs.cold.9+0x22/0x2f [11556.220670] ? watchdog_timer_fn.cold.6+0x62/0x10a [11556.222322] ? __hrtimer_run_queues+0x1c2/0x4d0 [11556.223478] ? watchdog+0x40/0x40 [11556.227001] ? hrtimer_interrupt+0x128/0x2c0 [11556.231084] ? smp_apic_timer_interrupt+0xa6/0x270 [11556.236124] ? apic_timer_interrupt+0xf/0x20 [11556.240732] [11556.241790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11556.244978] do_raw_spin_lock+0x177/0x190 [11556.246792] _raw_spin_lock+0x21/0x30 [11556.248014] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11556.252887] ? do_raw_spin_unlock+0x75/0x190 [11556.255181] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11556.260405] ? do_wait_intr+0xf0/0xf0 [11556.262912] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11556.266114] mdc_close+0x53b/0xd70 [mdc] [11556.276933] lmv_close+0x18d/0x490 [lmv] [11556.284898] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11556.292341] ll_md_real_close+0xa5/0x2a0 [lustre] [11556.299047] ll_md_close+0x274/0x800 [lustre] [11556.301789] ll_file_release+0x128/0xc50 [lustre] [11556.305060] __fput+0x10d/0x440 [11556.306811] ____fput+0x12/0x20 [11556.308839] task_work_run+0xc6/0x110 [11556.310983] exit_to_usermode_loop+0x1dd/0x1f0 [11556.315195] do_syscall_64+0x3d6/0x440 [11556.317153] entry_SYSCALL_64_after_hwframe+0x49/0xae [11556.321741] RIP: 0033:0x7fa749552cd8 [11556.325559] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11556.350154] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11556.369665] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11556.379573] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11556.392092] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11556.399175] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11556.408538] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11568.054027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11568.058759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11568.061030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11568.090100] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11568.090117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11568.090120] RIP: 0010:native_safe_halt+0x2e/0x50 [11568.090134] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11568.090138] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11568.090144] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11568.090147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11568.090149] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11568.090151] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11568.090154] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11568.090158] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11568.090161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.090163] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11568.090173] Call Trace: [11568.090366] [11568.090374] ? show_regs.cold.9+0x22/0x2f [11568.090385] ? watchdog_timer_fn.cold.6+0x62/0x10a [11568.090392] ? __hrtimer_run_queues+0x1c2/0x4d0 [11568.090396] ? watchdog+0x40/0x40 [11568.090470] ? hrtimer_interrupt+0x128/0x2c0 [11568.090475] ? smp_apic_timer_interrupt+0xa6/0x270 [11568.090482] ? apic_timer_interrupt+0xf/0x20 [11568.090488] [11568.090491] ? native_safe_halt+0x2e/0x50 [11568.090496] kvm_wait+0xcd/0xe0 [11568.090505] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11568.090515] do_raw_spin_lock+0x177/0x190 [11568.090520] _raw_spin_lock+0x21/0x30 [11568.090524] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11568.090785] ? do_raw_spin_unlock+0x75/0x190 [11568.090791] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11568.090990] ? do_wait_intr+0xf0/0xf0 [11568.092072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11568.092580] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11568.092808] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11568.094086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.094501] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11568.094542] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11568.094575] mdc_intent_lock+0x353/0x7e0 [mdc] [11568.094606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.098726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11568.098811] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11568.098819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11568.098822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11568.098835] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11568.098840] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11568.098845] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11568.098848] RDX: 0000000000007f03 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11568.098851] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11568.098853] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11568.098856] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11568.098859] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11568.098862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.098864] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11568.098873] Call Trace: [11568.098879] [11568.098885] ? show_regs.cold.9+0x22/0x2f [11568.098894] ? watchdog_timer_fn.cold.6+0x62/0x10a [11568.098901] ? __hrtimer_run_queues+0x1c2/0x4d0 [11568.098905] ? watchdog+0x40/0x40 [11568.098910] ? hrtimer_interrupt+0x128/0x2c0 [11568.098914] ? smp_apic_timer_interrupt+0xa6/0x270 [11568.098923] ? apic_timer_interrupt+0xf/0x20 [11568.098928] [11568.098931] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11568.098936] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11568.098941] do_raw_spin_lock+0x177/0x190 [11568.098946] _raw_spin_lock+0x21/0x30 [11568.098954] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11568.099884] obd_mod_free+0x46/0x140 [ptlrpc] [11568.100132] mdc_commit_open+0x91/0xb0 [mdc] [11568.100163] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11568.100344] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11568.104830] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11568.105266] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11568.105326] lmv_intent_open+0x338/0x1150 [lmv] [11568.105363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.105562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.105654] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11568.105753] ll_intent_lock+0xc5/0xaa0 [lustre] [11568.105847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.106080] ll_lookup_it+0xe0e/0x2270 [lustre] [11568.106172] ? do_raw_spin_unlock+0x75/0x190 [11568.106183] ? _raw_spin_unlock+0x12/0x30 [11568.106190] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11568.106215] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11568.106236] ll_atomic_open+0x36a/0xd70 [lustre] [11568.106339] lookup_open+0x356/0xb40 [11568.106349] do_last+0x313/0xfc0 [11568.106354] ? nd_jump_root+0xe5/0x160 [11568.106359] ? path_init+0x437/0x520 [11568.106364] path_openat+0xf7/0x500 [11568.106369] ? do_raw_spin_unlock+0x75/0x190 [11568.106374] do_filp_open+0x99/0x140 [11568.106381] ? getname_flags+0x6e/0x330 [11568.106385] ? __check_object_size+0xff/0x256 [11568.106394] ? do_raw_spin_unlock+0x75/0x190 [11568.106472] ? _raw_spin_unlock+0x12/0x30 [11568.106478] do_sys_openat2+0x2b4/0x410 [11568.106484] do_sys_open+0x73/0xa0 [11568.106489] __x64_sys_openat+0x24/0x30 [11568.106493] do_syscall_64+0xc1/0x440 [11568.106500] entry_SYSCALL_64_after_hwframe+0x49/0xae [11568.106507] RIP: 0033:0x7fa749552332 [11568.106514] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11568.106519] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11568.106524] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11568.106527] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11568.106530] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11568.106532] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11568.106535] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11568.469432] after_reply+0xe2c/0x1780 [ptlrpc] [11568.470849] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11568.472476] ? __next_timer_interrupt+0x160/0x160 [11568.473582] ptlrpcd+0xa82/0xe40 [ptlrpc] [11568.475252] ? do_wait_intr+0xf0/0xf0 [11568.476335] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11568.477801] kthread+0x1d1/0x200 [11568.478729] ? set_kthread_struct+0x70/0x70 [11568.479855] ret_from_fork+0x1f/0x30 [11571.172376] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9654s! [11571.182781] Showing busy workqueues and worker pools: [11571.185427] workqueue events: flags=0x0 [11571.189463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11571.191286] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11571.198190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11571.201220] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11571.205628] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.208580] pending: kfree_rcu_monitor [11571.210052] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11571.212708] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11571.216207] workqueue events_long: flags=0x0 [11571.218426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.221272] pending: lu_objects_flush [obdclass] [11571.224243] workqueue events_unbound: flags=0x2 [11571.226680] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11571.230763] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11571.236540] workqueue events_power_efficient: flags=0x80 [11571.239528] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.243389] pending: check_lifetime [11571.244776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.247757] pending: neigh_periodic_work [11571.249335] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.252912] pending: neigh_periodic_work [11571.254890] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.258703] pending: do_cache_clean [11571.261218] workqueue mm_percpu_wq: flags=0x8 [11571.263328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.266860] pending: vmstat_update [11571.268089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.272122] pending: vmstat_update [11571.273750] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11571.276157] pending: vmstat_update [11571.278353] workqueue writeback: flags=0x4a [11571.280448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11571.283252] pending: wb_workfn, wb_workfn [11571.285215] workqueue xprtiod: flags=0xa [11571.286698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11571.288931] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11571.292392] workqueue ipv6_addrconf: flags=0x40008 [11571.294514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11571.297190] pending: addrconf_verify_work [11571.299577] workqueue knbd0-recv: flags=0x1a [11571.301812] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11571.304793] in-flight: 651:recv_work [11571.306590] workqueue ptlrpc_pinger: flags=0x2 [11571.308675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11571.311024] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11571.313072] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9654s workers=4 idle: 37 31846 28897 [11571.316604] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11561s workers=2 idle: 653 [11572.057029] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11572.060928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11572.089338] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11572.094773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11572.099107] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11572.102304] RIP: 0010:native_safe_halt+0x2e/0x50 [11572.104812] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11572.114173] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11572.118118] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11572.121553] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11572.124839] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11572.128414] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11572.131798] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11572.135097] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11572.138838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11572.141598] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11572.145098] Call Trace: [11572.146263] [11572.147369] ? show_regs.cold.9+0x22/0x2f [11572.149199] ? watchdog_timer_fn.cold.6+0x62/0x10a [11572.151795] ? __hrtimer_run_queues+0x1c2/0x4d0 [11572.154120] ? watchdog+0x40/0x40 [11572.155353] ? hrtimer_interrupt+0x128/0x2c0 [11572.158135] ? smp_apic_timer_interrupt+0xa6/0x270 [11572.160594] ? apic_timer_interrupt+0xf/0x20 [11572.163248] [11572.164356] ? native_safe_halt+0x2e/0x50 [11572.166727] kvm_wait+0xcd/0xe0 [11572.170556] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11572.176251] do_raw_spin_lock+0x177/0x190 [11572.178161] _raw_spin_lock+0x21/0x30 [11572.182268] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11572.191778] process_one_work+0x2c8/0x700 [11572.194341] worker_thread+0x296/0x6e0 [11572.198403] ? rescuer_thread+0x570/0x570 [11572.201381] kthread+0x1d1/0x200 [11572.207545] ? set_kthread_struct+0x70/0x70 [11572.209433] ret_from_fork+0x1f/0x30 [11584.059038] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11584.063855] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11584.094679] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11584.099610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11584.103368] RIP: 0010:native_safe_halt+0x2e/0x50 [11584.104866] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11584.113650] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11584.117235] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11584.120442] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11584.124256] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11584.127486] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11584.130477] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11584.134451] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11584.142592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11584.147902] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11584.151422] Call Trace: [11584.152931] [11584.154484] ? show_regs.cold.9+0x22/0x2f [11584.156329] ? watchdog_timer_fn.cold.6+0x62/0x10a [11584.158783] ? __hrtimer_run_queues+0x1c2/0x4d0 [11584.160823] ? watchdog+0x40/0x40 [11584.162391] ? hrtimer_interrupt+0x128/0x2c0 [11584.164352] ? smp_apic_timer_interrupt+0xa6/0x270 [11584.166512] ? apic_timer_interrupt+0xf/0x20 [11584.168859] [11584.169623] ? native_safe_halt+0x2e/0x50 [11584.171612] kvm_wait+0xcd/0xe0 [11584.173020] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11584.175322] do_raw_spin_lock+0x177/0x190 [11584.177313] _raw_spin_lock+0x21/0x30 [11584.178767] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11584.181825] ? do_raw_spin_unlock+0x75/0x190 [11584.183749] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11584.185043] ? do_wait_intr+0xf0/0xf0 [11584.185900] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11584.188084] mdc_close+0x53b/0xd70 [mdc] [11584.189381] lmv_close+0x18d/0x490 [lmv] [11584.190598] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11584.195304] ll_md_real_close+0xa5/0x2a0 [lustre] [11584.197485] ll_md_close+0x274/0x800 [lustre] [11584.198998] ll_file_release+0x128/0xc50 [lustre] [11584.202970] __fput+0x10d/0x440 [11584.204844] ____fput+0x12/0x20 [11584.205639] task_work_run+0xc6/0x110 [11584.206555] exit_to_usermode_loop+0x1dd/0x1f0 [11584.209241] do_syscall_64+0x3d6/0x440 [11584.210550] entry_SYSCALL_64_after_hwframe+0x49/0xae [11584.212168] RIP: 0033:0x7fa749552cd8 [11584.213558] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11584.221515] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11584.224470] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11584.226381] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11584.228220] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11584.230667] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11584.233470] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11596.054559] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11596.060738] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [11596.060760] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11596.060764] lquota(O) [11596.060771] Modules linked in: [11596.072498] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix [11596.076743] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11596.076829] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11596.076836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11596.076839] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11596.076939] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11596.076944] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11596.076949] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11596.076951] RDX: 0000000000005c05 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11596.076954] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11596.076957] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11596.076959] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11596.076961] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11596.076964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11596.076966] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11596.076975] Call Trace: [11596.076980] [11596.076986] ? show_regs.cold.9+0x22/0x2f [11596.076995] ? watchdog_timer_fn.cold.6+0x62/0x10a [11596.077002] ? __hrtimer_run_queues+0x1c2/0x4d0 [11596.077006] ? watchdog+0x40/0x40 [11596.077010] ? hrtimer_interrupt+0x128/0x2c0 [11596.077015] ? smp_apic_timer_interrupt+0xa6/0x270 [11596.077024] ? apic_timer_interrupt+0xf/0x20 [11596.077030] [11596.077033] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11596.077039] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11596.077043] do_raw_spin_lock+0x177/0x190 [11596.077048] _raw_spin_lock+0x21/0x30 [11596.077055] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11596.077500] obd_mod_free+0x46/0x140 [ptlrpc] [11596.081821] serio_raw [11596.085381] mdc_commit_open+0x91/0xb0 [mdc] [11596.115804] libata [11596.147145] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11596.156624] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11596.156644] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11596.163206] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11596.163481] after_reply+0xe2c/0x1780 [ptlrpc] [11596.164027] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11596.164248] ? __next_timer_interrupt+0x160/0x160 [11596.164258] ptlrpcd+0xa82/0xe40 [ptlrpc] [11596.164455] ? do_wait_intr+0xf0/0xf0 [11596.164463] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11596.168756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11596.168766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11596.170312] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11596.170325] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11596.170332] RAX: 0000000000007f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11596.170334] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11596.170337] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11596.170339] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11596.170341] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11596.170344] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11596.170347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11596.170350] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11596.170359] Call Trace: [11596.170364] [11596.170371] ? show_regs.cold.9+0x22/0x2f [11596.170383] ? watchdog_timer_fn.cold.6+0x62/0x10a [11596.170389] ? __hrtimer_run_queues+0x1c2/0x4d0 [11596.170393] ? watchdog+0x40/0x40 [11596.170398] ? hrtimer_interrupt+0x128/0x2c0 [11596.170402] ? smp_apic_timer_interrupt+0xa6/0x270 [11596.170410] ? apic_timer_interrupt+0xf/0x20 [11596.170416] [11596.170419] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11596.170429] do_raw_spin_lock+0x177/0x190 [11596.170434] _raw_spin_lock+0x21/0x30 [11596.170438] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11596.171472] ? do_raw_spin_unlock+0x75/0x190 [11596.171483] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11596.171780] ? do_wait_intr+0xf0/0xf0 [11596.171789] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11596.172087] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11596.181758] kthread+0x1d1/0x200 [11596.181778] ? set_kthread_struct+0x70/0x70 [11596.181783] ret_from_fork+0x1f/0x30 [11596.483839] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11596.488230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.489978] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11596.494379] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11596.495884] mdc_intent_lock+0x353/0x7e0 [mdc] [11596.497286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.498977] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11596.500881] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11596.502674] lmv_intent_open+0x338/0x1150 [lmv] [11596.504154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.505974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.507759] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11596.509216] ll_intent_lock+0xc5/0xaa0 [lustre] [11596.510865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.514829] ll_lookup_it+0xe0e/0x2270 [lustre] [11596.516623] ? do_raw_spin_unlock+0x75/0x190 [11596.518418] ? _raw_spin_unlock+0x12/0x30 [11596.520838] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11596.523711] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11596.525158] ll_atomic_open+0x36a/0xd70 [lustre] [11596.533657] lookup_open+0x356/0xb40 [11596.535031] do_last+0x313/0xfc0 [11596.537185] ? nd_jump_root+0xe5/0x160 [11596.538465] ? path_init+0x437/0x520 [11596.539641] path_openat+0xf7/0x500 [11596.542895] ? do_raw_spin_unlock+0x75/0x190 [11596.545327] do_filp_open+0x99/0x140 [11596.547240] ? getname_flags+0x6e/0x330 [11596.549228] ? __check_object_size+0xff/0x256 [11596.555378] ? do_raw_spin_unlock+0x75/0x190 [11596.557395] ? _raw_spin_unlock+0x12/0x30 [11596.559433] do_sys_openat2+0x2b4/0x410 [11596.561268] do_sys_open+0x73/0xa0 [11596.563010] __x64_sys_openat+0x24/0x30 [11596.566003] do_syscall_64+0xc1/0x440 [11596.567872] entry_SYSCALL_64_after_hwframe+0x49/0xae [11596.569988] RIP: 0033:0x7fa749552332 [11596.571465] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11596.581542] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11596.596014] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11596.602536] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11596.609141] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11596.613303] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11596.622925] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11600.057053] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11600.059944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11600.115651] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11600.130002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11600.142582] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11600.152191] RIP: 0010:native_safe_halt+0x2e/0x50 [11600.158815] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11600.179747] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11600.189179] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11600.191638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11600.204254] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11600.211670] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11600.218681] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11600.228539] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11600.239826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11600.245674] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11600.254943] Call Trace: [11600.258975] [11600.261442] ? show_regs.cold.9+0x22/0x2f [11600.266960] ? watchdog_timer_fn.cold.6+0x62/0x10a [11600.270233] ? __hrtimer_run_queues+0x1c2/0x4d0 [11600.278896] ? watchdog+0x40/0x40 [11600.283961] ? hrtimer_interrupt+0x128/0x2c0 [11600.289994] ? smp_apic_timer_interrupt+0xa6/0x270 [11600.296749] ? apic_timer_interrupt+0xf/0x20 [11600.302059] [11600.303101] ? native_safe_halt+0x2e/0x50 [11600.309021] kvm_wait+0xcd/0xe0 [11600.311710] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11600.319221] do_raw_spin_lock+0x177/0x190 [11600.325821] _raw_spin_lock+0x21/0x30 [11600.327595] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11600.333759] process_one_work+0x2c8/0x700 [11600.338656] worker_thread+0x296/0x6e0 [11600.340246] ? rescuer_thread+0x570/0x570 [11600.345718] kthread+0x1d1/0x200 [11600.346966] ? set_kthread_struct+0x70/0x70 [11600.352847] ret_from_fork+0x1f/0x30 [11601.385368] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9684s! [11601.398503] Showing busy workqueues and worker pools: [11601.401561] workqueue events: flags=0x0 [11601.414083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11601.426557] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11601.442532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11601.455021] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11601.470926] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.479098] pending: kfree_rcu_monitor [11601.485130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11601.497379] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11601.507198] workqueue events_long: flags=0x0 [11601.519105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.539461] pending: lu_objects_flush [obdclass] [11601.551424] workqueue events_unbound: flags=0x2 [11601.560144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11601.567298] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11601.574843] workqueue events_power_efficient: flags=0x80 [11601.581696] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.589332] pending: check_lifetime [11601.592869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.602464] pending: neigh_periodic_work [11601.613306] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.627364] pending: neigh_periodic_work [11601.630912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.635324] pending: do_cache_clean [11601.638818] workqueue mm_percpu_wq: flags=0x8 [11601.642091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.645924] pending: vmstat_update [11601.648403] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.652172] pending: vmstat_update [11601.654123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11601.661959] pending: vmstat_update [11601.664257] workqueue writeback: flags=0x4a [11601.668297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11601.674459] pending: wb_workfn, wb_workfn [11601.678627] workqueue xprtiod: flags=0xa [11601.681909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11601.689053] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11601.695329] workqueue ipv6_addrconf: flags=0x40008 [11601.704083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11601.708929] pending: addrconf_verify_work [11601.715307] workqueue knbd0-recv: flags=0x1a [11601.719233] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11601.722878] in-flight: 651:recv_work [11601.726633] workqueue ptlrpc_pinger: flags=0x2 [11601.729559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11601.736596] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11601.744218] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9684s workers=4 idle: 37 31846 28897 [11601.752327] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11591s workers=2 idle: 653 [11612.062784] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11612.077237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11612.118062] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11612.126628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11612.133061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11612.138698] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11612.151685] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11612.155983] RAX: 0000000000007710 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11612.164021] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11612.176103] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11612.184241] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11612.195287] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11612.202696] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11612.211857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11612.215968] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11612.221044] Call Trace: [11612.222139] [11612.224447] ? show_regs.cold.9+0x22/0x2f [11612.226669] ? watchdog_timer_fn.cold.6+0x62/0x10a [11612.228610] ? __hrtimer_run_queues+0x1c2/0x4d0 [11612.231171] ? watchdog+0x40/0x40 [11612.232862] ? hrtimer_interrupt+0x128/0x2c0 [11612.235682] ? smp_apic_timer_interrupt+0xa6/0x270 [11612.237758] ? apic_timer_interrupt+0xf/0x20 [11612.239654] [11612.240678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11612.243139] do_raw_spin_lock+0x177/0x190 [11612.244764] _raw_spin_lock+0x21/0x30 [11612.246418] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11612.248328] ? do_raw_spin_unlock+0x75/0x190 [11612.249634] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11612.251450] ? do_wait_intr+0xf0/0xf0 [11612.252897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11612.255839] mdc_close+0x53b/0xd70 [mdc] [11612.258415] lmv_close+0x18d/0x490 [lmv] [11612.260761] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11612.263046] ll_md_real_close+0xa5/0x2a0 [lustre] [11612.265343] ll_md_close+0x274/0x800 [lustre] [11612.266943] ll_file_release+0x128/0xc50 [lustre] [11612.269112] __fput+0x10d/0x440 [11612.270595] ____fput+0x12/0x20 [11612.272353] task_work_run+0xc6/0x110 [11612.273822] exit_to_usermode_loop+0x1dd/0x1f0 [11612.275456] do_syscall_64+0x3d6/0x440 [11612.277615] entry_SYSCALL_64_after_hwframe+0x49/0xae [11612.279795] RIP: 0033:0x7fa749552cd8 [11612.281344] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11612.289009] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11612.293633] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11612.296708] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11612.300775] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11612.304092] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11612.307107] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11624.061599] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11624.065828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11624.109936] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11624.115243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11624.123536] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11624.127776] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11624.144395] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11624.152411] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11624.164296] RDX: 000000000000082b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11624.174442] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11624.182116] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11624.189676] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11624.197764] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11624.206080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11624.213462] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11624.222708] Call Trace: [11624.224314] [11624.227435] ? show_regs.cold.9+0x22/0x2f [11624.231718] ? watchdog_timer_fn.cold.6+0x62/0x10a [11624.238928] ? __hrtimer_run_queues+0x1c2/0x4d0 [11624.241128] ? watchdog+0x40/0x40 [11624.244663] ? hrtimer_interrupt+0x128/0x2c0 [11624.249094] ? smp_apic_timer_interrupt+0xa6/0x270 [11624.253460] ? apic_timer_interrupt+0xf/0x20 [11624.257032] [11624.261357] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11624.267309] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11624.274471] do_raw_spin_lock+0x177/0x190 [11624.279385] _raw_spin_lock+0x21/0x30 [11624.284384] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11624.290759] obd_mod_free+0x46/0x140 [ptlrpc] [11624.293736] mdc_commit_open+0x91/0xb0 [mdc] [11624.299403] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11624.305581] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11624.312375] after_reply+0xe2c/0x1780 [ptlrpc] [11624.315481] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11624.324923] ? __next_timer_interrupt+0x160/0x160 [11624.327746] ptlrpcd+0xa82/0xe40 [ptlrpc] [11624.332786] ? do_wait_intr+0xf0/0xf0 [11624.335977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11624.343101] kthread+0x1d1/0x200 [11624.344712] ? set_kthread_struct+0x70/0x70 [11624.351886] ret_from_fork+0x1f/0x30 [11628.057077] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11628.060627] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11628.063587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11628.063666] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11628.063673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11628.063679] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11628.063946] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11628.073574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11628.073652] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11628.073786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11628.073791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11628.073804] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11628.073809] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11628.073826] RAX: 0000000000005770 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11628.073829] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11628.073831] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11628.073833] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11628.073835] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11628.073838] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11628.073841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11628.073843] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11628.073850] Call Trace: [11628.073856] [11628.073862] ? show_regs.cold.9+0x22/0x2f [11628.073871] ? watchdog_timer_fn.cold.6+0x62/0x10a [11628.073877] ? __hrtimer_run_queues+0x1c2/0x4d0 [11628.073882] ? watchdog+0x40/0x40 [11628.073885] ? hrtimer_interrupt+0x128/0x2c0 [11628.073890] ? smp_apic_timer_interrupt+0xa6/0x270 [11628.073898] ? apic_timer_interrupt+0xf/0x20 [11628.073902] [11628.073905] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11628.073912] do_raw_spin_lock+0x177/0x190 [11628.073917] _raw_spin_lock+0x21/0x30 [11628.073922] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11628.132111] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11628.132126] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11628.132132] RAX: 0000000000000e90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11628.132134] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11628.132136] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11628.132138] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11628.132141] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11628.132144] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11628.132147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11628.132149] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11628.132220] Call Trace: [11628.132227] [11628.132241] ? show_regs.cold.9+0x22/0x2f [11628.132254] ? watchdog_timer_fn.cold.6+0x62/0x10a [11628.132260] ? __hrtimer_run_queues+0x1c2/0x4d0 [11628.132265] ? watchdog+0x40/0x40 [11628.132269] ? hrtimer_interrupt+0x128/0x2c0 [11628.132273] ? smp_apic_timer_interrupt+0xa6/0x270 [11628.132280] ? apic_timer_interrupt+0xf/0x20 [11628.132285] [11628.132287] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11628.132293] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11628.132301] do_raw_spin_lock+0x177/0x190 [11628.132306] _raw_spin_lock+0x21/0x30 [11628.132311] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11628.132571] process_one_work+0x2c8/0x700 [11628.132578] worker_thread+0x296/0x6e0 [11628.132583] ? rescuer_thread+0x570/0x570 [11628.132586] kthread+0x1d1/0x200 [11628.132590] ? set_kthread_struct+0x70/0x70 [11628.132593] ret_from_fork+0x1f/0x30 [11628.790285] ? do_raw_spin_unlock+0x75/0x190 [11628.793015] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11628.798315] ? do_wait_intr+0xf0/0xf0 [11628.806130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11628.815416] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11628.821339] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11628.826132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11628.836638] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11628.842699] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11628.852258] mdc_intent_lock+0x353/0x7e0 [mdc] [11628.858153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11628.866275] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11628.876666] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11628.878839] lmv_intent_open+0x338/0x1150 [lmv] [11628.884302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11628.893653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11628.895587] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11628.900318] ll_intent_lock+0xc5/0xaa0 [lustre] [11628.906608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11628.908706] ll_lookup_it+0xe0e/0x2270 [lustre] [11628.913768] ? do_raw_spin_unlock+0x75/0x190 [11628.916680] ? _raw_spin_unlock+0x12/0x30 [11628.925413] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11628.932889] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11628.937893] ll_atomic_open+0x36a/0xd70 [lustre] [11628.952024] lookup_open+0x356/0xb40 [11628.953732] do_last+0x313/0xfc0 [11628.963312] ? nd_jump_root+0xe5/0x160 [11628.968025] ? path_init+0x437/0x520 [11628.972404] path_openat+0xf7/0x500 [11628.976065] ? do_raw_spin_unlock+0x75/0x190 [11628.980342] do_filp_open+0x99/0x140 [11628.981600] ? getname_flags+0x6e/0x330 [11628.989413] ? __check_object_size+0xff/0x256 [11628.995754] ? do_raw_spin_unlock+0x75/0x190 [11629.001523] ? _raw_spin_unlock+0x12/0x30 [11629.006606] do_sys_openat2+0x2b4/0x410 [11629.008688] do_sys_open+0x73/0xa0 [11629.015108] __x64_sys_openat+0x24/0x30 [11629.016775] do_syscall_64+0xc1/0x440 [11629.022768] entry_SYSCALL_64_after_hwframe+0x49/0xae [11629.033005] RIP: 0033:0x7fa749552332 [11629.042763] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11629.066003] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11629.081085] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11629.096740] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11629.104530] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11629.111471] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11629.117354] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11632.103031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9714s! [11632.107572] Showing busy workqueues and worker pools: [11632.133192] workqueue events: flags=0x0 [11632.143222] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11632.163410] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11632.183559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11632.195743] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11632.205793] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.214061] pending: kfree_rcu_monitor [11632.219798] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11632.226750] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11632.232350] workqueue events_long: flags=0x0 [11632.240300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.260286] pending: lu_objects_flush [obdclass] [11632.269575] workqueue events_unbound: flags=0x2 [11632.279232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11632.292216] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11632.306813] workqueue events_power_efficient: flags=0x80 [11632.326310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.340664] pending: check_lifetime [11632.342062] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.363779] pending: neigh_periodic_work [11632.370970] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.373142] pending: neigh_periodic_work [11632.385782] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.395911] pending: do_cache_clean [11632.414181] workqueue mm_percpu_wq: flags=0x8 [11632.415863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.429962] pending: vmstat_update [11632.431444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.448665] pending: vmstat_update [11632.453826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11632.473808] pending: vmstat_update [11632.477857] workqueue writeback: flags=0x4a [11632.489313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11632.495341] pending: wb_workfn, wb_workfn [11632.502533] workqueue xprtiod: flags=0xa [11632.503940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11632.516856] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11632.528862] workqueue ipv6_addrconf: flags=0x40008 [11632.533993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11632.548500] pending: addrconf_verify_work [11632.558548] workqueue knbd0-recv: flags=0x1a [11632.565439] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11632.575636] in-flight: 651:recv_work [11632.580230] workqueue ptlrpc_pinger: flags=0x2 [11632.585523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11632.610439] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11632.621462] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9715s workers=4 idle: 37 31846 28897 [11632.636682] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11622s workers=2 idle: 653 [11640.059017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11640.063275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11640.094869] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11640.102369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11640.109184] RIP: 0010:native_safe_halt+0x2e/0x50 [11640.113987] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11640.135136] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11640.138509] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11640.142216] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11640.145039] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11640.147834] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11640.150904] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11640.153360] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11640.156610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11640.158206] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11640.160087] Call Trace: [11640.160728] [11640.161338] ? show_regs.cold.9+0x22/0x2f [11640.162427] ? watchdog_timer_fn.cold.6+0x62/0x10a [11640.163806] ? __hrtimer_run_queues+0x1c2/0x4d0 [11640.164973] ? watchdog+0x40/0x40 [11640.166090] ? hrtimer_interrupt+0x128/0x2c0 [11640.167173] ? smp_apic_timer_interrupt+0xa6/0x270 [11640.168496] ? apic_timer_interrupt+0xf/0x20 [11640.169580] [11640.170088] ? native_safe_halt+0x2e/0x50 [11640.171017] kvm_wait+0xcd/0xe0 [11640.171870] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11640.173103] do_raw_spin_lock+0x177/0x190 [11640.174362] _raw_spin_lock+0x21/0x30 [11640.175208] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11640.177016] ? do_raw_spin_unlock+0x75/0x190 [11640.178661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11640.180692] ? do_wait_intr+0xf0/0xf0 [11640.182014] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11640.184182] mdc_close+0x53b/0xd70 [mdc] [11640.185541] lmv_close+0x18d/0x490 [lmv] [11640.187168] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11640.189341] ll_md_real_close+0xa5/0x2a0 [lustre] [11640.190982] ll_md_close+0x274/0x800 [lustre] [11640.192471] ll_file_release+0x128/0xc50 [lustre] [11640.193842] __fput+0x10d/0x440 [11640.194604] ____fput+0x12/0x20 [11640.195529] task_work_run+0xc6/0x110 [11640.196523] exit_to_usermode_loop+0x1dd/0x1f0 [11640.197691] do_syscall_64+0x3d6/0x440 [11640.198701] entry_SYSCALL_64_after_hwframe+0x49/0xae [11640.200000] RIP: 0033:0x7fa749552cd8 [11640.201009] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11640.206395] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11640.208126] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11640.210351] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11640.213450] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11640.216590] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11640.219716] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11652.058008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11652.062523] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11652.108175] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11652.121287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11652.126472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11652.129114] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11652.142387] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11652.146426] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11652.152585] RDX: 00000000000015a9 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11652.157417] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11652.162150] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11652.167172] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11652.173259] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11652.177214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11652.181256] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11652.189180] Call Trace: [11652.190024] [11652.193905] ? show_regs.cold.9+0x22/0x2f [11652.195589] ? watchdog_timer_fn.cold.6+0x62/0x10a [11652.197329] ? __hrtimer_run_queues+0x1c2/0x4d0 [11652.199935] ? watchdog+0x40/0x40 [11652.202233] ? hrtimer_interrupt+0x128/0x2c0 [11652.204729] ? smp_apic_timer_interrupt+0xa6/0x270 [11652.206439] ? apic_timer_interrupt+0xf/0x20 [11652.209556] [11652.210396] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11652.215084] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11652.220358] do_raw_spin_lock+0x177/0x190 [11652.229782] _raw_spin_lock+0x21/0x30 [11652.234099] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11652.236551] obd_mod_free+0x46/0x140 [ptlrpc] [11652.241681] mdc_commit_open+0x91/0xb0 [mdc] [11652.247338] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11652.254573] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11652.264448] after_reply+0xe2c/0x1780 [ptlrpc] [11652.267396] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11652.274456] ? __next_timer_interrupt+0x160/0x160 [11652.281435] ptlrpcd+0xa82/0xe40 [ptlrpc] [11652.284995] ? do_wait_intr+0xf0/0xf0 [11652.291276] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11652.301280] kthread+0x1d1/0x200 [11652.303442] ? set_kthread_struct+0x70/0x70 [11652.306547] ret_from_fork+0x1f/0x30 [11656.057678] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11656.063509] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11656.115577] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11656.130950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11656.137382] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11656.141697] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11656.145311] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11656.165186] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11656.173448] RAX: 0000000000007a87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11656.177074] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11656.186133] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11656.193241] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11656.199233] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11656.204987] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11656.212010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11656.215207] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11656.219356] Call Trace: [11656.220892] [11656.221942] ? show_regs.cold.9+0x22/0x2f [11656.224149] ? watchdog_timer_fn.cold.6+0x62/0x10a [11656.228075] ? __hrtimer_run_queues+0x1c2/0x4d0 [11656.229344] ? watchdog+0x40/0x40 [11656.231124] ? hrtimer_interrupt+0x128/0x2c0 [11656.232716] ? smp_apic_timer_interrupt+0xa6/0x270 [11656.234298] ? apic_timer_interrupt+0xf/0x20 [11656.236884] [11656.237859] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11656.240030] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11656.242041] do_raw_spin_lock+0x177/0x190 [11656.243596] _raw_spin_lock+0x21/0x30 [11656.244861] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11656.246897] process_one_work+0x2c8/0x700 [11656.248661] worker_thread+0x296/0x6e0 [11656.250293] ? rescuer_thread+0x570/0x570 [11656.251845] kthread+0x1d1/0x200 [11656.252628] ? set_kthread_struct+0x70/0x70 [11656.253951] ret_from_fork+0x1f/0x30 [11660.054035] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11660.057695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11660.080617] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11660.085754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11660.099513] RIP: 0010:native_safe_halt+0x2e/0x50 [11660.103046] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11660.120191] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11660.126613] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11660.131990] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11660.140728] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11660.150496] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11660.157329] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11660.162296] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11660.173793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11660.177509] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11660.181616] Call Trace: [11660.182715] [11660.183684] ? show_regs.cold.9+0x22/0x2f [11660.185962] ? watchdog_timer_fn.cold.6+0x62/0x10a [11660.189761] ? __hrtimer_run_queues+0x1c2/0x4d0 [11660.191909] ? watchdog+0x40/0x40 [11660.193890] ? hrtimer_interrupt+0x128/0x2c0 [11660.195726] ? smp_apic_timer_interrupt+0xa6/0x270 [11660.202825] ? apic_timer_interrupt+0xf/0x20 [11660.207856] [11660.211121] ? native_safe_halt+0x2e/0x50 [11660.213114] kvm_wait+0xcd/0xe0 [11660.219764] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11660.223694] do_raw_spin_lock+0x177/0x190 [11660.226104] _raw_spin_lock+0x21/0x30 [11660.230053] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11660.233102] ? do_raw_spin_unlock+0x75/0x190 [11660.236497] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11660.243101] ? do_wait_intr+0xf0/0xf0 [11660.246291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11660.251261] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11660.254735] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11660.257514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.260811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11660.269751] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11660.275625] mdc_intent_lock+0x353/0x7e0 [mdc] [11660.280780] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.286294] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11660.295658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11660.307739] lmv_intent_open+0x338/0x1150 [lmv] [11660.312032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.325974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.330962] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11660.332473] ll_intent_lock+0xc5/0xaa0 [lustre] [11660.345041] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.354183] ll_lookup_it+0xe0e/0x2270 [lustre] [11660.360785] ? do_raw_spin_unlock+0x75/0x190 [11660.368169] ? _raw_spin_unlock+0x12/0x30 [11660.381814] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11660.391251] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11660.400533] ll_atomic_open+0x36a/0xd70 [lustre] [11660.402199] lookup_open+0x356/0xb40 [11660.409835] do_last+0x313/0xfc0 [11660.411020] ? nd_jump_root+0xe5/0x160 [11660.420827] ? path_init+0x437/0x520 [11660.424877] path_openat+0xf7/0x500 [11660.435168] ? do_raw_spin_unlock+0x75/0x190 [11660.436642] do_filp_open+0x99/0x140 [11660.443746] ? getname_flags+0x6e/0x330 [11660.445808] ? __check_object_size+0xff/0x256 [11660.454897] ? do_raw_spin_unlock+0x75/0x190 [11660.459955] ? _raw_spin_unlock+0x12/0x30 [11660.464487] do_sys_openat2+0x2b4/0x410 [11660.466950] do_sys_open+0x73/0xa0 [11660.470482] __x64_sys_openat+0x24/0x30 [11660.472739] do_syscall_64+0xc1/0x440 [11660.475852] entry_SYSCALL_64_after_hwframe+0x49/0xae [11660.479518] RIP: 0033:0x7fa749552332 [11660.481982] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11660.494018] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11660.496539] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11660.500337] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11660.514656] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11660.526927] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11660.536136] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11662.816239] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9745s! [11662.827076] Showing busy workqueues and worker pools: [11662.829709] workqueue events: flags=0x0 [11662.835260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11662.840460] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11662.848435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11662.855617] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11662.858146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.864409] pending: kfree_rcu_monitor [11662.867697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11662.872327] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11662.881879] workqueue events_long: flags=0x0 [11662.889255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.893665] pending: lu_objects_flush [obdclass] [11662.901430] workqueue events_unbound: flags=0x2 [11662.902780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11662.911715] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11662.921294] workqueue events_power_efficient: flags=0x80 [11662.922922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.928877] pending: check_lifetime [11662.930244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.936088] pending: neigh_periodic_work [11662.937380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.943646] pending: neigh_periodic_work [11662.947261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.953215] pending: do_cache_clean [11662.956131] workqueue mm_percpu_wq: flags=0x8 [11662.957686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.965922] pending: vmstat_update [11662.970733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.977620] pending: vmstat_update [11662.979935] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11662.985886] pending: vmstat_update [11662.991261] workqueue writeback: flags=0x4a [11662.994206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11663.000662] pending: wb_workfn, wb_workfn [11663.004984] workqueue xprtiod: flags=0xa [11663.008159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11663.014825] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11663.021967] workqueue ipv6_addrconf: flags=0x40008 [11663.027735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11663.034881] pending: addrconf_verify_work [11663.040286] workqueue knbd0-recv: flags=0x1a [11663.041913] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11663.048020] in-flight: 651:recv_work [11663.052869] workqueue ptlrpc_pinger: flags=0x2 [11663.054725] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11663.060849] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11663.066903] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9745s workers=4 idle: 37 31846 28897 [11663.073012] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11652s workers=2 idle: 653 [11668.061341] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11668.068453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11668.113999] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11668.121160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11668.130834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11668.139808] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11668.162039] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11668.175109] RAX: 0000000000005a83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11668.180423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11668.185850] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11668.201325] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11668.205723] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11668.209146] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11668.219336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11668.224298] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11668.231742] Call Trace: [11668.233192] [11668.235082] ? show_regs.cold.9+0x22/0x2f [11668.237441] ? watchdog_timer_fn.cold.6+0x62/0x10a [11668.244076] ? __hrtimer_run_queues+0x1c2/0x4d0 [11668.247769] ? watchdog+0x40/0x40 [11668.249646] ? hrtimer_interrupt+0x128/0x2c0 [11668.251841] ? smp_apic_timer_interrupt+0xa6/0x270 [11668.255138] ? apic_timer_interrupt+0xf/0x20 [11668.260338] [11668.262463] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11668.266578] do_raw_spin_lock+0x177/0x190 [11668.269477] _raw_spin_lock+0x21/0x30 [11668.270782] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11668.274881] ? do_raw_spin_unlock+0x75/0x190 [11668.278077] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11668.285086] ? do_wait_intr+0xf0/0xf0 [11668.288889] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11668.297395] mdc_close+0x53b/0xd70 [mdc] [11668.303780] lmv_close+0x18d/0x490 [lmv] [11668.308149] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11668.311073] ll_md_real_close+0xa5/0x2a0 [lustre] [11668.317416] ll_md_close+0x274/0x800 [lustre] [11668.320865] ll_file_release+0x128/0xc50 [lustre] [11668.324195] __fput+0x10d/0x440 [11668.327682] ____fput+0x12/0x20 [11668.333611] task_work_run+0xc6/0x110 [11668.338884] exit_to_usermode_loop+0x1dd/0x1f0 [11668.342309] do_syscall_64+0x3d6/0x440 [11668.344328] entry_SYSCALL_64_after_hwframe+0x49/0xae [11668.350376] RIP: 0033:0x7fa749552cd8 [11668.351461] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11668.367273] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11668.375509] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11668.383549] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11668.393719] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11668.400398] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11668.407841] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11680.058008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11680.063311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11680.088964] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11680.095517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11680.101838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11680.104856] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11680.114582] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11680.118542] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11680.128478] RDX: 0000000000002e88 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11680.131178] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11680.137761] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11680.144100] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11680.150912] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11680.154569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11680.164371] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11680.174524] Call Trace: [11680.176664] [11680.179874] ? show_regs.cold.9+0x22/0x2f [11680.183790] ? watchdog_timer_fn.cold.6+0x62/0x10a [11680.187622] ? __hrtimer_run_queues+0x1c2/0x4d0 [11680.189888] ? watchdog+0x40/0x40 [11680.194535] ? hrtimer_interrupt+0x128/0x2c0 [11680.197208] ? smp_apic_timer_interrupt+0xa6/0x270 [11680.202401] ? apic_timer_interrupt+0xf/0x20 [11680.206318] [11680.208367] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11680.213735] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11680.219218] do_raw_spin_lock+0x177/0x190 [11680.222667] _raw_spin_lock+0x21/0x30 [11680.226129] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11680.230884] obd_mod_free+0x46/0x140 [ptlrpc] [11680.235407] mdc_commit_open+0x91/0xb0 [mdc] [11680.239800] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11680.244241] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11680.249382] after_reply+0xe2c/0x1780 [ptlrpc] [11680.257881] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11680.268218] ? __next_timer_interrupt+0x160/0x160 [11680.274659] ptlrpcd+0xa82/0xe40 [ptlrpc] [11680.278609] ? do_wait_intr+0xf0/0xf0 [11680.280828] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11680.284586] kthread+0x1d1/0x200 [11680.286638] ? set_kthread_struct+0x70/0x70 [11680.290251] ret_from_fork+0x1f/0x30 [11684.057229] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11684.066490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11684.116551] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11684.138550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11684.148606] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11684.155846] RIP: 0010:native_safe_halt+0x2e/0x50 [11684.159997] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11684.179711] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11684.187778] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11684.198413] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11684.201860] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11684.207615] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11684.215225] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11684.223692] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11684.227709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11684.236510] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11684.244312] Call Trace: [11684.246779] [11684.248347] ? show_regs.cold.9+0x22/0x2f [11684.252866] ? watchdog_timer_fn.cold.6+0x62/0x10a [11684.257717] ? __hrtimer_run_queues+0x1c2/0x4d0 [11684.260229] ? watchdog+0x40/0x40 [11684.261934] ? hrtimer_interrupt+0x128/0x2c0 [11684.264054] ? smp_apic_timer_interrupt+0xa6/0x270 [11684.268783] ? apic_timer_interrupt+0xf/0x20 [11684.272807] [11684.276107] ? native_safe_halt+0x2e/0x50 [11684.279985] kvm_wait+0xcd/0xe0 [11684.281814] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11684.286640] do_raw_spin_lock+0x177/0x190 [11684.290077] _raw_spin_lock+0x21/0x30 [11684.291886] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11684.298809] process_one_work+0x2c8/0x700 [11684.301701] worker_thread+0x296/0x6e0 [11684.304955] ? rescuer_thread+0x570/0x570 [11684.306752] kthread+0x1d1/0x200 [11684.310089] ? set_kthread_struct+0x70/0x70 [11684.314435] ret_from_fork+0x1f/0x30 [11688.054868] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11688.067707] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11688.106233] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11688.118337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11688.123338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11688.127254] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11688.138695] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11688.143911] RAX: 0000000000001a8c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11688.148320] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11688.152257] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11688.156702] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11688.163676] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11688.170731] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11688.176429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11688.186560] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11688.196653] Call Trace: [11688.198572] [11688.200600] ? show_regs.cold.9+0x22/0x2f [11688.202773] ? watchdog_timer_fn.cold.6+0x62/0x10a [11688.209271] ? __hrtimer_run_queues+0x1c2/0x4d0 [11688.213712] ? watchdog+0x40/0x40 [11688.216521] ? hrtimer_interrupt+0x128/0x2c0 [11688.223257] ? smp_apic_timer_interrupt+0xa6/0x270 [11688.225029] ? apic_timer_interrupt+0xf/0x20 [11688.231175] [11688.231954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11688.238055] do_raw_spin_lock+0x177/0x190 [11688.239857] _raw_spin_lock+0x21/0x30 [11688.245406] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11688.251668] ? do_raw_spin_unlock+0x75/0x190 [11688.254247] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11688.260525] ? do_wait_intr+0xf0/0xf0 [11688.263127] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11688.266854] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11688.276599] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11688.283139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.285875] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11688.290767] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11688.296655] mdc_intent_lock+0x353/0x7e0 [mdc] [11688.301868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.303875] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11688.314689] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11688.318738] lmv_intent_open+0x338/0x1150 [lmv] [11688.321578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.328646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.333820] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11688.337331] ll_intent_lock+0xc5/0xaa0 [lustre] [11688.339623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.343724] ll_lookup_it+0xe0e/0x2270 [lustre] [11688.346964] ? do_raw_spin_unlock+0x75/0x190 [11688.349524] ? _raw_spin_unlock+0x12/0x30 [11688.352304] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11688.355390] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11688.362458] ll_atomic_open+0x36a/0xd70 [lustre] [11688.368234] lookup_open+0x356/0xb40 [11688.370398] do_last+0x313/0xfc0 [11688.371513] ? nd_jump_root+0xe5/0x160 [11688.372897] ? path_init+0x437/0x520 [11688.380696] path_openat+0xf7/0x500 [11688.382459] ? do_raw_spin_unlock+0x75/0x190 [11688.386814] do_filp_open+0x99/0x140 [11688.392734] ? getname_flags+0x6e/0x330 [11688.398594] ? __check_object_size+0xff/0x256 [11688.401288] ? do_raw_spin_unlock+0x75/0x190 [11688.406295] ? _raw_spin_unlock+0x12/0x30 [11688.408945] do_sys_openat2+0x2b4/0x410 [11688.410386] do_sys_open+0x73/0xa0 [11688.415863] __x64_sys_openat+0x24/0x30 [11688.418139] do_syscall_64+0xc1/0x440 [11688.419548] entry_SYSCALL_64_after_hwframe+0x49/0xae [11688.421478] RIP: 0033:0x7fa749552332 [11688.422846] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11688.430519] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11688.433739] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11688.436685] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11688.442534] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11688.444647] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11688.448745] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11693.539108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9776s! [11693.542169] Showing busy workqueues and worker pools: [11693.549374] workqueue events: flags=0x0 [11693.550645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11693.552606] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11693.561567] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11693.567056] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11693.571875] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.576610] pending: kfree_rcu_monitor [11693.579189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11693.594243] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11693.602575] workqueue events_long: flags=0x0 [11693.607891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.610308] pending: lu_objects_flush [obdclass] [11693.611904] workqueue events_unbound: flags=0x2 [11693.617391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11693.622364] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11693.631415] workqueue events_power_efficient: flags=0x80 [11693.637411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.650422] pending: check_lifetime [11693.655262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.662321] pending: neigh_periodic_work [11693.665627] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.676756] pending: neigh_periodic_work [11693.680449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.689389] pending: do_cache_clean [11693.696708] workqueue mm_percpu_wq: flags=0x8 [11693.700721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.702919] pending: vmstat_update [11693.712352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.725876] pending: vmstat_update [11693.727175] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11693.731419] pending: vmstat_update [11693.738728] workqueue writeback: flags=0x4a [11693.742793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11693.745467] pending: wb_workfn, wb_workfn [11693.749932] workqueue xprtiod: flags=0xa [11693.750937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11693.753876] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11693.762463] workqueue ipv6_addrconf: flags=0x40008 [11693.763988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11693.767228] pending: addrconf_verify_work [11693.770435] workqueue knbd0-recv: flags=0x1a [11693.772856] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11693.776411] in-flight: 651:recv_work [11693.778674] workqueue ptlrpc_pinger: flags=0x2 [11693.782730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11693.785751] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11693.792954] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9776s workers=4 idle: 37 31846 28897 [11693.797639] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11683s workers=2 idle: 653 [11693.899019] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [11693.901566] rcu: 2-...!: (5528085 ticks this GP) idle=4aa/1/0x4000000000000000 softirq=131143/131143 fqs=645 [11693.908283] (detected by 3, t=9780218 jiffies, g=258701, q=4349) [11693.910403] Sending NMI from CPU 3 to CPUs 2: [11693.916025] NMI backtrace for cpu 2 [11693.916034] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11693.916037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11693.916038] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11693.916040] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11693.916043] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 [11693.916046] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11693.916048] RDX: 0000000000005638 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11693.916330] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11693.916333] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11693.916334] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11693.916336] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11693.916338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11693.916339] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11693.916341] Call Trace: [11693.916342] [11693.916343] ? show_regs.cold.9+0x22/0x2f [11693.916343] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11693.916345] ? nmi_cpu_backtrace_handler+0x15/0x40 [11693.916346] ? nmi_handle+0xbd/0x210 [11693.916347] ? default_do_nmi+0x4b/0x1b0 [11693.916348] ? do_nmi+0x280/0x380 [11693.916349] ? end_repeat_nmi+0x16/0x1f [11693.916350] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11693.916351] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11693.916352] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11693.916353] [11693.916354] do_raw_spin_lock+0x177/0x190 [11693.916355] _raw_spin_lock+0x21/0x30 [11693.916356] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11693.916357] obd_mod_free+0x46/0x140 [ptlrpc] [11693.916358] mdc_commit_open+0x91/0xb0 [mdc] [11693.916359] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11693.916361] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11693.916362] after_reply+0xe2c/0x1780 [ptlrpc] [11693.916363] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11693.916365] ? __next_timer_interrupt+0x160/0x160 [11693.916366] ptlrpcd+0xa82/0xe40 [ptlrpc] [11693.916367] ? do_wait_intr+0xf0/0xf0 [11693.916368] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11693.916369] kthread+0x1d1/0x200 [11693.916370] ? set_kthread_struct+0x70/0x70 [11693.916371] ret_from_fork+0x1f/0x30 [11693.922114] rcu: rcu_sched kthread starved for 9777015 jiffies! g258701 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [11694.123539] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11694.132309] rcu: RCU grace-period kthread stack dump: [11694.139205] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11694.148289] Call Trace: [11694.149427] __schedule+0x351/0xcb0 [11694.150463] schedule+0xc0/0x180 [11694.158906] schedule_timeout+0xb4/0x190 [11694.160423] ? do_raw_spin_unlock+0x75/0x190 [11694.163260] ? __next_timer_interrupt+0x160/0x160 [11694.168388] ? prepare_to_swait_event+0xf3/0x140 [11694.173876] rcu_gp_kthread+0x8dd/0x15b0 [11694.176897] ? force_qs_rnp+0x2c0/0x2c0 [11694.179912] kthread+0x1d1/0x200 [11694.189946] ? set_kthread_struct+0x70/0x70 [11694.190982] ret_from_fork+0x1f/0x30 [11694.191891] rcu: Stack dump where RCU GP kthread last ran: [11694.196902] Sending NMI from CPU 3 to CPUs 1: [11694.203460] NMI backtrace for cpu 1 [11694.203467] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11694.203468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11694.203470] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11694.203472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.203474] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11694.203476] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 [11694.203478] RAX: 000000000000608f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11694.203479] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11694.203480] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11694.203481] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11694.203482] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11694.203493] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11694.203495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11694.203496] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11694.203498] Call Trace: [11694.203499] [11694.203500] ? show_regs.cold.9+0x22/0x2f [11694.203500] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11694.203502] ? nmi_cpu_backtrace_handler+0x15/0x40 [11694.203503] ? nmi_handle+0xbd/0x210 [11694.203504] ? default_do_nmi+0x4b/0x1b0 [11694.203505] ? do_nmi+0x280/0x380 [11694.203505] ? end_repeat_nmi+0x16/0x1f [11694.203506] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.203508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.203509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.203510] [11694.203511] do_raw_spin_lock+0x177/0x190 [11694.203512] _raw_spin_lock+0x21/0x30 [11694.203513] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11694.203514] process_one_work+0x2c8/0x700 [11694.203516] worker_thread+0x296/0x6e0 [11694.203517] ? rescuer_thread+0x570/0x570 [11694.203518] kthread+0x1d1/0x200 [11694.203519] ? set_kthread_struct+0x70/0x70 [11694.203520] ret_from_fork+0x1f/0x30 [11720.054094] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11720.057283] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11720.059027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11720.059039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.059108] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.059114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.059117] RIP: 0010:native_safe_halt+0x2e/0x50 [11720.059130] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11720.059135] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11720.059140] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11720.059142] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11720.059143] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11720.059145] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11720.059147] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11720.059151] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11720.059154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.059156] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11720.059165] Call Trace: [11720.059170] [11720.059176] ? show_regs.cold.9+0x22/0x2f [11720.059185] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.059192] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.059196] ? watchdog+0x40/0x40 [11720.059200] ? hrtimer_interrupt+0x128/0x2c0 [11720.059204] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.059211] ? apic_timer_interrupt+0xf/0x20 [11720.059215] [11720.059218] ? native_safe_halt+0x2e/0x50 [11720.059222] kvm_wait+0xcd/0xe0 [11720.059229] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11720.059238] do_raw_spin_lock+0x177/0x190 [11720.059242] _raw_spin_lock+0x21/0x30 [11720.059247] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11720.059531] ? do_raw_spin_unlock+0x75/0x190 [11720.059537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11720.059819] ? do_wait_intr+0xf0/0xf0 [11720.059827] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11720.060054] mdc_close+0x53b/0xd70 [mdc] [11720.060104] lmv_close+0x18d/0x490 [lmv] [11720.060137] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11720.060267] ll_md_real_close+0xa5/0x2a0 [lustre] [11720.060364] ll_md_close+0x274/0x800 [lustre] [11720.060516] ll_file_release+0x128/0xc50 [lustre] [11720.060666] __fput+0x10d/0x440 [11720.060680] ____fput+0x12/0x20 [11720.060688] task_work_run+0xc6/0x110 [11720.060698] exit_to_usermode_loop+0x1dd/0x1f0 [11720.060705] do_syscall_64+0x3d6/0x440 [11720.060710] entry_SYSCALL_64_after_hwframe+0x49/0xae [11720.060731] RIP: 0033:0x7fa749552cd8 [11720.060741] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11720.060745] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11720.060776] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11720.060778] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11720.060784] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11720.060786] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11720.060788] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11720.060853] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11720.060858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.060933] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.060938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.060941] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11720.060947] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11720.060951] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11720.060955] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11720.060957] RDX: 0000000000004ba0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11720.060958] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11720.060960] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11720.060962] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11720.060965] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11720.060967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.060969] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11720.060976] Call Trace: [11720.060980] [11720.060982] ? show_regs.cold.9+0x22/0x2f [11720.060990] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.060996] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.061012] ? watchdog+0x40/0x40 [11720.061017] ? hrtimer_interrupt+0x128/0x2c0 [11720.061020] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.061026] ? apic_timer_interrupt+0xf/0x20 [11720.061030] [11720.061033] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11720.061038] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11720.061043] do_raw_spin_lock+0x177/0x190 [11720.061048] _raw_spin_lock+0x21/0x30 [11720.061052] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11720.061285] obd_mod_free+0x46/0x140 [ptlrpc] [11720.061544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.061607] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.061613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.061615] RIP: 0010:native_safe_halt+0x2e/0x50 [11720.061634] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11720.061638] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11720.061643] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11720.061645] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11720.061647] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11720.061649] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11720.061652] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11720.061655] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11720.061658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.061661] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11720.061668] Call Trace: [11720.061671] [11720.061674] ? show_regs.cold.9+0x22/0x2f [11720.061681] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.061687] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.061691] ? watchdog+0x40/0x40 [11720.061696] ? hrtimer_interrupt+0x128/0x2c0 [11720.061700] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.061706] ? apic_timer_interrupt+0xf/0x20 [11720.061711] [11720.061714] ? native_safe_halt+0x2e/0x50 [11720.061719] kvm_wait+0xcd/0xe0 [11720.061729] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11720.061736] do_raw_spin_lock+0x177/0x190 [11720.061740] _raw_spin_lock+0x21/0x30 [11720.061744] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11720.062171] ? do_raw_spin_unlock+0x75/0x190 [11720.062177] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11720.062655] ? do_wait_intr+0xf0/0xf0 [11720.062664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11720.062894] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11720.063596] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11720.063822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.064077] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11720.064172] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11720.064201] mdc_intent_lock+0x353/0x7e0 [mdc] [11720.064758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.064856] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11720.065363] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11720.065394] lmv_intent_open+0x338/0x1150 [lmv] [11720.065423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.065714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.065800] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11720.065822] ll_intent_lock+0xc5/0xaa0 [lustre] [11720.066980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.067002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.067060] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.067068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.067075] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11720.067531] RIP: 0010:native_safe_halt+0x2e/0x50 [11720.067420] ll_lookup_it+0xe0e/0x2270 [lustre] [11720.067616] ? do_raw_spin_unlock+0x75/0x190 [11720.067637] ? _raw_spin_unlock+0x12/0x30 [11720.067642] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11720.067672] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11720.067691] ll_atomic_open+0x36a/0xd70 [lustre] [11720.067780] lookup_open+0x356/0xb40 [11720.067788] do_last+0x313/0xfc0 [11720.067794] ? nd_jump_root+0xe5/0x160 [11720.067798] ? path_init+0x437/0x520 [11720.067803] path_openat+0xf7/0x500 [11720.067807] ? do_raw_spin_unlock+0x75/0x190 [11720.067811] do_filp_open+0x99/0x140 [11720.067818] ? getname_flags+0x6e/0x330 [11720.067822] ? __check_object_size+0xff/0x256 [11720.067830] ? do_raw_spin_unlock+0x75/0x190 [11720.067835] ? _raw_spin_unlock+0x12/0x30 [11720.067841] do_sys_openat2+0x2b4/0x410 [11720.067845] do_sys_open+0x73/0xa0 [11720.067851] __x64_sys_openat+0x24/0x30 [11720.067855] do_syscall_64+0xc1/0x440 [11720.067862] entry_SYSCALL_64_after_hwframe+0x49/0xae [11720.067868] RIP: 0033:0x7fa749552332 [11720.067874] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11720.067878] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11720.067883] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11720.067885] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11720.067887] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11720.067889] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11720.067892] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11720.071746] mdc_commit_open+0x91/0xb0 [mdc] [11720.071796] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11720.072093] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11720.072304] after_reply+0xe2c/0x1780 [ptlrpc] [11720.072504] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11720.072727] ? __next_timer_interrupt+0x160/0x160 [11720.072740] ptlrpcd+0xa82/0xe40 [ptlrpc] [11720.072966] ? do_wait_intr+0xf0/0xf0 [11720.072977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11720.073173] kthread+0x1d1/0x200 [11720.073180] ? set_kthread_struct+0x70/0x70 [11720.073183] ret_from_fork+0x1f/0x30 [11720.800654] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11720.823569] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11720.828444] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11720.832157] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11720.834971] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11720.839694] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11720.843629] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11720.853012] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11720.861104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.864755] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11720.872097] Call Trace: [11720.876222] [11720.876993] ? show_regs.cold.9+0x22/0x2f [11720.878400] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.881962] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.887018] ? watchdog+0x40/0x40 [11720.889893] ? hrtimer_interrupt+0x128/0x2c0 [11720.892125] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.895875] ? apic_timer_interrupt+0xf/0x20 [11720.897993] [11720.900639] ? native_safe_halt+0x2e/0x50 [11720.903980] kvm_wait+0xcd/0xe0 [11720.907108] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11720.914083] do_raw_spin_lock+0x177/0x190 [11720.916433] _raw_spin_lock+0x21/0x30 [11720.917878] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11720.920769] process_one_work+0x2c8/0x700 [11720.926788] worker_thread+0x296/0x6e0 [11720.928093] ? rescuer_thread+0x570/0x570 [11720.929513] kthread+0x1d1/0x200 [11720.942830] ? set_kthread_struct+0x70/0x70 [11720.947505] ret_from_fork+0x1f/0x30 [11724.256272] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9807s! [11724.264048] Showing busy workqueues and worker pools: [11724.271611] workqueue events: flags=0x0 [11724.272926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11724.289097] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11724.303560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11724.315069] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11724.323536] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.334364] pending: kfree_rcu_monitor [11724.339169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11724.352420] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11724.361006] workqueue events_long: flags=0x0 [11724.365711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.376994] pending: lu_objects_flush [obdclass] [11724.382218] workqueue events_unbound: flags=0x2 [11724.387911] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11724.404242] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11724.422155] workqueue events_power_efficient: flags=0x80 [11724.431352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.441613] pending: check_lifetime [11724.445034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.456662] pending: neigh_periodic_work [11724.460033] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.466363] pending: neigh_periodic_work [11724.469543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.479740] pending: do_cache_clean [11724.481494] workqueue mm_percpu_wq: flags=0x8 [11724.485104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.489006] pending: vmstat_update [11724.492253] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.495961] pending: vmstat_update [11724.498447] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11724.502575] pending: vmstat_update [11724.506305] workqueue writeback: flags=0x4a [11724.507992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11724.513991] pending: wb_workfn, wb_workfn [11724.516030] workqueue xprtiod: flags=0xa [11724.517427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11724.524517] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11724.528721] workqueue ipv6_addrconf: flags=0x40008 [11724.534577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11724.539834] pending: addrconf_verify_work [11724.544108] workqueue knbd0-recv: flags=0x1a [11724.549905] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11724.563092] in-flight: 651:recv_work [11724.567973] workqueue ptlrpc_pinger: flags=0x2 [11724.581407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11724.592018] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11724.594296] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9807s workers=4 idle: 37 31846 28897 [11724.603228] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11714s workers=2 idle: 653 [11748.057093] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11748.058646] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11748.059690] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.059771] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.059778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.059785] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11748.060279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.060290] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11748.060294] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11748.060299] RAX: 000000000000322d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11748.060302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11748.060304] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11748.060307] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11748.060309] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11748.060313] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11748.060316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.060318] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11748.060326] Call Trace: [11748.060331] [11748.060336] ? show_regs.cold.9+0x22/0x2f [11748.060345] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.060351] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.060354] ? watchdog+0x40/0x40 [11748.060359] ? hrtimer_interrupt+0x128/0x2c0 [11748.060363] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.060371] ? apic_timer_interrupt+0xf/0x20 [11748.060375] [11748.060378] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.060383] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11748.060388] do_raw_spin_lock+0x177/0x190 [11748.060393] _raw_spin_lock+0x21/0x30 [11748.060398] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11748.060650] process_one_work+0x2c8/0x700 [11748.060658] worker_thread+0x296/0x6e0 [11748.060662] ? rescuer_thread+0x570/0x570 [11748.060666] kthread+0x1d1/0x200 [11748.060671] ? set_kthread_struct+0x70/0x70 [11748.060675] ret_from_fork+0x1f/0x30 [11748.066707] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11748.066720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.066795] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.066801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.066803] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.066823] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11748.066828] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11748.066832] RAX: 0000000000000792 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11748.066835] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11748.066837] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11748.066840] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11748.066842] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11748.066845] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11748.066848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.066851] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11748.066859] Call Trace: [11748.066865] [11748.066871] ? show_regs.cold.9+0x22/0x2f [11748.066880] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.066886] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.066891] ? watchdog+0x40/0x40 [11748.066896] ? hrtimer_interrupt+0x128/0x2c0 [11748.066901] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.066909] ? apic_timer_interrupt+0xf/0x20 [11748.066914] [11748.066917] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.066924] do_raw_spin_lock+0x177/0x190 [11748.066929] _raw_spin_lock+0x21/0x30 [11748.066934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11748.085570] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.085646] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.085654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.085656] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11748.085671] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11748.085676] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11748.085680] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11748.085682] RDX: 00000000000018c0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11748.085684] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11748.085698] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11748.085700] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11748.085703] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11748.085706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.085708] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11748.085716] Call Trace: [11748.085721] [11748.085727] ? show_regs.cold.9+0x22/0x2f [11748.085738] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.085744] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.085748] ? watchdog+0x40/0x40 [11748.085752] ? hrtimer_interrupt+0x128/0x2c0 [11748.085756] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.085765] ? apic_timer_interrupt+0xf/0x20 [11748.085769] [11748.085772] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11748.085777] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11748.085781] do_raw_spin_lock+0x177/0x190 [11748.085786] _raw_spin_lock+0x21/0x30 [11748.085792] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11748.086072] obd_mod_free+0x46/0x140 [ptlrpc] [11748.086329] mdc_commit_open+0x91/0xb0 [mdc] [11748.086387] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11748.086637] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11748.086851] after_reply+0xe2c/0x1780 [ptlrpc] [11748.087049] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11748.087553] ? __next_timer_interrupt+0x160/0x160 [11748.087563] ptlrpcd+0xa82/0xe40 [ptlrpc] [11748.087874] ? do_wait_intr+0xf0/0xf0 [11748.087884] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11748.088083] kthread+0x1d1/0x200 [11748.088088] ? set_kthread_struct+0x70/0x70 [11748.088092] ret_from_fork+0x1f/0x30 [11748.355435] ? do_raw_spin_unlock+0x75/0x190 [11748.357054] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11748.359092] ? do_wait_intr+0xf0/0xf0 [11748.360457] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11748.362069] mdc_close+0x53b/0xd70 [mdc] [11748.363153] lmv_close+0x18d/0x490 [lmv] [11748.364192] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11748.365760] ll_md_real_close+0xa5/0x2a0 [lustre] [11748.367041] ll_md_close+0x274/0x800 [lustre] [11748.368268] ll_file_release+0x128/0xc50 [lustre] [11748.369484] __fput+0x10d/0x440 [11748.370505] ____fput+0x12/0x20 [11748.371306] task_work_run+0xc6/0x110 [11748.372229] exit_to_usermode_loop+0x1dd/0x1f0 [11748.373716] do_syscall_64+0x3d6/0x440 [11748.374768] entry_SYSCALL_64_after_hwframe+0x49/0xae [11748.376035] RIP: 0033:0x7fa749552cd8 [11748.377192] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11748.387152] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11748.389814] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11748.392103] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11748.394058] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11748.395988] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11748.398528] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11752.061572] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11752.070481] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11752.148554] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11752.162683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11752.175614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11752.180648] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11752.210260] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11752.225806] RAX: 0000000000000a88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11752.233623] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11752.245371] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11752.253877] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11752.263397] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11752.274433] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11752.281262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11752.292846] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11752.294935] Call Trace: [11752.309597] [11752.310334] ? show_regs.cold.9+0x22/0x2f [11752.311745] ? watchdog_timer_fn.cold.6+0x62/0x10a [11752.321757] ? __hrtimer_run_queues+0x1c2/0x4d0 [11752.332845] ? watchdog+0x40/0x40 [11752.342720] ? hrtimer_interrupt+0x128/0x2c0 [11752.344219] ? smp_apic_timer_interrupt+0xa6/0x270 [11752.354655] ? apic_timer_interrupt+0xf/0x20 [11752.361494] [11752.365103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11752.372077] do_raw_spin_lock+0x177/0x190 [11752.376453] _raw_spin_lock+0x21/0x30 [11752.384696] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11752.390473] ? do_raw_spin_unlock+0x75/0x190 [11752.392475] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11752.396606] ? do_wait_intr+0xf0/0xf0 [11752.398483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11752.411840] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11752.413636] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11752.431513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11752.447306] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11752.457614] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11752.472744] mdc_intent_lock+0x353/0x7e0 [mdc] [11752.479915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11752.491700] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11752.503068] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11752.504644] lmv_intent_open+0x338/0x1150 [lmv] [11752.506658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11752.508693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11752.511272] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11752.513310] ll_intent_lock+0xc5/0xaa0 [lustre] [11752.515389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11752.517863] ll_lookup_it+0xe0e/0x2270 [lustre] [11752.520325] ? do_raw_spin_unlock+0x75/0x190 [11752.522342] ? _raw_spin_unlock+0x12/0x30 [11752.524118] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11752.527701] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11752.529526] ll_atomic_open+0x36a/0xd70 [lustre] [11752.531365] lookup_open+0x356/0xb40 [11752.532449] do_last+0x313/0xfc0 [11752.533394] ? nd_jump_root+0xe5/0x160 [11752.534727] ? path_init+0x437/0x520 [11752.536371] path_openat+0xf7/0x500 [11752.537519] ? do_raw_spin_unlock+0x75/0x190 [11752.539351] do_filp_open+0x99/0x140 [11752.540617] ? getname_flags+0x6e/0x330 [11752.542210] ? __check_object_size+0xff/0x256 [11752.543708] ? do_raw_spin_unlock+0x75/0x190 [11752.545299] ? _raw_spin_unlock+0x12/0x30 [11752.547034] do_sys_openat2+0x2b4/0x410 [11752.548739] do_sys_open+0x73/0xa0 [11752.551369] __x64_sys_openat+0x24/0x30 [11752.553419] do_syscall_64+0xc1/0x440 [11752.555386] entry_SYSCALL_64_after_hwframe+0x49/0xae [11752.557836] RIP: 0033:0x7fa749552332 [11752.559839] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11752.567306] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11752.570615] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11752.574258] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11752.578033] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11752.580692] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11752.583748] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11754.982195] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9837s! [11754.987510] Showing busy workqueues and worker pools: [11754.991042] workqueue events: flags=0x0 [11754.993773] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11754.998737] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11755.017569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11755.019891] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11755.036979] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.047348] pending: kfree_rcu_monitor [11755.051644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11755.073562] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11755.098039] workqueue events_long: flags=0x0 [11755.102041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.106502] pending: lu_objects_flush [obdclass] [11755.111680] workqueue events_unbound: flags=0x2 [11755.114671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11755.121339] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11755.127545] workqueue events_power_efficient: flags=0x80 [11755.131371] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.136516] pending: check_lifetime [11755.141226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.148491] pending: neigh_periodic_work [11755.150160] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.154457] pending: neigh_periodic_work [11755.165429] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.175531] pending: do_cache_clean [11755.179681] workqueue mm_percpu_wq: flags=0x8 [11755.182231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.184541] pending: vmstat_update [11755.188187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.200184] pending: vmstat_update [11755.202865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11755.206028] pending: vmstat_update [11755.207731] workqueue writeback: flags=0x4a [11755.212982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11755.216385] pending: wb_workfn, wb_workfn [11755.222067] workqueue xprtiod: flags=0xa [11755.226637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11755.231052] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11755.239598] workqueue ipv6_addrconf: flags=0x40008 [11755.242305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11755.251625] pending: addrconf_verify_work [11755.257092] workqueue knbd0-recv: flags=0x1a [11755.271486] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11755.287464] in-flight: 651:recv_work [11755.292549] workqueue ptlrpc_pinger: flags=0x2 [11755.300147] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11755.310510] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11755.321692] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9838s workers=4 idle: 37 31846 28897 [11755.343047] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11745s workers=2 idle: 653 [11776.057071] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [11776.059732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.060395] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11776.062669] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11776.062683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.064239] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.064246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.064249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11776.064262] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11776.064266] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11776.064270] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11776.064272] RDX: 00000000000055cf RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11776.064274] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11776.064276] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11776.064278] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11776.064281] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11776.064284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.064286] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11776.064295] Call Trace: [11776.064300] [11776.064306] ? show_regs.cold.9+0x22/0x2f [11776.064317] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.064328] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.064333] ? watchdog+0x40/0x40 [11776.064337] ? hrtimer_interrupt+0x128/0x2c0 [11776.064341] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.064350] ? apic_timer_interrupt+0xf/0x20 [11776.064355] [11776.064358] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11776.064363] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11776.064368] do_raw_spin_lock+0x177/0x190 [11776.064373] _raw_spin_lock+0x21/0x30 [11776.064380] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11776.067367] obd_mod_free+0x46/0x140 [ptlrpc] [11776.068612] mdc_commit_open+0x91/0xb0 [mdc] [11776.069523] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11776.071652] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11776.071862] after_reply+0xe2c/0x1780 [ptlrpc] [11776.072141] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11776.072359] ? __next_timer_interrupt+0x160/0x160 [11776.072370] ptlrpcd+0xa82/0xe40 [ptlrpc] [11776.072669] ? do_wait_intr+0xf0/0xf0 [11776.072824] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11776.073040] kthread+0x1d1/0x200 [11776.073046] ? set_kthread_struct+0x70/0x70 [11776.073050] ret_from_fork+0x1f/0x30 [11776.120383] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.120396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.120402] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11776.120695] RIP: 0010:native_safe_halt+0x2e/0x50 [11776.120705] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11776.120709] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11776.120714] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11776.120716] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11776.120718] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11776.120721] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11776.120723] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11776.120727] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11776.120740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.120743] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11776.120751] Call Trace: [11776.120756] [11776.120761] ? show_regs.cold.9+0x22/0x2f [11776.120770] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.120775] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.120780] ? watchdog+0x40/0x40 [11776.120785] ? hrtimer_interrupt+0x128/0x2c0 [11776.120789] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.120797] ? apic_timer_interrupt+0xf/0x20 [11776.120802] [11776.120805] ? native_safe_halt+0x2e/0x50 [11776.120809] kvm_wait+0xcd/0xe0 [11776.120816] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11776.120824] do_raw_spin_lock+0x177/0x190 [11776.120828] _raw_spin_lock+0x21/0x30 [11776.120833] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11776.121028] process_one_work+0x2c8/0x700 [11776.121035] worker_thread+0x296/0x6e0 [11776.121040] ? rescuer_thread+0x570/0x570 [11776.121044] kthread+0x1d1/0x200 [11776.121048] ? set_kthread_struct+0x70/0x70 [11776.121052] ret_from_fork+0x1f/0x30 [11776.442728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.479849] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.484224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.488808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.491974] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11776.500267] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11776.504851] RAX: 0000000000007e93 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11776.507662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11776.510772] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11776.513861] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11776.516940] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11776.520171] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11776.532482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.539618] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11776.546125] Call Trace: [11776.547234] [11776.548121] ? show_regs.cold.9+0x22/0x2f [11776.551756] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.557184] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.559701] ? watchdog+0x40/0x40 [11776.565267] ? hrtimer_interrupt+0x128/0x2c0 [11776.567915] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.570472] ? apic_timer_interrupt+0xf/0x20 [11776.572891] [11776.575142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.581246] do_raw_spin_lock+0x177/0x190 [11776.583024] _raw_spin_lock+0x21/0x30 [11776.586228] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11776.588851] ? do_raw_spin_unlock+0x75/0x190 [11776.591081] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11776.593647] ? do_wait_intr+0xf0/0xf0 [11776.600113] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11776.612130] mdc_close+0x53b/0xd70 [mdc] [11776.616681] lmv_close+0x18d/0x490 [lmv] [11776.621788] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11776.629425] ll_md_real_close+0xa5/0x2a0 [lustre] [11776.634933] ll_md_close+0x274/0x800 [lustre] [11776.650605] ll_file_release+0x128/0xc50 [lustre] [11776.655672] __fput+0x10d/0x440 [11776.657149] ____fput+0x12/0x20 [11776.660726] task_work_run+0xc6/0x110 [11776.664232] exit_to_usermode_loop+0x1dd/0x1f0 [11776.671921] do_syscall_64+0x3d6/0x440 [11776.677299] entry_SYSCALL_64_after_hwframe+0x49/0xae [11776.683004] RIP: 0033:0x7fa749552cd8 [11776.692262] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11776.712923] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11776.725727] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11776.731858] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11776.735602] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11776.742407] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11776.745037] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11780.056778] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11780.070522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11780.114664] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11780.124786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11780.130171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11780.136044] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11780.150483] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11780.155523] RAX: 0000000000005b85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11780.158245] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11780.162746] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11780.171006] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11780.179808] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11780.184652] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11780.189154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11780.196729] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11780.205582] Call Trace: [11780.207495] [11780.208272] ? show_regs.cold.9+0x22/0x2f [11780.210269] ? watchdog_timer_fn.cold.6+0x62/0x10a [11780.213648] ? __hrtimer_run_queues+0x1c2/0x4d0 [11780.215170] ? watchdog+0x40/0x40 [11780.221248] ? hrtimer_interrupt+0x128/0x2c0 [11780.222455] ? smp_apic_timer_interrupt+0xa6/0x270 [11780.224507] ? apic_timer_interrupt+0xf/0x20 [11780.227500] [11780.228883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11780.231901] do_raw_spin_lock+0x177/0x190 [11780.233698] _raw_spin_lock+0x21/0x30 [11780.234852] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11780.242821] ? do_raw_spin_unlock+0x75/0x190 [11780.247670] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11780.253142] ? do_wait_intr+0xf0/0xf0 [11780.256072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11780.258484] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11780.261220] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11780.263615] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11780.267637] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11780.270255] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11780.272345] mdc_intent_lock+0x353/0x7e0 [mdc] [11780.274196] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11780.278186] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11780.281941] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11780.284588] lmv_intent_open+0x338/0x1150 [lmv] [11780.286450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11780.289931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11780.293510] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11780.297537] ll_intent_lock+0xc5/0xaa0 [lustre] [11780.299770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11780.304646] ll_lookup_it+0xe0e/0x2270 [lustre] [11780.306404] ? do_raw_spin_unlock+0x75/0x190 [11780.313786] ? _raw_spin_unlock+0x12/0x30 [11780.320077] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11780.325261] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11780.329869] ll_atomic_open+0x36a/0xd70 [lustre] [11780.332125] lookup_open+0x356/0xb40 [11780.334060] do_last+0x313/0xfc0 [11780.335694] ? nd_jump_root+0xe5/0x160 [11780.337581] ? path_init+0x437/0x520 [11780.339147] path_openat+0xf7/0x500 [11780.340658] ? do_raw_spin_unlock+0x75/0x190 [11780.342466] do_filp_open+0x99/0x140 [11780.343883] ? getname_flags+0x6e/0x330 [11780.345511] ? __check_object_size+0xff/0x256 [11780.348254] ? do_raw_spin_unlock+0x75/0x190 [11780.353239] ? _raw_spin_unlock+0x12/0x30 [11780.356539] do_sys_openat2+0x2b4/0x410 [11780.357984] do_sys_open+0x73/0xa0 [11780.361369] __x64_sys_openat+0x24/0x30 [11780.363917] do_syscall_64+0xc1/0x440 [11780.369777] entry_SYSCALL_64_after_hwframe+0x49/0xae [11780.373213] RIP: 0033:0x7fa749552332 [11780.375789] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11780.397822] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11780.419630] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11780.432351] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11780.438175] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11780.446469] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11780.452964] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11785.705831] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9868s! [11785.710377] Showing busy workqueues and worker pools: [11785.712941] workqueue events: flags=0x0 [11785.719191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11785.723888] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11785.734580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11785.742215] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11785.752456] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.755547] pending: kfree_rcu_monitor [11785.757907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11785.761306] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11785.770610] workqueue events_long: flags=0x0 [11785.772635] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.780716] pending: lu_objects_flush [obdclass] [11785.790763] workqueue events_unbound: flags=0x2 [11785.797172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11785.808073] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11785.818522] workqueue events_power_efficient: flags=0x80 [11785.827110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.837570] pending: check_lifetime [11785.843372] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.849048] pending: neigh_periodic_work [11785.852860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.859551] pending: neigh_periodic_work [11785.863984] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.877740] pending: do_cache_clean [11785.883877] workqueue mm_percpu_wq: flags=0x8 [11785.892666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.904506] pending: vmstat_update [11785.908826] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.914729] pending: vmstat_update [11785.919300] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11785.922334] pending: vmstat_update [11785.926559] workqueue writeback: flags=0x4a [11785.929702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11785.936506] pending: wb_workfn, wb_workfn [11785.940869] workqueue xprtiod: flags=0xa [11785.943560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11785.949686] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11785.957840] workqueue ipv6_addrconf: flags=0x40008 [11785.962714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11785.966459] pending: addrconf_verify_work [11785.978760] workqueue knbd0-recv: flags=0x1a [11785.981577] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11785.987646] in-flight: 651:recv_work [11785.989794] workqueue ptlrpc_pinger: flags=0x2 [11785.992119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11785.996414] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11786.000391] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9868s workers=4 idle: 37 31846 28897 [11786.006179] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11775s workers=2 idle: 653 [11804.058106] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11804.061545] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11804.065423] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11804.065440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11804.065613] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11804.065619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11804.065622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.065641] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11804.065645] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11804.065650] RAX: 0000000000002e3c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11804.065652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11804.065654] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11804.065655] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11804.065657] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11804.065660] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11804.065663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11804.065665] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11804.065674] Call Trace: [11804.065687] [11804.065694] ? show_regs.cold.9+0x22/0x2f [11804.065704] ? watchdog_timer_fn.cold.6+0x62/0x10a [11804.065710] ? __hrtimer_run_queues+0x1c2/0x4d0 [11804.065714] ? watchdog+0x40/0x40 [11804.065718] ? hrtimer_interrupt+0x128/0x2c0 [11804.065722] ? smp_apic_timer_interrupt+0xa6/0x270 [11804.065729] ? apic_timer_interrupt+0xf/0x20 [11804.065734] [11804.065737] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.065743] do_raw_spin_lock+0x177/0x190 [11804.065747] _raw_spin_lock+0x21/0x30 [11804.065859] Modules linked in: lustre(O) osp(O) [11804.065752] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11804.065927] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11804.065982] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11804.065986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11804.065991] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11804.066218] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.066226] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11804.066230] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11804.066234] RAX: 0000000000000f12 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11804.066237] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11804.066239] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11804.066241] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11804.066244] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11804.066247] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11804.066250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11804.066252] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11804.066259] Call Trace: [11804.066261] [11804.066265] ? show_regs.cold.9+0x22/0x2f [11804.066271] ? watchdog_timer_fn.cold.6+0x62/0x10a [11804.066276] ? __hrtimer_run_queues+0x1c2/0x4d0 [11804.066280] ? watchdog+0x40/0x40 [11804.066284] ? hrtimer_interrupt+0x128/0x2c0 [11804.066289] ? smp_apic_timer_interrupt+0xa6/0x270 [11804.066294] ? apic_timer_interrupt+0xf/0x20 [11804.066299] [11804.066302] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.066307] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11804.066312] do_raw_spin_lock+0x177/0x190 [11804.066317] _raw_spin_lock+0x21/0x30 [11804.066321] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11804.066976] ? do_raw_spin_unlock+0x75/0x190 [11804.066985] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11804.067189] ? do_wait_intr+0xf0/0xf0 [11804.067198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11804.067658] process_one_work+0x2c8/0x700 [11804.067668] worker_thread+0x296/0x6e0 [11804.067672] ? rescuer_thread+0x570/0x570 [11804.067676] kthread+0x1d1/0x200 [11804.067681] ? set_kthread_struct+0x70/0x70 [11804.067685] ret_from_fork+0x1f/0x30 [11804.067845] mdc_close+0x53b/0xd70 [mdc] [11804.067880] lmv_close+0x18d/0x490 [lmv] [11804.067908] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11804.068069] ll_md_real_close+0xa5/0x2a0 [lustre] [11804.076266] Modules linked in: lustre(O) [11804.083695] ll_md_close+0x274/0x800 [lustre] [11804.083814] ll_file_release+0x128/0xc50 [lustre] [11804.086068] __fput+0x10d/0x440 [11804.086080] ____fput+0x12/0x20 [11804.086085] task_work_run+0xc6/0x110 [11804.086094] exit_to_usermode_loop+0x1dd/0x1f0 [11804.086101] do_syscall_64+0x3d6/0x440 [11804.086105] entry_SYSCALL_64_after_hwframe+0x49/0xae [11804.086114] RIP: 0033:0x7fa749552cd8 [11804.086174] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11804.086180] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11804.086186] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11804.086189] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11804.086191] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11804.086193] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11804.086195] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11804.916029] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11804.935311] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11804.940586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11804.944471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11804.946768] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11804.953794] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11804.957032] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11804.959773] RDX: 0000000000002ef1 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11804.963200] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11804.965690] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11804.969064] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11804.972135] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11804.975495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11804.977615] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11804.981389] Call Trace: [11804.982518] [11804.983277] ? show_regs.cold.9+0x22/0x2f [11804.985244] ? watchdog_timer_fn.cold.6+0x62/0x10a [11804.986869] ? __hrtimer_run_queues+0x1c2/0x4d0 [11804.989325] ? watchdog+0x40/0x40 [11804.990667] ? hrtimer_interrupt+0x128/0x2c0 [11804.992796] ? smp_apic_timer_interrupt+0xa6/0x270 [11804.994748] ? apic_timer_interrupt+0xf/0x20 [11804.996993] [11804.997759] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11804.999893] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11805.002699] do_raw_spin_lock+0x177/0x190 [11805.004679] _raw_spin_lock+0x21/0x30 [11805.006812] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11805.009709] obd_mod_free+0x46/0x140 [ptlrpc] [11805.012189] mdc_commit_open+0x91/0xb0 [mdc] [11805.014222] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11805.016518] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11805.019244] after_reply+0xe2c/0x1780 [ptlrpc] [11805.021481] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11805.024257] ? __next_timer_interrupt+0x160/0x160 [11805.026241] ptlrpcd+0xa82/0xe40 [ptlrpc] [11805.028588] ? do_wait_intr+0xf0/0xf0 [11805.048147] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11805.063379] kthread+0x1d1/0x200 [11805.067029] ? set_kthread_struct+0x70/0x70 [11805.069609] ret_from_fork+0x1f/0x30 [11812.054264] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11812.065693] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11812.124760] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11812.132978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11812.143279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11812.146519] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11812.157709] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11812.160932] RAX: 0000000000007e09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11812.164596] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11812.167545] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11812.171132] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11812.174216] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11812.181971] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11812.187793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11812.192308] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11812.197083] Call Trace: [11812.197937] [11812.204951] ? show_regs.cold.9+0x22/0x2f [11812.212915] ? watchdog_timer_fn.cold.6+0x62/0x10a [11812.219077] ? __hrtimer_run_queues+0x1c2/0x4d0 [11812.221965] ? watchdog+0x40/0x40 [11812.229962] ? hrtimer_interrupt+0x128/0x2c0 [11812.236085] ? smp_apic_timer_interrupt+0xa6/0x270 [11812.242066] ? apic_timer_interrupt+0xf/0x20 [11812.244896] [11812.247977] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11812.251841] do_raw_spin_lock+0x177/0x190 [11812.254176] _raw_spin_lock+0x21/0x30 [11812.256072] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11812.259134] ? do_raw_spin_unlock+0x75/0x190 [11812.261261] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11812.264462] ? do_wait_intr+0xf0/0xf0 [11812.266149] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11812.268669] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11812.271286] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11812.274351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11812.277086] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11812.279484] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11812.281270] mdc_intent_lock+0x353/0x7e0 [mdc] [11812.282982] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11812.285206] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11812.288187] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11812.290487] lmv_intent_open+0x338/0x1150 [lmv] [11812.292941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11812.295369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11812.297918] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11812.299860] ll_intent_lock+0xc5/0xaa0 [lustre] [11812.302262] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11812.304545] ll_lookup_it+0xe0e/0x2270 [lustre] [11812.307157] ? do_raw_spin_unlock+0x75/0x190 [11812.308638] ? _raw_spin_unlock+0x12/0x30 [11812.309993] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11812.312166] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11812.314413] ll_atomic_open+0x36a/0xd70 [lustre] [11812.316308] lookup_open+0x356/0xb40 [11812.317459] do_last+0x313/0xfc0 [11812.318544] ? nd_jump_root+0xe5/0x160 [11812.320251] ? path_init+0x437/0x520 [11812.321897] path_openat+0xf7/0x500 [11812.323271] ? do_raw_spin_unlock+0x75/0x190 [11812.325023] do_filp_open+0x99/0x140 [11812.326549] ? getname_flags+0x6e/0x330 [11812.328313] ? __check_object_size+0xff/0x256 [11812.330109] ? do_raw_spin_unlock+0x75/0x190 [11812.332072] ? _raw_spin_unlock+0x12/0x30 [11812.333711] do_sys_openat2+0x2b4/0x410 [11812.335513] do_sys_open+0x73/0xa0 [11812.337694] __x64_sys_openat+0x24/0x30 [11812.339288] do_syscall_64+0xc1/0x440 [11812.340935] entry_SYSCALL_64_after_hwframe+0x49/0xae [11812.342970] RIP: 0033:0x7fa749552332 [11812.345560] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11812.363053] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11812.365892] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11812.369083] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11812.373321] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11812.377049] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11812.381019] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11816.422676] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9899s! [11816.427569] Showing busy workqueues and worker pools: [11816.431599] workqueue events: flags=0x0 [11816.437261] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11816.443432] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11816.455430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11816.465396] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11816.476535] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.485987] pending: kfree_rcu_monitor [11816.490490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11816.498571] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11816.511017] workqueue events_long: flags=0x0 [11816.516454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.527110] pending: lu_objects_flush [obdclass] [11816.545369] workqueue events_unbound: flags=0x2 [11816.563249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11816.574552] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11816.578473] workqueue events_power_efficient: flags=0x80 [11816.596016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.598702] pending: check_lifetime [11816.610463] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.621583] pending: neigh_periodic_work [11816.626843] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.633912] pending: neigh_periodic_work [11816.643437] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.648409] pending: do_cache_clean [11816.651634] workqueue mm_percpu_wq: flags=0x8 [11816.655772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.658432] pending: vmstat_update [11816.660601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.667293] pending: vmstat_update [11816.669350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11816.673455] pending: vmstat_update [11816.678461] workqueue writeback: flags=0x4a [11816.680098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11816.691073] pending: wb_workfn, wb_workfn [11816.693641] workqueue xprtiod: flags=0xa [11816.699092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11816.703385] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11816.708762] workqueue ipv6_addrconf: flags=0x40008 [11816.712373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11816.715557] pending: addrconf_verify_work [11816.719274] workqueue knbd0-recv: flags=0x1a [11816.727253] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11816.740592] in-flight: 651:recv_work [11816.742602] workqueue ptlrpc_pinger: flags=0x2 [11816.749412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11816.755368] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11816.759526] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9899s workers=4 idle: 37 31846 28897 [11816.764592] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11806s workers=2 idle: 653 [11832.057774] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11832.059150] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11832.062677] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11832.062693] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [11832.062857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11832.062925] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11832.062931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11832.063136] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11832.063154] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11832.063158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11832.063160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.063172] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11832.063176] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11832.063181] RAX: 0000000000007f84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11832.063183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11832.063186] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11832.063188] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11832.063190] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11832.063194] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11832.063197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11832.063199] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11832.063209] Call Trace: [11832.063215] [11832.063222] ? show_regs.cold.9+0x22/0x2f [11832.063232] ? watchdog_timer_fn.cold.6+0x62/0x10a [11832.063239] ? __hrtimer_run_queues+0x1c2/0x4d0 [11832.063244] ? watchdog+0x40/0x40 [11832.063248] ? hrtimer_interrupt+0x128/0x2c0 [11832.063252] ? smp_apic_timer_interrupt+0xa6/0x270 [11832.063266] ? apic_timer_interrupt+0xf/0x20 [11832.063333] [11832.063337] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.063343] do_raw_spin_lock+0x177/0x190 [11832.063348] _raw_spin_lock+0x21/0x30 [11832.063353] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11832.063950] ? do_raw_spin_unlock+0x75/0x190 [11832.062938] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11832.063958] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11832.064359] [11832.064362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.064371] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11832.064375] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11832.064379] RAX: 0000000000007186 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11832.064381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11832.064383] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11832.064385] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11832.064388] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11832.064391] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11832.064393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11832.064396] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11832.064403] Call Trace: [11832.064406] [11832.064409] ? show_regs.cold.9+0x22/0x2f [11832.064417] ? watchdog_timer_fn.cold.6+0x62/0x10a [11832.064423] ? __hrtimer_run_queues+0x1c2/0x4d0 [11832.064426] ? watchdog+0x40/0x40 [11832.064431] ? hrtimer_interrupt+0x128/0x2c0 [11832.064435] ? smp_apic_timer_interrupt+0xa6/0x270 [11832.064441] ? apic_timer_interrupt+0xf/0x20 [11832.064445] [11832.064448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.064452] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11832.064457] do_raw_spin_lock+0x177/0x190 [11832.064461] _raw_spin_lock+0x21/0x30 [11832.064539] ? do_wait_intr+0xf0/0xf0 [11832.064465] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11832.064905] process_one_work+0x2c8/0x700 [11832.064913] worker_thread+0x296/0x6e0 [11832.064917] ? rescuer_thread+0x570/0x570 [11832.064921] kthread+0x1d1/0x200 [11832.064926] ? set_kthread_struct+0x70/0x70 [11832.064929] ret_from_fork+0x1f/0x30 [11832.064547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11832.072394] Modules linked in: [11832.081231] mdc_close+0x53b/0xd70 [mdc] [11832.081334] lmv_close+0x18d/0x490 [lmv] [11832.081365] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11832.081482] ll_md_real_close+0xa5/0x2a0 [lustre] [11832.081573] ll_md_close+0x274/0x800 [lustre] [11832.118090] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11832.118164] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11832.118172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11832.118175] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11832.118189] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11832.118193] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11832.118198] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11832.118200] RDX: 000000000000761d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11832.118202] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11832.118204] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11832.118206] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11832.118209] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11832.118212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11832.118214] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11832.118222] Call Trace: [11832.118227] [11832.118233] ? show_regs.cold.9+0x22/0x2f [11832.118242] ? watchdog_timer_fn.cold.6+0x62/0x10a [11832.118248] ? __hrtimer_run_queues+0x1c2/0x4d0 [11832.118252] ? watchdog+0x40/0x40 [11832.118256] ? hrtimer_interrupt+0x128/0x2c0 [11832.118260] ? smp_apic_timer_interrupt+0xa6/0x270 [11832.118268] ? apic_timer_interrupt+0xf/0x20 [11832.118272] [11832.118287] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11832.118293] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11832.118297] do_raw_spin_lock+0x177/0x190 [11832.118302] _raw_spin_lock+0x21/0x30 [11832.118308] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11832.118911] obd_mod_free+0x46/0x140 [ptlrpc] [11832.119337] mdc_commit_open+0x91/0xb0 [mdc] [11832.119448] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11832.119810] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11832.120003] after_reply+0xe2c/0x1780 [ptlrpc] [11832.120558] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11832.120983] ? __next_timer_interrupt+0x160/0x160 [11832.120994] ptlrpcd+0xa82/0xe40 [ptlrpc] [11832.121890] ? do_wait_intr+0xf0/0xf0 [11832.121900] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11832.122809] kthread+0x1d1/0x200 [11832.122816] ? set_kthread_struct+0x70/0x70 [11832.122820] ret_from_fork+0x1f/0x30 [11832.799682] ll_file_release+0x128/0xc50 [lustre] [11832.809108] __fput+0x10d/0x440 [11832.811245] ____fput+0x12/0x20 [11832.813153] task_work_run+0xc6/0x110 [11832.814900] exit_to_usermode_loop+0x1dd/0x1f0 [11832.818585] do_syscall_64+0x3d6/0x440 [11832.820102] entry_SYSCALL_64_after_hwframe+0x49/0xae [11832.823188] RIP: 0033:0x7fa749552cd8 [11832.825311] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11832.837518] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11832.845492] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11832.850247] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11832.860163] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11832.864135] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11832.870720] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11844.054013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11844.058651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11844.076642] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11844.092926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11844.104758] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11844.108870] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11844.128164] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11844.137335] RAX: 0000000000002c32 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11844.143285] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11844.149104] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11844.153384] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11844.161037] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11844.166725] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11844.169961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11844.172334] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11844.175170] Call Trace: [11844.176837] [11844.177969] ? show_regs.cold.9+0x22/0x2f [11844.180361] ? watchdog_timer_fn.cold.6+0x62/0x10a [11844.187738] ? __hrtimer_run_queues+0x1c2/0x4d0 [11844.190851] ? watchdog+0x40/0x40 [11844.192469] ? hrtimer_interrupt+0x128/0x2c0 [11844.194913] ? smp_apic_timer_interrupt+0xa6/0x270 [11844.200223] ? apic_timer_interrupt+0xf/0x20 [11844.206137] [11844.207105] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11844.212186] do_raw_spin_lock+0x177/0x190 [11844.215832] _raw_spin_lock+0x21/0x30 [11844.217921] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11844.220824] ? do_raw_spin_unlock+0x75/0x190 [11844.222838] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11844.225245] ? do_wait_intr+0xf0/0xf0 [11844.229007] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11844.233680] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11844.237238] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11844.240214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.243482] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11844.246217] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11844.248630] mdc_intent_lock+0x353/0x7e0 [mdc] [11844.251355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.254388] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11844.258195] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11844.260977] lmv_intent_open+0x338/0x1150 [lmv] [11844.263116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.266492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.269768] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11844.272794] ll_intent_lock+0xc5/0xaa0 [lustre] [11844.275707] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.278991] ll_lookup_it+0xe0e/0x2270 [lustre] [11844.281541] ? do_raw_spin_unlock+0x75/0x190 [11844.283762] ? _raw_spin_unlock+0x12/0x30 [11844.285996] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11844.288780] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11844.291137] ll_atomic_open+0x36a/0xd70 [lustre] [11844.293636] lookup_open+0x356/0xb40 [11844.295477] do_last+0x313/0xfc0 [11844.296794] ? nd_jump_root+0xe5/0x160 [11844.298699] ? path_init+0x437/0x520 [11844.300936] path_openat+0xf7/0x500 [11844.302824] ? do_raw_spin_unlock+0x75/0x190 [11844.305070] do_filp_open+0x99/0x140 [11844.306957] ? getname_flags+0x6e/0x330 [11844.308498] ? __check_object_size+0xff/0x256 [11844.310272] ? do_raw_spin_unlock+0x75/0x190 [11844.311542] ? _raw_spin_unlock+0x12/0x30 [11844.315093] do_sys_openat2+0x2b4/0x410 [11844.316325] do_sys_open+0x73/0xa0 [11844.318038] __x64_sys_openat+0x24/0x30 [11844.319228] do_syscall_64+0xc1/0x440 [11844.320172] entry_SYSCALL_64_after_hwframe+0x49/0xae [11844.321444] RIP: 0033:0x7fa749552332 [11844.322910] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11844.335865] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11844.340799] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11844.344091] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11844.348073] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11844.350288] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11844.352905] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11847.141123] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9929s! [11847.160663] Showing busy workqueues and worker pools: [11847.169128] workqueue events: flags=0x0 [11847.172975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11847.181550] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11847.190071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11847.200482] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11847.212537] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.218642] pending: kfree_rcu_monitor [11847.226270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11847.234219] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11847.246099] workqueue events_long: flags=0x0 [11847.249949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.256182] pending: lu_objects_flush [obdclass] [11847.259043] workqueue events_unbound: flags=0x2 [11847.269019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11847.273722] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11847.278963] workqueue events_power_efficient: flags=0x80 [11847.281743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.292604] pending: check_lifetime [11847.294721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.297830] pending: neigh_periodic_work [11847.299805] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.302626] pending: neigh_periodic_work [11847.304764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.307699] pending: do_cache_clean [11847.309537] workqueue mm_percpu_wq: flags=0x8 [11847.311448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.314705] pending: vmstat_update [11847.316169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.319568] pending: vmstat_update [11847.321446] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11847.324343] pending: vmstat_update [11847.326065] workqueue writeback: flags=0x4a [11847.327833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11847.330679] pending: wb_workfn, wb_workfn [11847.332558] workqueue xprtiod: flags=0xa [11847.336912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11847.343742] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11847.349605] workqueue ipv6_addrconf: flags=0x40008 [11847.353446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11847.360342] pending: addrconf_verify_work [11847.363522] workqueue knbd0-recv: flags=0x1a [11847.366086] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11847.370973] in-flight: 651:recv_work [11847.375268] workqueue ptlrpc_pinger: flags=0x2 [11847.380283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11847.386091] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11847.391772] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9930s workers=4 idle: 37 31846 28897 [11847.395893] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11837s workers=2 idle: 653 [11860.060592] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11860.061605] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11860.065080] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11860.065095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11860.065182] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11860.065187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11860.065194] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11860.067125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11860.067223] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11860.067234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11860.067237] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11860.067250] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11860.067254] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11860.067398] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11860.067400] RDX: 0000000000002939 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11860.067403] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11860.067408] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11860.067411] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11860.067413] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11860.067415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11860.067418] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11860.067436] Call Trace: [11860.067442] [11860.067451] ? show_regs.cold.9+0x22/0x2f [11860.067461] ? watchdog_timer_fn.cold.6+0x62/0x10a [11860.067471] ? __hrtimer_run_queues+0x1c2/0x4d0 [11860.067474] ? watchdog+0x40/0x40 [11860.067478] ? hrtimer_interrupt+0x128/0x2c0 [11860.067483] ? smp_apic_timer_interrupt+0xa6/0x270 [11860.067494] ? apic_timer_interrupt+0xf/0x20 [11860.067498] [11860.067501] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11860.067505] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11860.067510] do_raw_spin_lock+0x177/0x190 [11860.067514] _raw_spin_lock+0x21/0x30 [11860.067519] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11860.068609] [11860.068612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.068622] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11860.068626] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11860.068631] RAX: 0000000000007d11 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11860.068633] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11860.068635] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11860.068638] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11860.068640] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11860.068643] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11860.068646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11860.068648] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11860.068655] Call Trace: [11860.068659] [11860.068663] ? show_regs.cold.9+0x22/0x2f [11860.068670] ? watchdog_timer_fn.cold.6+0x62/0x10a [11860.068676] ? __hrtimer_run_queues+0x1c2/0x4d0 [11860.068679] ? watchdog+0x40/0x40 [11860.068683] ? hrtimer_interrupt+0x128/0x2c0 [11860.068687] ? smp_apic_timer_interrupt+0xa6/0x270 [11860.068694] ? apic_timer_interrupt+0xf/0x20 [11860.068705] [11860.068708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.068713] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11860.068719] do_raw_spin_lock+0x177/0x190 [11860.068724] _raw_spin_lock+0x21/0x30 [11860.068728] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11860.068980] process_one_work+0x2c8/0x700 [11860.068987] worker_thread+0x296/0x6e0 [11860.068991] ? rescuer_thread+0x570/0x570 [11860.068995] kthread+0x1d1/0x200 [11860.068999] ? set_kthread_struct+0x70/0x70 [11860.069003] ret_from_fork+0x1f/0x30 [11860.067806] obd_mod_free+0x46/0x140 [ptlrpc] [11860.070255] mdc_commit_open+0x91/0xb0 [mdc] [11860.070358] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11860.077343] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11860.077427] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11860.077436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11860.077440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.077456] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11860.077460] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11860.077465] RAX: 0000000000007f89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11860.077468] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11860.077470] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11860.077472] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11860.077475] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11860.077479] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11860.077482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11860.077485] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11860.077493] Call Trace: [11860.077499] [11860.077505] ? show_regs.cold.9+0x22/0x2f [11860.077515] ? watchdog_timer_fn.cold.6+0x62/0x10a [11860.077522] ? __hrtimer_run_queues+0x1c2/0x4d0 [11860.077527] ? watchdog+0x40/0x40 [11860.077532] ? hrtimer_interrupt+0x128/0x2c0 [11860.077536] ? smp_apic_timer_interrupt+0xa6/0x270 [11860.077545] ? apic_timer_interrupt+0xf/0x20 [11860.077550] [11860.077552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.077559] do_raw_spin_lock+0x177/0x190 [11860.077564] _raw_spin_lock+0x21/0x30 [11860.077570] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11860.084691] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11860.087796] after_reply+0xe2c/0x1780 [ptlrpc] [11860.088114] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11860.088712] ? __next_timer_interrupt+0x160/0x160 [11860.088723] ptlrpcd+0xa82/0xe40 [ptlrpc] [11860.089034] ? do_wait_intr+0xf0/0xf0 [11860.089044] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11860.089850] kthread+0x1d1/0x200 [11860.089857] ? set_kthread_struct+0x70/0x70 [11860.089862] ret_from_fork+0x1f/0x30 [11860.724963] ? do_raw_spin_unlock+0x75/0x190 [11860.726521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11860.729710] ? do_wait_intr+0xf0/0xf0 [11860.731660] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11860.739860] mdc_close+0x53b/0xd70 [mdc] [11860.742732] lmv_close+0x18d/0x490 [lmv] [11860.746463] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11860.749555] ll_md_real_close+0xa5/0x2a0 [lustre] [11860.755853] ll_md_close+0x274/0x800 [lustre] [11860.759811] ll_file_release+0x128/0xc50 [lustre] [11860.762488] __fput+0x10d/0x440 [11860.765710] ____fput+0x12/0x20 [11860.767095] task_work_run+0xc6/0x110 [11860.771862] exit_to_usermode_loop+0x1dd/0x1f0 [11860.777376] do_syscall_64+0x3d6/0x440 [11860.781495] entry_SYSCALL_64_after_hwframe+0x49/0xae [11860.783650] RIP: 0033:0x7fa749552cd8 [11860.786676] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11860.797530] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11860.804291] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11860.812730] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11860.819142] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11860.823555] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11860.828437] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11872.054933] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11872.063986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11872.110271] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11872.119040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11872.129872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11872.136481] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11872.154589] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11872.160766] RAX: 0000000000002911 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11872.167905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11872.175424] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11872.179344] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11872.182674] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11872.186687] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11872.190296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11872.193931] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11872.197172] Call Trace: [11872.198511] [11872.199438] ? show_regs.cold.9+0x22/0x2f [11872.201872] ? watchdog_timer_fn.cold.6+0x62/0x10a [11872.203853] ? __hrtimer_run_queues+0x1c2/0x4d0 [11872.206092] ? watchdog+0x40/0x40 [11872.207622] ? hrtimer_interrupt+0x128/0x2c0 [11872.209640] ? smp_apic_timer_interrupt+0xa6/0x270 [11872.212141] ? apic_timer_interrupt+0xf/0x20 [11872.214221] [11872.215153] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11872.218347] do_raw_spin_lock+0x177/0x190 [11872.220026] _raw_spin_lock+0x21/0x30 [11872.222081] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11872.224167] ? do_raw_spin_unlock+0x75/0x190 [11872.226055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11872.228357] ? do_wait_intr+0xf0/0xf0 [11872.229550] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11872.232931] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11872.236300] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11872.239861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.242973] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11872.246082] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11872.248463] mdc_intent_lock+0x353/0x7e0 [mdc] [11872.250372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.253460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11872.256380] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11872.258722] lmv_intent_open+0x338/0x1150 [lmv] [11872.260636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.262884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.265384] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11872.267377] ll_intent_lock+0xc5/0xaa0 [lustre] [11872.269261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.271547] ll_lookup_it+0xe0e/0x2270 [lustre] [11872.274358] ? do_raw_spin_unlock+0x75/0x190 [11872.276396] ? _raw_spin_unlock+0x12/0x30 [11872.278583] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11872.285940] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11872.291305] ll_atomic_open+0x36a/0xd70 [lustre] [11872.294545] lookup_open+0x356/0xb40 [11872.296064] do_last+0x313/0xfc0 [11872.297480] ? nd_jump_root+0xe5/0x160 [11872.302823] ? path_init+0x437/0x520 [11872.305287] path_openat+0xf7/0x500 [11872.306777] ? do_raw_spin_unlock+0x75/0x190 [11872.308472] do_filp_open+0x99/0x140 [11872.314074] ? getname_flags+0x6e/0x330 [11872.319240] ? __check_object_size+0xff/0x256 [11872.325042] ? do_raw_spin_unlock+0x75/0x190 [11872.331456] ? _raw_spin_unlock+0x12/0x30 [11872.337080] do_sys_openat2+0x2b4/0x410 [11872.342868] do_sys_open+0x73/0xa0 [11872.345320] __x64_sys_openat+0x24/0x30 [11872.349214] do_syscall_64+0xc1/0x440 [11872.352010] entry_SYSCALL_64_after_hwframe+0x49/0xae [11872.356106] RIP: 0033:0x7fa749552332 [11872.362056] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11872.387826] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11872.396808] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11872.404014] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11872.414349] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11872.424111] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11872.433890] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11873.906236] rcu: INFO: rcu_sched self-detected stall on CPU [11873.917596] rcu: 2-...!: (5624439 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [11873.927287] (t=9960217 jiffies g=258701 q=4414) [11873.931799] rcu: rcu_sched kthread starved for 9957012 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [11873.943410] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11873.949419] rcu: RCU grace-period kthread stack dump: [11873.953454] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11873.961111] Call Trace: [11873.962457] __schedule+0x351/0xcb0 [11873.964495] schedule+0xc0/0x180 [11873.967625] schedule_timeout+0xb4/0x190 [11873.969513] ? do_raw_spin_unlock+0x75/0x190 [11873.973275] ? __next_timer_interrupt+0x160/0x160 [11873.977506] ? prepare_to_swait_event+0xf3/0x140 [11873.983244] rcu_gp_kthread+0x8dd/0x15b0 [11873.985710] ? force_qs_rnp+0x2c0/0x2c0 [11873.990837] kthread+0x1d1/0x200 [11873.993069] ? set_kthread_struct+0x70/0x70 [11873.995874] ret_from_fork+0x1f/0x30 [11873.997992] rcu: Stack dump where RCU GP kthread last ran: [11874.003489] Sending NMI from CPU 2 to CPUs 3: [11874.010244] NMI backtrace for cpu 3 [11874.010252] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11874.010253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11874.010255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11874.010257] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11874.010259] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 [11874.010262] RAX: 00000000000001c7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11874.010264] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11874.010265] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11874.010266] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11874.010268] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11874.010269] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11874.010270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11874.010271] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11874.010273] Call Trace: [11874.010273] [11874.010274] ? show_regs.cold.9+0x22/0x2f [11874.010275] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11874.010277] ? nmi_cpu_backtrace_handler+0x15/0x40 [11874.010277] ? nmi_handle+0xbd/0x210 [11874.010279] ? default_do_nmi+0x4b/0x1b0 [11874.010280] ? do_nmi+0x280/0x380 [11874.010281] ? end_repeat_nmi+0x16/0x1f [11874.010283] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11874.010284] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11874.010285] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11874.010286] [11874.010287] do_raw_spin_lock+0x177/0x190 [11874.010288] _raw_spin_lock+0x21/0x30 [11874.010289] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11874.010290] ? do_raw_spin_unlock+0x75/0x190 [11874.010291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11874.010292] ? do_wait_intr+0xf0/0xf0 [11874.010293] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11874.010294] mdc_close+0x53b/0xd70 [mdc] [11874.010295] lmv_close+0x18d/0x490 [lmv] [11874.010296] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11874.010297] ll_md_real_close+0xa5/0x2a0 [lustre] [11874.010298] ll_md_close+0x274/0x800 [lustre] [11874.010299] ll_file_release+0x128/0xc50 [lustre] [11874.010300] __fput+0x10d/0x440 [11874.010301] ____fput+0x12/0x20 [11874.010302] task_work_run+0xc6/0x110 [11874.010303] exit_to_usermode_loop+0x1dd/0x1f0 [11874.010304] do_syscall_64+0x3d6/0x440 [11874.010305] entry_SYSCALL_64_after_hwframe+0x49/0xae [11874.010306] RIP: 0033:0x7fa749552cd8 [11874.010307] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11874.010309] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11874.010312] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11874.010313] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11874.010314] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11874.010316] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11874.010317] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11874.016088] NMI backtrace for cpu 2 [11874.220427] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11874.231396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11874.236462] Call Trace: [11874.237879] [11874.238972] ? dump_stack+0xbb/0x10e [11874.240115] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11874.246622] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11874.250143] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11874.252384] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11874.258027] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11874.264418] ? rcu_dump_cpu_stacks+0x11b/0x171 [11874.268068] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11874.273846] ? update_process_times+0x6d/0xc0 [11874.278738] ? tick_sched_handle.isra.18+0x27/0x90 [11874.281342] ? tick_sched_timer+0x53/0xb0 [11874.283038] ? __hrtimer_run_queues+0x1c2/0x4d0 [11874.285648] ? tick_sched_handle.isra.18+0x90/0x90 [11874.287327] ? hrtimer_interrupt+0x128/0x2c0 [11874.292539] ? smp_apic_timer_interrupt+0xa6/0x270 [11874.294711] ? apic_timer_interrupt+0xf/0x20 [11874.296460] [11874.300704] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11874.305991] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11874.310125] ? do_raw_spin_lock+0x177/0x190 [11874.315967] ? _raw_spin_lock+0x21/0x30 [11874.319999] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11874.325472] ? obd_mod_free+0x46/0x140 [ptlrpc] [11874.331403] ? mdc_commit_open+0x91/0xb0 [mdc] [11874.334120] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11874.337123] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11874.344405] ? after_reply+0xe2c/0x1780 [ptlrpc] [11874.350781] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11874.353555] ? __next_timer_interrupt+0x160/0x160 [11874.357471] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11874.361442] ? do_wait_intr+0xf0/0xf0 [11874.363750] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11874.366539] ? kthread+0x1d1/0x200 [11874.369003] ? set_kthread_struct+0x70/0x70 [11874.370965] ? ret_from_fork+0x1f/0x30 [11877.857590] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9960s! [11877.864306] Showing busy workqueues and worker pools: [11877.867469] workqueue events: flags=0x0 [11877.869779] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11877.874289] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11877.880489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11877.883846] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11877.887739] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11877.890472] pending: kfree_rcu_monitor [11877.891955] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11877.894884] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11877.898970] workqueue events_long: flags=0x0 [11877.900711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11877.904018] pending: lu_objects_flush [obdclass] [11877.906827] workqueue events_unbound: flags=0x2 [11877.909089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11877.914011] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11877.926968] workqueue events_power_efficient: flags=0x80 [11877.930898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11877.941081] pending: check_lifetime [11877.947953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11877.956790] pending: neigh_periodic_work [11877.963876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11877.973629] pending: neigh_periodic_work [11877.980131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11877.988669] pending: do_cache_clean [11877.993123] workqueue mm_percpu_wq: flags=0x8 [11877.997566] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11878.006801] pending: vmstat_update [11878.014268] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11878.023485] pending: vmstat_update [11878.025324] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11878.033288] pending: vmstat_update [11878.039496] workqueue writeback: flags=0x4a [11878.048928] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11878.054695] pending: wb_workfn, wb_workfn [11878.060112] workqueue xprtiod: flags=0xa [11878.064119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11878.069041] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11878.081951] workqueue ipv6_addrconf: flags=0x40008 [11878.091508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11878.097114] pending: addrconf_verify_work [11878.101794] workqueue knbd0-recv: flags=0x1a [11878.109091] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11878.127661] in-flight: 651:recv_work [11878.133165] workqueue ptlrpc_pinger: flags=0x2 [11878.136801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11878.146087] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11878.154434] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9960s workers=4 idle: 37 31846 28897 [11878.168135] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11868s workers=2 idle: 653 [11888.058800] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11888.068903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11888.109572] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11888.118026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11888.125010] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11888.129204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11888.133860] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11888.149117] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11888.157617] RAX: 0000000000006e8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11888.164588] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11888.169168] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11888.172081] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11888.174656] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11888.177192] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11888.180653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11888.184048] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11888.186739] Call Trace: [11888.187791] [11888.188609] ? show_regs.cold.9+0x22/0x2f [11888.190128] ? watchdog_timer_fn.cold.6+0x62/0x10a [11888.191962] ? __hrtimer_run_queues+0x1c2/0x4d0 [11888.193670] ? watchdog+0x40/0x40 [11888.194940] ? hrtimer_interrupt+0x128/0x2c0 [11888.196715] ? smp_apic_timer_interrupt+0xa6/0x270 [11888.198821] ? apic_timer_interrupt+0xf/0x20 [11888.200478] [11888.201176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11888.203241] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11888.205348] do_raw_spin_lock+0x177/0x190 [11888.207445] _raw_spin_lock+0x21/0x30 [11888.208650] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11888.210720] process_one_work+0x2c8/0x700 [11888.212221] worker_thread+0x296/0x6e0 [11888.213824] ? rescuer_thread+0x570/0x570 [11888.215605] kthread+0x1d1/0x200 [11888.216736] ? set_kthread_struct+0x70/0x70 [11888.218014] ret_from_fork+0x1f/0x30 [11900.060856] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11900.064096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11900.064330] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11900.089055] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11900.089071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11900.089074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11900.089090] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11900.089095] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11900.089101] RAX: 0000000000007c64 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11900.089103] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11900.089105] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11900.089108] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11900.089110] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11900.089114] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11900.089117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11900.089120] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11900.089129] Call Trace: [11900.089135] [11900.089142] ? show_regs.cold.9+0x22/0x2f [11900.089152] ? watchdog_timer_fn.cold.6+0x62/0x10a [11900.089158] ? __hrtimer_run_queues+0x1c2/0x4d0 [11900.089163] ? watchdog+0x40/0x40 [11900.089167] ? hrtimer_interrupt+0x128/0x2c0 [11900.089172] ? smp_apic_timer_interrupt+0xa6/0x270 [11900.089180] ? apic_timer_interrupt+0xf/0x20 [11900.089253] [11900.089258] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11900.089267] do_raw_spin_lock+0x177/0x190 [11900.089273] _raw_spin_lock+0x21/0x30 [11900.089278] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11900.089915] ? do_raw_spin_unlock+0x75/0x190 [11900.089921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11900.090120] ? do_wait_intr+0xf0/0xf0 [11900.090128] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11900.090348] mdc_close+0x53b/0xd70 [mdc] [11900.090385] lmv_close+0x18d/0x490 [lmv] [11900.090430] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11900.090726] ll_md_real_close+0xa5/0x2a0 [lustre] [11900.090806] ll_md_close+0x274/0x800 [lustre] [11900.090888] ll_file_release+0x128/0xc50 [lustre] [11900.090970] __fput+0x10d/0x440 [11900.090980] ____fput+0x12/0x20 [11900.090985] task_work_run+0xc6/0x110 [11900.090992] exit_to_usermode_loop+0x1dd/0x1f0 [11900.090999] do_syscall_64+0x3d6/0x440 [11900.091003] entry_SYSCALL_64_after_hwframe+0x49/0xae [11900.091011] RIP: 0033:0x7fa749552cd8 [11900.091018] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11900.091022] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11900.091027] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11900.091030] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11900.091032] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11900.091034] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11900.091037] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11900.258932] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11900.290495] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11900.299743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11900.306079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11900.309414] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11900.316092] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11900.319682] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11900.322303] RDX: 00000000000063be RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11900.328005] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11900.334613] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11900.338717] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11900.343501] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11900.351053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11900.354002] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11900.356655] Call Trace: [11900.361765] [11900.362634] ? show_regs.cold.9+0x22/0x2f [11900.365569] ? watchdog_timer_fn.cold.6+0x62/0x10a [11900.371012] ? __hrtimer_run_queues+0x1c2/0x4d0 [11900.373354] ? watchdog+0x40/0x40 [11900.375263] ? hrtimer_interrupt+0x128/0x2c0 [11900.376349] ? smp_apic_timer_interrupt+0xa6/0x270 [11900.381622] ? apic_timer_interrupt+0xf/0x20 [11900.383951] [11900.384917] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11900.386785] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11900.388769] do_raw_spin_lock+0x177/0x190 [11900.390591] _raw_spin_lock+0x21/0x30 [11900.391768] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11900.393866] obd_mod_free+0x46/0x140 [ptlrpc] [11900.395774] mdc_commit_open+0x91/0xb0 [mdc] [11900.397412] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11900.399440] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11900.401566] after_reply+0xe2c/0x1780 [ptlrpc] [11900.403374] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11900.405540] ? __next_timer_interrupt+0x160/0x160 [11900.406994] ptlrpcd+0xa82/0xe40 [ptlrpc] [11900.408690] ? do_wait_intr+0xf0/0xf0 [11900.409812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11900.411690] kthread+0x1d1/0x200 [11900.412570] ? set_kthread_struct+0x70/0x70 [11900.413798] ret_from_fork+0x1f/0x30 [11904.054164] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11904.061035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11904.098088] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11904.116091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11904.123910] RIP: 0010:native_safe_halt+0x2e/0x50 [11904.129317] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11904.145924] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11904.153254] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11904.157321] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11904.161301] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11904.167889] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11904.177268] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11904.189227] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11904.194398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11904.196319] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11904.213926] Call Trace: [11904.214882] [11904.215591] ? show_regs.cold.9+0x22/0x2f [11904.216840] ? watchdog_timer_fn.cold.6+0x62/0x10a [11904.230011] ? __hrtimer_run_queues+0x1c2/0x4d0 [11904.232786] ? watchdog+0x40/0x40 [11904.234488] ? hrtimer_interrupt+0x128/0x2c0 [11904.237694] ? smp_apic_timer_interrupt+0xa6/0x270 [11904.246564] ? apic_timer_interrupt+0xf/0x20 [11904.253423] [11904.254223] ? native_safe_halt+0x2e/0x50 [11904.255660] kvm_wait+0xcd/0xe0 [11904.263269] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11904.272454] do_raw_spin_lock+0x177/0x190 [11904.274950] _raw_spin_lock+0x21/0x30 [11904.277383] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11904.280681] ? do_raw_spin_unlock+0x75/0x190 [11904.284940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11904.291720] ? do_wait_intr+0xf0/0xf0 [11904.294029] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11904.296929] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11904.300707] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11904.306291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11904.313208] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11904.317977] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11904.322475] mdc_intent_lock+0x353/0x7e0 [mdc] [11904.326612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11904.328041] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11904.333444] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11904.341697] lmv_intent_open+0x338/0x1150 [lmv] [11904.345238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11904.351377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11904.354645] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11904.364353] ll_intent_lock+0xc5/0xaa0 [lustre] [11904.370441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11904.376511] ll_lookup_it+0xe0e/0x2270 [lustre] [11904.385070] ? do_raw_spin_unlock+0x75/0x190 [11904.391051] ? _raw_spin_unlock+0x12/0x30 [11904.392485] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11904.394256] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11904.405014] ll_atomic_open+0x36a/0xd70 [lustre] [11904.409541] lookup_open+0x356/0xb40 [11904.412463] do_last+0x313/0xfc0 [11904.414323] ? nd_jump_root+0xe5/0x160 [11904.416957] ? path_init+0x437/0x520 [11904.418599] path_openat+0xf7/0x500 [11904.421572] ? do_raw_spin_unlock+0x75/0x190 [11904.424998] do_filp_open+0x99/0x140 [11904.430638] ? getname_flags+0x6e/0x330 [11904.437958] ? __check_object_size+0xff/0x256 [11904.442159] ? do_raw_spin_unlock+0x75/0x190 [11904.446713] ? _raw_spin_unlock+0x12/0x30 [11904.450208] do_sys_openat2+0x2b4/0x410 [11904.454495] do_sys_open+0x73/0xa0 [11904.457301] __x64_sys_openat+0x24/0x30 [11904.467727] do_syscall_64+0xc1/0x440 [11904.469004] entry_SYSCALL_64_after_hwframe+0x49/0xae [11904.474914] RIP: 0033:0x7fa749552332 [11904.476176] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11904.488347] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11904.494075] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11904.498140] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11904.504535] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11904.509154] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11904.516737] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11908.580800] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9991s! [11908.590811] Showing busy workqueues and worker pools: [11908.596864] workqueue events: flags=0x0 [11908.601216] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11908.608296] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11908.615831] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11908.623998] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11908.630529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.638848] pending: kfree_rcu_monitor [11908.644033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11908.647793] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11908.656642] workqueue events_long: flags=0x0 [11908.659423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.670153] pending: lu_objects_flush [obdclass] [11908.676526] workqueue events_unbound: flags=0x2 [11908.681156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11908.687460] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11908.695819] workqueue events_power_efficient: flags=0x80 [11908.706814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.724831] pending: check_lifetime [11908.726234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.739035] pending: neigh_periodic_work [11908.740586] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.745434] pending: neigh_periodic_work [11908.747785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.753505] pending: do_cache_clean [11908.755918] workqueue mm_percpu_wq: flags=0x8 [11908.761200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.765254] pending: vmstat_update [11908.769314] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.775833] pending: vmstat_update [11908.777887] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11908.784826] pending: vmstat_update [11908.787117] workqueue writeback: flags=0x4a [11908.792939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11908.800093] pending: wb_workfn, wb_workfn [11908.806711] workqueue xprtiod: flags=0xa [11908.808108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11908.815885] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11908.822009] workqueue ipv6_addrconf: flags=0x40008 [11908.824842] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11908.830549] pending: addrconf_verify_work [11908.835314] workqueue knbd0-recv: flags=0x1a [11908.839927] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11908.844915] in-flight: 651:recv_work [11908.850115] workqueue ptlrpc_pinger: flags=0x2 [11908.856016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11908.862391] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11908.866987] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9991s workers=4 idle: 37 31846 28897 [11908.872188] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11898s workers=2 idle: 653 [11916.057024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11916.060035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11916.081547] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11916.087245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11916.096312] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11916.104441] RIP: 0010:native_safe_halt+0x2e/0x50 [11916.107336] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11916.119891] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11916.123789] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11916.128411] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11916.133637] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11916.138488] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11916.146020] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11916.150224] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11916.153483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11916.156651] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11916.159869] Call Trace: [11916.161295] [11916.162377] ? show_regs.cold.9+0x22/0x2f [11916.165501] ? watchdog_timer_fn.cold.6+0x62/0x10a [11916.173336] ? __hrtimer_run_queues+0x1c2/0x4d0 [11916.177764] ? watchdog+0x40/0x40 [11916.179347] ? hrtimer_interrupt+0x128/0x2c0 [11916.182179] ? smp_apic_timer_interrupt+0xa6/0x270 [11916.184127] ? apic_timer_interrupt+0xf/0x20 [11916.185851] [11916.186887] ? native_safe_halt+0x2e/0x50 [11916.188358] kvm_wait+0xcd/0xe0 [11916.189755] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11916.191745] do_raw_spin_lock+0x177/0x190 [11916.193455] _raw_spin_lock+0x21/0x30 [11916.195094] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11916.197530] process_one_work+0x2c8/0x700 [11916.199101] worker_thread+0x296/0x6e0 [11916.201072] ? rescuer_thread+0x570/0x570 [11916.202777] kthread+0x1d1/0x200 [11916.204187] ? set_kthread_struct+0x70/0x70 [11916.207144] ret_from_fork+0x1f/0x30 [11928.058627] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11928.060550] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11928.069416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11928.069527] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11928.074007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11928.074108] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11928.074116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11928.074118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.074137] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11928.074142] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11928.074150] RAX: 0000000000005708 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11928.074152] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11928.074172] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11928.074174] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11928.074176] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11928.074180] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11928.074183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11928.074192] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11928.074199] Call Trace: [11928.074204] [11928.074210] ? show_regs.cold.9+0x22/0x2f [11928.074220] ? watchdog_timer_fn.cold.6+0x62/0x10a [11928.074236] ? __hrtimer_run_queues+0x1c2/0x4d0 [11928.074241] ? watchdog+0x40/0x40 [11928.074245] ? hrtimer_interrupt+0x128/0x2c0 [11928.074250] ? smp_apic_timer_interrupt+0xa6/0x270 [11928.074257] ? apic_timer_interrupt+0xf/0x20 [11928.074262] [11928.074265] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.074271] do_raw_spin_lock+0x177/0x190 [11928.074275] _raw_spin_lock+0x21/0x30 [11928.074280] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11928.074564] ? do_raw_spin_unlock+0x75/0x190 [11928.074570] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11928.075067] ? do_wait_intr+0xf0/0xf0 [11928.075075] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11928.075454] mdc_close+0x53b/0xd70 [mdc] [11928.075502] lmv_close+0x18d/0x490 [lmv] [11928.075539] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11928.075665] ll_md_real_close+0xa5/0x2a0 [lustre] [11928.075869] ll_md_close+0x274/0x800 [lustre] [11928.075954] ll_file_release+0x128/0xc50 [lustre] [11928.076048] __fput+0x10d/0x440 [11928.076059] ____fput+0x12/0x20 [11928.076063] task_work_run+0xc6/0x110 [11928.076070] exit_to_usermode_loop+0x1dd/0x1f0 [11928.076077] do_syscall_64+0x3d6/0x440 [11928.076082] entry_SYSCALL_64_after_hwframe+0x49/0xae [11928.076090] RIP: 0033:0x7fa749552cd8 [11928.076097] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11928.076101] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11928.076106] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11928.076109] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11928.076111] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11928.076113] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11928.076115] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11928.541646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11928.550742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11928.557151] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11928.597127] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11928.607117] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11928.621277] RDX: 000000000000284e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11928.625016] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11928.640266] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11928.659244] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11928.667323] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11928.677782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11928.688811] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11928.702379] Call Trace: [11928.703439] [11928.704815] ? show_regs.cold.9+0x22/0x2f [11928.718161] ? watchdog_timer_fn.cold.6+0x62/0x10a [11928.724297] ? __hrtimer_run_queues+0x1c2/0x4d0 [11928.727520] ? watchdog+0x40/0x40 [11928.728593] ? hrtimer_interrupt+0x128/0x2c0 [11928.739893] ? smp_apic_timer_interrupt+0xa6/0x270 [11928.749461] ? apic_timer_interrupt+0xf/0x20 [11928.753554] [11928.754106] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11928.772076] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11928.783240] do_raw_spin_lock+0x177/0x190 [11928.785886] _raw_spin_lock+0x21/0x30 [11928.787426] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11928.792766] obd_mod_free+0x46/0x140 [ptlrpc] [11928.804419] mdc_commit_open+0x91/0xb0 [mdc] [11928.815489] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11928.825114] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11928.835439] after_reply+0xe2c/0x1780 [ptlrpc] [11928.840201] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11928.848142] ? __next_timer_interrupt+0x160/0x160 [11928.851363] ptlrpcd+0xa82/0xe40 [ptlrpc] [11928.854492] ? do_wait_intr+0xf0/0xf0 [11928.861220] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11928.869403] kthread+0x1d1/0x200 [11928.870305] ? set_kthread_struct+0x70/0x70 [11928.888626] ret_from_fork+0x1f/0x30 [11936.055793] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11936.064303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11936.135177] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11936.144443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11936.153879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11936.156497] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11936.177410] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11936.187543] RAX: 0000000000002b87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11936.190892] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11936.197720] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11936.205850] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11936.211228] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11936.218911] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11936.225998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11936.232563] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11936.235391] Call Trace: [11936.236515] [11936.237545] ? show_regs.cold.9+0x22/0x2f [11936.243251] ? watchdog_timer_fn.cold.6+0x62/0x10a [11936.247145] ? __hrtimer_run_queues+0x1c2/0x4d0 [11936.251456] ? watchdog+0x40/0x40 [11936.255507] ? hrtimer_interrupt+0x128/0x2c0 [11936.258601] ? smp_apic_timer_interrupt+0xa6/0x270 [11936.270485] ? apic_timer_interrupt+0xf/0x20 [11936.280695] [11936.283951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11936.295383] do_raw_spin_lock+0x177/0x190 [11936.298047] _raw_spin_lock+0x21/0x30 [11936.305987] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11936.319057] ? do_raw_spin_unlock+0x75/0x190 [11936.322304] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11936.334106] ? do_wait_intr+0xf0/0xf0 [11936.341323] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11936.348367] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11936.361157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11936.370670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11936.375423] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11936.381353] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11936.392427] mdc_intent_lock+0x353/0x7e0 [mdc] [11936.395460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11936.404210] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11936.412827] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11936.420481] lmv_intent_open+0x338/0x1150 [lmv] [11936.428524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11936.431490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11936.441024] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11936.445229] ll_intent_lock+0xc5/0xaa0 [lustre] [11936.453470] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11936.463493] ll_lookup_it+0xe0e/0x2270 [lustre] [11936.469277] ? do_raw_spin_unlock+0x75/0x190 [11936.474490] ? _raw_spin_unlock+0x12/0x30 [11936.486737] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11936.500431] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11936.506620] ll_atomic_open+0x36a/0xd70 [lustre] [11936.516545] lookup_open+0x356/0xb40 [11936.520764] do_last+0x313/0xfc0 [11936.530637] ? nd_jump_root+0xe5/0x160 [11936.533691] ? path_init+0x437/0x520 [11936.542272] path_openat+0xf7/0x500 [11936.553896] ? do_raw_spin_unlock+0x75/0x190 [11936.572926] do_filp_open+0x99/0x140 [11936.581456] ? getname_flags+0x6e/0x330 [11936.583613] ? __check_object_size+0xff/0x256 [11936.587460] ? do_raw_spin_unlock+0x75/0x190 [11936.589062] ? _raw_spin_unlock+0x12/0x30 [11936.599750] do_sys_openat2+0x2b4/0x410 [11936.604655] do_sys_open+0x73/0xa0 [11936.611283] __x64_sys_openat+0x24/0x30 [11936.613428] do_syscall_64+0xc1/0x440 [11936.617937] entry_SYSCALL_64_after_hwframe+0x49/0xae [11936.625615] RIP: 0033:0x7fa749552332 [11936.626981] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11936.654223] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11936.660702] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11936.669590] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11936.679723] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11936.690683] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11936.698369] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11939.300168] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10022s! [11939.303422] Showing busy workqueues and worker pools: [11939.307743] workqueue events: flags=0x0 [11939.310574] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11939.317622] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11939.331290] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11939.335140] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11939.341535] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.346824] pending: kfree_rcu_monitor [11939.348055] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11939.350697] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11939.355452] workqueue events_long: flags=0x0 [11939.356778] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.362812] pending: lu_objects_flush [obdclass] [11939.366146] workqueue events_unbound: flags=0x2 [11939.373697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11939.377854] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11939.384476] workqueue events_power_efficient: flags=0x80 [11939.386193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.391109] pending: check_lifetime [11939.393629] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.397476] pending: neigh_periodic_work [11939.403719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.407728] pending: neigh_periodic_work [11939.410292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.414222] pending: do_cache_clean [11939.417124] workqueue mm_percpu_wq: flags=0x8 [11939.421015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.425105] pending: vmstat_update [11939.427070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.431002] pending: vmstat_update [11939.434290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11939.438144] pending: vmstat_update [11939.441015] workqueue writeback: flags=0x4a [11939.443096] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11939.447250] pending: wb_workfn, wb_workfn [11939.449967] workqueue xprtiod: flags=0xa [11939.452695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11939.457821] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11939.466019] workqueue ipv6_addrconf: flags=0x40008 [11939.468621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11939.475837] pending: addrconf_verify_work [11939.482616] workqueue knbd0-recv: flags=0x1a [11939.484660] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11939.491720] in-flight: 651:recv_work [11939.497719] workqueue ptlrpc_pinger: flags=0x2 [11939.501048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11939.504179] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11939.508881] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10022s workers=4 idle: 37 31846 28897 [11939.515309] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11929s workers=2 idle: 653 [11944.058241] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11944.067414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11944.097200] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11944.104882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11944.109836] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11944.113696] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11944.116398] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11944.124739] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11944.128238] RAX: 00000000000041d3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11944.131938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11944.134798] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11944.138507] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11944.141032] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11944.143277] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11944.146116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11944.148729] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11944.151051] Call Trace: [11944.152149] [11944.153218] ? show_regs.cold.9+0x22/0x2f [11944.155342] ? watchdog_timer_fn.cold.6+0x62/0x10a [11944.157847] ? __hrtimer_run_queues+0x1c2/0x4d0 [11944.159537] ? watchdog+0x40/0x40 [11944.161217] ? hrtimer_interrupt+0x128/0x2c0 [11944.162850] ? smp_apic_timer_interrupt+0xa6/0x270 [11944.164474] ? apic_timer_interrupt+0xf/0x20 [11944.166193] [11944.167101] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11944.168920] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11944.171554] do_raw_spin_lock+0x177/0x190 [11944.173159] _raw_spin_lock+0x21/0x30 [11944.175258] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11944.178941] process_one_work+0x2c8/0x700 [11944.180740] worker_thread+0x296/0x6e0 [11944.182564] ? rescuer_thread+0x570/0x570 [11944.184300] kthread+0x1d1/0x200 [11944.185918] ? set_kthread_struct+0x70/0x70 [11944.188050] ret_from_fork+0x1f/0x30 [11956.059073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [11956.062715] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [11956.064566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11956.064642] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11956.064650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11956.064652] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11956.064666] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11956.064671] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11956.064675] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11956.064677] RDX: 0000000000000792 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11956.064679] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11956.064681] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11956.064683] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11956.064686] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11956.064689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11956.064691] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11956.064700] Call Trace: [11956.064706] [11956.064711] ? show_regs.cold.9+0x22/0x2f [11956.064719] ? watchdog_timer_fn.cold.6+0x62/0x10a [11956.064726] ? __hrtimer_run_queues+0x1c2/0x4d0 [11956.064730] ? watchdog+0x40/0x40 [11956.064734] ? hrtimer_interrupt+0x128/0x2c0 [11956.064739] ? smp_apic_timer_interrupt+0xa6/0x270 [11956.064746] ? apic_timer_interrupt+0xf/0x20 [11956.064751] [11956.064754] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11956.064759] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11956.064763] do_raw_spin_lock+0x177/0x190 [11956.064769] _raw_spin_lock+0x21/0x30 [11956.064775] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11956.066080] obd_mod_free+0x46/0x140 [ptlrpc] [11956.067642] mdc_commit_open+0x91/0xb0 [mdc] [11956.067687] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11956.067914] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11956.074906] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11956.074993] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11956.075009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11956.075013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11956.075028] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11956.075032] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11956.075037] RAX: 0000000000005272 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11956.075040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11956.075042] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11956.075045] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11956.075048] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11956.075051] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11956.075054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11956.075057] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11956.075065] Call Trace: [11956.075071] [11956.075077] ? show_regs.cold.9+0x22/0x2f [11956.075086] ? watchdog_timer_fn.cold.6+0x62/0x10a [11956.075099] ? __hrtimer_run_queues+0x1c2/0x4d0 [11956.075104] ? watchdog+0x40/0x40 [11956.075109] ? hrtimer_interrupt+0x128/0x2c0 [11956.075113] ? smp_apic_timer_interrupt+0xa6/0x270 [11956.075126] ? apic_timer_interrupt+0xf/0x20 [11956.075132] [11956.075135] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11956.075142] do_raw_spin_lock+0x177/0x190 [11956.075148] _raw_spin_lock+0x21/0x30 [11956.075153] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11956.075680] ? do_raw_spin_unlock+0x75/0x190 [11956.075686] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11956.076117] ? do_wait_intr+0xf0/0xf0 [11956.076126] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11956.076619] mdc_close+0x53b/0xd70 [mdc] [11956.076669] lmv_close+0x18d/0x490 [lmv] [11956.076701] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11956.076833] ll_md_real_close+0xa5/0x2a0 [lustre] [11956.077197] ll_md_close+0x274/0x800 [lustre] [11956.077476] ll_file_release+0x128/0xc50 [lustre] [11956.077570] __fput+0x10d/0x440 [11956.077581] ____fput+0x12/0x20 [11956.077585] task_work_run+0xc6/0x110 [11956.077592] exit_to_usermode_loop+0x1dd/0x1f0 [11956.077599] do_syscall_64+0x3d6/0x440 [11956.077604] entry_SYSCALL_64_after_hwframe+0x49/0xae [11956.077613] RIP: 0033:0x7fa749552cd8 [11956.077620] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11956.077624] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11956.077630] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11956.077632] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11956.077634] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11956.077636] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11956.077638] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11956.533687] after_reply+0xe2c/0x1780 [ptlrpc] [11956.536153] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11956.538829] ? __next_timer_interrupt+0x160/0x160 [11956.540714] ptlrpcd+0xa82/0xe40 [ptlrpc] [11956.542436] ? do_wait_intr+0xf0/0xf0 [11956.544064] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11956.545960] kthread+0x1d1/0x200 [11956.547540] ? set_kthread_struct+0x70/0x70 [11956.548859] ret_from_fork+0x1f/0x30 [11964.054531] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [11964.060813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11964.078269] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11964.083017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11964.087594] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11964.090520] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11964.099416] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11964.103053] RAX: 0000000000001913 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11964.106831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [11964.110825] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11964.113971] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11964.117241] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11964.120987] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11964.124999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11964.127369] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11964.130750] Call Trace: [11964.131834] [11964.132883] ? show_regs.cold.9+0x22/0x2f [11964.135213] ? watchdog_timer_fn.cold.6+0x62/0x10a [11964.137751] ? __hrtimer_run_queues+0x1c2/0x4d0 [11964.139595] ? watchdog+0x40/0x40 [11964.141225] ? hrtimer_interrupt+0x128/0x2c0 [11964.142860] ? smp_apic_timer_interrupt+0xa6/0x270 [11964.145130] ? apic_timer_interrupt+0xf/0x20 [11964.147172] [11964.148046] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11964.150564] do_raw_spin_lock+0x177/0x190 [11964.152099] _raw_spin_lock+0x21/0x30 [11964.153493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11964.156569] ? do_raw_spin_unlock+0x75/0x190 [11964.158666] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11964.160435] ? do_wait_intr+0xf0/0xf0 [11964.162368] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11964.164777] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11964.167178] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11964.170025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11964.172529] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11964.175461] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11964.177838] mdc_intent_lock+0x353/0x7e0 [mdc] [11964.179767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11964.182314] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11964.185261] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11964.187563] lmv_intent_open+0x338/0x1150 [lmv] [11964.189703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11964.192494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11964.195653] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11964.198042] ll_intent_lock+0xc5/0xaa0 [lustre] [11964.199554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11964.201593] ll_lookup_it+0xe0e/0x2270 [lustre] [11964.203873] ? do_raw_spin_unlock+0x75/0x190 [11964.205989] ? _raw_spin_unlock+0x12/0x30 [11964.207775] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11964.210233] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11964.212408] ll_atomic_open+0x36a/0xd70 [lustre] [11964.214338] lookup_open+0x356/0xb40 [11964.215903] do_last+0x313/0xfc0 [11964.217219] ? nd_jump_root+0xe5/0x160 [11964.219013] ? path_init+0x437/0x520 [11964.220504] path_openat+0xf7/0x500 [11964.222116] ? do_raw_spin_unlock+0x75/0x190 [11964.223908] do_filp_open+0x99/0x140 [11964.225551] ? getname_flags+0x6e/0x330 [11964.227207] ? __check_object_size+0xff/0x256 [11964.229214] ? do_raw_spin_unlock+0x75/0x190 [11964.231188] ? _raw_spin_unlock+0x12/0x30 [11964.232883] do_sys_openat2+0x2b4/0x410 [11964.234631] do_sys_open+0x73/0xa0 [11964.236007] __x64_sys_openat+0x24/0x30 [11964.237736] do_syscall_64+0xc1/0x440 [11964.239266] entry_SYSCALL_64_after_hwframe+0x49/0xae [11964.241406] RIP: 0033:0x7fa749552332 [11964.243718] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11964.249632] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11964.252603] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11964.255203] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11964.258517] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11964.261312] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11964.264004] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [11970.018353] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10052s! [11970.025714] Showing busy workqueues and worker pools: [11970.029720] workqueue events: flags=0x0 [11970.031417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11970.034499] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [11970.043616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11970.047077] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11970.052779] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.057817] pending: kfree_rcu_monitor [11970.060404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11970.063187] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [11970.068219] workqueue events_long: flags=0x0 [11970.073088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.075484] pending: lu_objects_flush [obdclass] [11970.079946] workqueue events_unbound: flags=0x2 [11970.083390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [11970.095868] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11970.102171] workqueue events_power_efficient: flags=0x80 [11970.107378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.110900] pending: check_lifetime [11970.112090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.115149] pending: neigh_periodic_work [11970.117535] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.121187] pending: neigh_periodic_work [11970.123749] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.127714] pending: do_cache_clean [11970.129937] workqueue mm_percpu_wq: flags=0x8 [11970.132512] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.136628] pending: vmstat_update [11970.138763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.144222] pending: vmstat_update [11970.147387] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11970.152621] pending: vmstat_update [11970.156103] workqueue writeback: flags=0x4a [11970.159705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11970.162845] pending: wb_workfn, wb_workfn [11970.167897] workqueue xprtiod: flags=0xa [11970.171510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11970.181649] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11970.185174] workqueue ipv6_addrconf: flags=0x40008 [11970.188902] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11970.197039] pending: addrconf_verify_work [11970.198659] workqueue knbd0-recv: flags=0x1a [11970.207436] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11970.210951] in-flight: 651:recv_work [11970.213692] workqueue ptlrpc_pinger: flags=0x2 [11970.216441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11970.220523] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [11970.224878] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10053s workers=4 idle: 37 31846 28897 [11970.227296] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11960s workers=2 idle: 653 [11972.060597] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [11972.067681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11972.106657] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11972.121668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11972.126418] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11972.133979] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11972.137604] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11972.153932] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11972.156121] RAX: 0000000000007a9e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11972.171290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [11972.175395] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [11972.186431] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [11972.188357] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [11972.200849] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [11972.211892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11972.220916] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [11972.224613] Call Trace: [11972.230498] [11972.231384] ? show_regs.cold.9+0x22/0x2f [11972.236527] ? watchdog_timer_fn.cold.6+0x62/0x10a [11972.240495] ? __hrtimer_run_queues+0x1c2/0x4d0 [11972.246187] ? watchdog+0x40/0x40 [11972.250007] ? hrtimer_interrupt+0x128/0x2c0 [11972.253545] ? smp_apic_timer_interrupt+0xa6/0x270 [11972.258087] ? apic_timer_interrupt+0xf/0x20 [11972.262090] [11972.264827] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11972.269357] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11972.273166] do_raw_spin_lock+0x177/0x190 [11972.280435] _raw_spin_lock+0x21/0x30 [11972.286726] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11972.289456] process_one_work+0x2c8/0x700 [11972.293555] worker_thread+0x296/0x6e0 [11972.295603] ? rescuer_thread+0x570/0x570 [11972.299301] kthread+0x1d1/0x200 [11972.301408] ? set_kthread_struct+0x70/0x70 [11972.305883] ret_from_fork+0x1f/0x30 [11984.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [11984.059293] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [11984.062200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11984.062374] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11984.062382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11984.062384] RIP: 0010:native_safe_halt+0x2e/0x50 [11984.062408] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11984.062412] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11984.062416] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11984.062419] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [11984.062421] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [11984.062423] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [11984.062426] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [11984.062430] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [11984.062432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11984.062434] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [11984.062442] Call Trace: [11984.062458] [11984.062465] ? show_regs.cold.9+0x22/0x2f [11984.062476] ? watchdog_timer_fn.cold.6+0x62/0x10a [11984.062484] ? __hrtimer_run_queues+0x1c2/0x4d0 [11984.062489] ? watchdog+0x40/0x40 [11984.062493] ? hrtimer_interrupt+0x128/0x2c0 [11984.062498] ? smp_apic_timer_interrupt+0xa6/0x270 [11984.062504] ? apic_timer_interrupt+0xf/0x20 [11984.062508] [11984.062510] ? native_safe_halt+0x2e/0x50 [11984.062514] kvm_wait+0xcd/0xe0 [11984.062522] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11984.062532] do_raw_spin_lock+0x177/0x190 [11984.062536] _raw_spin_lock+0x21/0x30 [11984.062541] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11984.062835] ? do_raw_spin_unlock+0x75/0x190 [11984.062842] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11984.063048] ? do_wait_intr+0xf0/0xf0 [11984.063056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11984.063248] mdc_close+0x53b/0xd70 [mdc] [11984.063288] lmv_close+0x18d/0x490 [lmv] [11984.063319] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11984.063434] ll_md_real_close+0xa5/0x2a0 [lustre] [11984.063516] ll_md_close+0x274/0x800 [lustre] [11984.063600] ll_file_release+0x128/0xc50 [lustre] [11984.063683] __fput+0x10d/0x440 [11984.063693] ____fput+0x12/0x20 [11984.063697] task_work_run+0xc6/0x110 [11984.063705] exit_to_usermode_loop+0x1dd/0x1f0 [11984.063712] do_syscall_64+0x3d6/0x440 [11984.063718] entry_SYSCALL_64_after_hwframe+0x49/0xae [11984.063724] RIP: 0033:0x7fa749552cd8 [11984.063749] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11984.063754] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11984.063758] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [11984.063761] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [11984.063763] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [11984.063766] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [11984.063768] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11984.347538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11984.382541] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11984.391073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11984.397445] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11984.400546] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11984.414604] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11984.424715] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [11984.436584] RDX: 0000000000001823 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [11984.443706] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [11984.451951] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [11984.454806] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11984.459627] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [11984.466396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11984.476053] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [11984.482409] Call Trace: [11984.484243] [11984.486685] ? show_regs.cold.9+0x22/0x2f [11984.488632] ? watchdog_timer_fn.cold.6+0x62/0x10a [11984.493403] ? __hrtimer_run_queues+0x1c2/0x4d0 [11984.499121] ? watchdog+0x40/0x40 [11984.506554] ? hrtimer_interrupt+0x128/0x2c0 [11984.508032] ? smp_apic_timer_interrupt+0xa6/0x270 [11984.516523] ? apic_timer_interrupt+0xf/0x20 [11984.521855] [11984.524155] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11984.528475] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11984.531893] do_raw_spin_lock+0x177/0x190 [11984.534833] _raw_spin_lock+0x21/0x30 [11984.537502] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11984.540036] obd_mod_free+0x46/0x140 [ptlrpc] [11984.544169] mdc_commit_open+0x91/0xb0 [mdc] [11984.552164] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11984.558638] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11984.564018] after_reply+0xe2c/0x1780 [ptlrpc] [11984.565369] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11984.569807] ? __next_timer_interrupt+0x160/0x160 [11984.572168] ptlrpcd+0xa82/0xe40 [ptlrpc] [11984.574434] ? do_wait_intr+0xf0/0xf0 [11984.578085] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11984.583615] kthread+0x1d1/0x200 [11984.587901] ? set_kthread_struct+0x70/0x70 [11984.593515] ret_from_fork+0x1f/0x30 [11996.054039] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [11996.057298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11996.077426] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11996.083016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11996.086981] RIP: 0010:native_safe_halt+0x2e/0x50 [11996.089154] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11996.097184] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11996.100500] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [11996.103758] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [11996.107611] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [11996.110183] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [11996.113195] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [11996.116532] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [11996.120195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11996.122628] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [11996.126135] Call Trace: [11996.127272] [11996.128123] ? show_regs.cold.9+0x22/0x2f [11996.130032] ? watchdog_timer_fn.cold.6+0x62/0x10a [11996.131726] ? __hrtimer_run_queues+0x1c2/0x4d0 [11996.134223] ? watchdog+0x40/0x40 [11996.135658] ? hrtimer_interrupt+0x128/0x2c0 [11996.137517] ? smp_apic_timer_interrupt+0xa6/0x270 [11996.139755] ? apic_timer_interrupt+0xf/0x20 [11996.141715] [11996.142815] ? native_safe_halt+0x2e/0x50 [11996.144472] kvm_wait+0xcd/0xe0 [11996.145891] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11996.148189] do_raw_spin_lock+0x177/0x190 [11996.149871] _raw_spin_lock+0x21/0x30 [11996.151463] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11996.154127] ? do_raw_spin_unlock+0x75/0x190 [11996.156049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11996.159022] ? do_wait_intr+0xf0/0xf0 [11996.160365] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11996.163058] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11996.165052] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11996.167266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.169200] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11996.171256] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11996.172884] mdc_intent_lock+0x353/0x7e0 [mdc] [11996.174386] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.176242] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11996.178386] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11996.180212] lmv_intent_open+0x338/0x1150 [lmv] [11996.181774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.183803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.186409] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11996.188238] ll_intent_lock+0xc5/0xaa0 [lustre] [11996.190028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.191835] ll_lookup_it+0xe0e/0x2270 [lustre] [11996.193861] ? do_raw_spin_unlock+0x75/0x190 [11996.195433] ? _raw_spin_unlock+0x12/0x30 [11996.196593] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11996.198718] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11996.200701] ll_atomic_open+0x36a/0xd70 [lustre] [11996.203037] lookup_open+0x356/0xb40 [11996.204552] do_last+0x313/0xfc0 [11996.206231] ? nd_jump_root+0xe5/0x160 [11996.207837] ? path_init+0x437/0x520 [11996.209889] path_openat+0xf7/0x500 [11996.211781] ? do_raw_spin_unlock+0x75/0x190 [11996.213855] do_filp_open+0x99/0x140 [11996.215493] ? getname_flags+0x6e/0x330 [11996.217312] ? __check_object_size+0xff/0x256 [11996.219263] ? do_raw_spin_unlock+0x75/0x190 [11996.221339] ? _raw_spin_unlock+0x12/0x30 [11996.223439] do_sys_openat2+0x2b4/0x410 [11996.225179] do_sys_open+0x73/0xa0 [11996.226812] __x64_sys_openat+0x24/0x30 [11996.228255] do_syscall_64+0xc1/0x440 [11996.229859] entry_SYSCALL_64_after_hwframe+0x49/0xae [11996.232152] RIP: 0033:0x7fa749552332 [11996.233643] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11996.242332] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11996.245730] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [11996.248628] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [11996.251983] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [11996.254846] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [11996.257935] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12000.057137] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12000.063939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12000.100930] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12000.109174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12000.113953] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12000.116483] RIP: 0010:native_safe_halt+0x2e/0x50 [12000.117914] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12000.128235] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12000.131661] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12000.136564] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12000.139824] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12000.143209] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12000.148175] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12000.151207] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12000.154816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12000.161348] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12000.166277] Call Trace: [12000.167178] [12000.167688] ? show_regs.cold.9+0x22/0x2f [12000.172114] ? watchdog_timer_fn.cold.6+0x62/0x10a [12000.177694] ? __hrtimer_run_queues+0x1c2/0x4d0 [12000.183526] ? watchdog+0x40/0x40 [12000.184808] ? hrtimer_interrupt+0x128/0x2c0 [12000.186235] ? smp_apic_timer_interrupt+0xa6/0x270 [12000.187503] ? apic_timer_interrupt+0xf/0x20 [12000.188569] [12000.189213] ? native_safe_halt+0x2e/0x50 [12000.190799] kvm_wait+0xcd/0xe0 [12000.191977] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12000.194695] do_raw_spin_lock+0x177/0x190 [12000.196588] _raw_spin_lock+0x21/0x30 [12000.200811] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12000.203862] process_one_work+0x2c8/0x700 [12000.208772] worker_thread+0x296/0x6e0 [12000.210237] ? rescuer_thread+0x570/0x570 [12000.213273] kthread+0x1d1/0x200 [12000.218304] ? set_kthread_struct+0x70/0x70 [12000.227823] ret_from_fork+0x1f/0x30 [12000.740512] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10083s! [12000.749566] Showing busy workqueues and worker pools: [12000.752637] workqueue events: flags=0x0 [12000.754546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12000.760321] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12000.771413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12000.776113] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12000.783047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.789224] pending: kfree_rcu_monitor [12000.795094] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12000.804087] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12000.812423] workqueue events_long: flags=0x0 [12000.814052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.818823] pending: lu_objects_flush [obdclass] [12000.823388] workqueue events_unbound: flags=0x2 [12000.830679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12000.833853] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12000.848146] workqueue events_power_efficient: flags=0x80 [12000.857834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.862950] pending: check_lifetime [12000.865779] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.874289] pending: neigh_periodic_work [12000.879239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.887574] pending: neigh_periodic_work [12000.894092] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.902565] pending: do_cache_clean [12000.904098] workqueue mm_percpu_wq: flags=0x8 [12000.911234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.922667] pending: vmstat_update [12000.925271] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.931876] pending: vmstat_update [12000.935459] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12000.937357] pending: vmstat_update [12000.938616] workqueue writeback: flags=0x4a [12000.946586] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12000.956115] pending: wb_workfn, wb_workfn [12000.961391] workqueue xprtiod: flags=0xa [12000.964378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12000.973641] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12000.986266] workqueue ipv6_addrconf: flags=0x40008 [12000.993418] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12000.995945] pending: addrconf_verify_work [12001.000800] workqueue knbd0-recv: flags=0x1a [12001.002372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12001.009090] in-flight: 651:recv_work [12001.012032] workqueue ptlrpc_pinger: flags=0x2 [12001.014142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12001.017270] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12001.021066] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10083s workers=4 idle: 37 31846 28897 [12001.025309] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11990s workers=2 idle: 653 [12012.058951] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12012.059130] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12012.062696] Modules linked in: [12012.065882] Modules linked in: [12012.065885] lustre(O) osp(O) ofd(O) [12012.067204] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12012.067278] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12012.067286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12012.067289] RIP: 0010:native_safe_halt+0x2e/0x50 [12012.068308] lod(O) [12012.069959] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12012.092351] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12012.092423] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12012.092431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12012.092434] RIP: 0010:native_safe_halt+0x2e/0x50 [12012.092447] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12012.092452] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12012.092456] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12012.092458] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12012.092461] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12012.092463] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12012.092466] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12012.092469] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12012.092472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12012.092474] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12012.092484] Call Trace: [12012.092492] [12012.092499] ? show_regs.cold.9+0x22/0x2f [12012.092509] ? watchdog_timer_fn.cold.6+0x62/0x10a [12012.092527] ? __hrtimer_run_queues+0x1c2/0x4d0 [12012.092532] ? watchdog+0x40/0x40 [12012.092537] ? hrtimer_interrupt+0x128/0x2c0 [12012.092541] ? smp_apic_timer_interrupt+0xa6/0x270 [12012.092548] ? apic_timer_interrupt+0xf/0x20 [12012.092552] [12012.092556] ? native_safe_halt+0x2e/0x50 [12012.092560] kvm_wait+0xcd/0xe0 [12012.092568] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12012.092575] do_raw_spin_lock+0x177/0x190 [12012.092579] _raw_spin_lock+0x21/0x30 [12012.092585] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12012.092972] obd_mod_free+0x46/0x140 [ptlrpc] [12012.093505] mdc_commit_open+0x91/0xb0 [mdc] [12012.099480] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 [12012.104816] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12012.108394] ORIG_RAX: ffffffffffffff13 [12012.108405] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12012.108408] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12012.108410] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12012.108413] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12012.108415] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12012.108419] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12012.108422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12012.108425] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12012.108434] Call Trace: [12012.108441] [12012.108447] ? show_regs.cold.9+0x22/0x2f [12012.108460] ? watchdog_timer_fn.cold.6+0x62/0x10a [12012.108466] ? __hrtimer_run_queues+0x1c2/0x4d0 [12012.108471] ? watchdog+0x40/0x40 [12012.108476] ? hrtimer_interrupt+0x128/0x2c0 [12012.108480] ? smp_apic_timer_interrupt+0xa6/0x270 [12012.108489] ? apic_timer_interrupt+0xf/0x20 [12012.108493] [12012.108496] ? native_safe_halt+0x2e/0x50 [12012.108500] kvm_wait+0xcd/0xe0 [12012.108519] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12012.108531] do_raw_spin_lock+0x177/0x190 [12012.108535] _raw_spin_lock+0x21/0x30 [12012.108541] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12012.108898] ? do_raw_spin_unlock+0x75/0x190 [12012.108905] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12012.109165] ? do_wait_intr+0xf0/0xf0 [12012.109174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12012.109440] mdc_close+0x53b/0xd70 [mdc] [12012.109481] lmv_close+0x18d/0x490 [lmv] [12012.109522] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12012.109716] ll_md_real_close+0xa5/0x2a0 [lustre] [12012.109797] ll_md_close+0x274/0x800 [lustre] [12012.109877] ll_file_release+0x128/0xc50 [lustre] [12012.109958] __fput+0x10d/0x440 [12012.109968] ____fput+0x12/0x20 [12012.109972] task_work_run+0xc6/0x110 [12012.109979] exit_to_usermode_loop+0x1dd/0x1f0 [12012.109987] do_syscall_64+0x3d6/0x440 [12012.109991] entry_SYSCALL_64_after_hwframe+0x49/0xae [12012.110048] RIP: 0033:0x7fa749552cd8 [12012.110057] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12012.110060] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12012.110065] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12012.110067] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12012.110070] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12012.110072] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12012.110074] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12012.518938] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12012.522845] after_reply+0xe2c/0x1780 [ptlrpc] [12012.527482] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12012.532276] ? __next_timer_interrupt+0x160/0x160 [12012.536207] ptlrpcd+0xa82/0xe40 [ptlrpc] [12012.539362] ? do_wait_intr+0xf0/0xf0 [12012.541912] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12012.544989] kthread+0x1d1/0x200 [12012.546782] ? set_kthread_struct+0x70/0x70 [12012.548811] ret_from_fork+0x1f/0x30 [12028.054571] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12028.057046] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12028.057366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12028.057435] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12028.057441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12028.057444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12028.057460] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12028.057464] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12028.057469] RAX: 0000000000006384 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12028.057472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12028.057474] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12028.057476] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12028.057478] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12028.057482] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12028.057485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12028.057487] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12028.057496] Call Trace: [12028.057502] [12028.057508] ? show_regs.cold.9+0x22/0x2f [12028.057517] ? watchdog_timer_fn.cold.6+0x62/0x10a [12028.057523] ? __hrtimer_run_queues+0x1c2/0x4d0 [12028.057527] ? watchdog+0x40/0x40 [12028.057531] ? hrtimer_interrupt+0x128/0x2c0 [12028.057535] ? smp_apic_timer_interrupt+0xa6/0x270 [12028.057543] ? apic_timer_interrupt+0xf/0x20 [12028.057548] [12028.057551] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12028.057558] do_raw_spin_lock+0x177/0x190 [12028.057563] _raw_spin_lock+0x21/0x30 [12028.057568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12028.064429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12028.064506] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12028.064513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12028.064519] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12028.066040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12028.066054] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12028.066058] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12028.066064] RAX: 0000000000004f8c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12028.066066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12028.066069] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12028.066071] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12028.066073] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12028.066077] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12028.066080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12028.066082] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12028.066092] Call Trace: [12028.066098] [12028.066105] ? show_regs.cold.9+0x22/0x2f [12028.066117] ? watchdog_timer_fn.cold.6+0x62/0x10a [12028.066123] ? __hrtimer_run_queues+0x1c2/0x4d0 [12028.066127] ? watchdog+0x40/0x40 [12028.066132] ? hrtimer_interrupt+0x128/0x2c0 [12028.066136] ? smp_apic_timer_interrupt+0xa6/0x270 [12028.139481] ? do_raw_spin_unlock+0x75/0x190 [12028.139503] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12028.140887] ? do_wait_intr+0xf0/0xf0 [12028.140901] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12028.141796] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12028.142392] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12028.142941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12028.143920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12028.143990] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12028.144085] mdc_intent_lock+0x353/0x7e0 [mdc] [12028.144124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12028.144245] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12028.157144] ? apic_timer_interrupt+0xf/0x20 [12028.157171] [12028.157176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12028.157183] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12028.157189] do_raw_spin_lock+0x177/0x190 [12028.157195] _raw_spin_lock+0x21/0x30 [12028.157200] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12028.157501] process_one_work+0x2c8/0x700 [12028.157507] worker_thread+0x296/0x6e0 [12028.157512] ? rescuer_thread+0x570/0x570 [12028.157516] kthread+0x1d1/0x200 [12028.157520] ? set_kthread_struct+0x70/0x70 [12028.157524] ret_from_fork+0x1f/0x30 [12028.373400] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12028.376423] lmv_intent_open+0x338/0x1150 [lmv] [12028.378544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12028.381212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12028.383574] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12028.385667] ll_intent_lock+0xc5/0xaa0 [lustre] [12028.388078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12028.390472] ll_lookup_it+0xe0e/0x2270 [lustre] [12028.393147] ? do_raw_spin_unlock+0x75/0x190 [12028.395217] ? _raw_spin_unlock+0x12/0x30 [12028.396914] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12028.399141] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12028.401276] ll_atomic_open+0x36a/0xd70 [lustre] [12028.403662] lookup_open+0x356/0xb40 [12028.405368] do_last+0x313/0xfc0 [12028.406774] ? nd_jump_root+0xe5/0x160 [12028.408000] ? path_init+0x437/0x520 [12028.409050] path_openat+0xf7/0x500 [12028.410098] ? do_raw_spin_unlock+0x75/0x190 [12028.411814] do_filp_open+0x99/0x140 [12028.413273] ? getname_flags+0x6e/0x330 [12028.414845] ? __check_object_size+0xff/0x256 [12028.416813] ? do_raw_spin_unlock+0x75/0x190 [12028.418654] ? _raw_spin_unlock+0x12/0x30 [12028.420483] do_sys_openat2+0x2b4/0x410 [12028.422191] do_sys_open+0x73/0xa0 [12028.423476] __x64_sys_openat+0x24/0x30 [12028.425157] do_syscall_64+0xc1/0x440 [12028.426762] entry_SYSCALL_64_after_hwframe+0x49/0xae [12028.428903] RIP: 0033:0x7fa749552332 [12028.430726] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12028.439070] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12028.443022] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12028.446109] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12028.449914] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12028.452952] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12028.456061] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12031.458484] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10114s! [12031.465140] Showing busy workqueues and worker pools: [12031.470263] workqueue events: flags=0x0 [12031.474279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12031.478840] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12031.487227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12031.494390] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12031.504124] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.509877] pending: kfree_rcu_monitor [12031.512258] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12031.516560] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12031.526723] workqueue events_long: flags=0x0 [12031.528301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.531460] pending: lu_objects_flush [obdclass] [12031.534728] workqueue events_unbound: flags=0x2 [12031.536823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12031.541289] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12031.546925] workqueue events_power_efficient: flags=0x80 [12031.550217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.554340] pending: check_lifetime [12031.562972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.572630] pending: neigh_periodic_work [12031.574286] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.584738] pending: neigh_periodic_work [12031.585867] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.589647] pending: do_cache_clean [12031.592080] workqueue mm_percpu_wq: flags=0x8 [12031.594336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.597819] pending: vmstat_update [12031.599157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.602544] pending: vmstat_update [12031.604580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12031.607202] pending: vmstat_update [12031.609241] workqueue writeback: flags=0x4a [12031.611441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12031.613456] pending: wb_workfn, wb_workfn [12031.615450] workqueue xprtiod: flags=0xa [12031.618067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12031.620935] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12031.624013] workqueue ipv6_addrconf: flags=0x40008 [12031.626446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12031.629477] pending: addrconf_verify_work [12031.631599] workqueue knbd0-recv: flags=0x1a [12031.633585] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12031.636718] in-flight: 651:recv_work [12031.638749] workqueue ptlrpc_pinger: flags=0x2 [12031.640499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12031.645595] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12031.653027] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10114s workers=4 idle: 37 31846 28897 [12031.663153] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12021s workers=2 idle: 653 [12040.058009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12040.060996] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12040.063466] Modules linked in: [12040.069305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [12040.070805] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12040.070884] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12040.070891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12040.070895] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12040.070913] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12040.070917] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12040.070921] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12040.070923] RDX: 0000000000001b63 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12040.070925] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12040.070927] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12040.070929] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12040.070932] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12040.070935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12040.070937] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12040.070944] Call Trace: [12040.070950] [12040.070955] ? show_regs.cold.9+0x22/0x2f [12040.078401] lfsck(O) [12040.126382] ? watchdog_timer_fn.cold.6+0x62/0x10a [12040.126403] ? __hrtimer_run_queues+0x1c2/0x4d0 [12040.126408] ? watchdog+0x40/0x40 [12040.126412] ? hrtimer_interrupt+0x128/0x2c0 [12040.126417] ? smp_apic_timer_interrupt+0xa6/0x270 [12040.126427] ? apic_timer_interrupt+0xf/0x20 [12040.126432] [12040.126437] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12040.126444] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12040.126449] do_raw_spin_lock+0x177/0x190 [12040.126455] _raw_spin_lock+0x21/0x30 [12040.126462] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12040.127000] obd_mod_free+0x46/0x140 [ptlrpc] [12040.135470] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12040.135534] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12040.135541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12040.135545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12040.135559] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12040.135563] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12040.135568] RAX: 0000000000007d86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12040.135571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12040.135573] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12040.135575] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12040.135577] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12040.135640] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12040.135644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12040.135646] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12040.135654] Call Trace: [12040.135727] [12040.135778] ? show_regs.cold.9+0x22/0x2f [12040.135789] ? watchdog_timer_fn.cold.6+0x62/0x10a [12040.135796] ? __hrtimer_run_queues+0x1c2/0x4d0 [12040.135800] ? watchdog+0x40/0x40 [12040.135804] ? hrtimer_interrupt+0x128/0x2c0 [12040.135808] ? smp_apic_timer_interrupt+0xa6/0x270 [12040.135816] ? apic_timer_interrupt+0xf/0x20 [12040.135820] [12040.135823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12040.135830] do_raw_spin_lock+0x177/0x190 [12040.135835] _raw_spin_lock+0x21/0x30 [12040.135839] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12040.136133] ? do_raw_spin_unlock+0x75/0x190 [12040.136139] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12040.136339] ? do_wait_intr+0xf0/0xf0 [12040.136348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12040.136547] mdc_close+0x53b/0xd70 [mdc] [12040.136587] lmv_close+0x18d/0x490 [lmv] [12040.136619] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12040.137580] ll_md_real_close+0xa5/0x2a0 [lustre] [12040.137742] ll_md_close+0x274/0x800 [lustre] [12040.137899] ll_file_release+0x128/0xc50 [lustre] [12040.138002] __fput+0x10d/0x440 [12040.138016] ____fput+0x12/0x20 [12040.138031] task_work_run+0xc6/0x110 [12040.138043] exit_to_usermode_loop+0x1dd/0x1f0 [12040.138051] do_syscall_64+0x3d6/0x440 [12040.138055] entry_SYSCALL_64_after_hwframe+0x49/0xae [12040.138063] RIP: 0033:0x7fa749552cd8 [12040.138071] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12040.138075] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12040.138080] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12040.138082] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12040.138084] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12040.138087] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12040.138089] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12040.585365] mdc_commit_open+0x91/0xb0 [mdc] [12040.591055] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12040.595969] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12040.599827] after_reply+0xe2c/0x1780 [ptlrpc] [12040.602950] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12040.606638] ? __next_timer_interrupt+0x160/0x160 [12040.609524] ptlrpcd+0xa82/0xe40 [ptlrpc] [12040.612097] ? do_wait_intr+0xf0/0xf0 [12040.614233] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12040.617384] kthread+0x1d1/0x200 [12040.619633] ? set_kthread_struct+0x70/0x70 [12040.621939] ret_from_fork+0x1f/0x30 [12053.912072] rcu: INFO: rcu_sched self-detected stall on CPU [12053.924121] rcu: 2-...!: (5733154 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [12053.943744] (t=10140223 jiffies g=258701 q=4447) [12053.952082] rcu: rcu_sched kthread starved for 10137018 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12053.964050] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12053.989271] rcu: RCU grace-period kthread stack dump: [12053.994787] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12054.010289] Call Trace: [12054.013074] __schedule+0x351/0xcb0 [12054.020310] schedule+0xc0/0x180 [12054.025458] schedule_timeout+0xb4/0x190 [12054.035937] ? do_raw_spin_unlock+0x75/0x190 [12054.041943] ? __next_timer_interrupt+0x160/0x160 [12054.046431] ? prepare_to_swait_event+0xf3/0x140 [12054.051698] rcu_gp_kthread+0x8dd/0x15b0 [12054.053307] ? force_qs_rnp+0x2c0/0x2c0 [12054.054663] kthread+0x1d1/0x200 [12054.056376] ? set_kthread_struct+0x70/0x70 [12054.058271] ret_from_fork+0x1f/0x30 [12054.059861] rcu: Stack dump where RCU GP kthread last ran: [12054.062237] Sending NMI from CPU 2 to CPUs 0: [12054.064145] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [12054.065886] NMI backtrace for cpu 2 [12054.072141] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12054.080312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12054.086891] Call Trace: [12054.088314] [12054.089168] ? dump_stack+0xbb/0x10e [12054.091041] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12054.094856] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12054.101266] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12054.104744] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12054.107843] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12054.109920] ? rcu_dump_cpu_stacks+0x11b/0x171 [12054.112036] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12054.116223] ? update_process_times+0x6d/0xc0 [12054.121855] ? tick_sched_handle.isra.18+0x27/0x90 [12054.123267] ? tick_sched_timer+0x53/0xb0 [12054.125322] ? __hrtimer_run_queues+0x1c2/0x4d0 [12054.129103] ? tick_sched_handle.isra.18+0x90/0x90 [12054.135268] ? hrtimer_interrupt+0x128/0x2c0 [12054.139017] ? smp_apic_timer_interrupt+0xa6/0x270 [12054.141097] ? apic_timer_interrupt+0xf/0x20 [12054.144577] [12054.148148] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12054.150561] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12054.152289] ? do_raw_spin_lock+0x177/0x190 [12054.154472] ? _raw_spin_lock+0x21/0x30 [12054.156670] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12054.161810] ? obd_mod_free+0x46/0x140 [ptlrpc] [12054.165088] ? mdc_commit_open+0x91/0xb0 [mdc] [12054.170913] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12054.177444] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12054.182440] ? after_reply+0xe2c/0x1780 [ptlrpc] [12054.192606] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12054.196846] ? __next_timer_interrupt+0x160/0x160 [12054.198945] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12054.207868] ? do_wait_intr+0xf0/0xf0 [12054.209962] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12054.216268] ? kthread+0x1d1/0x200 [12054.218016] ? set_kthread_struct+0x70/0x70 [12054.224033] ? ret_from_fork+0x1f/0x30 [12056.057058] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12056.061044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12056.086007] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12056.096012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12056.104010] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12056.109588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12056.113278] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12056.123888] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12056.128985] RAX: 0000000000000ec7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12056.133127] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12056.137402] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12056.142187] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12056.145882] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12056.148965] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12056.152860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12056.156169] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12056.159762] Call Trace: [12056.161094] [12056.162331] ? show_regs.cold.9+0x22/0x2f [12056.164013] ? watchdog_timer_fn.cold.6+0x62/0x10a [12056.166587] ? __hrtimer_run_queues+0x1c2/0x4d0 [12056.168837] ? watchdog+0x40/0x40 [12056.170860] ? hrtimer_interrupt+0x128/0x2c0 [12056.172981] ? smp_apic_timer_interrupt+0xa6/0x270 [12056.175608] ? apic_timer_interrupt+0xf/0x20 [12056.178223] [12056.179693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12056.182210] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12056.184433] do_raw_spin_lock+0x177/0x190 [12056.186086] _raw_spin_lock+0x21/0x30 [12056.187891] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12056.190759] process_one_work+0x2c8/0x700 [12056.192571] worker_thread+0x296/0x6e0 [12056.194313] ? rescuer_thread+0x570/0x570 [12056.196356] kthread+0x1d1/0x200 [12056.197946] ? set_kthread_struct+0x70/0x70 [12056.200074] ret_from_fork+0x1f/0x30 [12062.176504] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10145s! [12062.186503] Showing busy workqueues and worker pools: [12062.188412] workqueue events: flags=0x0 [12062.191203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12062.195824] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12062.203483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12062.213747] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12062.222814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.229896] pending: kfree_rcu_monitor [12062.236691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12062.245836] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12062.256535] workqueue events_long: flags=0x0 [12062.266670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.270882] pending: lu_objects_flush [obdclass] [12062.274965] workqueue events_unbound: flags=0x2 [12062.276569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12062.287777] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12062.304160] workqueue events_power_efficient: flags=0x80 [12062.307191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.319679] pending: check_lifetime [12062.324755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.329159] pending: neigh_periodic_work [12062.333308] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.342615] pending: neigh_periodic_work [12062.347265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.355997] pending: do_cache_clean [12062.359156] workqueue mm_percpu_wq: flags=0x8 [12062.361059] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.366837] pending: vmstat_update [12062.369010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.377595] pending: vmstat_update [12062.379038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12062.390047] pending: vmstat_update [12062.392255] workqueue writeback: flags=0x4a [12062.396481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12062.400712] pending: wb_workfn, wb_workfn [12062.402376] workqueue xprtiod: flags=0xa [12062.410758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12062.418788] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12062.430880] workqueue ipv6_addrconf: flags=0x40008 [12062.432693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12062.443820] pending: addrconf_verify_work [12062.449693] workqueue knbd0-recv: flags=0x1a [12062.454425] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12062.462091] in-flight: 651:recv_work [12062.465825] workqueue ptlrpc_pinger: flags=0x2 [12062.472333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12062.478114] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12062.481899] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10145s workers=4 idle: 37 31846 28897 [12062.486049] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12052s workers=2 idle: 653 [12068.059575] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12068.061853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12068.075987] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12068.079004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12068.082438] RIP: 0010:native_safe_halt+0x2e/0x50 [12068.083941] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12068.092650] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12068.096670] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12068.100004] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12068.103359] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12068.106978] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12068.111725] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12068.114949] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12068.118650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12068.121174] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12068.124512] Call Trace: [12068.125895] [12068.127206] ? show_regs.cold.9+0x22/0x2f [12068.128923] ? watchdog_timer_fn.cold.6+0x62/0x10a [12068.131505] ? __hrtimer_run_queues+0x1c2/0x4d0 [12068.133390] ? watchdog+0x40/0x40 [12068.135023] ? hrtimer_interrupt+0x128/0x2c0 [12068.137008] ? smp_apic_timer_interrupt+0xa6/0x270 [12068.139118] ? apic_timer_interrupt+0xf/0x20 [12068.141193] [12068.142108] ? native_safe_halt+0x2e/0x50 [12068.143689] kvm_wait+0xcd/0xe0 [12068.145044] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12068.147409] do_raw_spin_lock+0x177/0x190 [12068.149085] _raw_spin_lock+0x21/0x30 [12068.150798] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12068.153910] ? do_raw_spin_unlock+0x75/0x190 [12068.155883] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12068.158452] ? do_wait_intr+0xf0/0xf0 [12068.160073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12068.163143] mdc_close+0x53b/0xd70 [mdc] [12068.165363] lmv_close+0x18d/0x490 [lmv] [12068.167805] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12068.171055] ll_md_real_close+0xa5/0x2a0 [lustre] [12068.173151] ll_md_close+0x274/0x800 [lustre] [12068.175793] ll_file_release+0x128/0xc50 [lustre] [12068.178715] __fput+0x10d/0x440 [12068.179977] ____fput+0x12/0x20 [12068.181305] task_work_run+0xc6/0x110 [12068.182919] exit_to_usermode_loop+0x1dd/0x1f0 [12068.184761] do_syscall_64+0x3d6/0x440 [12068.186592] entry_SYSCALL_64_after_hwframe+0x49/0xae [12068.188933] RIP: 0033:0x7fa749552cd8 [12068.190410] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12068.198401] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12068.202762] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12068.205955] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12068.209436] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12068.212910] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12068.215341] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12080.058018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12080.062331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12080.082191] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12080.087421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12080.092283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12080.095300] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12080.102911] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12080.106162] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12080.109255] RDX: 0000000000006d71 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12080.113010] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12080.116628] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12080.119962] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12080.123882] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12080.127926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12080.130710] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12080.135044] Call Trace: [12080.136232] [12080.137033] ? show_regs.cold.9+0x22/0x2f [12080.138837] ? watchdog_timer_fn.cold.6+0x62/0x10a [12080.140854] ? __hrtimer_run_queues+0x1c2/0x4d0 [12080.142603] ? watchdog+0x40/0x40 [12080.143980] ? hrtimer_interrupt+0x128/0x2c0 [12080.145996] ? smp_apic_timer_interrupt+0xa6/0x270 [12080.147698] ? apic_timer_interrupt+0xf/0x20 [12080.151475] [12080.152198] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12080.154709] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12080.156680] do_raw_spin_lock+0x177/0x190 [12080.160122] _raw_spin_lock+0x21/0x30 [12080.164268] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12080.169864] obd_mod_free+0x46/0x140 [ptlrpc] [12080.174213] mdc_commit_open+0x91/0xb0 [mdc] [12080.178997] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12080.186671] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12080.193403] after_reply+0xe2c/0x1780 [ptlrpc] [12080.196917] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12080.199311] ? __next_timer_interrupt+0x160/0x160 [12080.201758] ptlrpcd+0xa82/0xe40 [ptlrpc] [12080.209996] ? do_wait_intr+0xf0/0xf0 [12080.211982] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12080.218024] kthread+0x1d1/0x200 [12080.222149] ? set_kthread_struct+0x70/0x70 [12080.226108] ret_from_fork+0x1f/0x30 [12084.057032] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12084.060040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12084.083721] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12084.090597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12084.093891] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12084.096995] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12084.103334] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12084.114875] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12084.117623] RAX: 0000000000004783 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12084.127046] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12084.135426] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12084.141827] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12084.145505] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12084.152013] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12084.156549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12084.159394] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12084.169189] Call Trace: [12084.170600] [12084.171786] ? show_regs.cold.9+0x22/0x2f [12084.175064] ? watchdog_timer_fn.cold.6+0x62/0x10a [12084.177549] ? __hrtimer_run_queues+0x1c2/0x4d0 [12084.182887] ? watchdog+0x40/0x40 [12084.186587] ? hrtimer_interrupt+0x128/0x2c0 [12084.189704] ? smp_apic_timer_interrupt+0xa6/0x270 [12084.194953] ? apic_timer_interrupt+0xf/0x20 [12084.197654] [12084.198636] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12084.201533] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12084.204672] do_raw_spin_lock+0x177/0x190 [12084.207619] _raw_spin_lock+0x21/0x30 [12084.211727] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12084.219538] process_one_work+0x2c8/0x700 [12084.221826] worker_thread+0x296/0x6e0 [12084.223817] ? rescuer_thread+0x570/0x570 [12084.226238] kthread+0x1d1/0x200 [12084.228531] ? set_kthread_struct+0x70/0x70 [12084.230008] ret_from_fork+0x1f/0x30 [12088.054030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12088.059343] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12088.102993] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12088.108651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12088.114333] RIP: 0010:native_safe_halt+0x2e/0x50 [12088.116988] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12088.132037] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12088.135225] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12088.141813] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12088.144352] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12088.151197] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12088.158325] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12088.162330] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12088.168237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12088.171015] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12088.174095] Call Trace: [12088.175278] [12088.176099] ? show_regs.cold.9+0x22/0x2f [12088.177647] ? watchdog_timer_fn.cold.6+0x62/0x10a [12088.184692] ? __hrtimer_run_queues+0x1c2/0x4d0 [12088.185861] ? watchdog+0x40/0x40 [12088.186704] ? hrtimer_interrupt+0x128/0x2c0 [12088.190539] ? smp_apic_timer_interrupt+0xa6/0x270 [12088.192957] ? apic_timer_interrupt+0xf/0x20 [12088.194020] [12088.194573] ? native_safe_halt+0x2e/0x50 [12088.198154] kvm_wait+0xcd/0xe0 [12088.198998] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12088.200308] do_raw_spin_lock+0x177/0x190 [12088.203362] _raw_spin_lock+0x21/0x30 [12088.204567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12088.206830] ? do_raw_spin_unlock+0x75/0x190 [12088.209716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12088.212503] ? do_wait_intr+0xf0/0xf0 [12088.213848] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12088.219093] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12088.222251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12088.227503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12088.230115] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12088.235673] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12088.237296] mdc_intent_lock+0x353/0x7e0 [mdc] [12088.238500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12088.243333] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12088.246164] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12088.250826] lmv_intent_open+0x338/0x1150 [lmv] [12088.254535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12088.256234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12088.258108] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12088.260996] ll_intent_lock+0xc5/0xaa0 [lustre] [12088.262880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12088.266359] ll_lookup_it+0xe0e/0x2270 [lustre] [12088.268339] ? do_raw_spin_unlock+0x75/0x190 [12088.269768] ? _raw_spin_unlock+0x12/0x30 [12088.278825] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12088.280599] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12088.288646] ll_atomic_open+0x36a/0xd70 [lustre] [12088.296420] lookup_open+0x356/0xb40 [12088.300074] do_last+0x313/0xfc0 [12088.301095] ? nd_jump_root+0xe5/0x160 [12088.302948] ? path_init+0x437/0x520 [12088.304261] path_openat+0xf7/0x500 [12088.307637] ? do_raw_spin_unlock+0x75/0x190 [12088.309824] do_filp_open+0x99/0x140 [12088.312624] ? getname_flags+0x6e/0x330 [12088.314411] ? __check_object_size+0xff/0x256 [12088.319208] ? do_raw_spin_unlock+0x75/0x190 [12088.321044] ? _raw_spin_unlock+0x12/0x30 [12088.323510] do_sys_openat2+0x2b4/0x410 [12088.327013] do_sys_open+0x73/0xa0 [12088.328741] __x64_sys_openat+0x24/0x30 [12088.333961] do_syscall_64+0xc1/0x440 [12088.337954] entry_SYSCALL_64_after_hwframe+0x49/0xae [12088.340854] RIP: 0033:0x7fa749552332 [12088.343344] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12088.362955] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12088.369959] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12088.373124] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12088.375690] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12088.384512] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12088.397730] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12092.899074] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10175s! [12092.908168] Showing busy workqueues and worker pools: [12092.911650] workqueue events: flags=0x0 [12092.914199] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12092.917390] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12092.925757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12092.929627] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12092.944766] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12092.952882] pending: kfree_rcu_monitor [12092.955042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12092.958295] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12092.967309] workqueue events_long: flags=0x0 [12092.969407] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12092.973573] pending: lu_objects_flush [obdclass] [12092.974939] workqueue events_unbound: flags=0x2 [12092.981368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12092.985426] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12092.990308] workqueue events_power_efficient: flags=0x80 [12092.992331] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12092.995182] pending: check_lifetime [12092.998070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.006168] pending: neigh_periodic_work [12093.009055] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.014539] pending: neigh_periodic_work [12093.015793] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.021696] pending: do_cache_clean [12093.024229] workqueue mm_percpu_wq: flags=0x8 [12093.026615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.030469] pending: vmstat_update [12093.032513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.036013] pending: vmstat_update [12093.037643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.041326] pending: vmstat_update [12093.043306] workqueue writeback: flags=0x4a [12093.044790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12093.047365] pending: wb_workfn, wb_workfn [12093.050447] workqueue xprtiod: flags=0xa [12093.054992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12093.059966] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12093.067019] workqueue ipv6_addrconf: flags=0x40008 [12093.070110] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12093.074001] pending: addrconf_verify_work [12093.076452] workqueue knbd0-recv: flags=0x1a [12093.078933] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12093.085126] in-flight: 651:recv_work [12093.089293] workqueue ptlrpc_pinger: flags=0x2 [12093.092714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12093.095759] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12093.098055] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10175s workers=4 idle: 37 31846 28897 [12093.101818] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12083s workers=2 idle: 653 [12096.059282] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12096.065504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12096.097528] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12096.106236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12096.114027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12096.122977] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12096.144523] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12096.153888] RAX: 0000000000006112 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12096.159067] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12096.167009] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12096.176892] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12096.187233] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12096.191952] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12096.198397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12096.200700] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12096.207017] Call Trace: [12096.208985] [12096.209714] ? show_regs.cold.9+0x22/0x2f [12096.214855] ? watchdog_timer_fn.cold.6+0x62/0x10a [12096.220299] ? __hrtimer_run_queues+0x1c2/0x4d0 [12096.225095] ? watchdog+0x40/0x40 [12096.228291] ? hrtimer_interrupt+0x128/0x2c0 [12096.234338] ? smp_apic_timer_interrupt+0xa6/0x270 [12096.239645] ? apic_timer_interrupt+0xf/0x20 [12096.240999] [12096.243257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12096.248181] do_raw_spin_lock+0x177/0x190 [12096.253172] _raw_spin_lock+0x21/0x30 [12096.258453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12096.264312] ? do_raw_spin_unlock+0x75/0x190 [12096.268821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12096.273178] ? do_wait_intr+0xf0/0xf0 [12096.274734] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12096.278387] mdc_close+0x53b/0xd70 [mdc] [12096.281512] lmv_close+0x18d/0x490 [lmv] [12096.286397] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12096.291288] ll_md_real_close+0xa5/0x2a0 [lustre] [12096.298721] ll_md_close+0x274/0x800 [lustre] [12096.305562] ll_file_release+0x128/0xc50 [lustre] [12096.311369] __fput+0x10d/0x440 [12096.314591] ____fput+0x12/0x20 [12096.320916] task_work_run+0xc6/0x110 [12096.324248] exit_to_usermode_loop+0x1dd/0x1f0 [12096.331532] do_syscall_64+0x3d6/0x440 [12096.335871] entry_SYSCALL_64_after_hwframe+0x49/0xae [12096.343236] RIP: 0033:0x7fa749552cd8 [12096.345331] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12096.356699] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12096.363240] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12096.366168] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12096.368958] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12096.372314] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12096.376704] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12108.058040] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12108.068775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12108.100409] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12108.113974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12108.119210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12108.122434] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12108.134157] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12108.138038] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12108.142398] RDX: 000000000000645f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12108.149496] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12108.164773] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12108.166699] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12108.175572] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12108.180340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12108.184430] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12108.188274] Call Trace: [12108.189167] [12108.189783] ? show_regs.cold.9+0x22/0x2f [12108.192480] ? watchdog_timer_fn.cold.6+0x62/0x10a [12108.195404] ? __hrtimer_run_queues+0x1c2/0x4d0 [12108.196795] ? watchdog+0x40/0x40 [12108.198311] ? hrtimer_interrupt+0x128/0x2c0 [12108.203844] ? smp_apic_timer_interrupt+0xa6/0x270 [12108.206729] ? apic_timer_interrupt+0xf/0x20 [12108.210316] [12108.211800] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12108.215430] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12108.218499] do_raw_spin_lock+0x177/0x190 [12108.221112] _raw_spin_lock+0x21/0x30 [12108.223453] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12108.226606] obd_mod_free+0x46/0x140 [ptlrpc] [12108.230091] mdc_commit_open+0x91/0xb0 [mdc] [12108.232762] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12108.236783] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12108.244332] after_reply+0xe2c/0x1780 [ptlrpc] [12108.247077] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12108.248836] ? __next_timer_interrupt+0x160/0x160 [12108.250467] ptlrpcd+0xa82/0xe40 [ptlrpc] [12108.254601] ? do_wait_intr+0xf0/0xf0 [12108.257659] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12108.260916] kthread+0x1d1/0x200 [12108.262293] ? set_kthread_struct+0x70/0x70 [12108.264145] ret_from_fork+0x1f/0x30 [12112.059242] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12112.063680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12112.087819] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12112.093073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12112.096795] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12112.100738] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12112.103584] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12112.112310] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12112.115712] RAX: 0000000000007c83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12112.118958] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12112.122277] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12112.125690] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12112.129156] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12112.132583] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12112.136629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12112.141182] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12112.144398] Call Trace: [12112.145949] [12112.146947] ? show_regs.cold.9+0x22/0x2f [12112.149090] ? watchdog_timer_fn.cold.6+0x62/0x10a [12112.151684] ? __hrtimer_run_queues+0x1c2/0x4d0 [12112.154256] ? watchdog+0x40/0x40 [12112.156357] ? hrtimer_interrupt+0x128/0x2c0 [12112.158307] ? smp_apic_timer_interrupt+0xa6/0x270 [12112.160885] ? apic_timer_interrupt+0xf/0x20 [12112.162772] [12112.163875] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12112.166577] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12112.169138] do_raw_spin_lock+0x177/0x190 [12112.171338] _raw_spin_lock+0x21/0x30 [12112.172920] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12112.176184] process_one_work+0x2c8/0x700 [12112.178054] worker_thread+0x296/0x6e0 [12112.180385] ? rescuer_thread+0x570/0x570 [12112.182694] kthread+0x1d1/0x200 [12112.184339] ? set_kthread_struct+0x70/0x70 [12112.186705] ret_from_fork+0x1f/0x30 [12120.054012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12120.056143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12120.081418] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12120.086896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12120.090614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [12120.093905] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [12120.103453] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12120.112660] RAX: 0000000000000e44 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12120.120434] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12120.124757] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12120.132328] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12120.139064] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12120.144024] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12120.147212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12120.151713] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12120.155649] Call Trace: [12120.156250] [12120.159003] ? show_regs.cold.9+0x22/0x2f [12120.164605] ? watchdog_timer_fn.cold.6+0x62/0x10a [12120.166657] ? __hrtimer_run_queues+0x1c2/0x4d0 [12120.170280] ? watchdog+0x40/0x40 [12120.171817] ? hrtimer_interrupt+0x128/0x2c0 [12120.174047] ? smp_apic_timer_interrupt+0xa6/0x270 [12120.176014] ? apic_timer_interrupt+0xf/0x20 [12120.178892] [12120.179681] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12120.181433] do_raw_spin_lock+0x177/0x190 [12120.184969] _raw_spin_lock+0x21/0x30 [12120.187648] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12120.189900] ? do_raw_spin_unlock+0x75/0x190 [12120.191111] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12120.193882] ? do_wait_intr+0xf0/0xf0 [12120.197007] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12120.199643] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12120.204613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12120.207086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12120.209136] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12120.213007] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12120.215265] mdc_intent_lock+0x353/0x7e0 [mdc] [12120.219200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12120.224021] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12120.227484] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12120.230266] lmv_intent_open+0x338/0x1150 [lmv] [12120.231984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12120.236686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12120.239265] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12120.243043] ll_intent_lock+0xc5/0xaa0 [lustre] [12120.244954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12120.252876] ll_lookup_it+0xe0e/0x2270 [lustre] [12120.259328] ? do_raw_spin_unlock+0x75/0x190 [12120.263135] ? _raw_spin_unlock+0x12/0x30 [12120.264516] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12120.268320] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12120.271984] ll_atomic_open+0x36a/0xd70 [lustre] [12120.274882] lookup_open+0x356/0xb40 [12120.277900] do_last+0x313/0xfc0 [12120.279344] ? nd_jump_root+0xe5/0x160 [12120.282940] ? path_init+0x437/0x520 [12120.284685] path_openat+0xf7/0x500 [12120.287531] ? do_raw_spin_unlock+0x75/0x190 [12120.289760] do_filp_open+0x99/0x140 [12120.293231] ? getname_flags+0x6e/0x330 [12120.295370] ? __check_object_size+0xff/0x256 [12120.297260] ? do_raw_spin_unlock+0x75/0x190 [12120.299986] ? _raw_spin_unlock+0x12/0x30 [12120.302149] do_sys_openat2+0x2b4/0x410 [12120.308325] do_sys_open+0x73/0xa0 [12120.313935] __x64_sys_openat+0x24/0x30 [12120.317630] do_syscall_64+0xc1/0x440 [12120.320813] entry_SYSCALL_64_after_hwframe+0x49/0xae [12120.323795] RIP: 0033:0x7fa749552332 [12120.327320] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12120.344782] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12120.349657] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12120.355876] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12120.359788] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12120.362215] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12120.369476] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12123.105727] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10205s! [12123.111372] Showing busy workqueues and worker pools: [12123.117099] workqueue events: flags=0x0 [12123.119347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12123.124860] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12123.133507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12123.137682] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12123.141907] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.143923] pending: kfree_rcu_monitor [12123.145141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12123.159195] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12123.166822] workqueue events_long: flags=0x0 [12123.171709] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.182155] pending: lu_objects_flush [obdclass] [12123.191701] workqueue events_unbound: flags=0x2 [12123.197028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12123.201809] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12123.216126] workqueue events_power_efficient: flags=0x80 [12123.219869] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.224706] pending: check_lifetime [12123.227783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.230682] pending: neigh_periodic_work [12123.236068] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.246286] pending: neigh_periodic_work [12123.249558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.252283] pending: do_cache_clean [12123.259299] workqueue mm_percpu_wq: flags=0x8 [12123.262505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.271090] pending: vmstat_update [12123.273734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.279439] pending: vmstat_update [12123.281440] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.286744] pending: vmstat_update [12123.289486] workqueue writeback: flags=0x4a [12123.291265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12123.296614] pending: wb_workfn, wb_workfn [12123.297892] workqueue xprtiod: flags=0xa [12123.301091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12123.305468] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12123.309539] workqueue ipv6_addrconf: flags=0x40008 [12123.315246] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12123.317633] pending: addrconf_verify_work [12123.319457] workqueue knbd0-recv: flags=0x1a [12123.321779] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12123.323862] in-flight: 651:recv_work [12123.326513] workqueue ptlrpc_pinger: flags=0x2 [12123.329154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12123.332032] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12123.335714] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10206s workers=4 idle: 37 31846 28897 [12123.338979] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12113s workers=2 idle: 653 [12124.061640] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12124.073792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12124.122389] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12124.128078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12124.131070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12124.132955] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12124.140185] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12124.142093] RAX: 0000000000006e0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12124.146591] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12124.149495] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12124.151675] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12124.153944] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12124.156794] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12124.159837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12124.161839] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12124.164056] Call Trace: [12124.164655] [12124.165163] ? show_regs.cold.9+0x22/0x2f [12124.166180] ? watchdog_timer_fn.cold.6+0x62/0x10a [12124.167758] ? __hrtimer_run_queues+0x1c2/0x4d0 [12124.169631] ? watchdog+0x40/0x40 [12124.172256] ? hrtimer_interrupt+0x128/0x2c0 [12124.174192] ? smp_apic_timer_interrupt+0xa6/0x270 [12124.177488] ? apic_timer_interrupt+0xf/0x20 [12124.179746] [12124.180684] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12124.188709] do_raw_spin_lock+0x177/0x190 [12124.191368] _raw_spin_lock+0x21/0x30 [12124.195843] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12124.205580] ? do_raw_spin_unlock+0x75/0x190 [12124.208462] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12124.210617] ? do_wait_intr+0xf0/0xf0 [12124.218564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12124.223855] mdc_close+0x53b/0xd70 [mdc] [12124.228528] lmv_close+0x18d/0x490 [lmv] [12124.232560] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12124.238404] ll_md_real_close+0xa5/0x2a0 [lustre] [12124.246622] ll_md_close+0x274/0x800 [lustre] [12124.249635] ll_file_release+0x128/0xc50 [lustre] [12124.253800] __fput+0x10d/0x440 [12124.257156] ____fput+0x12/0x20 [12124.258304] task_work_run+0xc6/0x110 [12124.264752] exit_to_usermode_loop+0x1dd/0x1f0 [12124.268444] do_syscall_64+0x3d6/0x440 [12124.275450] entry_SYSCALL_64_after_hwframe+0x49/0xae [12124.281580] RIP: 0033:0x7fa749552cd8 [12124.284376] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12124.304474] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12124.312078] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12124.317615] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12124.327950] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12124.333720] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12124.342106] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12136.070545] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12136.079360] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12136.130614] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12136.136325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12136.144729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12136.149514] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12136.164393] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12136.168287] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12136.175931] RDX: 0000000000005757 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12136.181719] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12136.186149] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12136.190885] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12136.195025] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12136.202316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12136.207437] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12136.222259] Call Trace: [12136.226278] [12136.228520] ? show_regs.cold.9+0x22/0x2f [12136.231822] ? watchdog_timer_fn.cold.6+0x62/0x10a [12136.240172] ? __hrtimer_run_queues+0x1c2/0x4d0 [12136.245423] ? watchdog+0x40/0x40 [12136.248867] ? hrtimer_interrupt+0x128/0x2c0 [12136.256632] ? smp_apic_timer_interrupt+0xa6/0x270 [12136.260594] ? apic_timer_interrupt+0xf/0x20 [12136.269985] [12136.270773] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12136.279505] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12136.283630] do_raw_spin_lock+0x177/0x190 [12136.289015] _raw_spin_lock+0x21/0x30 [12136.290262] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12136.293644] obd_mod_free+0x46/0x140 [ptlrpc] [12136.296347] mdc_commit_open+0x91/0xb0 [mdc] [12136.300870] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12136.310404] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12136.315605] after_reply+0xe2c/0x1780 [ptlrpc] [12136.322371] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12136.325501] ? __next_timer_interrupt+0x160/0x160 [12136.327389] ptlrpcd+0xa82/0xe40 [ptlrpc] [12136.329514] ? do_wait_intr+0xf0/0xf0 [12136.335914] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12136.344540] kthread+0x1d1/0x200 [12136.351595] ? set_kthread_struct+0x70/0x70 [12136.357163] ret_from_fork+0x1f/0x30 [12140.057484] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12140.069370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12140.112879] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12140.125267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12140.132039] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12140.137278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12140.147433] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12140.163342] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12140.167980] RAX: 0000000000007d76 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12140.173750] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12140.183349] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12140.190407] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12140.200963] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12140.224573] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12140.248553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12140.254807] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12140.265281] Call Trace: [12140.266346] [12140.268577] ? show_regs.cold.9+0x22/0x2f [12140.271467] ? watchdog_timer_fn.cold.6+0x62/0x10a [12140.275477] ? __hrtimer_run_queues+0x1c2/0x4d0 [12140.277716] ? watchdog+0x40/0x40 [12140.278881] ? hrtimer_interrupt+0x128/0x2c0 [12140.280158] ? smp_apic_timer_interrupt+0xa6/0x270 [12140.287226] ? apic_timer_interrupt+0xf/0x20 [12140.291634] [12140.292733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12140.295918] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12140.300421] do_raw_spin_lock+0x177/0x190 [12140.304603] _raw_spin_lock+0x21/0x30 [12140.310851] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12140.315590] process_one_work+0x2c8/0x700 [12140.322451] worker_thread+0x296/0x6e0 [12140.324591] ? rescuer_thread+0x570/0x570 [12140.333475] kthread+0x1d1/0x200 [12140.335830] ? set_kthread_struct+0x70/0x70 [12140.340396] ret_from_fork+0x1f/0x30 [12148.057074] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12148.066079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12148.105487] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12148.115361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12148.128320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12148.131486] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12148.146809] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12148.150587] RAX: 0000000000006b11 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12148.153823] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12148.157281] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12148.160567] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12148.162433] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12148.164434] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12148.168038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12148.170494] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12148.175115] Call Trace: [12148.179519] [12148.180322] ? show_regs.cold.9+0x22/0x2f [12148.185632] ? watchdog_timer_fn.cold.6+0x62/0x10a [12148.189091] ? __hrtimer_run_queues+0x1c2/0x4d0 [12148.192930] ? watchdog+0x40/0x40 [12148.195410] ? hrtimer_interrupt+0x128/0x2c0 [12148.197820] ? smp_apic_timer_interrupt+0xa6/0x270 [12148.200661] ? apic_timer_interrupt+0xf/0x20 [12148.203502] [12148.204096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12148.206530] do_raw_spin_lock+0x177/0x190 [12148.208057] _raw_spin_lock+0x21/0x30 [12148.209192] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12148.211698] ? do_raw_spin_unlock+0x75/0x190 [12148.214830] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12148.216833] ? do_wait_intr+0xf0/0xf0 [12148.224785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12148.227057] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12148.230562] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12148.234562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12148.237665] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12148.242503] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12148.245327] mdc_intent_lock+0x353/0x7e0 [mdc] [12148.246755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12148.250787] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12148.256297] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12148.259037] lmv_intent_open+0x338/0x1150 [lmv] [12148.261174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12148.269090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12148.279123] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12148.283016] ll_intent_lock+0xc5/0xaa0 [lustre] [12148.286344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12148.288940] ll_lookup_it+0xe0e/0x2270 [lustre] [12148.295372] ? do_raw_spin_unlock+0x75/0x190 [12148.298574] ? _raw_spin_unlock+0x12/0x30 [12148.301187] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12148.303838] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12148.310239] ll_atomic_open+0x36a/0xd70 [lustre] [12148.315678] lookup_open+0x356/0xb40 [12148.316967] do_last+0x313/0xfc0 [12148.326815] ? nd_jump_root+0xe5/0x160 [12148.331245] ? path_init+0x437/0x520 [12148.333389] path_openat+0xf7/0x500 [12148.334971] ? do_raw_spin_unlock+0x75/0x190 [12148.337226] do_filp_open+0x99/0x140 [12148.338810] ? getname_flags+0x6e/0x330 [12148.341551] ? __check_object_size+0xff/0x256 [12148.344915] ? do_raw_spin_unlock+0x75/0x190 [12148.351361] ? _raw_spin_unlock+0x12/0x30 [12148.352859] do_sys_openat2+0x2b4/0x410 [12148.359378] do_sys_open+0x73/0xa0 [12148.361729] __x64_sys_openat+0x24/0x30 [12148.365081] do_syscall_64+0xc1/0x440 [12148.366412] entry_SYSCALL_64_after_hwframe+0x49/0xae [12148.377564] RIP: 0033:0x7fa749552332 [12148.378595] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12148.389365] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12148.397895] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12148.401362] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12148.410875] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12148.422334] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12148.428375] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12152.060888] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12152.066596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12152.094492] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12152.102593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12152.108217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12152.112877] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12152.120058] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12152.123610] RAX: 0000000000005e90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12152.128348] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12152.133891] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12152.135782] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12152.140030] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12152.143159] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12152.146934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12152.148453] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12152.153768] Call Trace: [12152.155731] [12152.157247] ? show_regs.cold.9+0x22/0x2f [12152.159547] ? watchdog_timer_fn.cold.6+0x62/0x10a [12152.162759] ? __hrtimer_run_queues+0x1c2/0x4d0 [12152.164957] ? watchdog+0x40/0x40 [12152.166501] ? hrtimer_interrupt+0x128/0x2c0 [12152.168718] ? smp_apic_timer_interrupt+0xa6/0x270 [12152.171591] ? apic_timer_interrupt+0xf/0x20 [12152.172842] [12152.173374] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12152.178705] do_raw_spin_lock+0x177/0x190 [12152.181539] _raw_spin_lock+0x21/0x30 [12152.186555] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12152.192989] ? do_raw_spin_unlock+0x75/0x190 [12152.195351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12152.197486] ? do_wait_intr+0xf0/0xf0 [12152.199343] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12152.203657] mdc_close+0x53b/0xd70 [mdc] [12152.204803] lmv_close+0x18d/0x490 [lmv] [12152.205895] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12152.213036] ll_md_real_close+0xa5/0x2a0 [lustre] [12152.216746] ll_md_close+0x274/0x800 [lustre] [12152.218547] ll_file_release+0x128/0xc50 [lustre] [12152.220688] __fput+0x10d/0x440 [12152.221975] ____fput+0x12/0x20 [12152.223717] task_work_run+0xc6/0x110 [12152.225274] exit_to_usermode_loop+0x1dd/0x1f0 [12152.227529] do_syscall_64+0x3d6/0x440 [12152.232135] entry_SYSCALL_64_after_hwframe+0x49/0xae [12152.234646] RIP: 0033:0x7fa749552cd8 [12152.236867] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12152.248319] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12152.251688] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12152.255717] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12152.261341] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12152.264067] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12152.266928] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12153.826076] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10236s! [12153.829308] Showing busy workqueues and worker pools: [12153.832028] workqueue events: flags=0x0 [12153.833881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12153.836513] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12153.842093] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12153.845014] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12153.848910] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.852096] pending: kfree_rcu_monitor [12153.854022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12153.856836] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12153.861418] workqueue events_long: flags=0x0 [12153.863367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.866478] pending: lu_objects_flush [obdclass] [12153.868860] workqueue events_unbound: flags=0x2 [12153.870530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12153.873334] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12153.881574] workqueue events_power_efficient: flags=0x80 [12153.886899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.893416] pending: check_lifetime [12153.898009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.902811] pending: neigh_periodic_work [12153.909528] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.914507] pending: neigh_periodic_work [12153.919113] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.923879] pending: do_cache_clean [12153.929883] workqueue mm_percpu_wq: flags=0x8 [12153.932805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.937411] pending: vmstat_update [12153.942134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.945618] pending: vmstat_update [12153.951403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.956874] pending: vmstat_update [12153.959376] workqueue writeback: flags=0x4a [12153.965613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12153.968973] pending: wb_workfn, wb_workfn [12153.974356] workqueue xprtiod: flags=0xa [12153.977417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12153.983763] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12153.991351] workqueue ipv6_addrconf: flags=0x40008 [12153.996582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12154.001197] pending: addrconf_verify_work [12154.007379] workqueue knbd0-recv: flags=0x1a [12154.010606] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12154.016995] in-flight: 651:recv_work [12154.023192] workqueue ptlrpc_pinger: flags=0x2 [12154.024533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12154.028937] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12154.035323] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10236s workers=4 idle: 37 31846 28897 [12154.043614] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12143s workers=2 idle: 653 [12164.058360] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12164.061733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12164.097506] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12164.110202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12164.117391] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12164.120045] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12164.133804] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12164.139748] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12164.149949] RDX: 00000000000053f7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12164.152952] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12164.157057] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12164.165526] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12164.170789] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12164.175063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12164.177611] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12164.188494] Call Trace: [12164.191510] [12164.192145] ? show_regs.cold.9+0x22/0x2f [12164.193498] ? watchdog_timer_fn.cold.6+0x62/0x10a [12164.200914] ? __hrtimer_run_queues+0x1c2/0x4d0 [12164.209724] ? watchdog+0x40/0x40 [12164.211784] ? hrtimer_interrupt+0x128/0x2c0 [12164.213981] ? smp_apic_timer_interrupt+0xa6/0x270 [12164.216156] ? apic_timer_interrupt+0xf/0x20 [12164.218798] [12164.220737] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12164.227786] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12164.231769] do_raw_spin_lock+0x177/0x190 [12164.233480] _raw_spin_lock+0x21/0x30 [12164.236929] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12164.245592] obd_mod_free+0x46/0x140 [ptlrpc] [12164.248703] mdc_commit_open+0x91/0xb0 [mdc] [12164.250184] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12164.254013] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12164.258985] after_reply+0xe2c/0x1780 [ptlrpc] [12164.261882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12164.265490] ? __next_timer_interrupt+0x160/0x160 [12164.272533] ptlrpcd+0xa82/0xe40 [ptlrpc] [12164.278906] ? do_wait_intr+0xf0/0xf0 [12164.283178] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12164.292296] kthread+0x1d1/0x200 [12164.297152] ? set_kthread_struct+0x70/0x70 [12164.304371] ret_from_fork+0x1f/0x30 [12168.057044] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12168.061080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12168.082119] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12168.087327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12168.093474] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12168.098292] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12168.100842] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12168.113955] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12168.121200] RAX: 0000000000000d08 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12168.131470] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12168.135993] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12168.139285] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12168.141864] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12168.143505] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12168.146511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12168.149236] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12168.152853] Call Trace: [12168.154176] [12168.155594] ? show_regs.cold.9+0x22/0x2f [12168.157411] ? watchdog_timer_fn.cold.6+0x62/0x10a [12168.159843] ? __hrtimer_run_queues+0x1c2/0x4d0 [12168.161886] ? watchdog+0x40/0x40 [12168.162978] ? hrtimer_interrupt+0x128/0x2c0 [12168.165286] ? smp_apic_timer_interrupt+0xa6/0x270 [12168.167456] ? apic_timer_interrupt+0xf/0x20 [12168.169702] [12168.170810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12168.173216] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12168.175512] do_raw_spin_lock+0x177/0x190 [12168.177294] _raw_spin_lock+0x21/0x30 [12168.178866] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12168.181429] process_one_work+0x2c8/0x700 [12168.183376] worker_thread+0x296/0x6e0 [12168.184981] ? rescuer_thread+0x570/0x570 [12168.186995] kthread+0x1d1/0x200 [12168.189911] ? set_kthread_struct+0x70/0x70 [12168.192022] ret_from_fork+0x1f/0x30 [12180.056247] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12180.059022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12180.065219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12180.065309] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12180.065316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12180.065318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12180.065332] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12180.065335] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12180.065339] RAX: 0000000000000d54 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12180.065342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12180.065344] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12180.065346] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12180.065349] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12180.065352] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12180.065354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12180.065357] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12180.065365] Call Trace: [12180.065371] [12180.065377] ? show_regs.cold.9+0x22/0x2f [12180.065387] ? watchdog_timer_fn.cold.6+0x62/0x10a [12180.065394] ? __hrtimer_run_queues+0x1c2/0x4d0 [12180.065399] ? watchdog+0x40/0x40 [12180.065403] ? hrtimer_interrupt+0x128/0x2c0 [12180.065408] ? smp_apic_timer_interrupt+0xa6/0x270 [12180.065416] ? apic_timer_interrupt+0xf/0x20 [12180.065421] [12180.065424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12180.065431] do_raw_spin_lock+0x177/0x190 [12180.065436] _raw_spin_lock+0x21/0x30 [12180.065442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12180.065750] ? do_raw_spin_unlock+0x75/0x190 [12180.065755] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12180.065939] ? do_wait_intr+0xf0/0xf0 [12180.065947] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12180.066162] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12180.066377] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12180.070936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12180.071015] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12180.071022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12180.071025] RIP: 0010:native_safe_halt+0x2e/0x50 [12180.071038] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12180.071042] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12180.071047] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12180.071050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12180.071052] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12180.071054] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12180.071057] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12180.071061] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12180.071064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12180.071066] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12180.071078] Call Trace: [12180.071086] [12180.071092] ? show_regs.cold.9+0x22/0x2f [12180.071104] ? watchdog_timer_fn.cold.6+0x62/0x10a [12180.071112] ? __hrtimer_run_queues+0x1c2/0x4d0 [12180.071117] ? watchdog+0x40/0x40 [12180.071122] ? hrtimer_interrupt+0x128/0x2c0 [12180.071125] ? smp_apic_timer_interrupt+0xa6/0x270 [12180.071139] ? apic_timer_interrupt+0xf/0x20 [12180.071144] [12180.071146] ? native_safe_halt+0x2e/0x50 [12180.071150] kvm_wait+0xcd/0xe0 [12180.071158] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12180.071167] do_raw_spin_lock+0x177/0x190 [12180.071171] _raw_spin_lock+0x21/0x30 [12180.071176] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12180.071766] ? do_raw_spin_unlock+0x75/0x190 [12180.071773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12180.071997] ? do_wait_intr+0xf0/0xf0 [12180.072007] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12180.072258] mdc_close+0x53b/0xd70 [mdc] [12180.072304] lmv_close+0x18d/0x490 [lmv] [12180.072419] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12180.072776] ll_md_real_close+0xa5/0x2a0 [lustre] [12180.072922] ll_md_close+0x274/0x800 [lustre] [12180.073010] ll_file_release+0x128/0xc50 [lustre] [12180.073098] __fput+0x10d/0x440 [12180.073130] ____fput+0x12/0x20 [12180.073135] task_work_run+0xc6/0x110 [12180.073142] exit_to_usermode_loop+0x1dd/0x1f0 [12180.073148] do_syscall_64+0x3d6/0x440 [12180.073153] entry_SYSCALL_64_after_hwframe+0x49/0xae [12180.073161] RIP: 0033:0x7fa749552cd8 [12180.073169] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12180.073173] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12180.073178] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12180.073180] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12180.073183] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12180.073186] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12180.073188] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12180.593362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12180.602806] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12180.607118] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12180.609268] mdc_intent_lock+0x353/0x7e0 [mdc] [12180.610955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12180.617861] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12180.633869] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12180.640422] lmv_intent_open+0x338/0x1150 [lmv] [12180.644594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12180.648519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12180.651788] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12180.654727] ll_intent_lock+0xc5/0xaa0 [lustre] [12180.661520] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12180.667837] ll_lookup_it+0xe0e/0x2270 [lustre] [12180.675828] ? do_raw_spin_unlock+0x75/0x190 [12180.677542] ? _raw_spin_unlock+0x12/0x30 [12180.679572] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12180.689512] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12180.691389] ll_atomic_open+0x36a/0xd70 [lustre] [12180.699428] lookup_open+0x356/0xb40 [12180.701503] do_last+0x313/0xfc0 [12180.708219] ? nd_jump_root+0xe5/0x160 [12180.712156] ? path_init+0x437/0x520 [12180.717216] path_openat+0xf7/0x500 [12180.722781] ? do_raw_spin_unlock+0x75/0x190 [12180.729342] do_filp_open+0x99/0x140 [12180.739064] ? getname_flags+0x6e/0x330 [12180.746079] ? __check_object_size+0xff/0x256 [12180.748930] ? do_raw_spin_unlock+0x75/0x190 [12180.752356] ? _raw_spin_unlock+0x12/0x30 [12180.756935] do_sys_openat2+0x2b4/0x410 [12180.759171] do_sys_open+0x73/0xa0 [12180.761200] __x64_sys_openat+0x24/0x30 [12180.763222] do_syscall_64+0xc1/0x440 [12180.765337] entry_SYSCALL_64_after_hwframe+0x49/0xae [12180.770529] RIP: 0033:0x7fa749552332 [12180.771871] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12180.780004] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12180.783230] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12180.786864] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12180.790268] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12180.793779] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12180.798981] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12184.551028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10267s! [12184.554614] Showing busy workqueues and worker pools: [12184.556710] workqueue events: flags=0x0 [12184.559103] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12184.563153] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12184.568961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12184.572085] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12184.576728] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.579477] pending: kfree_rcu_monitor [12184.581164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12184.585354] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12184.589348] workqueue events_long: flags=0x0 [12184.591724] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.594819] pending: lu_objects_flush [obdclass] [12184.597021] workqueue events_unbound: flags=0x2 [12184.598522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12184.601199] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12184.605564] workqueue events_power_efficient: flags=0x80 [12184.608010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.610823] pending: check_lifetime [12184.612306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.615146] pending: neigh_periodic_work [12184.616844] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.619203] pending: neigh_periodic_work [12184.621314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.624630] pending: do_cache_clean [12184.625944] workqueue mm_percpu_wq: flags=0x8 [12184.627849] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.630363] pending: vmstat_update [12184.632105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.634616] pending: vmstat_update [12184.635925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.638572] pending: vmstat_update [12184.640322] workqueue writeback: flags=0x4a [12184.642163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12184.645315] pending: wb_workfn, wb_workfn [12184.647582] workqueue xprtiod: flags=0xa [12184.652774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12184.656604] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12184.661146] workqueue ipv6_addrconf: flags=0x40008 [12184.666630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12184.673368] pending: addrconf_verify_work [12184.677486] workqueue knbd0-recv: flags=0x1a [12184.681366] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12184.683712] in-flight: 651:recv_work [12184.685423] workqueue ptlrpc_pinger: flags=0x2 [12184.687155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12184.689743] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12184.692121] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10267s workers=4 idle: 37 31846 28897 [12184.695435] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12174s workers=2 idle: 653 [12192.058451] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12192.064896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12192.085667] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12192.091240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12192.095301] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12192.097880] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12192.106652] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12192.109916] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12192.114434] RDX: 0000000000007d82 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12192.117866] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12192.121083] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12192.124641] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12192.127181] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12192.130699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12192.133326] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12192.136500] Call Trace: [12192.137579] [12192.138530] ? show_regs.cold.9+0x22/0x2f [12192.140291] ? watchdog_timer_fn.cold.6+0x62/0x10a [12192.142442] ? __hrtimer_run_queues+0x1c2/0x4d0 [12192.144856] ? watchdog+0x40/0x40 [12192.146304] ? hrtimer_interrupt+0x128/0x2c0 [12192.147909] ? smp_apic_timer_interrupt+0xa6/0x270 [12192.149957] ? apic_timer_interrupt+0xf/0x20 [12192.152317] [12192.153260] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12192.156253] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12192.158719] do_raw_spin_lock+0x177/0x190 [12192.160882] _raw_spin_lock+0x21/0x30 [12192.162638] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12192.165766] obd_mod_free+0x46/0x140 [ptlrpc] [12192.167890] mdc_commit_open+0x91/0xb0 [mdc] [12192.170183] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12192.175613] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12192.181455] after_reply+0xe2c/0x1780 [ptlrpc] [12192.183864] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12192.186681] ? __next_timer_interrupt+0x160/0x160 [12192.188553] ptlrpcd+0xa82/0xe40 [ptlrpc] [12192.190464] ? do_wait_intr+0xf0/0xf0 [12192.191911] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12192.194017] kthread+0x1d1/0x200 [12192.195084] ? set_kthread_struct+0x70/0x70 [12192.199395] ret_from_fork+0x1f/0x30 [12196.057070] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12196.060858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12196.080748] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12196.091528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12196.095453] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12196.098281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12196.102275] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12196.117439] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12196.120921] RAX: 0000000000006c17 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12196.124801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12196.129067] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12196.141402] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12196.143552] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12196.151765] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12196.155498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12196.157733] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12196.163027] Call Trace: [12196.164083] [12196.165475] ? show_regs.cold.9+0x22/0x2f [12196.169151] ? watchdog_timer_fn.cold.6+0x62/0x10a [12196.176977] ? __hrtimer_run_queues+0x1c2/0x4d0 [12196.183309] ? watchdog+0x40/0x40 [12196.191251] ? hrtimer_interrupt+0x128/0x2c0 [12196.194597] ? smp_apic_timer_interrupt+0xa6/0x270 [12196.201411] ? apic_timer_interrupt+0xf/0x20 [12196.206725] [12196.210326] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12196.214259] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12196.221736] do_raw_spin_lock+0x177/0x190 [12196.225593] _raw_spin_lock+0x21/0x30 [12196.231463] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12196.240916] process_one_work+0x2c8/0x700 [12196.253239] worker_thread+0x296/0x6e0 [12196.254456] ? rescuer_thread+0x570/0x570 [12196.262466] kthread+0x1d1/0x200 [12196.265824] ? set_kthread_struct+0x70/0x70 [12196.269497] ret_from_fork+0x1f/0x30 [12208.059075] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12208.061761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12208.082030] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12208.086014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12208.089402] RIP: 0010:native_safe_halt+0x2e/0x50 [12208.091332] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12208.098905] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12208.102064] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12208.105004] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12208.107609] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12208.109640] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12208.112996] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12208.115796] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12208.119245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12208.121938] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12208.125120] Call Trace: [12208.126385] [12208.127355] ? show_regs.cold.9+0x22/0x2f [12208.129211] ? watchdog_timer_fn.cold.6+0x62/0x10a [12208.131319] ? __hrtimer_run_queues+0x1c2/0x4d0 [12208.133884] ? watchdog+0x40/0x40 [12208.135332] ? hrtimer_interrupt+0x128/0x2c0 [12208.137302] ? smp_apic_timer_interrupt+0xa6/0x270 [12208.139556] ? apic_timer_interrupt+0xf/0x20 [12208.141497] [12208.142425] ? native_safe_halt+0x2e/0x50 [12208.143884] kvm_wait+0xcd/0xe0 [12208.144784] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12208.146416] do_raw_spin_lock+0x177/0x190 [12208.148093] _raw_spin_lock+0x21/0x30 [12208.149811] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12208.152595] ? do_raw_spin_unlock+0x75/0x190 [12208.154862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12208.157360] ? do_wait_intr+0xf0/0xf0 [12208.159165] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12208.161452] mdc_close+0x53b/0xd70 [mdc] [12208.163294] lmv_close+0x18d/0x490 [lmv] [12208.164825] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12208.166951] ll_md_real_close+0xa5/0x2a0 [lustre] [12208.169373] ll_md_close+0x274/0x800 [lustre] [12208.171898] ll_file_release+0x128/0xc50 [lustre] [12208.174149] __fput+0x10d/0x440 [12208.175453] ____fput+0x12/0x20 [12208.176938] task_work_run+0xc6/0x110 [12208.178414] exit_to_usermode_loop+0x1dd/0x1f0 [12208.180471] do_syscall_64+0x3d6/0x440 [12208.182297] entry_SYSCALL_64_after_hwframe+0x49/0xae [12208.184342] RIP: 0033:0x7fa749552cd8 [12208.185995] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12208.194529] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12208.197640] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12208.200772] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12208.204201] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12208.206733] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12208.210063] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12212.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12212.062402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12212.098320] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12212.103630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12212.108539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12212.111464] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12212.119950] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12212.122908] RAX: 0000000000002583 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12212.126438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12212.129675] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12212.132872] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12212.137073] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12212.140715] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12212.145203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12212.147927] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12212.153006] Call Trace: [12212.155227] [12212.156129] ? show_regs.cold.9+0x22/0x2f [12212.157934] ? watchdog_timer_fn.cold.6+0x62/0x10a [12212.160162] ? __hrtimer_run_queues+0x1c2/0x4d0 [12212.162480] ? watchdog+0x40/0x40 [12212.164126] ? hrtimer_interrupt+0x128/0x2c0 [12212.166021] ? smp_apic_timer_interrupt+0xa6/0x270 [12212.168123] ? apic_timer_interrupt+0xf/0x20 [12212.170121] [12212.171100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12212.173611] do_raw_spin_lock+0x177/0x190 [12212.175531] _raw_spin_lock+0x21/0x30 [12212.177077] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12212.179709] ? do_raw_spin_unlock+0x75/0x190 [12212.181569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12212.184005] ? do_wait_intr+0xf0/0xf0 [12212.185691] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12212.188152] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12212.191629] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12212.194432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12212.196906] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12212.199403] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12212.201442] mdc_intent_lock+0x353/0x7e0 [mdc] [12212.203531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12212.206019] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12212.208910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12212.211495] lmv_intent_open+0x338/0x1150 [lmv] [12212.213785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12212.216270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12212.219157] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12212.221313] ll_intent_lock+0xc5/0xaa0 [lustre] [12212.223491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12212.226514] ll_lookup_it+0xe0e/0x2270 [lustre] [12212.229070] ? do_raw_spin_unlock+0x75/0x190 [12212.230972] ? _raw_spin_unlock+0x12/0x30 [12212.233147] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12212.235194] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12212.237606] ll_atomic_open+0x36a/0xd70 [lustre] [12212.239927] lookup_open+0x356/0xb40 [12212.241593] do_last+0x313/0xfc0 [12212.243015] ? nd_jump_root+0xe5/0x160 [12212.244826] ? path_init+0x437/0x520 [12212.246809] path_openat+0xf7/0x500 [12212.248619] ? do_raw_spin_unlock+0x75/0x190 [12212.250838] do_filp_open+0x99/0x140 [12212.252510] ? getname_flags+0x6e/0x330 [12212.253789] ? __check_object_size+0xff/0x256 [12212.256254] ? do_raw_spin_unlock+0x75/0x190 [12212.258472] ? _raw_spin_unlock+0x12/0x30 [12212.260409] do_sys_openat2+0x2b4/0x410 [12212.262250] do_sys_open+0x73/0xa0 [12212.264004] __x64_sys_openat+0x24/0x30 [12212.265866] do_syscall_64+0xc1/0x440 [12212.267730] entry_SYSCALL_64_after_hwframe+0x49/0xae [12212.269817] RIP: 0033:0x7fa749552332 [12212.271293] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12212.279291] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12212.282757] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12212.285465] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12212.288334] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12212.291552] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12212.294482] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12214.760986] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10297s! [12214.770739] Showing busy workqueues and worker pools: [12214.776683] workqueue events: flags=0x0 [12214.789611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12214.799754] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12214.810245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12214.817625] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12214.840168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12214.845388] pending: kfree_rcu_monitor [12214.856994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12214.867988] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12214.875868] workqueue events_long: flags=0x0 [12214.880850] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12214.883840] pending: lu_objects_flush [obdclass] [12214.892834] workqueue events_unbound: flags=0x2 [12214.902746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12214.905017] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12214.924270] workqueue events_power_efficient: flags=0x80 [12214.927439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12214.933594] pending: check_lifetime [12214.935397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12214.950113] pending: neigh_periodic_work [12214.964161] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12214.972553] pending: neigh_periodic_work [12214.975242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12214.987409] pending: do_cache_clean [12215.002461] workqueue mm_percpu_wq: flags=0x8 [12215.007822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12215.015226] pending: vmstat_update [12215.020556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12215.028883] pending: vmstat_update [12215.032604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12215.046836] pending: vmstat_update [12215.054464] workqueue writeback: flags=0x4a [12215.056022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12215.060122] pending: wb_workfn, wb_workfn [12215.062627] workqueue xprtiod: flags=0xa [12215.067215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12215.081179] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12215.088733] workqueue ipv6_addrconf: flags=0x40008 [12215.096189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12215.106797] pending: addrconf_verify_work [12215.110815] workqueue knbd0-recv: flags=0x1a [12215.112239] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12215.119203] in-flight: 651:recv_work [12215.122461] workqueue ptlrpc_pinger: flags=0x2 [12215.124314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12215.128620] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12215.131045] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10297s workers=4 idle: 37 31846 28897 [12215.134406] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12205s workers=2 idle: 653 [12220.060729] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12220.070244] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12220.120411] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12220.145615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12220.154267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12220.172157] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12220.202915] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12220.222428] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12220.235485] RDX: 0000000000007e9b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12220.245446] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12220.258419] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12220.270112] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12220.281080] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12220.291470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12220.304147] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12220.306506] Call Trace: [12220.313138] [12220.314773] ? show_regs.cold.9+0x22/0x2f [12220.322435] ? watchdog_timer_fn.cold.6+0x62/0x10a [12220.325663] ? __hrtimer_run_queues+0x1c2/0x4d0 [12220.328413] ? watchdog+0x40/0x40 [12220.335823] ? hrtimer_interrupt+0x128/0x2c0 [12220.337323] ? smp_apic_timer_interrupt+0xa6/0x270 [12220.340811] ? apic_timer_interrupt+0xf/0x20 [12220.342540] [12220.344077] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12220.345973] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12220.355239] do_raw_spin_lock+0x177/0x190 [12220.360332] _raw_spin_lock+0x21/0x30 [12220.362936] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12220.366833] obd_mod_free+0x46/0x140 [ptlrpc] [12220.368536] mdc_commit_open+0x91/0xb0 [mdc] [12220.371562] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12220.375427] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12220.380537] after_reply+0xe2c/0x1780 [ptlrpc] [12220.385962] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12220.390993] ? __next_timer_interrupt+0x160/0x160 [12220.394223] ptlrpcd+0xa82/0xe40 [ptlrpc] [12220.397193] ? do_wait_intr+0xf0/0xf0 [12220.398568] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12220.403546] kthread+0x1d1/0x200 [12220.405633] ? set_kthread_struct+0x70/0x70 [12220.407045] ret_from_fork+0x1f/0x30 [12224.070004] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12224.076513] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12224.162054] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12224.183420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12224.189926] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12224.207371] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12224.227276] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12224.247789] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12224.272700] RAX: 0000000000006592 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12224.281481] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12224.291399] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12224.296575] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12224.306869] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12224.311198] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12224.315370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12224.318409] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12224.321115] Call Trace: [12224.323472] [12224.327270] ? show_regs.cold.9+0x22/0x2f [12224.328605] ? watchdog_timer_fn.cold.6+0x62/0x10a [12224.335382] ? __hrtimer_run_queues+0x1c2/0x4d0 [12224.341859] ? watchdog+0x40/0x40 [12224.343985] ? hrtimer_interrupt+0x128/0x2c0 [12224.346478] ? smp_apic_timer_interrupt+0xa6/0x270 [12224.349903] ? apic_timer_interrupt+0xf/0x20 [12224.351437] [12224.352208] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12224.356499] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12224.362617] do_raw_spin_lock+0x177/0x190 [12224.364966] _raw_spin_lock+0x21/0x30 [12224.367509] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12224.370317] process_one_work+0x2c8/0x700 [12224.374774] worker_thread+0x296/0x6e0 [12224.377068] ? rescuer_thread+0x570/0x570 [12224.387847] kthread+0x1d1/0x200 [12224.395410] ? set_kthread_struct+0x70/0x70 [12224.405444] ret_from_fork+0x1f/0x30 [12233.915356] rcu: INFO: rcu_sched self-detected stall on CPU [12233.920025] rcu: 2-...!: (5838124 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [12233.928818] (t=10320226 jiffies g=258701 q=4479) [12233.935912] rcu: rcu_sched kthread starved for 10317021 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [12233.950563] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12233.962026] rcu: RCU grace-period kthread stack dump: [12233.968938] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12233.985014] Call Trace: [12233.985895] __schedule+0x351/0xcb0 [12233.991120] schedule+0xc0/0x180 [12233.992481] schedule_timeout+0xb4/0x190 [12234.000434] ? do_raw_spin_unlock+0x75/0x190 [12234.002712] ? __next_timer_interrupt+0x160/0x160 [12234.005488] ? prepare_to_swait_event+0xf3/0x140 [12234.013646] rcu_gp_kthread+0x8dd/0x15b0 [12234.019973] ? force_qs_rnp+0x2c0/0x2c0 [12234.024483] kthread+0x1d1/0x200 [12234.027175] ? set_kthread_struct+0x70/0x70 [12234.032214] ret_from_fork+0x1f/0x30 [12234.038583] rcu: Stack dump where RCU GP kthread last ran: [12234.041480] NMI backtrace for cpu 2 [12234.042870] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12234.059606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12234.072232] Call Trace: [12234.075587] [12234.076776] ? dump_stack+0xbb/0x10e [12234.082724] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12234.093113] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12234.098941] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12234.105091] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12234.106335] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12234.116888] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [12234.125620] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [12234.131076] ? update_process_times+0x6d/0xc0 [12234.139121] ? tick_sched_handle.isra.18+0x27/0x90 [12234.143814] ? tick_sched_timer+0x53/0xb0 [12234.149561] ? __hrtimer_run_queues+0x1c2/0x4d0 [12234.158510] ? tick_sched_handle.isra.18+0x90/0x90 [12234.170076] ? hrtimer_interrupt+0x128/0x2c0 [12234.175116] ? smp_apic_timer_interrupt+0xa6/0x270 [12234.178409] ? apic_timer_interrupt+0xf/0x20 [12234.179891] [12234.180577] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12234.200103] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12234.204850] ? do_raw_spin_lock+0x177/0x190 [12234.210806] ? _raw_spin_lock+0x21/0x30 [12234.212216] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12234.214201] ? obd_mod_free+0x46/0x140 [ptlrpc] [12234.221374] ? mdc_commit_open+0x91/0xb0 [mdc] [12234.222954] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12234.228247] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12234.231975] ? after_reply+0xe2c/0x1780 [ptlrpc] [12234.233734] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12234.236580] ? __next_timer_interrupt+0x160/0x160 [12234.238587] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12234.240718] ? do_wait_intr+0xf0/0xf0 [12234.247436] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12234.250437] ? kthread+0x1d1/0x200 [12234.259023] ? set_kthread_struct+0x70/0x70 [12234.263797] ? ret_from_fork+0x1f/0x30 [12234.266657] NMI backtrace for cpu 2 [12234.269135] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12234.276683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12234.285562] Call Trace: [12234.296345] [12234.298826] ? dump_stack+0xbb/0x10e [12234.308635] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12234.313501] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12234.322005] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12234.326259] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12234.329053] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12234.337908] ? rcu_dump_cpu_stacks+0x11b/0x171 [12234.345700] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12234.353795] ? update_process_times+0x6d/0xc0 [12234.366865] ? tick_sched_handle.isra.18+0x27/0x90 [12234.374559] ? tick_sched_timer+0x53/0xb0 [12234.383934] ? __hrtimer_run_queues+0x1c2/0x4d0 [12234.391287] ? tick_sched_handle.isra.18+0x90/0x90 [12234.397971] ? hrtimer_interrupt+0x128/0x2c0 [12234.401566] ? smp_apic_timer_interrupt+0xa6/0x270 [12234.408248] ? apic_timer_interrupt+0xf/0x20 [12234.409721] [12234.410197] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12234.411364] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12234.428724] ? do_raw_spin_lock+0x177/0x190 [12234.433866] ? _raw_spin_lock+0x21/0x30 [12234.436484] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12234.440762] ? obd_mod_free+0x46/0x140 [ptlrpc] [12234.457115] ? mdc_commit_open+0x91/0xb0 [mdc] [12234.463128] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12234.476891] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12234.480788] ? after_reply+0xe2c/0x1780 [ptlrpc] [12234.498148] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12234.513222] ? __next_timer_interrupt+0x160/0x160 [12234.514955] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12234.530768] ? do_wait_intr+0xf0/0xf0 [12234.532318] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12234.540728] ? kthread+0x1d1/0x200 [12234.551564] ? set_kthread_struct+0x70/0x70 [12234.559590] ? ret_from_fork+0x1f/0x30 [12236.059247] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12236.063370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12236.102662] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12236.108845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12236.113135] RIP: 0010:native_safe_halt+0x2e/0x50 [12236.116206] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12236.130833] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12236.134729] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12236.138324] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12236.142313] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12236.146025] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12236.149132] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12236.152705] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12236.156456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12236.159390] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12236.162728] Call Trace: [12236.163812] [12236.164861] ? show_regs.cold.9+0x22/0x2f [12236.166801] ? watchdog_timer_fn.cold.6+0x62/0x10a [12236.171181] ? __hrtimer_run_queues+0x1c2/0x4d0 [12236.174372] ? watchdog+0x40/0x40 [12236.176237] ? hrtimer_interrupt+0x128/0x2c0 [12236.179198] ? smp_apic_timer_interrupt+0xa6/0x270 [12236.182086] ? apic_timer_interrupt+0xf/0x20 [12236.184740] [12236.186277] ? native_safe_halt+0x2e/0x50 [12236.190995] kvm_wait+0xcd/0xe0 [12236.192893] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12236.196104] do_raw_spin_lock+0x177/0x190 [12236.198379] _raw_spin_lock+0x21/0x30 [12236.199900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12236.203136] ? do_raw_spin_unlock+0x75/0x190 [12236.205381] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12236.208416] ? do_wait_intr+0xf0/0xf0 [12236.211088] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12236.213991] mdc_close+0x53b/0xd70 [mdc] [12236.216045] lmv_close+0x18d/0x490 [lmv] [12236.218131] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12236.221934] ll_md_real_close+0xa5/0x2a0 [lustre] [12236.224210] ll_md_close+0x274/0x800 [lustre] [12236.226957] ll_file_release+0x128/0xc50 [lustre] [12236.229000] __fput+0x10d/0x440 [12236.231133] ____fput+0x12/0x20 [12236.234453] task_work_run+0xc6/0x110 [12236.237078] exit_to_usermode_loop+0x1dd/0x1f0 [12236.239539] do_syscall_64+0x3d6/0x440 [12236.241870] entry_SYSCALL_64_after_hwframe+0x49/0xae [12236.245077] RIP: 0033:0x7fa749552cd8 [12236.246867] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12236.256912] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12236.263255] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12236.269019] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12236.273380] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12236.276953] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12236.282813] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12240.060000] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12240.067990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12240.138827] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12240.156121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12240.169997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12240.178117] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12240.214828] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12240.231959] RAX: 000000000000681e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12240.252536] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12240.267952] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12240.271116] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12240.281528] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12240.292824] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12240.311296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12240.317907] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12240.338343] Call Trace: [12240.349056] [12240.355311] ? show_regs.cold.9+0x22/0x2f [12240.364200] ? watchdog_timer_fn.cold.6+0x62/0x10a [12240.376974] ? __hrtimer_run_queues+0x1c2/0x4d0 [12240.382193] ? watchdog+0x40/0x40 [12240.388405] ? hrtimer_interrupt+0x128/0x2c0 [12240.394005] ? smp_apic_timer_interrupt+0xa6/0x270 [12240.401109] ? apic_timer_interrupt+0xf/0x20 [12240.403031] [12240.404492] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12240.412399] do_raw_spin_lock+0x177/0x190 [12240.418448] _raw_spin_lock+0x21/0x30 [12240.422020] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12240.439789] ? do_raw_spin_unlock+0x75/0x190 [12240.457486] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12240.473348] ? do_wait_intr+0xf0/0xf0 [12240.484009] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12240.499001] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12240.510200] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12240.521097] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12240.538309] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12240.551952] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12240.563214] mdc_intent_lock+0x353/0x7e0 [mdc] [12240.576777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12240.589776] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12240.603600] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12240.605293] lmv_intent_open+0x338/0x1150 [lmv] [12240.617186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12240.636160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12240.649072] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12240.650240] ll_intent_lock+0xc5/0xaa0 [lustre] [12240.665608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12240.684027] ll_lookup_it+0xe0e/0x2270 [lustre] [12240.685680] ? do_raw_spin_unlock+0x75/0x190 [12240.700368] ? _raw_spin_unlock+0x12/0x30 [12240.701758] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12240.721309] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12240.736852] ll_atomic_open+0x36a/0xd70 [lustre] [12240.752033] lookup_open+0x356/0xb40 [12240.761231] do_last+0x313/0xfc0 [12240.762304] ? nd_jump_root+0xe5/0x160 [12240.778041] ? path_init+0x437/0x520 [12240.779374] path_openat+0xf7/0x500 [12240.780597] ? do_raw_spin_unlock+0x75/0x190 [12240.802295] do_filp_open+0x99/0x140 [12240.804101] ? getname_flags+0x6e/0x330 [12240.817980] ? __check_object_size+0xff/0x256 [12240.831147] ? do_raw_spin_unlock+0x75/0x190 [12240.838699] ? _raw_spin_unlock+0x12/0x30 [12240.849906] do_sys_openat2+0x2b4/0x410 [12240.856415] do_sys_open+0x73/0xa0 [12240.861313] __x64_sys_openat+0x24/0x30 [12240.869270] do_syscall_64+0xc1/0x440 [12240.870537] entry_SYSCALL_64_after_hwframe+0x49/0xae [12240.877114] RIP: 0033:0x7fa749552332 [12240.879876] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12240.907769] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12240.929087] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12240.937452] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12240.960555] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12240.972517] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12240.981122] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12245.472101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10328s! [12245.475659] Showing busy workqueues and worker pools: [12245.478723] workqueue events: flags=0x0 [12245.482402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12245.485610] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12245.497726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12245.500912] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12245.510626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.513490] pending: kfree_rcu_monitor [12245.516280] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12245.520526] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12245.524376] workqueue events_long: flags=0x0 [12245.525912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.528491] pending: lu_objects_flush [obdclass] [12245.530773] workqueue events_unbound: flags=0x2 [12245.532847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12245.534739] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12245.538975] workqueue events_power_efficient: flags=0x80 [12245.540960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.544124] pending: check_lifetime [12245.545484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.547830] pending: neigh_periodic_work [12245.549469] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.551621] pending: neigh_periodic_work [12245.552779] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.555206] pending: do_cache_clean [12245.556616] workqueue mm_percpu_wq: flags=0x8 [12245.568209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.573919] pending: vmstat_update [12245.582121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.587884] pending: vmstat_update [12245.590672] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12245.598593] pending: vmstat_update [12245.606612] workqueue writeback: flags=0x4a [12245.610809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12245.613873] pending: wb_workfn, wb_workfn [12245.616306] workqueue xprtiod: flags=0xa [12245.617869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12245.622903] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12245.646794] workqueue ipv6_addrconf: flags=0x40008 [12245.653063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12245.657704] pending: addrconf_verify_work [12245.663878] workqueue knbd0-recv: flags=0x1a [12245.666229] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12245.669926] in-flight: 651:recv_work [12245.671772] workqueue ptlrpc_pinger: flags=0x2 [12245.673573] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12245.675372] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12245.678295] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10328s workers=4 idle: 37 31846 28897 [12245.682227] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12235s workers=2 idle: 653 [12252.057078] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12252.065440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12252.103593] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12252.109553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12252.116421] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12252.122970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12252.132992] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12252.145182] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12252.151243] RAX: 0000000000005307 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12252.155160] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12252.158589] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12252.161695] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12252.164824] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12252.167789] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12252.170933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12252.173257] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12252.177236] Call Trace: [12252.178933] [12252.179803] ? show_regs.cold.9+0x22/0x2f [12252.184842] ? watchdog_timer_fn.cold.6+0x62/0x10a [12252.194123] ? __hrtimer_run_queues+0x1c2/0x4d0 [12252.202658] ? watchdog+0x40/0x40 [12252.204028] ? hrtimer_interrupt+0x128/0x2c0 [12252.209051] ? smp_apic_timer_interrupt+0xa6/0x270 [12252.217360] ? apic_timer_interrupt+0xf/0x20 [12252.218774] [12252.221476] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12252.225267] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12252.227879] do_raw_spin_lock+0x177/0x190 [12252.229759] _raw_spin_lock+0x21/0x30 [12252.231582] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12252.234418] process_one_work+0x2c8/0x700 [12252.236261] worker_thread+0x296/0x6e0 [12252.238330] ? rescuer_thread+0x570/0x570 [12252.240008] kthread+0x1d1/0x200 [12252.241445] ? set_kthread_struct+0x70/0x70 [12252.243558] ret_from_fork+0x1f/0x30 [12260.058199] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12260.063512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12260.128788] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12260.141126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12260.153081] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12260.156529] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12260.176711] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12260.191161] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12260.196375] RDX: 00000000000045e3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12260.205204] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12260.210794] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12260.217798] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12260.230704] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12260.238986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12260.246994] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12260.250987] Call Trace: [12260.252282] [12260.257870] ? show_regs.cold.9+0x22/0x2f [12260.262815] ? watchdog_timer_fn.cold.6+0x62/0x10a [12260.265057] ? __hrtimer_run_queues+0x1c2/0x4d0 [12260.267233] ? watchdog+0x40/0x40 [12260.269606] ? hrtimer_interrupt+0x128/0x2c0 [12260.271620] ? smp_apic_timer_interrupt+0xa6/0x270 [12260.273865] ? apic_timer_interrupt+0xf/0x20 [12260.276110] [12260.277206] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12260.280031] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12260.282709] do_raw_spin_lock+0x177/0x190 [12260.285909] _raw_spin_lock+0x21/0x30 [12260.289835] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12260.294756] obd_mod_free+0x46/0x140 [ptlrpc] [12260.302923] mdc_commit_open+0x91/0xb0 [mdc] [12260.305084] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12260.311162] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12260.318281] after_reply+0xe2c/0x1780 [ptlrpc] [12260.321730] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12260.325113] ? __next_timer_interrupt+0x160/0x160 [12260.332147] ptlrpcd+0xa82/0xe40 [ptlrpc] [12260.337822] ? do_wait_intr+0xf0/0xf0 [12260.341097] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12260.347136] kthread+0x1d1/0x200 [12260.348712] ? set_kthread_struct+0x70/0x70 [12260.351824] ret_from_fork+0x1f/0x30 [12264.060502] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12264.065386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12264.090016] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12264.096089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12264.099692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12264.102364] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12264.110413] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12264.115914] RAX: 0000000000002dfd RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12264.118971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12264.121690] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12264.124699] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12264.128320] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12264.131426] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12264.135293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12264.137938] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12264.141145] Call Trace: [12264.142046] [12264.142798] ? show_regs.cold.9+0x22/0x2f [12264.144396] ? watchdog_timer_fn.cold.6+0x62/0x10a [12264.146070] ? __hrtimer_run_queues+0x1c2/0x4d0 [12264.148048] ? watchdog+0x40/0x40 [12264.149324] ? hrtimer_interrupt+0x128/0x2c0 [12264.150645] ? smp_apic_timer_interrupt+0xa6/0x270 [12264.153057] ? apic_timer_interrupt+0xf/0x20 [12264.155120] [12264.155970] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12264.158843] do_raw_spin_lock+0x177/0x190 [12264.160567] _raw_spin_lock+0x21/0x30 [12264.162313] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12264.165054] ? do_raw_spin_unlock+0x75/0x190 [12264.166417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12264.168951] ? do_wait_intr+0xf0/0xf0 [12264.170502] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12264.173212] mdc_close+0x53b/0xd70 [mdc] [12264.174904] lmv_close+0x18d/0x490 [lmv] [12264.176370] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12264.178996] ll_md_real_close+0xa5/0x2a0 [lustre] [12264.180724] ll_md_close+0x274/0x800 [lustre] [12264.182395] ll_file_release+0x128/0xc50 [lustre] [12264.183797] __fput+0x10d/0x440 [12264.185250] ____fput+0x12/0x20 [12264.186452] task_work_run+0xc6/0x110 [12264.188418] exit_to_usermode_loop+0x1dd/0x1f0 [12264.189526] do_syscall_64+0x3d6/0x440 [12264.190640] entry_SYSCALL_64_after_hwframe+0x49/0xae [12264.192879] RIP: 0033:0x7fa749552cd8 [12264.194479] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12264.201776] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12264.205523] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12264.208863] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12264.211956] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12264.215328] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12264.217540] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12272.054429] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [12272.067621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12272.100165] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12272.114161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12272.122071] RIP: 0010:native_safe_halt+0x2e/0x50 [12272.127054] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12272.138869] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12272.144052] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12272.151140] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [12272.162883] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12272.170050] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12272.180579] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12272.191375] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12272.201847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12272.210024] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12272.216754] Call Trace: [12272.217818] [12272.219854] ? show_regs.cold.9+0x22/0x2f [12272.223475] ? watchdog_timer_fn.cold.6+0x62/0x10a [12272.225206] ? __hrtimer_run_queues+0x1c2/0x4d0 [12272.227530] ? watchdog+0x40/0x40 [12272.229009] ? hrtimer_interrupt+0x128/0x2c0 [12272.230779] ? smp_apic_timer_interrupt+0xa6/0x270 [12272.233088] ? apic_timer_interrupt+0xf/0x20 [12272.242092] [12272.245559] ? native_safe_halt+0x2e/0x50 [12272.246549] kvm_wait+0xcd/0xe0 [12272.248163] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12272.255502] do_raw_spin_lock+0x177/0x190 [12272.257501] _raw_spin_lock+0x21/0x30 [12272.261598] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12272.268904] ? do_raw_spin_unlock+0x75/0x190 [12272.272428] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12272.280601] ? do_wait_intr+0xf0/0xf0 [12272.283921] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12272.287022] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12272.291076] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12272.297000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12272.301534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12272.307246] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12272.311942] mdc_intent_lock+0x353/0x7e0 [mdc] [12272.316025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12272.320681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12272.324119] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12272.329354] lmv_intent_open+0x338/0x1150 [lmv] [12272.330532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12272.335480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12272.338384] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12272.340277] ll_intent_lock+0xc5/0xaa0 [lustre] [12272.345433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12272.348404] ll_lookup_it+0xe0e/0x2270 [lustre] [12272.353123] ? do_raw_spin_unlock+0x75/0x190 [12272.356687] ? _raw_spin_unlock+0x12/0x30 [12272.360230] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12272.362636] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12272.365008] ll_atomic_open+0x36a/0xd70 [lustre] [12272.367648] lookup_open+0x356/0xb40 [12272.371202] do_last+0x313/0xfc0 [12272.375505] ? nd_jump_root+0xe5/0x160 [12272.381804] ? path_init+0x437/0x520 [12272.392084] path_openat+0xf7/0x500 [12272.396660] ? do_raw_spin_unlock+0x75/0x190 [12272.400605] do_filp_open+0x99/0x140 [12272.405856] ? getname_flags+0x6e/0x330 [12272.411606] ? __check_object_size+0xff/0x256 [12272.417527] ? do_raw_spin_unlock+0x75/0x190 [12272.422401] ? _raw_spin_unlock+0x12/0x30 [12272.432500] do_sys_openat2+0x2b4/0x410 [12272.438719] do_sys_open+0x73/0xa0 [12272.444478] __x64_sys_openat+0x24/0x30 [12272.448063] do_syscall_64+0xc1/0x440 [12272.450505] entry_SYSCALL_64_after_hwframe+0x49/0xae [12272.460092] RIP: 0033:0x7fa749552332 [12272.470078] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12272.491203] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12272.493886] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12272.509140] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12272.515681] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12272.521971] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12272.529219] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12276.195978] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10359s! [12276.206093] Showing busy workqueues and worker pools: [12276.208029] workqueue events: flags=0x0 [12276.211143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12276.216184] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12276.222947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12276.228005] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12276.236536] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.243141] pending: kfree_rcu_monitor [12276.245872] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12276.250235] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12276.259966] workqueue events_long: flags=0x0 [12276.261354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.270580] pending: lu_objects_flush [obdclass] [12276.276849] workqueue events_unbound: flags=0x2 [12276.280184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12276.284161] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12276.293569] workqueue events_power_efficient: flags=0x80 [12276.297486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.301779] pending: check_lifetime [12276.308229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.323596] pending: neigh_periodic_work [12276.325226] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.336542] pending: neigh_periodic_work [12276.339341] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.344287] pending: do_cache_clean [12276.345637] workqueue mm_percpu_wq: flags=0x8 [12276.350162] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.354611] pending: vmstat_update [12276.358449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.367181] pending: vmstat_update [12276.370177] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.374363] pending: vmstat_update [12276.375667] workqueue writeback: flags=0x4a [12276.387656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12276.391376] pending: wb_workfn, wb_workfn [12276.393302] workqueue xprtiod: flags=0xa [12276.395814] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12276.399267] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12276.402572] workqueue ipv6_addrconf: flags=0x40008 [12276.404544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12276.411532] pending: addrconf_verify_work [12276.413479] workqueue knbd0-recv: flags=0x1a [12276.417105] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12276.422426] in-flight: 651:recv_work [12276.424856] workqueue ptlrpc_pinger: flags=0x2 [12276.427308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12276.430562] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12276.434441] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10359s workers=4 idle: 37 31846 28897 [12276.438558] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12266s workers=2 idle: 653 [12280.057091] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12280.060788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12280.087344] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12280.093120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12280.097546] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12280.101635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12280.104815] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12280.114600] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12280.118489] RAX: 0000000000005579 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12280.121755] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12280.125281] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12280.128750] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12280.132267] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12280.134242] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12280.138438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12280.141518] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12280.144492] Call Trace: [12280.145792] [12280.146951] ? show_regs.cold.9+0x22/0x2f [12280.148725] ? watchdog_timer_fn.cold.6+0x62/0x10a [12280.150781] ? __hrtimer_run_queues+0x1c2/0x4d0 [12280.153433] ? watchdog+0x40/0x40 [12280.155023] ? hrtimer_interrupt+0x128/0x2c0 [12280.157755] ? smp_apic_timer_interrupt+0xa6/0x270 [12280.159940] ? apic_timer_interrupt+0xf/0x20 [12280.162117] [12280.163351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12280.166084] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12280.168736] do_raw_spin_lock+0x177/0x190 [12280.170731] _raw_spin_lock+0x21/0x30 [12280.172830] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12280.175542] process_one_work+0x2c8/0x700 [12280.177835] worker_thread+0x296/0x6e0 [12280.179433] ? rescuer_thread+0x570/0x570 [12280.181669] kthread+0x1d1/0x200 [12280.183211] ? set_kthread_struct+0x70/0x70 [12280.185794] ret_from_fork+0x1f/0x30 [12288.058377] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12288.060746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12288.078909] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12288.083139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12288.086303] RIP: 0010:native_safe_halt+0x2e/0x50 [12288.088103] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12288.094502] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12288.097239] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12288.099657] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12288.101935] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12288.104131] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12288.106581] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12288.108624] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12288.111241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12288.113172] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12288.115912] Call Trace: [12288.116879] [12288.117648] ? show_regs.cold.9+0x22/0x2f [12288.118935] ? watchdog_timer_fn.cold.6+0x62/0x10a [12288.120565] ? __hrtimer_run_queues+0x1c2/0x4d0 [12288.122202] ? watchdog+0x40/0x40 [12288.123374] ? hrtimer_interrupt+0x128/0x2c0 [12288.125207] ? smp_apic_timer_interrupt+0xa6/0x270 [12288.127186] ? apic_timer_interrupt+0xf/0x20 [12288.128613] [12288.139128] ? native_safe_halt+0x2e/0x50 [12288.142208] kvm_wait+0xcd/0xe0 [12288.146633] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12288.151339] do_raw_spin_lock+0x177/0x190 [12288.155338] _raw_spin_lock+0x21/0x30 [12288.159347] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12288.165081] obd_mod_free+0x46/0x140 [ptlrpc] [12288.171195] mdc_commit_open+0x91/0xb0 [mdc] [12288.172387] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12288.176361] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12288.182499] after_reply+0xe2c/0x1780 [ptlrpc] [12288.187949] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12288.192350] ? __next_timer_interrupt+0x160/0x160 [12288.197956] ptlrpcd+0xa82/0xe40 [ptlrpc] [12288.202504] ? do_wait_intr+0xf0/0xf0 [12288.204094] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12288.210328] kthread+0x1d1/0x200 [12288.211356] ? set_kthread_struct+0x70/0x70 [12288.212520] ret_from_fork+0x1f/0x30 [12292.059460] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12292.062507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12292.088685] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12292.095479] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12292.099275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12292.102175] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12292.111139] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12292.114492] RAX: 0000000000007066 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12292.117580] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12292.121025] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12292.124337] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12292.127662] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12292.131319] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12292.134904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12292.137459] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12292.141026] Call Trace: [12292.142166] [12292.143200] ? show_regs.cold.9+0x22/0x2f [12292.146985] ? watchdog_timer_fn.cold.6+0x62/0x10a [12292.149808] ? __hrtimer_run_queues+0x1c2/0x4d0 [12292.151838] ? watchdog+0x40/0x40 [12292.153617] ? hrtimer_interrupt+0x128/0x2c0 [12292.155856] ? smp_apic_timer_interrupt+0xa6/0x270 [12292.158805] ? apic_timer_interrupt+0xf/0x20 [12292.161506] [12292.163457] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12292.167204] do_raw_spin_lock+0x177/0x190 [12292.170653] _raw_spin_lock+0x21/0x30 [12292.173764] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12292.176797] ? do_raw_spin_unlock+0x75/0x190 [12292.179176] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12292.181517] ? do_wait_intr+0xf0/0xf0 [12292.183482] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12292.186297] mdc_close+0x53b/0xd70 [mdc] [12292.189376] lmv_close+0x18d/0x490 [lmv] [12292.191191] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12292.194694] ll_md_real_close+0xa5/0x2a0 [lustre] [12292.197434] ll_md_close+0x274/0x800 [lustre] [12292.199530] ll_file_release+0x128/0xc50 [lustre] [12292.202656] __fput+0x10d/0x440 [12292.204318] ____fput+0x12/0x20 [12292.205946] task_work_run+0xc6/0x110 [12292.208574] exit_to_usermode_loop+0x1dd/0x1f0 [12292.210972] do_syscall_64+0x3d6/0x440 [12292.212919] entry_SYSCALL_64_after_hwframe+0x49/0xae [12292.215941] RIP: 0033:0x7fa749552cd8 [12292.217741] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12292.226639] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12292.230474] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12292.234227] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12292.237712] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12292.241134] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12292.243527] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12304.060346] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [12304.071945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12304.129911] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12304.145942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12304.154101] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12304.163884] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12304.202046] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12304.210544] RAX: 0000000000007837 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12304.221692] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12304.233676] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12304.238510] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12304.243290] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12304.247860] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12304.252133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12304.255267] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12304.259543] Call Trace: [12304.261087] [12304.262375] ? show_regs.cold.9+0x22/0x2f [12304.264562] ? watchdog_timer_fn.cold.6+0x62/0x10a [12304.267203] ? __hrtimer_run_queues+0x1c2/0x4d0 [12304.269620] ? watchdog+0x40/0x40 [12304.271358] ? hrtimer_interrupt+0x128/0x2c0 [12304.274496] ? smp_apic_timer_interrupt+0xa6/0x270 [12304.282908] ? apic_timer_interrupt+0xf/0x20 [12304.284524] [12304.285306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12304.288051] do_raw_spin_lock+0x177/0x190 [12304.289651] _raw_spin_lock+0x21/0x30 [12304.291192] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12304.293527] ? do_raw_spin_unlock+0x75/0x190 [12304.295274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12304.297494] ? do_wait_intr+0xf0/0xf0 [12304.299044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12304.301478] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12304.303729] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12304.307754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.318539] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12304.321489] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12304.325443] mdc_intent_lock+0x353/0x7e0 [mdc] [12304.328111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.330610] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12304.333891] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12304.341354] lmv_intent_open+0x338/0x1150 [lmv] [12304.344457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.350223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.352357] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12304.353958] ll_intent_lock+0xc5/0xaa0 [lustre] [12304.358581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.361475] ll_lookup_it+0xe0e/0x2270 [lustre] [12304.364706] ? do_raw_spin_unlock+0x75/0x190 [12304.371631] ? _raw_spin_unlock+0x12/0x30 [12304.373690] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12304.377201] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12304.380592] ll_atomic_open+0x36a/0xd70 [lustre] [12304.386181] lookup_open+0x356/0xb40 [12304.388743] do_last+0x313/0xfc0 [12304.390623] ? nd_jump_root+0xe5/0x160 [12304.400150] ? path_init+0x437/0x520 [12304.401639] path_openat+0xf7/0x500 [12304.403388] ? do_raw_spin_unlock+0x75/0x190 [12304.407012] do_filp_open+0x99/0x140 [12304.410699] ? getname_flags+0x6e/0x330 [12304.419094] ? __check_object_size+0xff/0x256 [12304.420746] ? do_raw_spin_unlock+0x75/0x190 [12304.442003] ? _raw_spin_unlock+0x12/0x30 [12304.443539] do_sys_openat2+0x2b4/0x410 [12304.449265] do_sys_open+0x73/0xa0 [12304.450522] __x64_sys_openat+0x24/0x30 [12304.451938] do_syscall_64+0xc1/0x440 [12304.456896] entry_SYSCALL_64_after_hwframe+0x49/0xae [12304.461311] RIP: 0033:0x7fa749552332 [12304.465568] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12304.491714] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12304.498689] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12304.510383] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12304.522362] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12304.524867] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12304.536779] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12306.917027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10389s! [12306.926059] Showing busy workqueues and worker pools: [12306.927742] workqueue events: flags=0x0 [12306.928965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12306.939439] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12306.945226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12306.956767] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12306.976407] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12306.984410] pending: kfree_rcu_monitor [12306.989490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12306.992713] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12307.003625] workqueue events_long: flags=0x0 [12307.006951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12307.013321] pending: lu_objects_flush [obdclass] [12307.018526] workqueue events_unbound: flags=0x2 [12307.022593] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12307.032518] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12307.046014] workqueue events_power_efficient: flags=0x80 [12307.051338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12307.056590] pending: check_lifetime [12307.059141] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12307.067572] pending: neigh_periodic_work [12307.069097] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12307.076289] pending: neigh_periodic_work [12307.079678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12307.083593] pending: do_cache_clean [12307.085815] workqueue mm_percpu_wq: flags=0x8 [12307.088540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12307.091411] pending: vmstat_update [12307.092609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12307.095391] pending: vmstat_update [12307.097364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12307.099482] pending: vmstat_update [12307.100940] workqueue writeback: flags=0x4a [12307.102344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12307.105163] pending: wb_workfn, wb_workfn [12307.107791] workqueue xprtiod: flags=0xa [12307.109606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12307.112226] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12307.116835] workqueue ipv6_addrconf: flags=0x40008 [12307.125903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12307.130496] pending: addrconf_verify_work [12307.141907] workqueue knbd0-recv: flags=0x1a [12307.144183] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12307.151766] in-flight: 651:recv_work [12307.153039] workqueue ptlrpc_pinger: flags=0x2 [12307.164190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12307.172778] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12307.180054] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10390s workers=4 idle: 37 31846 28897 [12307.183312] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12297s workers=2 idle: 653 [12308.057045] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12308.064478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12308.121027] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12308.134142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12308.139536] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12308.144581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12308.150986] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12308.161346] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12308.175819] RAX: 0000000000007e19 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12308.177584] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12308.187296] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12308.194529] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12308.200314] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12308.204256] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12308.213702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12308.217363] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12308.222653] Call Trace: [12308.223979] [12308.224851] ? show_regs.cold.9+0x22/0x2f [12308.226953] ? watchdog_timer_fn.cold.6+0x62/0x10a [12308.230194] ? __hrtimer_run_queues+0x1c2/0x4d0 [12308.233768] ? watchdog+0x40/0x40 [12308.239426] ? hrtimer_interrupt+0x128/0x2c0 [12308.244412] ? smp_apic_timer_interrupt+0xa6/0x270 [12308.251713] ? apic_timer_interrupt+0xf/0x20 [12308.256820] [12308.257716] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12308.262865] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12308.271469] do_raw_spin_lock+0x177/0x190 [12308.272882] _raw_spin_lock+0x21/0x30 [12308.278118] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12308.281539] process_one_work+0x2c8/0x700 [12308.286643] worker_thread+0x296/0x6e0 [12308.289031] ? rescuer_thread+0x570/0x570 [12308.294520] kthread+0x1d1/0x200 [12308.297996] ? set_kthread_struct+0x70/0x70 [12308.304417] ret_from_fork+0x1f/0x30 [12316.058804] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12316.068469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12316.098300] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12316.104875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12316.109462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12316.112812] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12316.120450] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12316.123280] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12316.126840] RDX: 0000000000007ca5 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12316.137832] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12316.142552] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12316.144778] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12316.149134] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12316.153487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12316.157733] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12316.163284] Call Trace: [12316.165017] [12316.166926] ? show_regs.cold.9+0x22/0x2f [12316.169556] ? watchdog_timer_fn.cold.6+0x62/0x10a [12316.173338] ? __hrtimer_run_queues+0x1c2/0x4d0 [12316.175505] ? watchdog+0x40/0x40 [12316.176324] ? hrtimer_interrupt+0x128/0x2c0 [12316.178183] ? smp_apic_timer_interrupt+0xa6/0x270 [12316.181045] ? apic_timer_interrupt+0xf/0x20 [12316.185524] [12316.187617] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12316.191209] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12316.194233] do_raw_spin_lock+0x177/0x190 [12316.197812] _raw_spin_lock+0x21/0x30 [12316.199173] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12316.202022] obd_mod_free+0x46/0x140 [ptlrpc] [12316.207438] mdc_commit_open+0x91/0xb0 [mdc] [12316.209433] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12316.215223] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12316.217837] after_reply+0xe2c/0x1780 [ptlrpc] [12316.220215] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12316.222779] ? __next_timer_interrupt+0x160/0x160 [12316.224816] ptlrpcd+0xa82/0xe40 [ptlrpc] [12316.230632] ? do_wait_intr+0xf0/0xf0 [12316.232276] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12316.235739] kthread+0x1d1/0x200 [12316.238061] ? set_kthread_struct+0x70/0x70 [12316.240932] ret_from_fork+0x1f/0x30 [12320.059116] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12320.062356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12320.100201] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12320.105041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12320.113683] RIP: 0010:load_balance+0x725/0x1090 [12320.118224] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [12320.132919] RSP: 0018:ffffb893c0d8cde0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12320.136408] RAX: 0000000000000006 RBX: ffff8d73442000b0 RCX: ffff8d73821b3440 [12320.142237] RDX: 0000000000000008 RSI: ffff8d7344200000 RDI: 0000000000000206 [12320.148347] RBP: ffffb893c0d8ce90 R08: ffff8d734134cf80 R09: 0000000000000000 [12320.154746] R10: ffff8d7344200098 R11: 00000000000098cd R12: ffff8d7344200000 [12320.159129] R13: ffff8d7382133e08 R14: 0000000000000002 R15: ffff8d7382133380 [12320.163740] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12320.170592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12320.176060] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12320.178139] Call Trace: [12320.186449] [12320.186985] ? show_regs.cold.9+0x22/0x2f [12320.192466] ? watchdog_timer_fn.cold.6+0x62/0x10a [12320.196533] ? __hrtimer_run_queues+0x1c2/0x4d0 [12320.199761] ? watchdog+0x40/0x40 [12320.202717] ? hrtimer_interrupt+0x128/0x2c0 [12320.204183] ? smp_apic_timer_interrupt+0xa6/0x270 [12320.207167] ? apic_timer_interrupt+0xf/0x20 [12320.211902] ? apic_timer_interrupt+0xa/0x20 [12320.216955] ? load_balance+0x725/0x1090 [12320.222088] rebalance_domains+0x348/0x580 [12320.225599] ? kvm_clock_get_cycles+0x2c/0x50 [12320.230324] run_rebalance_domains+0x68/0xa0 [12320.234742] __do_softirq+0x134/0x4e8 [12320.236969] irq_exit_rcu+0x1c6/0x1e0 [12320.239128] irq_exit+0x12/0x30 [12320.240392] smp_apic_timer_interrupt+0xe0/0x270 [12320.242450] apic_timer_interrupt+0xf/0x20 [12320.246412] [12320.247606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12320.253524] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12320.264954] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12320.267914] RAX: 00000000000063b2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12320.272030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12320.278042] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12320.280726] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12320.286256] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12320.293304] do_raw_spin_lock+0x177/0x190 [12320.295600] _raw_spin_lock+0x21/0x30 [12320.299050] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12320.306792] ? do_raw_spin_unlock+0x75/0x190 [12320.308608] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12320.312029] ? do_wait_intr+0xf0/0xf0 [12320.314608] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12320.317439] mdc_close+0x53b/0xd70 [mdc] [12320.319542] lmv_close+0x18d/0x490 [lmv] [12320.321439] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12320.324054] ll_md_real_close+0xa5/0x2a0 [lustre] [12320.326212] ll_md_close+0x274/0x800 [lustre] [12320.327723] ll_file_release+0x128/0xc50 [lustre] [12320.329543] __fput+0x10d/0x440 [12320.330732] ____fput+0x12/0x20 [12320.331938] task_work_run+0xc6/0x110 [12320.333380] exit_to_usermode_loop+0x1dd/0x1f0 [12320.334825] do_syscall_64+0x3d6/0x440 [12320.336251] entry_SYSCALL_64_after_hwframe+0x49/0xae [12320.338006] RIP: 0033:0x7fa749552cd8 [12320.339052] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12320.346209] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12320.349784] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12320.355562] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12320.357762] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12320.360468] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12320.366091] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12332.054831] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [12332.059025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12332.088268] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12332.101194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12332.104789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12332.108713] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12332.117490] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12332.120898] RAX: 000000000000698d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12332.124929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12332.130747] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12332.134119] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12332.137390] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12332.140421] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12332.149724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12332.154839] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12332.158779] Call Trace: [12332.159950] [12332.161081] ? show_regs.cold.9+0x22/0x2f [12332.162927] ? watchdog_timer_fn.cold.6+0x62/0x10a [12332.165240] ? __hrtimer_run_queues+0x1c2/0x4d0 [12332.169147] ? watchdog+0x40/0x40 [12332.170939] ? hrtimer_interrupt+0x128/0x2c0 [12332.173071] ? smp_apic_timer_interrupt+0xa6/0x270 [12332.175192] ? apic_timer_interrupt+0xf/0x20 [12332.177175] [12332.178215] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12332.183252] do_raw_spin_lock+0x177/0x190 [12332.188241] _raw_spin_lock+0x21/0x30 [12332.190760] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12332.194886] ? do_raw_spin_unlock+0x75/0x190 [12332.198820] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12332.206848] ? do_wait_intr+0xf0/0xf0 [12332.211831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12332.215446] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12332.218469] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12332.223967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.227132] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12332.229884] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12332.232135] mdc_intent_lock+0x353/0x7e0 [mdc] [12332.234217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.244197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12332.250131] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12332.254809] lmv_intent_open+0x338/0x1150 [lmv] [12332.258943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.265890] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.270059] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12332.272274] ll_intent_lock+0xc5/0xaa0 [lustre] [12332.280109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.285745] ll_lookup_it+0xe0e/0x2270 [lustre] [12332.290900] ? do_raw_spin_unlock+0x75/0x190 [12332.293115] ? _raw_spin_unlock+0x12/0x30 [12332.298125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12332.303208] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12332.306814] ll_atomic_open+0x36a/0xd70 [lustre] [12332.313061] lookup_open+0x356/0xb40 [12332.314815] do_last+0x313/0xfc0 [12332.318705] ? nd_jump_root+0xe5/0x160 [12332.320225] ? path_init+0x437/0x520 [12332.326437] path_openat+0xf7/0x500 [12332.329104] ? do_raw_spin_unlock+0x75/0x190 [12332.334273] do_filp_open+0x99/0x140 [12332.339525] ? getname_flags+0x6e/0x330 [12332.347102] ? __check_object_size+0xff/0x256 [12332.355076] ? do_raw_spin_unlock+0x75/0x190 [12332.356834] ? _raw_spin_unlock+0x12/0x30 [12332.366538] do_sys_openat2+0x2b4/0x410 [12332.371969] do_sys_open+0x73/0xa0 [12332.376525] __x64_sys_openat+0x24/0x30 [12332.381293] do_syscall_64+0xc1/0x440 [12332.383166] entry_SYSCALL_64_after_hwframe+0x49/0xae [12332.387168] RIP: 0033:0x7fa749552332 [12332.396546] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12332.418873] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12332.431958] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12332.441736] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12332.449962] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12332.462613] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12332.471766] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12336.057011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12336.060377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12336.081833] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12336.087190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12336.091393] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12336.095818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12336.099714] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12336.113193] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12336.117121] RAX: 000000000000090f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12336.121176] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12336.124803] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12336.128488] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12336.132193] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12336.135888] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12336.139472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12336.142058] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12336.145813] Call Trace: [12336.148217] [12336.149164] ? show_regs.cold.9+0x22/0x2f [12336.151499] ? watchdog_timer_fn.cold.6+0x62/0x10a [12336.154472] ? __hrtimer_run_queues+0x1c2/0x4d0 [12336.157154] ? watchdog+0x40/0x40 [12336.158803] ? hrtimer_interrupt+0x128/0x2c0 [12336.161131] ? smp_apic_timer_interrupt+0xa6/0x270 [12336.164142] ? apic_timer_interrupt+0xf/0x20 [12336.165924] [12336.166797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12336.169394] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12336.172356] do_raw_spin_lock+0x177/0x190 [12336.174586] _raw_spin_lock+0x21/0x30 [12336.176639] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12336.179522] process_one_work+0x2c8/0x700 [12336.182666] worker_thread+0x296/0x6e0 [12336.184420] ? rescuer_thread+0x570/0x570 [12336.186551] kthread+0x1d1/0x200 [12336.187992] ? set_kthread_struct+0x70/0x70 [12336.189860] ret_from_fork+0x1f/0x30 [12337.637031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10420s! [12337.641796] Showing busy workqueues and worker pools: [12337.644640] workqueue events: flags=0x0 [12337.646448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12337.651180] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12337.659846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12337.663866] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12337.669556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.671627] pending: kfree_rcu_monitor [12337.673145] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12337.676510] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12337.681478] workqueue events_long: flags=0x0 [12337.685838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.690301] pending: lu_objects_flush [obdclass] [12337.694695] workqueue events_unbound: flags=0x2 [12337.695946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12337.697977] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12337.703731] workqueue events_power_efficient: flags=0x80 [12337.705979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.713229] pending: check_lifetime [12337.717144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.720476] pending: neigh_periodic_work [12337.723832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.728068] pending: neigh_periodic_work [12337.731075] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.737201] pending: do_cache_clean [12337.739229] workqueue mm_percpu_wq: flags=0x8 [12337.743016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.746048] pending: vmstat_update [12337.749020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.753077] pending: vmstat_update [12337.755158] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.762503] pending: vmstat_update [12337.764244] workqueue writeback: flags=0x4a [12337.766669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12337.771046] pending: wb_workfn, wb_workfn [12337.775021] workqueue xprtiod: flags=0xa [12337.778164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12337.781527] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12337.794137] workqueue ipv6_addrconf: flags=0x40008 [12337.796227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12337.801864] pending: addrconf_verify_work [12337.808382] workqueue knbd0-recv: flags=0x1a [12337.811352] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12337.822211] in-flight: 651:recv_work [12337.825429] workqueue ptlrpc_pinger: flags=0x2 [12337.828165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12337.832233] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12337.836948] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10420s workers=4 idle: 37 31846 28897 [12337.841522] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12327s workers=2 idle: 653 [12344.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12344.061301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12344.081166] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12344.085979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12344.088951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12344.091496] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12344.100577] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12344.104256] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12344.107194] RDX: 00000000000001cf RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12344.109616] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12344.112188] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12344.115158] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12344.118023] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12344.121122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12344.123395] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12344.126256] Call Trace: [12344.127212] [12344.128101] ? show_regs.cold.9+0x22/0x2f [12344.129726] ? watchdog_timer_fn.cold.6+0x62/0x10a [12344.131368] ? __hrtimer_run_queues+0x1c2/0x4d0 [12344.133130] ? watchdog+0x40/0x40 [12344.134280] ? hrtimer_interrupt+0x128/0x2c0 [12344.136030] ? smp_apic_timer_interrupt+0xa6/0x270 [12344.138146] ? apic_timer_interrupt+0xf/0x20 [12344.139807] [12344.140936] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12344.142930] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12344.145065] do_raw_spin_lock+0x177/0x190 [12344.147016] _raw_spin_lock+0x21/0x30 [12344.148681] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12344.150944] obd_mod_free+0x46/0x140 [ptlrpc] [12344.152801] mdc_commit_open+0x91/0xb0 [mdc] [12344.154212] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12344.155998] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12344.157986] after_reply+0xe2c/0x1780 [ptlrpc] [12344.159799] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12344.161585] ? __next_timer_interrupt+0x160/0x160 [12344.163339] ptlrpcd+0xa82/0xe40 [ptlrpc] [12344.164891] ? do_wait_intr+0xf0/0xf0 [12344.165810] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12344.167568] kthread+0x1d1/0x200 [12344.168846] ? set_kthread_struct+0x70/0x70 [12344.170514] ret_from_fork+0x1f/0x30 [12348.061059] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12348.066091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12348.106916] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12348.115835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12348.120752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12348.128209] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12348.146378] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12348.154535] RAX: 0000000000007e0f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12348.160795] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12348.165508] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12348.170312] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12348.178413] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12348.184682] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12348.193715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12348.197107] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12348.202006] Call Trace: [12348.203007] [12348.204927] ? show_regs.cold.9+0x22/0x2f [12348.209982] ? watchdog_timer_fn.cold.6+0x62/0x10a [12348.212731] ? __hrtimer_run_queues+0x1c2/0x4d0 [12348.216053] ? watchdog+0x40/0x40 [12348.217867] ? hrtimer_interrupt+0x128/0x2c0 [12348.219201] ? smp_apic_timer_interrupt+0xa6/0x270 [12348.220856] ? apic_timer_interrupt+0xf/0x20 [12348.226838] [12348.227325] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12348.230268] do_raw_spin_lock+0x177/0x190 [12348.231808] _raw_spin_lock+0x21/0x30 [12348.233900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12348.236916] ? do_raw_spin_unlock+0x75/0x190 [12348.237956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12348.239285] ? do_wait_intr+0xf0/0xf0 [12348.246787] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12348.248050] mdc_close+0x53b/0xd70 [mdc] [12348.249068] lmv_close+0x18d/0x490 [lmv] [12348.253731] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12348.256117] ll_md_real_close+0xa5/0x2a0 [lustre] [12348.264317] ll_md_close+0x274/0x800 [lustre] [12348.275856] ll_file_release+0x128/0xc50 [lustre] [12348.277702] __fput+0x10d/0x440 [12348.286218] ____fput+0x12/0x20 [12348.288309] task_work_run+0xc6/0x110 [12348.290484] exit_to_usermode_loop+0x1dd/0x1f0 [12348.295391] do_syscall_64+0x3d6/0x440 [12348.304157] entry_SYSCALL_64_after_hwframe+0x49/0xae [12348.307252] RIP: 0033:0x7fa749552cd8 [12348.309047] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12348.318148] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12348.321587] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12348.324806] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12348.327451] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12348.330706] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12348.333851] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12364.054316] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12364.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12364.058131] Modules linked in: lustre(O) [12364.061580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12364.061638] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12364.061644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12364.061649] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12364.061884] RIP: 0010:native_safe_halt+0x2e/0x50 [12364.061892] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12364.061895] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12364.061898] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12364.061900] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12364.061902] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12364.061903] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12364.061905] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12364.061907] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12364.061909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12364.061911] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12364.061918] Call Trace: [12364.061923] [12364.061928] ? show_regs.cold.9+0x22/0x2f [12364.061936] ? watchdog_timer_fn.cold.6+0x62/0x10a [12364.061941] ? __hrtimer_run_queues+0x1c2/0x4d0 [12364.061944] ? watchdog+0x40/0x40 [12364.061947] ? hrtimer_interrupt+0x128/0x2c0 [12364.061949] ? smp_apic_timer_interrupt+0xa6/0x270 [12364.061955] ? apic_timer_interrupt+0xf/0x20 [12364.061958] [12364.061959] ? native_safe_halt+0x2e/0x50 [12364.061962] kvm_wait+0xcd/0xe0 [12364.061968] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12364.061973] do_raw_spin_lock+0x177/0x190 [12364.061976] _raw_spin_lock+0x21/0x30 [12364.061979] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12364.062217] process_one_work+0x2c8/0x700 [12364.062224] worker_thread+0x296/0x6e0 [12364.062227] ? rescuer_thread+0x570/0x570 [12364.062231] kthread+0x1d1/0x200 [12364.062235] ? set_kthread_struct+0x70/0x70 [12364.062238] ret_from_fork+0x1f/0x30 [12364.201889] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12364.232810] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12364.241016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12364.244292] RIP: 0010:native_safe_halt+0x2e/0x50 [12364.245917] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12364.268555] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12364.281010] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12364.290212] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [12364.300890] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12364.303924] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12364.308201] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12364.314879] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12364.325044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12364.327905] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12364.331008] Call Trace: [12364.331854] [12364.333008] ? show_regs.cold.9+0x22/0x2f [12364.336918] ? watchdog_timer_fn.cold.6+0x62/0x10a [12364.339295] ? __hrtimer_run_queues+0x1c2/0x4d0 [12364.342961] ? watchdog+0x40/0x40 [12364.344174] ? hrtimer_interrupt+0x128/0x2c0 [12364.346791] ? smp_apic_timer_interrupt+0xa6/0x270 [12364.354012] ? apic_timer_interrupt+0xf/0x20 [12364.358236] [12364.358901] ? native_safe_halt+0x2e/0x50 [12364.363609] kvm_wait+0xcd/0xe0 [12364.370113] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12364.375642] do_raw_spin_lock+0x177/0x190 [12364.380600] _raw_spin_lock+0x21/0x30 [12364.381826] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12364.385975] ? do_raw_spin_unlock+0x75/0x190 [12364.389369] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12364.393237] ? do_wait_intr+0xf0/0xf0 [12364.394384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12364.398534] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12364.400226] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12364.404856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12364.413408] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12364.415792] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12364.421605] mdc_intent_lock+0x353/0x7e0 [mdc] [12364.423053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12364.426831] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12364.432306] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12364.434799] lmv_intent_open+0x338/0x1150 [lmv] [12364.439327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12364.440935] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12364.449162] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12364.453676] ll_intent_lock+0xc5/0xaa0 [lustre] [12364.456301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12364.462049] ll_lookup_it+0xe0e/0x2270 [lustre] [12364.463740] ? do_raw_spin_unlock+0x75/0x190 [12364.468133] ? _raw_spin_unlock+0x12/0x30 [12364.471276] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12364.481841] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12364.491729] ll_atomic_open+0x36a/0xd70 [lustre] [12364.493882] lookup_open+0x356/0xb40 [12364.495254] do_last+0x313/0xfc0 [12364.499083] ? nd_jump_root+0xe5/0x160 [12364.500756] ? path_init+0x437/0x520 [12364.502598] path_openat+0xf7/0x500 [12364.507811] ? do_raw_spin_unlock+0x75/0x190 [12364.512488] do_filp_open+0x99/0x140 [12364.524060] ? getname_flags+0x6e/0x330 [12364.525644] ? __check_object_size+0xff/0x256 [12364.528903] ? do_raw_spin_unlock+0x75/0x190 [12364.538304] ? _raw_spin_unlock+0x12/0x30 [12364.541150] do_sys_openat2+0x2b4/0x410 [12364.544061] do_sys_open+0x73/0xa0 [12364.547943] __x64_sys_openat+0x24/0x30 [12364.558073] do_syscall_64+0xc1/0x440 [12364.559513] entry_SYSCALL_64_after_hwframe+0x49/0xae [12364.564009] RIP: 0033:0x7fa749552332 [12364.571361] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12364.582479] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12364.586713] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12364.593913] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12364.598084] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12364.610721] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12364.621665] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12368.352603] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10451s! [12368.355904] Showing busy workqueues and worker pools: [12368.358959] workqueue events: flags=0x0 [12368.363381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12368.370210] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12368.376041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12368.380499] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12368.388101] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.393764] pending: kfree_rcu_monitor [12368.395957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12368.398410] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12368.401675] workqueue events_long: flags=0x0 [12368.402889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.408538] pending: lu_objects_flush [obdclass] [12368.411761] workqueue events_unbound: flags=0x2 [12368.413330] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12368.416421] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12368.420880] workqueue events_power_efficient: flags=0x80 [12368.423126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.425813] pending: check_lifetime [12368.427334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.430133] pending: neigh_periodic_work [12368.436646] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.440118] pending: neigh_periodic_work [12368.448363] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.454074] pending: do_cache_clean [12368.457302] workqueue mm_percpu_wq: flags=0x8 [12368.461625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.465146] pending: vmstat_update [12368.466262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.470627] pending: vmstat_update [12368.472634] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12368.478059] pending: vmstat_update [12368.480072] workqueue writeback: flags=0x4a [12368.485108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12368.488601] pending: wb_workfn, wb_workfn [12368.492621] workqueue xprtiod: flags=0xa [12368.496729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12368.500660] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12368.502527] workqueue ipv6_addrconf: flags=0x40008 [12368.505603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12368.508843] pending: addrconf_verify_work [12368.516385] workqueue knbd0-recv: flags=0x1a [12368.518043] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12368.524505] in-flight: 651:recv_work [12368.527095] workqueue ptlrpc_pinger: flags=0x2 [12368.530089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12368.538427] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12368.541740] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10451s workers=4 idle: 37 31846 28897 [12368.545479] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12358s workers=2 idle: 653 [12372.058022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12372.065504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12372.104346] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12372.108880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12372.113349] RIP: 0010:native_safe_halt+0x2e/0x50 [12372.118697] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12372.128851] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12372.141370] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12372.143831] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12372.154524] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12372.158816] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12372.162721] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12372.168727] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12372.174452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12372.179623] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12372.183237] Call Trace: [12372.185717] [12372.188627] ? show_regs.cold.9+0x22/0x2f [12372.190887] ? watchdog_timer_fn.cold.6+0x62/0x10a [12372.192527] ? __hrtimer_run_queues+0x1c2/0x4d0 [12372.200505] ? watchdog+0x40/0x40 [12372.202475] ? hrtimer_interrupt+0x128/0x2c0 [12372.209790] ? smp_apic_timer_interrupt+0xa6/0x270 [12372.211604] ? apic_timer_interrupt+0xf/0x20 [12372.214590] [12372.217831] ? native_safe_halt+0x2e/0x50 [12372.221492] kvm_wait+0xcd/0xe0 [12372.224587] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12372.229058] do_raw_spin_lock+0x177/0x190 [12372.234758] _raw_spin_lock+0x21/0x30 [12372.237599] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12372.242454] obd_mod_free+0x46/0x140 [ptlrpc] [12372.245660] mdc_commit_open+0x91/0xb0 [mdc] [12372.247143] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12372.250222] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12372.255187] after_reply+0xe2c/0x1780 [ptlrpc] [12372.257416] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12372.262532] ? __next_timer_interrupt+0x160/0x160 [12372.264586] ptlrpcd+0xa82/0xe40 [ptlrpc] [12372.267418] ? do_wait_intr+0xf0/0xf0 [12372.268715] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12372.272165] kthread+0x1d1/0x200 [12372.274575] ? set_kthread_struct+0x70/0x70 [12372.276770] ret_from_fork+0x1f/0x30 [12376.059027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12376.061644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12376.087737] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12376.092927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12376.096589] RIP: 0010:native_safe_halt+0x2e/0x50 [12376.098068] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12376.108987] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12376.115436] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12376.118633] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12376.122326] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12376.124782] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12376.129554] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12376.131313] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12376.137705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12376.140028] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12376.143385] Call Trace: [12376.144531] [12376.145837] ? show_regs.cold.9+0x22/0x2f [12376.146895] ? watchdog_timer_fn.cold.6+0x62/0x10a [12376.148128] ? __hrtimer_run_queues+0x1c2/0x4d0 [12376.149809] ? watchdog+0x40/0x40 [12376.152754] ? hrtimer_interrupt+0x128/0x2c0 [12376.155032] ? smp_apic_timer_interrupt+0xa6/0x270 [12376.156234] ? apic_timer_interrupt+0xf/0x20 [12376.157828] [12376.160437] ? native_safe_halt+0x2e/0x50 [12376.162607] kvm_wait+0xcd/0xe0 [12376.164100] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12376.166350] do_raw_spin_lock+0x177/0x190 [12376.169443] _raw_spin_lock+0x21/0x30 [12376.171106] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12376.173211] ? do_raw_spin_unlock+0x75/0x190 [12376.175115] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12376.177385] ? do_wait_intr+0xf0/0xf0 [12376.178651] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12376.185134] mdc_close+0x53b/0xd70 [mdc] [12376.187540] lmv_close+0x18d/0x490 [lmv] [12376.189792] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12376.191575] ll_md_real_close+0xa5/0x2a0 [lustre] [12376.195052] ll_md_close+0x274/0x800 [lustre] [12376.196941] ll_file_release+0x128/0xc50 [lustre] [12376.198498] __fput+0x10d/0x440 [12376.203557] ____fput+0x12/0x20 [12376.204718] task_work_run+0xc6/0x110 [12376.206374] exit_to_usermode_loop+0x1dd/0x1f0 [12376.214741] do_syscall_64+0x3d6/0x440 [12376.215991] entry_SYSCALL_64_after_hwframe+0x49/0xae [12376.217269] RIP: 0033:0x7fa749552cd8 [12376.218124] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12376.233600] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12376.237065] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12376.241394] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12376.244604] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12376.249146] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12376.256529] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12392.057027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12392.060051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12392.086421] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12392.092774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12392.097834] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12392.101759] RIP: 0010:native_safe_halt+0x2e/0x50 [12392.103967] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12392.115324] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12392.118916] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12392.122734] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12392.126889] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12392.130815] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12392.135017] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12392.139274] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12392.143380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12392.146324] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12392.149337] Call Trace: [12392.150474] [12392.151347] ? show_regs.cold.9+0x22/0x2f [12392.153816] ? watchdog_timer_fn.cold.6+0x62/0x10a [12392.156386] ? __hrtimer_run_queues+0x1c2/0x4d0 [12392.158964] ? watchdog+0x40/0x40 [12392.160806] ? hrtimer_interrupt+0x128/0x2c0 [12392.163047] ? smp_apic_timer_interrupt+0xa6/0x270 [12392.165757] ? apic_timer_interrupt+0xf/0x20 [12392.167896] [12392.169000] ? native_safe_halt+0x2e/0x50 [12392.170900] kvm_wait+0xcd/0xe0 [12392.172026] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12392.175837] do_raw_spin_lock+0x177/0x190 [12392.177306] _raw_spin_lock+0x21/0x30 [12392.179233] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12392.182052] process_one_work+0x2c8/0x700 [12392.183835] worker_thread+0x296/0x6e0 [12392.185796] ? rescuer_thread+0x570/0x570 [12392.187682] kthread+0x1d1/0x200 [12392.189030] ? set_kthread_struct+0x70/0x70 [12392.190920] ret_from_fork+0x1f/0x30 [12396.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12396.056847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12396.085879] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12396.091016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12396.098478] RIP: 0010:native_safe_halt+0x2e/0x50 [12396.101364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12396.111330] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12396.116716] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12396.119959] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [12396.123247] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12396.126247] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12396.129903] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12396.133098] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12396.137402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12396.140061] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12396.143297] Call Trace: [12396.144276] [12396.145315] ? show_regs.cold.9+0x22/0x2f [12396.147997] ? watchdog_timer_fn.cold.6+0x62/0x10a [12396.150605] ? __hrtimer_run_queues+0x1c2/0x4d0 [12396.153752] ? watchdog+0x40/0x40 [12396.155282] ? hrtimer_interrupt+0x128/0x2c0 [12396.157257] ? smp_apic_timer_interrupt+0xa6/0x270 [12396.159324] ? apic_timer_interrupt+0xf/0x20 [12396.160998] [12396.162971] ? native_safe_halt+0x2e/0x50 [12396.167029] kvm_wait+0xcd/0xe0 [12396.168483] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12396.171388] do_raw_spin_lock+0x177/0x190 [12396.174107] _raw_spin_lock+0x21/0x30 [12396.176651] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12396.179549] ? do_raw_spin_unlock+0x75/0x190 [12396.181285] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12396.184594] ? do_wait_intr+0xf0/0xf0 [12396.186089] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12396.188964] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12396.192049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12396.194988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12396.198698] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12396.200886] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12396.202861] mdc_intent_lock+0x353/0x7e0 [mdc] [12396.204804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12396.207130] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12396.210600] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12396.212763] lmv_intent_open+0x338/0x1150 [lmv] [12396.214568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12396.216694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12396.218952] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12396.220599] ll_intent_lock+0xc5/0xaa0 [lustre] [12396.222290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12396.224476] ll_lookup_it+0xe0e/0x2270 [lustre] [12396.226286] ? do_raw_spin_unlock+0x75/0x190 [12396.228360] ? _raw_spin_unlock+0x12/0x30 [12396.234624] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12396.240017] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12396.244007] ll_atomic_open+0x36a/0xd70 [lustre] [12396.246046] lookup_open+0x356/0xb40 [12396.247394] do_last+0x313/0xfc0 [12396.248837] ? nd_jump_root+0xe5/0x160 [12396.250078] ? path_init+0x437/0x520 [12396.251727] path_openat+0xf7/0x500 [12396.253065] ? do_raw_spin_unlock+0x75/0x190 [12396.255105] do_filp_open+0x99/0x140 [12396.257297] ? getname_flags+0x6e/0x330 [12396.258779] ? __check_object_size+0xff/0x256 [12396.260403] ? do_raw_spin_unlock+0x75/0x190 [12396.261994] ? _raw_spin_unlock+0x12/0x30 [12396.263431] do_sys_openat2+0x2b4/0x410 [12396.264897] do_sys_open+0x73/0xa0 [12396.266246] __x64_sys_openat+0x24/0x30 [12396.267673] do_syscall_64+0xc1/0x440 [12396.269280] entry_SYSCALL_64_after_hwframe+0x49/0xae [12396.271117] RIP: 0033:0x7fa749552332 [12396.272438] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12396.279441] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12396.282267] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12396.284675] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12396.287258] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12396.289673] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12396.292281] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12398.564452] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10481s! [12398.569608] Showing busy workqueues and worker pools: [12398.573238] workqueue events: flags=0x0 [12398.575803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12398.579708] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12398.587958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12398.592177] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12398.597920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.601972] pending: kfree_rcu_monitor [12398.604446] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12398.607943] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12398.612401] workqueue events_long: flags=0x0 [12398.614901] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.618430] pending: lu_objects_flush [obdclass] [12398.621629] workqueue events_unbound: flags=0x2 [12398.624169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12398.627634] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12398.633788] workqueue events_power_efficient: flags=0x80 [12398.636819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.642173] pending: check_lifetime [12398.644630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.648819] pending: neigh_periodic_work [12398.651614] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.655846] pending: neigh_periodic_work [12398.658347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.661706] pending: do_cache_clean [12398.665183] workqueue mm_percpu_wq: flags=0x8 [12398.668422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.674389] pending: vmstat_update [12398.677901] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.684438] pending: vmstat_update [12398.690378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.700150] pending: vmstat_update [12398.705181] workqueue writeback: flags=0x4a [12398.709152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12398.713705] pending: wb_workfn, wb_workfn [12398.717929] workqueue xprtiod: flags=0xa [12398.721901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12398.726783] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12398.732863] workqueue ipv6_addrconf: flags=0x40008 [12398.737384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12398.743404] pending: addrconf_verify_work [12398.746978] workqueue knbd0-recv: flags=0x1a [12398.751267] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12398.756936] in-flight: 651:recv_work [12398.761847] workqueue ptlrpc_pinger: flags=0x2 [12398.765938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12398.774247] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12398.783761] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10481s workers=4 idle: 37 31846 28897 [12398.791961] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12388s workers=2 idle: 653 [12400.061657] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12400.069677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12400.110909] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12400.127081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12400.136563] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12400.139402] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12400.158295] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12400.163163] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12400.166540] RDX: 000000000000767f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12400.169353] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12400.171925] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12400.183477] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12400.187214] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12400.191203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12400.202937] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12400.210571] Call Trace: [12400.211858] [12400.214074] ? show_regs.cold.9+0x22/0x2f [12400.220239] ? watchdog_timer_fn.cold.6+0x62/0x10a [12400.222408] ? __hrtimer_run_queues+0x1c2/0x4d0 [12400.225832] ? watchdog+0x40/0x40 [12400.228235] ? hrtimer_interrupt+0x128/0x2c0 [12400.230672] ? smp_apic_timer_interrupt+0xa6/0x270 [12400.236390] ? apic_timer_interrupt+0xf/0x20 [12400.237954] [12400.243861] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12400.249776] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12400.256909] do_raw_spin_lock+0x177/0x190 [12400.259733] _raw_spin_lock+0x21/0x30 [12400.261889] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12400.269812] obd_mod_free+0x46/0x140 [ptlrpc] [12400.271673] mdc_commit_open+0x91/0xb0 [mdc] [12400.277265] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12400.286615] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12400.289331] after_reply+0xe2c/0x1780 [ptlrpc] [12400.291777] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12400.295782] ? __next_timer_interrupt+0x160/0x160 [12400.299423] ptlrpcd+0xa82/0xe40 [ptlrpc] [12400.302361] ? do_wait_intr+0xf0/0xf0 [12400.304387] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12400.310680] kthread+0x1d1/0x200 [12400.312247] ? set_kthread_struct+0x70/0x70 [12400.314769] ret_from_fork+0x1f/0x30 [12404.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12404.061433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12404.078981] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12404.084374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12404.087562] RIP: 0010:native_safe_halt+0x2e/0x50 [12404.088994] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12404.095817] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12404.098532] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12404.100886] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12404.103609] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12404.106374] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12404.108839] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12404.111799] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12404.114840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12404.117636] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12404.120956] Call Trace: [12404.122741] [12404.123530] ? show_regs.cold.9+0x22/0x2f [12404.125744] ? watchdog_timer_fn.cold.6+0x62/0x10a [12404.128288] ? __hrtimer_run_queues+0x1c2/0x4d0 [12404.130227] ? watchdog+0x40/0x40 [12404.131452] ? hrtimer_interrupt+0x128/0x2c0 [12404.133007] ? smp_apic_timer_interrupt+0xa6/0x270 [12404.134920] ? apic_timer_interrupt+0xf/0x20 [12404.137088] [12404.137855] ? native_safe_halt+0x2e/0x50 [12404.139335] kvm_wait+0xcd/0xe0 [12404.140331] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12404.141965] do_raw_spin_lock+0x177/0x190 [12404.143477] _raw_spin_lock+0x21/0x30 [12404.145239] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12404.147386] ? do_raw_spin_unlock+0x75/0x190 [12404.148719] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12404.151705] ? do_wait_intr+0xf0/0xf0 [12404.152925] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12404.155253] mdc_close+0x53b/0xd70 [mdc] [12404.157373] lmv_close+0x18d/0x490 [lmv] [12404.158561] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12404.160704] ll_md_real_close+0xa5/0x2a0 [lustre] [12404.162355] ll_md_close+0x274/0x800 [lustre] [12404.164535] ll_file_release+0x128/0xc50 [lustre] [12404.165982] __fput+0x10d/0x440 [12404.167315] ____fput+0x12/0x20 [12404.168903] task_work_run+0xc6/0x110 [12404.172162] exit_to_usermode_loop+0x1dd/0x1f0 [12404.174553] do_syscall_64+0x3d6/0x440 [12404.176857] entry_SYSCALL_64_after_hwframe+0x49/0xae [12404.178780] RIP: 0033:0x7fa749552cd8 [12404.180769] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12404.188745] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12404.192139] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12404.193927] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12404.197152] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12404.199849] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12404.202396] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12413.922769] rcu: INFO: rcu_sched self-detected stall on CPU [12413.929608] rcu: 2-...!: (5950523 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [12413.937205] (t=10500233 jiffies g=258701 q=4541) [12413.943702] rcu: rcu_sched kthread starved for 10497028 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [12413.946317] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12413.958113] rcu: RCU grace-period kthread stack dump: [12413.962015] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12413.966952] Call Trace: [12413.967560] __schedule+0x351/0xcb0 [12413.968359] schedule+0xc0/0x180 [12413.969115] schedule_timeout+0xb4/0x190 [12413.980254] ? do_raw_spin_unlock+0x75/0x190 [12413.983218] ? __next_timer_interrupt+0x160/0x160 [12413.990099] ? prepare_to_swait_event+0xf3/0x140 [12413.998342] rcu_gp_kthread+0x8dd/0x15b0 [12414.008172] ? force_qs_rnp+0x2c0/0x2c0 [12414.012235] kthread+0x1d1/0x200 [12414.014284] ? set_kthread_struct+0x70/0x70 [12414.019356] ret_from_fork+0x1f/0x30 [12414.031875] rcu: Stack dump where RCU GP kthread last ran: [12414.038042] NMI backtrace for cpu 2 [12414.042599] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12414.065216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12414.082504] Call Trace: [12414.083410] [12414.084133] ? dump_stack+0xbb/0x10e [12414.098320] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12414.109129] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12414.120111] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12414.127795] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12414.129177] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12414.140357] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [12414.143146] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [12414.145811] ? update_process_times+0x6d/0xc0 [12414.148620] ? tick_sched_handle.isra.18+0x27/0x90 [12414.150516] ? tick_sched_timer+0x53/0xb0 [12414.153100] ? __hrtimer_run_queues+0x1c2/0x4d0 [12414.159196] ? tick_sched_handle.isra.18+0x90/0x90 [12414.162715] ? hrtimer_interrupt+0x128/0x2c0 [12414.163834] ? smp_apic_timer_interrupt+0xa6/0x270 [12414.168363] ? apic_timer_interrupt+0xf/0x20 [12414.172036] [12414.175897] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12414.179163] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12414.186401] ? do_raw_spin_lock+0x177/0x190 [12414.194584] ? _raw_spin_lock+0x21/0x30 [12414.195913] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12414.203321] ? obd_mod_free+0x46/0x140 [ptlrpc] [12414.207843] ? mdc_commit_open+0x91/0xb0 [mdc] [12414.215852] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12414.225092] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12414.231924] ? after_reply+0xe2c/0x1780 [ptlrpc] [12414.235406] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12414.239789] ? __next_timer_interrupt+0x160/0x160 [12414.244014] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12414.250105] ? do_wait_intr+0xf0/0xf0 [12414.254388] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12414.257584] ? kthread+0x1d1/0x200 [12414.258799] ? set_kthread_struct+0x70/0x70 [12414.263075] ? ret_from_fork+0x1f/0x30 [12414.265352] NMI backtrace for cpu 2 [12414.267438] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12414.274897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12414.287451] Call Trace: [12414.294341] [12414.296178] ? dump_stack+0xbb/0x10e [12414.298759] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12414.302813] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12414.306036] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12414.312478] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12414.316921] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12414.321896] ? rcu_dump_cpu_stacks+0x11b/0x171 [12414.324380] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12414.327881] ? update_process_times+0x6d/0xc0 [12414.336125] ? tick_sched_handle.isra.18+0x27/0x90 [12414.340041] ? tick_sched_timer+0x53/0xb0 [12414.344992] ? __hrtimer_run_queues+0x1c2/0x4d0 [12414.352962] ? tick_sched_handle.isra.18+0x90/0x90 [12414.356811] ? hrtimer_interrupt+0x128/0x2c0 [12414.359972] ? smp_apic_timer_interrupt+0xa6/0x270 [12414.364905] ? apic_timer_interrupt+0xf/0x20 [12414.366378] [12414.377424] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12414.394551] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12414.403938] ? do_raw_spin_lock+0x177/0x190 [12414.409010] ? _raw_spin_lock+0x21/0x30 [12414.412100] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12414.421303] ? obd_mod_free+0x46/0x140 [ptlrpc] [12414.428722] ? mdc_commit_open+0x91/0xb0 [mdc] [12414.440427] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12414.457634] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12414.467853] ? after_reply+0xe2c/0x1780 [ptlrpc] [12414.475713] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12414.484333] ? __next_timer_interrupt+0x160/0x160 [12414.487105] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12414.494843] ? do_wait_intr+0xf0/0xf0 [12414.498935] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12414.510383] ? kthread+0x1d1/0x200 [12414.517226] ? set_kthread_struct+0x70/0x70 [12414.519155] ? ret_from_fork+0x1f/0x30 [12420.066834] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12420.069571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12420.094620] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12420.099634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12420.103813] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12420.107303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12420.109889] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12420.119234] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12420.123861] RAX: 000000000000774b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12420.127915] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12420.131552] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12420.134835] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12420.138212] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12420.141423] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12420.144892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12420.147781] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12420.150984] Call Trace: [12420.152131] [12420.153274] ? show_regs.cold.9+0x22/0x2f [12420.155005] ? watchdog_timer_fn.cold.6+0x62/0x10a [12420.157748] ? __hrtimer_run_queues+0x1c2/0x4d0 [12420.159848] ? watchdog+0x40/0x40 [12420.161237] ? hrtimer_interrupt+0x128/0x2c0 [12420.163149] ? smp_apic_timer_interrupt+0xa6/0x270 [12420.165445] ? apic_timer_interrupt+0xf/0x20 [12420.167578] [12420.168650] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12420.170948] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12420.174114] do_raw_spin_lock+0x177/0x190 [12420.175996] _raw_spin_lock+0x21/0x30 [12420.177735] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12420.180098] process_one_work+0x2c8/0x700 [12420.182057] worker_thread+0x296/0x6e0 [12420.183583] ? rescuer_thread+0x570/0x570 [12420.185009] kthread+0x1d1/0x200 [12420.186691] ? set_kthread_struct+0x70/0x70 [12420.188200] ret_from_fork+0x1f/0x30 [12424.056455] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12424.065683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12424.091026] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12424.097230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12424.101876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12424.104841] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12424.113678] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12424.116896] RAX: 0000000000007b83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12424.120314] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12424.123924] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12424.127368] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12424.130399] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12424.134951] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12424.144640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12424.149889] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12424.155807] Call Trace: [12424.157438] [12424.161903] ? show_regs.cold.9+0x22/0x2f [12424.166804] ? watchdog_timer_fn.cold.6+0x62/0x10a [12424.171978] ? __hrtimer_run_queues+0x1c2/0x4d0 [12424.177825] ? watchdog+0x40/0x40 [12424.181415] ? hrtimer_interrupt+0x128/0x2c0 [12424.184438] ? smp_apic_timer_interrupt+0xa6/0x270 [12424.188678] ? apic_timer_interrupt+0xf/0x20 [12424.192773] [12424.193505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12424.199754] do_raw_spin_lock+0x177/0x190 [12424.209758] _raw_spin_lock+0x21/0x30 [12424.212319] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12424.218381] ? do_raw_spin_unlock+0x75/0x190 [12424.230577] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12424.237892] ? do_wait_intr+0xf0/0xf0 [12424.240635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12424.249871] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12424.258231] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12424.270959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12424.276705] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12424.281647] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12424.286850] mdc_intent_lock+0x353/0x7e0 [mdc] [12424.292579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12424.303437] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12424.310806] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12424.314886] lmv_intent_open+0x338/0x1150 [lmv] [12424.318216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12424.326939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12424.333925] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12424.337742] ll_intent_lock+0xc5/0xaa0 [lustre] [12424.341412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12424.348342] ll_lookup_it+0xe0e/0x2270 [lustre] [12424.356874] ? do_raw_spin_unlock+0x75/0x190 [12424.362348] ? _raw_spin_unlock+0x12/0x30 [12424.365893] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12424.370535] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12424.374975] ll_atomic_open+0x36a/0xd70 [lustre] [12424.379319] lookup_open+0x356/0xb40 [12424.382760] do_last+0x313/0xfc0 [12424.385912] ? nd_jump_root+0xe5/0x160 [12424.395002] ? path_init+0x437/0x520 [12424.398217] path_openat+0xf7/0x500 [12424.402679] ? do_raw_spin_unlock+0x75/0x190 [12424.407990] do_filp_open+0x99/0x140 [12424.409461] ? getname_flags+0x6e/0x330 [12424.419108] ? __check_object_size+0xff/0x256 [12424.423372] ? do_raw_spin_unlock+0x75/0x190 [12424.429656] ? _raw_spin_unlock+0x12/0x30 [12424.439633] do_sys_openat2+0x2b4/0x410 [12424.442837] do_sys_open+0x73/0xa0 [12424.447866] __x64_sys_openat+0x24/0x30 [12424.449673] do_syscall_64+0xc1/0x440 [12424.456066] entry_SYSCALL_64_after_hwframe+0x49/0xae [12424.461139] RIP: 0033:0x7fa749552332 [12424.463081] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12424.477775] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12424.487840] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12424.495493] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12424.503164] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12424.511432] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12424.518840] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12429.285097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10512s! [12429.290125] Showing busy workqueues and worker pools: [12429.293667] workqueue events: flags=0x0 [12429.295805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12429.299693] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12429.310221] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12429.314157] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12429.320355] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.326333] pending: kfree_rcu_monitor [12429.329389] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12429.336840] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12429.349715] workqueue events_long: flags=0x0 [12429.351897] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.355675] pending: lu_objects_flush [obdclass] [12429.359191] workqueue events_unbound: flags=0x2 [12429.361509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12429.364262] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12429.369608] workqueue events_power_efficient: flags=0x80 [12429.372895] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.377182] pending: check_lifetime [12429.380200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.386289] pending: neigh_periodic_work [12429.389540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.392531] pending: neigh_periodic_work [12429.395239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.397316] pending: do_cache_clean [12429.398469] workqueue mm_percpu_wq: flags=0x8 [12429.400768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.405154] pending: vmstat_update [12429.406011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.408673] pending: vmstat_update [12429.410423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.412840] pending: vmstat_update [12429.414277] workqueue writeback: flags=0x4a [12429.415864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12429.418452] pending: wb_workfn, wb_workfn [12429.420159] workqueue xprtiod: flags=0xa [12429.421605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12429.423900] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12429.426531] workqueue ipv6_addrconf: flags=0x40008 [12429.428216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12429.430647] pending: addrconf_verify_work [12429.432764] workqueue knbd0-recv: flags=0x1a [12429.434361] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12429.436721] in-flight: 651:recv_work [12429.438054] workqueue ptlrpc_pinger: flags=0x2 [12429.439648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12429.441313] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12429.443435] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10512s workers=4 idle: 37 31846 28897 [12429.446390] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12419s workers=2 idle: 653 [12432.060470] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12432.069214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12432.113461] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12432.127139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12432.132779] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12432.142915] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12432.156839] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12432.167803] RAX: 0000000000007d1c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12432.174053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12432.180422] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12432.187062] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12432.199149] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12432.206629] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12432.214483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12432.223522] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12432.230041] Call Trace: [12432.231635] [12432.236547] ? show_regs.cold.9+0x22/0x2f [12432.240537] ? watchdog_timer_fn.cold.6+0x62/0x10a [12432.244629] ? __hrtimer_run_queues+0x1c2/0x4d0 [12432.249194] ? watchdog+0x40/0x40 [12432.254840] ? hrtimer_interrupt+0x128/0x2c0 [12432.260647] ? smp_apic_timer_interrupt+0xa6/0x270 [12432.268636] ? apic_timer_interrupt+0xf/0x20 [12432.276057] [12432.277155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12432.280311] do_raw_spin_lock+0x177/0x190 [12432.283177] _raw_spin_lock+0x21/0x30 [12432.287841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12432.294043] ? do_raw_spin_unlock+0x75/0x190 [12432.297861] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12432.299593] ? do_wait_intr+0xf0/0xf0 [12432.305858] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12432.312850] mdc_close+0x53b/0xd70 [mdc] [12432.315841] lmv_close+0x18d/0x490 [lmv] [12432.323699] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12432.328102] ll_md_real_close+0xa5/0x2a0 [lustre] [12432.331391] ll_md_close+0x274/0x800 [lustre] [12432.337597] ll_file_release+0x128/0xc50 [lustre] [12432.340615] __fput+0x10d/0x440 [12432.342320] ____fput+0x12/0x20 [12432.346712] task_work_run+0xc6/0x110 [12432.351784] exit_to_usermode_loop+0x1dd/0x1f0 [12432.354373] do_syscall_64+0x3d6/0x440 [12432.356721] entry_SYSCALL_64_after_hwframe+0x49/0xae [12432.359696] RIP: 0033:0x7fa749552cd8 [12432.368424] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12432.379299] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12432.383755] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12432.392455] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12432.399706] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12432.402640] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12432.410096] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12440.058426] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12440.063511] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12440.102448] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12440.112502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12440.115819] RIP: 0010:native_safe_halt+0x2e/0x50 [12440.117688] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12440.124533] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12440.127940] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12440.131034] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12440.133294] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12440.135970] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12440.139015] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12440.142708] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12440.146260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12440.149152] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12440.152157] Call Trace: [12440.153549] [12440.154536] ? show_regs.cold.9+0x22/0x2f [12440.156257] ? watchdog_timer_fn.cold.6+0x62/0x10a [12440.158198] ? __hrtimer_run_queues+0x1c2/0x4d0 [12440.159679] ? watchdog+0x40/0x40 [12440.161078] ? hrtimer_interrupt+0x128/0x2c0 [12440.165000] ? smp_apic_timer_interrupt+0xa6/0x270 [12440.166964] ? apic_timer_interrupt+0xf/0x20 [12440.170341] [12440.171263] ? native_safe_halt+0x2e/0x50 [12440.174517] kvm_wait+0xcd/0xe0 [12440.175614] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12440.177556] do_raw_spin_lock+0x177/0x190 [12440.181369] _raw_spin_lock+0x21/0x30 [12440.182974] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12440.194612] obd_mod_free+0x46/0x140 [ptlrpc] [12440.200243] mdc_commit_open+0x91/0xb0 [mdc] [12440.202050] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12440.207405] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12440.211144] after_reply+0xe2c/0x1780 [ptlrpc] [12440.215155] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12440.223692] ? __next_timer_interrupt+0x160/0x160 [12440.229650] ptlrpcd+0xa82/0xe40 [ptlrpc] [12440.236291] ? do_wait_intr+0xf0/0xf0 [12440.238503] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12440.244375] kthread+0x1d1/0x200 [12440.245627] ? set_kthread_struct+0x70/0x70 [12440.251464] ret_from_fork+0x1f/0x30 [12448.057896] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12448.062162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12448.096828] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12448.102828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12448.107080] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12448.110054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12448.113255] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12448.122299] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12448.131943] RAX: 0000000000003c87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12448.135612] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12448.142930] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12448.149126] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12448.156555] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12448.164414] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12448.169897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12448.174380] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12448.181414] Call Trace: [12448.185654] [12448.187003] ? show_regs.cold.9+0x22/0x2f [12448.188458] ? watchdog_timer_fn.cold.6+0x62/0x10a [12448.193671] ? __hrtimer_run_queues+0x1c2/0x4d0 [12448.195142] ? watchdog+0x40/0x40 [12448.201119] ? hrtimer_interrupt+0x128/0x2c0 [12448.205247] ? smp_apic_timer_interrupt+0xa6/0x270 [12448.208255] ? apic_timer_interrupt+0xf/0x20 [12448.211968] [12448.212847] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12448.215487] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12448.218873] do_raw_spin_lock+0x177/0x190 [12448.220227] _raw_spin_lock+0x21/0x30 [12448.221318] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12448.223387] process_one_work+0x2c8/0x700 [12448.224992] worker_thread+0x296/0x6e0 [12448.228557] ? rescuer_thread+0x570/0x570 [12448.231201] kthread+0x1d1/0x200 [12448.236085] ? set_kthread_struct+0x70/0x70 [12448.239985] ret_from_fork+0x1f/0x30 [12456.054587] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12456.060308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12456.105075] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12456.114445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12456.120284] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12456.126265] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12456.145899] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12456.150540] RAX: 000000000000258b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12456.156184] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12456.160259] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12456.164200] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12456.167503] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12456.171351] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12456.176880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12456.180951] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12456.185550] Call Trace: [12456.186568] [12456.187394] ? show_regs.cold.9+0x22/0x2f [12456.190537] ? watchdog_timer_fn.cold.6+0x62/0x10a [12456.192709] ? __hrtimer_run_queues+0x1c2/0x4d0 [12456.194642] ? watchdog+0x40/0x40 [12456.196168] ? hrtimer_interrupt+0x128/0x2c0 [12456.198894] ? smp_apic_timer_interrupt+0xa6/0x270 [12456.202227] ? apic_timer_interrupt+0xf/0x20 [12456.206889] [12456.207891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12456.210591] do_raw_spin_lock+0x177/0x190 [12456.214112] _raw_spin_lock+0x21/0x30 [12456.217262] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12456.221766] ? do_raw_spin_unlock+0x75/0x190 [12456.224922] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12456.232183] ? do_wait_intr+0xf0/0xf0 [12456.238160] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12456.243919] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12456.254122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12456.259723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12456.264518] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12456.270071] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12456.274155] mdc_intent_lock+0x353/0x7e0 [mdc] [12456.278768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12456.282193] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12456.294252] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12456.304395] lmv_intent_open+0x338/0x1150 [lmv] [12456.308607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12456.312359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12456.317527] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12456.321076] ll_intent_lock+0xc5/0xaa0 [lustre] [12456.329765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12456.333060] ll_lookup_it+0xe0e/0x2270 [lustre] [12456.337908] ? do_raw_spin_unlock+0x75/0x190 [12456.344111] ? _raw_spin_unlock+0x12/0x30 [12456.347791] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12456.351690] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12456.355549] ll_atomic_open+0x36a/0xd70 [lustre] [12456.358318] lookup_open+0x356/0xb40 [12456.359585] do_last+0x313/0xfc0 [12456.362184] ? nd_jump_root+0xe5/0x160 [12456.366169] ? path_init+0x437/0x520 [12456.367936] path_openat+0xf7/0x500 [12456.371873] ? do_raw_spin_unlock+0x75/0x190 [12456.375685] do_filp_open+0x99/0x140 [12456.378152] ? getname_flags+0x6e/0x330 [12456.379586] ? __check_object_size+0xff/0x256 [12456.381283] ? do_raw_spin_unlock+0x75/0x190 [12456.383572] ? _raw_spin_unlock+0x12/0x30 [12456.386213] do_sys_openat2+0x2b4/0x410 [12456.388646] do_sys_open+0x73/0xa0 [12456.390263] __x64_sys_openat+0x24/0x30 [12456.392457] do_syscall_64+0xc1/0x440 [12456.394582] entry_SYSCALL_64_after_hwframe+0x49/0xae [12456.397948] RIP: 0033:0x7fa749552332 [12456.400657] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12456.409196] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12456.413764] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12456.417309] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12456.424253] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12456.429305] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12456.434691] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12459.492075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10542s! [12459.496938] Showing busy workqueues and worker pools: [12459.500210] workqueue events: flags=0x0 [12459.502699] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12459.507596] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12459.513167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12459.516636] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12459.521210] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.524733] pending: kfree_rcu_monitor [12459.532532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12459.538731] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12459.547776] workqueue events_long: flags=0x0 [12459.549865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.560369] pending: lu_objects_flush [obdclass] [12459.565074] workqueue events_unbound: flags=0x2 [12459.571449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12459.576843] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12459.586871] workqueue events_power_efficient: flags=0x80 [12459.591084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.595112] pending: check_lifetime [12459.598898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.604539] pending: neigh_periodic_work [12459.614199] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.618886] pending: neigh_periodic_work [12459.625656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.631025] pending: do_cache_clean [12459.633539] workqueue mm_percpu_wq: flags=0x8 [12459.635742] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.644809] pending: vmstat_update [12459.651206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.663129] pending: vmstat_update [12459.672458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.681877] pending: vmstat_update [12459.686684] workqueue writeback: flags=0x4a [12459.688801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12459.693506] pending: wb_workfn, wb_workfn [12459.696226] workqueue xprtiod: flags=0xa [12459.698924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12459.703172] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12459.706900] workqueue ipv6_addrconf: flags=0x40008 [12459.708864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12459.711708] pending: addrconf_verify_work [12459.713945] workqueue knbd0-recv: flags=0x1a [12459.715326] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12459.717603] in-flight: 651:recv_work [12459.719897] workqueue ptlrpc_pinger: flags=0x2 [12459.721211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12459.724108] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12459.725619] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10542s workers=4 idle: 37 31846 28897 [12459.732048] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12449s workers=2 idle: 653 [12460.059369] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12460.072583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12460.136358] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12460.145875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12460.160655] RIP: 0010:native_safe_halt+0x2e/0x50 [12460.166973] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12460.187618] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12460.193516] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12460.196408] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12460.201286] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12460.207000] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12460.214372] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12460.228822] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12460.238621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12460.250706] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12460.258148] Call Trace: [12460.260351] [12460.262264] ? show_regs.cold.9+0x22/0x2f [12460.264366] ? watchdog_timer_fn.cold.6+0x62/0x10a [12460.265768] ? __hrtimer_run_queues+0x1c2/0x4d0 [12460.267096] ? watchdog+0x40/0x40 [12460.272366] ? hrtimer_interrupt+0x128/0x2c0 [12460.274185] ? smp_apic_timer_interrupt+0xa6/0x270 [12460.280878] ? apic_timer_interrupt+0xf/0x20 [12460.284924] [12460.286956] ? native_safe_halt+0x2e/0x50 [12460.288926] kvm_wait+0xcd/0xe0 [12460.291257] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12460.295251] do_raw_spin_lock+0x177/0x190 [12460.297350] _raw_spin_lock+0x21/0x30 [12460.300346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12460.302827] ? do_raw_spin_unlock+0x75/0x190 [12460.304161] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12460.306527] ? do_wait_intr+0xf0/0xf0 [12460.308178] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12460.317727] mdc_close+0x53b/0xd70 [mdc] [12460.320321] lmv_close+0x18d/0x490 [lmv] [12460.325013] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12460.328983] ll_md_real_close+0xa5/0x2a0 [lustre] [12460.331872] ll_md_close+0x274/0x800 [lustre] [12460.334715] ll_file_release+0x128/0xc50 [lustre] [12460.341852] __fput+0x10d/0x440 [12460.344558] ____fput+0x12/0x20 [12460.349505] task_work_run+0xc6/0x110 [12460.352532] exit_to_usermode_loop+0x1dd/0x1f0 [12460.356747] do_syscall_64+0x3d6/0x440 [12460.360025] entry_SYSCALL_64_after_hwframe+0x49/0xae [12460.363116] RIP: 0033:0x7fa749552cd8 [12460.365729] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12460.381562] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12460.385987] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12460.388406] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12460.402573] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12460.411798] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12460.422239] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12468.064619] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12468.075860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12468.120155] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12468.137512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12468.152702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12468.158543] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12468.175470] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12468.178554] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12468.180646] RDX: 0000000000007851 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12468.183341] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12468.187518] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12468.201886] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12468.204357] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12468.218768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12468.221305] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12468.227796] Call Trace: [12468.228640] [12468.229388] ? show_regs.cold.9+0x22/0x2f [12468.237209] ? watchdog_timer_fn.cold.6+0x62/0x10a [12468.243195] ? __hrtimer_run_queues+0x1c2/0x4d0 [12468.253732] ? watchdog+0x40/0x40 [12468.259199] ? hrtimer_interrupt+0x128/0x2c0 [12468.260684] ? smp_apic_timer_interrupt+0xa6/0x270 [12468.268405] ? apic_timer_interrupt+0xf/0x20 [12468.270543] [12468.271307] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12468.274601] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12468.280691] do_raw_spin_lock+0x177/0x190 [12468.284560] _raw_spin_lock+0x21/0x30 [12468.287694] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12468.293366] obd_mod_free+0x46/0x140 [ptlrpc] [12468.295943] mdc_commit_open+0x91/0xb0 [mdc] [12468.299950] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12468.302010] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12468.312824] after_reply+0xe2c/0x1780 [ptlrpc] [12468.317903] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12468.319949] ? __next_timer_interrupt+0x160/0x160 [12468.330408] ptlrpcd+0xa82/0xe40 [ptlrpc] [12468.334630] ? do_wait_intr+0xf0/0xf0 [12468.346663] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12468.356036] kthread+0x1d1/0x200 [12468.363100] ? set_kthread_struct+0x70/0x70 [12468.366400] ret_from_fork+0x1f/0x30 [12476.058459] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12476.062326] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12476.113198] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12476.122165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12476.126905] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12476.129885] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12476.132975] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12476.141842] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12476.145237] RAX: 0000000000007e18 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12476.150919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12476.155099] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12476.158360] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12476.173882] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12476.181254] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12476.190133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12476.194772] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12476.199115] Call Trace: [12476.200569] [12476.203166] ? show_regs.cold.9+0x22/0x2f [12476.206210] ? watchdog_timer_fn.cold.6+0x62/0x10a [12476.207877] ? __hrtimer_run_queues+0x1c2/0x4d0 [12476.209972] ? watchdog+0x40/0x40 [12476.217597] ? hrtimer_interrupt+0x128/0x2c0 [12476.219990] ? smp_apic_timer_interrupt+0xa6/0x270 [12476.224447] ? apic_timer_interrupt+0xf/0x20 [12476.228580] [12476.230793] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12476.236430] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12476.239368] do_raw_spin_lock+0x177/0x190 [12476.242720] _raw_spin_lock+0x21/0x30 [12476.247336] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12476.251409] process_one_work+0x2c8/0x700 [12476.255772] worker_thread+0x296/0x6e0 [12476.264878] ? rescuer_thread+0x570/0x570 [12476.269258] kthread+0x1d1/0x200 [12476.272096] ? set_kthread_struct+0x70/0x70 [12476.274884] ret_from_fork+0x1f/0x30 [12484.054039] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12484.056589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12484.074662] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12484.079760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12484.083549] RIP: 0010:native_safe_halt+0x2e/0x50 [12484.085685] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12484.093887] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12484.097211] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12484.100317] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [12484.103279] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12484.105932] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12484.110356] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12484.112827] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12484.115951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12484.118483] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12484.121607] Call Trace: [12484.122693] [12484.123400] ? show_regs.cold.9+0x22/0x2f [12484.124916] ? watchdog_timer_fn.cold.6+0x62/0x10a [12484.127106] ? __hrtimer_run_queues+0x1c2/0x4d0 [12484.129043] ? watchdog+0x40/0x40 [12484.130468] ? hrtimer_interrupt+0x128/0x2c0 [12484.132456] ? smp_apic_timer_interrupt+0xa6/0x270 [12484.134554] ? apic_timer_interrupt+0xf/0x20 [12484.136585] [12484.137433] ? native_safe_halt+0x2e/0x50 [12484.139094] kvm_wait+0xcd/0xe0 [12484.140275] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12484.142468] do_raw_spin_lock+0x177/0x190 [12484.144417] _raw_spin_lock+0x21/0x30 [12484.145824] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12484.148314] ? do_raw_spin_unlock+0x75/0x190 [12484.150221] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12484.152575] ? do_wait_intr+0xf0/0xf0 [12484.154203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12484.156566] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12484.158996] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12484.161597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12484.163862] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12484.166067] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12484.167861] mdc_intent_lock+0x353/0x7e0 [mdc] [12484.169844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12484.172228] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12484.174895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12484.177427] lmv_intent_open+0x338/0x1150 [lmv] [12484.179336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12484.181584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12484.184229] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12484.185890] ll_intent_lock+0xc5/0xaa0 [lustre] [12484.188009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12484.190510] ll_lookup_it+0xe0e/0x2270 [lustre] [12484.192430] ? do_raw_spin_unlock+0x75/0x190 [12484.194184] ? _raw_spin_unlock+0x12/0x30 [12484.195941] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12484.197926] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12484.200186] ll_atomic_open+0x36a/0xd70 [lustre] [12484.202067] lookup_open+0x356/0xb40 [12484.203564] do_last+0x313/0xfc0 [12484.204784] ? nd_jump_root+0xe5/0x160 [12484.206480] ? path_init+0x437/0x520 [12484.208092] path_openat+0xf7/0x500 [12484.209517] ? do_raw_spin_unlock+0x75/0x190 [12484.211321] do_filp_open+0x99/0x140 [12484.212850] ? getname_flags+0x6e/0x330 [12484.214522] ? __check_object_size+0xff/0x256 [12484.216379] ? do_raw_spin_unlock+0x75/0x190 [12484.218256] ? _raw_spin_unlock+0x12/0x30 [12484.220094] do_sys_openat2+0x2b4/0x410 [12484.221458] do_sys_open+0x73/0xa0 [12484.222759] __x64_sys_openat+0x24/0x30 [12484.224358] do_syscall_64+0xc1/0x440 [12484.226029] entry_SYSCALL_64_after_hwframe+0x49/0xae [12484.227857] RIP: 0033:0x7fa749552332 [12484.229533] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12484.237434] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12484.240703] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12484.243096] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12484.246258] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12484.248959] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12484.252129] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12488.059025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12488.061809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12488.120291] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12488.132837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12488.143264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12488.158441] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12488.180498] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12488.183807] RAX: 0000000000006386 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12488.191276] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12488.198236] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12488.205328] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12488.211853] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12488.218558] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12488.226457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12488.232468] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12488.241705] Call Trace: [12488.248320] [12488.249053] ? show_regs.cold.9+0x22/0x2f [12488.250472] ? watchdog_timer_fn.cold.6+0x62/0x10a [12488.253843] ? __hrtimer_run_queues+0x1c2/0x4d0 [12488.256428] ? watchdog+0x40/0x40 [12488.258234] ? hrtimer_interrupt+0x128/0x2c0 [12488.260466] ? smp_apic_timer_interrupt+0xa6/0x270 [12488.263649] ? apic_timer_interrupt+0xf/0x20 [12488.266414] [12488.267750] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12488.271160] do_raw_spin_lock+0x177/0x190 [12488.272903] _raw_spin_lock+0x21/0x30 [12488.274735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12488.278151] ? do_raw_spin_unlock+0x75/0x190 [12488.280308] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12488.283509] ? do_wait_intr+0xf0/0xf0 [12488.285404] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12488.288258] mdc_close+0x53b/0xd70 [mdc] [12488.290422] lmv_close+0x18d/0x490 [lmv] [12488.292363] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12488.296316] ll_md_real_close+0xa5/0x2a0 [lustre] [12488.300758] ll_md_close+0x274/0x800 [lustre] [12488.305956] ll_file_release+0x128/0xc50 [lustre] [12488.311824] __fput+0x10d/0x440 [12488.318015] ____fput+0x12/0x20 [12488.320508] task_work_run+0xc6/0x110 [12488.325171] exit_to_usermode_loop+0x1dd/0x1f0 [12488.329138] do_syscall_64+0x3d6/0x440 [12488.330439] entry_SYSCALL_64_after_hwframe+0x49/0xae [12488.337201] RIP: 0033:0x7fa749552cd8 [12488.338410] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12488.356368] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12488.366126] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12488.369908] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12488.377788] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12488.385541] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12488.392519] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12490.216573] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10573s! [12490.224352] Showing busy workqueues and worker pools: [12490.227225] workqueue events: flags=0x0 [12490.235158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12490.239804] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12490.248683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12490.251812] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12490.256592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.259927] pending: kfree_rcu_monitor [12490.261826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12490.269398] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12490.278398] workqueue events_long: flags=0x0 [12490.280040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.283032] pending: lu_objects_flush [obdclass] [12490.286248] workqueue events_unbound: flags=0x2 [12490.288654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12490.296316] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12490.310371] workqueue events_power_efficient: flags=0x80 [12490.315047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.320054] pending: check_lifetime [12490.322225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.327134] pending: neigh_periodic_work [12490.335332] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.344244] pending: neigh_periodic_work [12490.345528] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.355017] pending: do_cache_clean [12490.359104] workqueue mm_percpu_wq: flags=0x8 [12490.361388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.366169] pending: vmstat_update [12490.368824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.372188] pending: vmstat_update [12490.374661] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.377895] pending: vmstat_update [12490.381412] workqueue writeback: flags=0x4a [12490.385171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12490.391055] pending: wb_workfn, wb_workfn [12490.397672] workqueue xprtiod: flags=0xa [12490.399617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12490.406353] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12490.411653] workqueue ipv6_addrconf: flags=0x40008 [12490.416615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12490.422931] pending: addrconf_verify_work [12490.425181] workqueue knbd0-recv: flags=0x1a [12490.429794] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12490.436562] in-flight: 651:recv_work [12490.439103] workqueue ptlrpc_pinger: flags=0x2 [12490.441744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12490.445325] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12490.448098] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10573s workers=4 idle: 37 31846 28897 [12490.455227] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12480s workers=2 idle: 653 [12496.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12496.063750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12496.090414] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12496.095441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12496.097982] RIP: 0010:native_safe_halt+0x2e/0x50 [12496.100125] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12496.106336] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12496.110058] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12496.113106] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12496.116585] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12496.121019] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12496.123906] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12496.128695] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12496.133854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12496.137636] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12496.142815] Call Trace: [12496.143714] [12496.144455] ? show_regs.cold.9+0x22/0x2f [12496.146804] ? watchdog_timer_fn.cold.6+0x62/0x10a [12496.149998] ? __hrtimer_run_queues+0x1c2/0x4d0 [12496.151504] ? watchdog+0x40/0x40 [12496.152620] ? hrtimer_interrupt+0x128/0x2c0 [12496.157616] ? smp_apic_timer_interrupt+0xa6/0x270 [12496.162471] ? apic_timer_interrupt+0xf/0x20 [12496.168480] [12496.171035] ? native_safe_halt+0x2e/0x50 [12496.174577] kvm_wait+0xcd/0xe0 [12496.179187] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12496.184603] do_raw_spin_lock+0x177/0x190 [12496.187558] _raw_spin_lock+0x21/0x30 [12496.190447] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12496.193642] obd_mod_free+0x46/0x140 [ptlrpc] [12496.197403] mdc_commit_open+0x91/0xb0 [mdc] [12496.199516] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12496.202854] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12496.209026] after_reply+0xe2c/0x1780 [ptlrpc] [12496.223126] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12496.237012] ? __next_timer_interrupt+0x160/0x160 [12496.243513] ptlrpcd+0xa82/0xe40 [ptlrpc] [12496.246165] ? do_wait_intr+0xf0/0xf0 [12496.247695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12496.250895] kthread+0x1d1/0x200 [12496.252418] ? set_kthread_struct+0x70/0x70 [12496.256316] ret_from_fork+0x1f/0x30 [12504.057196] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12504.060083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12504.093307] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12504.102604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12504.115991] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12504.123226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [12504.126194] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [12504.134410] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12504.137327] RAX: 0000000000000459 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12504.141249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12504.148896] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12504.153337] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12504.158218] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12504.164233] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12504.171670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12504.184705] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12504.201294] Call Trace: [12504.202278] [12504.203025] ? show_regs.cold.9+0x22/0x2f [12504.204591] ? watchdog_timer_fn.cold.6+0x62/0x10a [12504.215209] ? __hrtimer_run_queues+0x1c2/0x4d0 [12504.217565] ? watchdog+0x40/0x40 [12504.219742] ? hrtimer_interrupt+0x128/0x2c0 [12504.226676] ? smp_apic_timer_interrupt+0xa6/0x270 [12504.232265] ? apic_timer_interrupt+0xf/0x20 [12504.236274] [12504.239103] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12504.242915] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12504.251117] do_raw_spin_lock+0x177/0x190 [12504.253320] _raw_spin_lock+0x21/0x30 [12504.255533] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12504.266715] process_one_work+0x2c8/0x700 [12504.268224] worker_thread+0x296/0x6e0 [12504.269616] ? rescuer_thread+0x570/0x570 [12504.283964] kthread+0x1d1/0x200 [12504.287406] ? set_kthread_struct+0x70/0x70 [12504.290240] ret_from_fork+0x1f/0x30 [12516.059110] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12516.062663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12516.062851] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12516.141285] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12516.143267] Modules linked in: lustre(O) [12516.163867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12516.163871] osp(O) ofd(O) lod(O) [12516.172492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12516.175507] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12516.175578] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12516.175586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12516.175589] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12516.175604] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12516.175610] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12516.175614] RAX: 000000000000083b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12516.175616] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12516.175619] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12516.175621] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12516.175624] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12516.175628] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12516.175631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12516.175633] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12516.175643] Call Trace: [12516.175648] [12516.175655] ? show_regs.cold.9+0x22/0x2f [12516.175665] ? watchdog_timer_fn.cold.6+0x62/0x10a [12516.175672] ? __hrtimer_run_queues+0x1c2/0x4d0 [12516.175678] ? watchdog+0x40/0x40 [12516.175683] ? hrtimer_interrupt+0x128/0x2c0 [12516.175701] ? smp_apic_timer_interrupt+0xa6/0x270 [12516.175710] ? apic_timer_interrupt+0xf/0x20 [12516.175715] [12516.175718] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12516.175724] do_raw_spin_lock+0x177/0x190 [12516.175729] _raw_spin_lock+0x21/0x30 [12516.175734] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12516.176022] ? do_raw_spin_unlock+0x75/0x190 [12516.176028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12516.176223] ? do_wait_intr+0xf0/0xf0 [12516.176232] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12516.176438] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12516.176651] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12516.176858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12516.176952] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12516.176983] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12516.177001] mdc_intent_lock+0x353/0x7e0 [mdc] [12516.177018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12516.177092] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12516.177255] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12516.177275] lmv_intent_open+0x338/0x1150 [lmv] [12516.177299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12516.177355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12516.177436] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12516.177460] ll_intent_lock+0xc5/0xaa0 [lustre] [12516.177564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12516.177647] ll_lookup_it+0xe0e/0x2270 [lustre] [12516.177745] ? do_raw_spin_unlock+0x75/0x190 [12516.177751] ? _raw_spin_unlock+0x12/0x30 [12516.177758] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12516.177785] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12516.177804] ll_atomic_open+0x36a/0xd70 [lustre] [12516.177889] lookup_open+0x356/0xb40 [12516.177898] do_last+0x313/0xfc0 [12516.177903] ? nd_jump_root+0xe5/0x160 [12516.177907] ? path_init+0x437/0x520 [12516.177912] path_openat+0xf7/0x500 [12516.177916] ? do_raw_spin_unlock+0x75/0x190 [12516.177920] do_filp_open+0x99/0x140 [12516.177926] ? getname_flags+0x6e/0x330 [12516.177931] ? __check_object_size+0xff/0x256 [12516.177938] ? do_raw_spin_unlock+0x75/0x190 [12516.177941] ? _raw_spin_unlock+0x12/0x30 [12516.177946] do_sys_openat2+0x2b4/0x410 [12516.177950] do_sys_open+0x73/0xa0 [12516.177954] __x64_sys_openat+0x24/0x30 [12516.177957] do_syscall_64+0xc1/0x440 [12516.177963] entry_SYSCALL_64_after_hwframe+0x49/0xae [12516.177969] RIP: 0033:0x7fa749552332 [12516.178020] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12516.178024] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12516.178029] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12516.178032] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12516.178034] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12516.178036] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12516.178038] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12516.939844] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12516.960970] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12516.968207] RAX: 0000000000007f37 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12516.971710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12516.980270] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12516.986303] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12516.992743] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12516.998705] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12517.008353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12517.014316] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12517.021711] Call Trace: [12517.024488] [12517.025917] ? show_regs.cold.9+0x22/0x2f [12517.029994] ? watchdog_timer_fn.cold.6+0x62/0x10a [12517.035792] ? __hrtimer_run_queues+0x1c2/0x4d0 [12517.041408] ? watchdog+0x40/0x40 [12517.042705] ? hrtimer_interrupt+0x128/0x2c0 [12517.044879] ? smp_apic_timer_interrupt+0xa6/0x270 [12517.047861] ? apic_timer_interrupt+0xf/0x20 [12517.051776] [12517.053891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12517.058032] do_raw_spin_lock+0x177/0x190 [12517.062990] _raw_spin_lock+0x21/0x30 [12517.068803] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12517.078827] ? do_raw_spin_unlock+0x75/0x190 [12517.080368] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12517.096179] ? do_wait_intr+0xf0/0xf0 [12517.102691] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12517.117266] mdc_close+0x53b/0xd70 [mdc] [12517.118915] lmv_close+0x18d/0x490 [lmv] [12517.122048] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12517.128015] ll_md_real_close+0xa5/0x2a0 [lustre] [12517.130318] ll_md_close+0x274/0x800 [lustre] [12517.132972] ll_file_release+0x128/0xc50 [lustre] [12517.137366] __fput+0x10d/0x440 [12517.139704] ____fput+0x12/0x20 [12517.144995] task_work_run+0xc6/0x110 [12517.149075] exit_to_usermode_loop+0x1dd/0x1f0 [12517.162378] do_syscall_64+0x3d6/0x440 [12517.163731] entry_SYSCALL_64_after_hwframe+0x49/0xae [12517.177839] RIP: 0033:0x7fa749552cd8 [12517.179170] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12517.216326] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12517.222753] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12517.228586] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12517.233597] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12517.236328] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12517.252140] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12520.928218] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10603s! [12520.936472] Showing busy workqueues and worker pools: [12520.938464] workqueue events: flags=0x0 [12520.942039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12520.947907] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12520.957456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12520.963076] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12520.969011] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12520.978988] pending: kfree_rcu_monitor [12520.981554] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12520.990031] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12520.996027] workqueue events_long: flags=0x0 [12520.997770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12521.002061] pending: lu_objects_flush [obdclass] [12521.008105] workqueue events_unbound: flags=0x2 [12521.013633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12521.022071] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12521.031842] workqueue events_power_efficient: flags=0x80 [12521.037076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12521.043713] pending: check_lifetime [12521.048030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12521.051984] pending: neigh_periodic_work [12521.053307] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12521.056571] pending: neigh_periodic_work [12521.060000] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12521.065906] pending: do_cache_clean [12521.069138] workqueue mm_percpu_wq: flags=0x8 [12521.073446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12521.083537] pending: vmstat_update [12521.086317] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12521.092068] pending: vmstat_update [12521.093370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12521.096798] pending: vmstat_update [12521.098985] workqueue writeback: flags=0x4a [12521.101016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12521.103148] pending: wb_workfn, wb_workfn [12521.111044] workqueue xprtiod: flags=0xa [12521.113475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12521.117301] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12521.121913] workqueue ipv6_addrconf: flags=0x40008 [12521.127739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12521.132324] pending: addrconf_verify_work [12521.134970] workqueue knbd0-recv: flags=0x1a [12521.137839] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12521.141786] in-flight: 651:recv_work [12521.142910] workqueue ptlrpc_pinger: flags=0x2 [12521.149794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12521.153742] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12521.157920] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10604s workers=4 idle: 37 31846 28897 [12521.167288] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12511s workers=2 idle: 653 [12524.061845] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12524.072606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12524.113762] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12524.122925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12524.130985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12524.133705] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12524.148090] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12524.156661] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12524.160724] RDX: 0000000000004ead RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12524.163976] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12524.167505] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12524.170937] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12524.174706] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12524.178257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12524.180915] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12524.183780] Call Trace: [12524.184950] [12524.186015] ? show_regs.cold.9+0x22/0x2f [12524.187751] ? watchdog_timer_fn.cold.6+0x62/0x10a [12524.189764] ? __hrtimer_run_queues+0x1c2/0x4d0 [12524.192094] ? watchdog+0x40/0x40 [12524.193611] ? hrtimer_interrupt+0x128/0x2c0 [12524.195601] ? smp_apic_timer_interrupt+0xa6/0x270 [12524.198438] ? apic_timer_interrupt+0xf/0x20 [12524.205382] [12524.206610] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12524.209429] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12524.213367] do_raw_spin_lock+0x177/0x190 [12524.219421] _raw_spin_lock+0x21/0x30 [12524.223369] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12524.229703] obd_mod_free+0x46/0x140 [ptlrpc] [12524.237034] mdc_commit_open+0x91/0xb0 [mdc] [12524.241805] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12524.251781] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12524.258378] after_reply+0xe2c/0x1780 [ptlrpc] [12524.261514] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12524.264050] ? __next_timer_interrupt+0x160/0x160 [12524.266455] ptlrpcd+0xa82/0xe40 [ptlrpc] [12524.268885] ? do_wait_intr+0xf0/0xf0 [12524.270698] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12524.276066] kthread+0x1d1/0x200 [12524.277875] ? set_kthread_struct+0x70/0x70 [12524.279607] ret_from_fork+0x1f/0x30 [12532.057029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12532.059235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12532.081323] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12532.085292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12532.088770] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12532.091909] RIP: 0010:native_safe_halt+0x2e/0x50 [12532.093685] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12532.101038] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12532.104209] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12532.107119] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12532.110387] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12532.113287] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12532.117751] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12532.121355] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12532.125377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12532.128372] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12532.132258] Call Trace: [12532.133375] [12532.134536] ? show_regs.cold.9+0x22/0x2f [12532.136403] ? watchdog_timer_fn.cold.6+0x62/0x10a [12532.139077] ? __hrtimer_run_queues+0x1c2/0x4d0 [12532.140871] ? watchdog+0x40/0x40 [12532.142009] ? hrtimer_interrupt+0x128/0x2c0 [12532.144008] ? smp_apic_timer_interrupt+0xa6/0x270 [12532.146608] ? apic_timer_interrupt+0xf/0x20 [12532.148744] [12532.149889] ? native_safe_halt+0x2e/0x50 [12532.151757] kvm_wait+0xcd/0xe0 [12532.152836] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12532.154445] do_raw_spin_lock+0x177/0x190 [12532.155764] _raw_spin_lock+0x21/0x30 [12532.157754] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12532.159656] process_one_work+0x2c8/0x700 [12532.160910] worker_thread+0x296/0x6e0 [12532.162529] ? rescuer_thread+0x570/0x570 [12532.163866] kthread+0x1d1/0x200 [12532.164930] ? set_kthread_struct+0x70/0x70 [12532.167647] ret_from_fork+0x1f/0x30 [12544.066995] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12544.074131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12544.150821] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12544.168876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12544.178678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12544.185052] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12544.221787] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12544.234863] RAX: 0000000000007872 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12544.238648] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12544.251906] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12544.255356] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12544.260555] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12544.272073] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12544.279341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12544.282561] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12544.285988] Call Trace: [12544.287075] [12544.288221] ? show_regs.cold.9+0x22/0x2f [12544.297590] ? watchdog_timer_fn.cold.6+0x62/0x10a [12544.305943] ? __hrtimer_run_queues+0x1c2/0x4d0 [12544.307835] ? watchdog+0x40/0x40 [12544.317640] ? hrtimer_interrupt+0x128/0x2c0 [12544.320675] ? smp_apic_timer_interrupt+0xa6/0x270 [12544.332856] ? apic_timer_interrupt+0xf/0x20 [12544.341890] [12544.342631] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12544.348018] do_raw_spin_lock+0x177/0x190 [12544.353092] _raw_spin_lock+0x21/0x30 [12544.359713] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12544.366791] ? do_raw_spin_unlock+0x75/0x190 [12544.368774] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12544.376272] ? do_wait_intr+0xf0/0xf0 [12544.382994] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12544.388333] mdc_close+0x53b/0xd70 [mdc] [12544.393012] lmv_close+0x18d/0x490 [lmv] [12544.398063] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12544.404382] ll_md_real_close+0xa5/0x2a0 [lustre] [12544.407082] ll_md_close+0x274/0x800 [lustre] [12544.413810] ll_file_release+0x128/0xc50 [lustre] [12544.420538] __fput+0x10d/0x440 [12544.423973] ____fput+0x12/0x20 [12544.426484] task_work_run+0xc6/0x110 [12544.429989] exit_to_usermode_loop+0x1dd/0x1f0 [12544.431902] do_syscall_64+0x3d6/0x440 [12544.434089] entry_SYSCALL_64_after_hwframe+0x49/0xae [12544.437597] RIP: 0033:0x7fa749552cd8 [12544.439635] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12544.452135] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12544.456627] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12544.460396] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12544.463872] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12544.471249] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12544.474662] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12548.054927] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [12548.067121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12548.106162] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12548.110481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12548.114245] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12548.117108] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12548.126213] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12548.133347] RAX: 000000000000510e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12548.140406] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12548.144238] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12548.148678] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12548.153088] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12548.157447] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12548.161608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12548.168248] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12548.173674] Call Trace: [12548.180225] [12548.181826] ? show_regs.cold.9+0x22/0x2f [12548.187467] ? watchdog_timer_fn.cold.6+0x62/0x10a [12548.192586] ? __hrtimer_run_queues+0x1c2/0x4d0 [12548.196765] ? watchdog+0x40/0x40 [12548.197739] ? hrtimer_interrupt+0x128/0x2c0 [12548.206088] ? smp_apic_timer_interrupt+0xa6/0x270 [12548.210989] ? apic_timer_interrupt+0xf/0x20 [12548.216108] [12548.216829] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12548.224544] do_raw_spin_lock+0x177/0x190 [12548.227685] _raw_spin_lock+0x21/0x30 [12548.231820] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12548.242515] ? do_raw_spin_unlock+0x75/0x190 [12548.249359] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12548.253385] ? do_wait_intr+0xf0/0xf0 [12548.262465] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12548.267314] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12548.275168] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12548.284216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12548.289444] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12548.293766] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12548.296007] mdc_intent_lock+0x353/0x7e0 [mdc] [12548.301533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12548.304999] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12548.311611] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12548.316373] lmv_intent_open+0x338/0x1150 [lmv] [12548.318662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12548.328546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12548.335233] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12548.340577] ll_intent_lock+0xc5/0xaa0 [lustre] [12548.343509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12548.346015] ll_lookup_it+0xe0e/0x2270 [lustre] [12548.348581] ? do_raw_spin_unlock+0x75/0x190 [12548.350368] ? _raw_spin_unlock+0x12/0x30 [12548.352947] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12548.357280] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12548.358860] ll_atomic_open+0x36a/0xd70 [lustre] [12548.364392] lookup_open+0x356/0xb40 [12548.365789] do_last+0x313/0xfc0 [12548.366762] ? nd_jump_root+0xe5/0x160 [12548.371415] ? path_init+0x437/0x520 [12548.374538] path_openat+0xf7/0x500 [12548.377732] ? do_raw_spin_unlock+0x75/0x190 [12548.379710] do_filp_open+0x99/0x140 [12548.380725] ? getname_flags+0x6e/0x330 [12548.381721] ? __check_object_size+0xff/0x256 [12548.388929] ? do_raw_spin_unlock+0x75/0x190 [12548.390617] ? _raw_spin_unlock+0x12/0x30 [12548.395137] do_sys_openat2+0x2b4/0x410 [12548.396412] do_sys_open+0x73/0xa0 [12548.399299] __x64_sys_openat+0x24/0x30 [12548.400788] do_syscall_64+0xc1/0x440 [12548.405135] entry_SYSCALL_64_after_hwframe+0x49/0xae [12548.413538] RIP: 0033:0x7fa749552332 [12548.416632] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12548.438474] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12548.445019] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12548.451723] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12548.455622] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12548.459006] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12548.466095] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12551.655389] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10634s! [12551.660176] Showing busy workqueues and worker pools: [12551.663266] workqueue events: flags=0x0 [12551.666544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12551.677517] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12551.694628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12551.701591] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12551.718385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.727301] pending: kfree_rcu_monitor [12551.730753] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12551.738011] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12551.747386] workqueue events_long: flags=0x0 [12551.752190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.760433] pending: lu_objects_flush [obdclass] [12551.762749] workqueue events_unbound: flags=0x2 [12551.766166] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12551.773870] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12551.790530] workqueue events_power_efficient: flags=0x80 [12551.799448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.806247] pending: check_lifetime [12551.814079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.820166] pending: neigh_periodic_work [12551.826763] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.830755] pending: neigh_periodic_work [12551.836900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.847571] pending: do_cache_clean [12551.848537] workqueue mm_percpu_wq: flags=0x8 [12551.850017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.856464] pending: vmstat_update [12551.858042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.872332] pending: vmstat_update [12551.876287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.881651] pending: vmstat_update [12551.882988] workqueue writeback: flags=0x4a [12551.889545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12551.894236] pending: wb_workfn, wb_workfn [12551.897145] workqueue xprtiod: flags=0xa [12551.900444] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12551.908028] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12551.924168] workqueue ipv6_addrconf: flags=0x40008 [12551.925906] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12551.935669] pending: addrconf_verify_work [12551.938676] workqueue knbd0-recv: flags=0x1a [12551.943052] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12551.948943] in-flight: 651:recv_work [12551.954350] workqueue ptlrpc_pinger: flags=0x2 [12551.957013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12551.960730] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12551.964201] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10634s workers=4 idle: 37 31846 28897 [12551.969327] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12541s workers=2 idle: 653 [12552.058026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12552.060935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12552.093926] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12552.104054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12552.112353] RIP: 0010:native_safe_halt+0x2e/0x50 [12552.117874] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12552.140159] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12552.143091] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12552.152304] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12552.157600] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12552.177400] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12552.184820] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12552.189931] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12552.197470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12552.200609] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12552.206606] Call Trace: [12552.210475] [12552.211548] ? show_regs.cold.9+0x22/0x2f [12552.214573] ? watchdog_timer_fn.cold.6+0x62/0x10a [12552.221577] ? __hrtimer_run_queues+0x1c2/0x4d0 [12552.223949] ? watchdog+0x40/0x40 [12552.226223] ? hrtimer_interrupt+0x128/0x2c0 [12552.231702] ? smp_apic_timer_interrupt+0xa6/0x270 [12552.234605] ? apic_timer_interrupt+0xf/0x20 [12552.236967] [12552.238576] ? native_safe_halt+0x2e/0x50 [12552.242732] kvm_wait+0xcd/0xe0 [12552.247539] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12552.257152] do_raw_spin_lock+0x177/0x190 [12552.258163] _raw_spin_lock+0x21/0x30 [12552.259448] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12552.262915] obd_mod_free+0x46/0x140 [ptlrpc] [12552.277969] mdc_commit_open+0x91/0xb0 [mdc] [12552.281214] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12552.285432] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12552.288337] after_reply+0xe2c/0x1780 [ptlrpc] [12552.290862] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12552.294510] ? __next_timer_interrupt+0x160/0x160 [12552.300649] ptlrpcd+0xa82/0xe40 [ptlrpc] [12552.302781] ? do_wait_intr+0xf0/0xf0 [12552.305706] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12552.308862] kthread+0x1d1/0x200 [12552.311762] ? set_kthread_struct+0x70/0x70 [12552.313944] ret_from_fork+0x1f/0x30 [12560.057009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12560.061318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12560.077709] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12560.081572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12560.085082] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12560.088259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12560.091009] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12560.100599] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12560.104705] RAX: 00000000000022c6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12560.107402] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12560.110898] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12560.114065] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12560.120834] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12560.123995] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12560.133612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12560.136024] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12560.139144] Call Trace: [12560.140375] [12560.141452] ? show_regs.cold.9+0x22/0x2f [12560.143308] ? watchdog_timer_fn.cold.6+0x62/0x10a [12560.145426] ? __hrtimer_run_queues+0x1c2/0x4d0 [12560.148196] ? watchdog+0x40/0x40 [12560.149672] ? hrtimer_interrupt+0x128/0x2c0 [12560.152474] ? smp_apic_timer_interrupt+0xa6/0x270 [12560.154581] ? apic_timer_interrupt+0xf/0x20 [12560.157685] [12560.158594] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12560.160531] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12560.162358] do_raw_spin_lock+0x177/0x190 [12560.163742] _raw_spin_lock+0x21/0x30 [12560.165241] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12560.166955] process_one_work+0x2c8/0x700 [12560.168322] worker_thread+0x296/0x6e0 [12560.169671] ? rescuer_thread+0x570/0x570 [12560.170994] kthread+0x1d1/0x200 [12560.172464] ? set_kthread_struct+0x70/0x70 [12560.174290] ret_from_fork+0x1f/0x30 [12572.060759] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12572.071760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12572.137326] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12572.152771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12572.162194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12572.167550] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12572.191051] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12572.205945] RAX: 0000000000004c87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12572.212443] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12572.220407] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12572.226945] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12572.229383] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12572.236213] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12572.243062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12572.249502] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12572.251966] Call Trace: [12572.253081] [12572.253883] ? show_regs.cold.9+0x22/0x2f [12572.255927] ? watchdog_timer_fn.cold.6+0x62/0x10a [12572.260390] ? __hrtimer_run_queues+0x1c2/0x4d0 [12572.276983] ? watchdog+0x40/0x40 [12572.280188] ? hrtimer_interrupt+0x128/0x2c0 [12572.281944] ? smp_apic_timer_interrupt+0xa6/0x270 [12572.285342] ? apic_timer_interrupt+0xf/0x20 [12572.286901] [12572.287704] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12572.290232] do_raw_spin_lock+0x177/0x190 [12572.292095] _raw_spin_lock+0x21/0x30 [12572.293876] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12572.296657] ? do_raw_spin_unlock+0x75/0x190 [12572.298768] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12572.301556] ? do_wait_intr+0xf0/0xf0 [12572.303640] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12572.306218] mdc_close+0x53b/0xd70 [mdc] [12572.311717] lmv_close+0x18d/0x490 [lmv] [12572.315864] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12572.322890] ll_md_real_close+0xa5/0x2a0 [lustre] [12572.324612] ll_md_close+0x274/0x800 [lustre] [12572.331614] ll_file_release+0x128/0xc50 [lustre] [12572.335366] __fput+0x10d/0x440 [12572.336461] ____fput+0x12/0x20 [12572.337536] task_work_run+0xc6/0x110 [12572.345331] exit_to_usermode_loop+0x1dd/0x1f0 [12572.348622] do_syscall_64+0x3d6/0x440 [12572.356759] entry_SYSCALL_64_after_hwframe+0x49/0xae [12572.363070] RIP: 0033:0x7fa749552cd8 [12572.364341] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12572.385414] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12572.389734] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12572.395925] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12572.402734] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12572.408825] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12572.421575] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12576.057720] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12576.065587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12576.118879] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12576.125101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12576.137725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12576.149403] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12576.159463] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12576.163080] RAX: 0000000000003d8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12576.168335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12576.172094] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12576.175822] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12576.178970] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12576.183063] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12576.190181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12576.193596] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12576.198618] Call Trace: [12576.201213] [12576.202694] ? show_regs.cold.9+0x22/0x2f [12576.207069] ? watchdog_timer_fn.cold.6+0x62/0x10a [12576.209617] ? __hrtimer_run_queues+0x1c2/0x4d0 [12576.212178] ? watchdog+0x40/0x40 [12576.214810] ? hrtimer_interrupt+0x128/0x2c0 [12576.218564] ? smp_apic_timer_interrupt+0xa6/0x270 [12576.222039] ? apic_timer_interrupt+0xf/0x20 [12576.224328] [12576.226065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12576.229201] do_raw_spin_lock+0x177/0x190 [12576.231677] _raw_spin_lock+0x21/0x30 [12576.233705] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12576.237327] ? do_raw_spin_unlock+0x75/0x190 [12576.240187] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12576.243296] ? do_wait_intr+0xf0/0xf0 [12576.245158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12576.248243] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12576.251661] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12576.257100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12576.264016] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12576.271263] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12576.274233] mdc_intent_lock+0x353/0x7e0 [mdc] [12576.277419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12576.281891] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12576.284960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12576.288197] lmv_intent_open+0x338/0x1150 [lmv] [12576.290432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12576.293217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12576.295957] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12576.298134] ll_intent_lock+0xc5/0xaa0 [lustre] [12576.300393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12576.302913] ll_lookup_it+0xe0e/0x2270 [lustre] [12576.305842] ? do_raw_spin_unlock+0x75/0x190 [12576.316261] ? _raw_spin_unlock+0x12/0x30 [12576.319240] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12576.321169] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12576.325711] ll_atomic_open+0x36a/0xd70 [lustre] [12576.331103] lookup_open+0x356/0xb40 [12576.333065] do_last+0x313/0xfc0 [12576.336138] ? nd_jump_root+0xe5/0x160 [12576.341594] ? path_init+0x437/0x520 [12576.343722] path_openat+0xf7/0x500 [12576.347007] ? do_raw_spin_unlock+0x75/0x190 [12576.350305] do_filp_open+0x99/0x140 [12576.353317] ? getname_flags+0x6e/0x330 [12576.361039] ? __check_object_size+0xff/0x256 [12576.364610] ? do_raw_spin_unlock+0x75/0x190 [12576.367782] ? _raw_spin_unlock+0x12/0x30 [12576.370323] do_sys_openat2+0x2b4/0x410 [12576.376269] do_sys_open+0x73/0xa0 [12576.377510] __x64_sys_openat+0x24/0x30 [12576.379269] do_syscall_64+0xc1/0x440 [12576.380926] entry_SYSCALL_64_after_hwframe+0x49/0xae [12576.383568] RIP: 0033:0x7fa749552332 [12576.385249] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12576.392227] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12576.395616] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12576.398744] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12576.401923] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12576.404819] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12576.407907] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12580.058405] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12580.060925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12580.083491] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12580.089560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12580.094351] RIP: 0010:native_safe_halt+0x2e/0x50 [12580.095966] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12580.104851] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12580.109262] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12580.113918] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12580.117532] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12580.120964] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12580.124550] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12580.128873] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12580.132510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12580.134893] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12580.137986] Call Trace: [12580.138809] [12580.139541] ? show_regs.cold.9+0x22/0x2f [12580.140823] ? watchdog_timer_fn.cold.6+0x62/0x10a [12580.142779] ? __hrtimer_run_queues+0x1c2/0x4d0 [12580.144851] ? watchdog+0x40/0x40 [12580.146323] ? hrtimer_interrupt+0x128/0x2c0 [12580.149421] ? smp_apic_timer_interrupt+0xa6/0x270 [12580.151603] ? apic_timer_interrupt+0xf/0x20 [12580.152992] [12580.154591] ? native_safe_halt+0x2e/0x50 [12580.156260] kvm_wait+0xcd/0xe0 [12580.158889] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12580.161398] do_raw_spin_lock+0x177/0x190 [12580.163540] _raw_spin_lock+0x21/0x30 [12580.165105] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12580.167767] obd_mod_free+0x46/0x140 [ptlrpc] [12580.170472] mdc_commit_open+0x91/0xb0 [mdc] [12580.172813] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12580.175827] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12580.178469] after_reply+0xe2c/0x1780 [ptlrpc] [12580.180749] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12580.183904] ? __next_timer_interrupt+0x160/0x160 [12580.187207] ptlrpcd+0xa82/0xe40 [ptlrpc] [12580.189678] ? do_wait_intr+0xf0/0xf0 [12580.191548] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12580.194163] kthread+0x1d1/0x200 [12580.195739] ? set_kthread_struct+0x70/0x70 [12580.198492] ret_from_fork+0x1f/0x30 [12582.371938] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10665s! [12582.376984] Showing busy workqueues and worker pools: [12582.382802] workqueue events: flags=0x0 [12582.388134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12582.392261] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12582.400151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12582.403119] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12582.405944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.407975] pending: kfree_rcu_monitor [12582.409514] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12582.412367] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12582.421793] workqueue events_long: flags=0x0 [12582.423357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.430900] pending: lu_objects_flush [obdclass] [12582.435282] workqueue events_unbound: flags=0x2 [12582.437906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12582.442471] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12582.451883] workqueue events_power_efficient: flags=0x80 [12582.455139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.458204] pending: check_lifetime [12582.460141] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.462770] pending: neigh_periodic_work [12582.463837] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.467182] pending: neigh_periodic_work [12582.469178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.472442] pending: do_cache_clean [12582.474131] workqueue mm_percpu_wq: flags=0x8 [12582.475625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.480191] pending: vmstat_update [12582.483616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.487454] pending: vmstat_update [12582.488893] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12582.491866] pending: vmstat_update [12582.494974] workqueue writeback: flags=0x4a [12582.499014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12582.502806] pending: wb_workfn, wb_workfn [12582.504770] workqueue xprtiod: flags=0xa [12582.506759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12582.511223] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12582.516937] workqueue ipv6_addrconf: flags=0x40008 [12582.518771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12582.521223] pending: addrconf_verify_work [12582.523477] workqueue knbd0-recv: flags=0x1a [12582.525576] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12582.528385] in-flight: 651:recv_work [12582.530012] workqueue ptlrpc_pinger: flags=0x2 [12582.531570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12582.533961] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12582.536423] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10665s workers=4 idle: 37 31846 28897 [12582.539336] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12572s workers=2 idle: 653 [12588.059502] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12588.065842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12588.105720] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12588.111997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12588.117411] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12588.120939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12588.123864] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12588.133424] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12588.138195] RAX: 00000000000056c2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12588.141884] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12588.146016] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12588.149379] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12588.153751] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12588.157122] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12588.162242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12588.164842] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12588.168742] Call Trace: [12588.169981] [12588.170869] ? show_regs.cold.9+0x22/0x2f [12588.172822] ? watchdog_timer_fn.cold.6+0x62/0x10a [12588.175110] ? __hrtimer_run_queues+0x1c2/0x4d0 [12588.177075] ? watchdog+0x40/0x40 [12588.178614] ? hrtimer_interrupt+0x128/0x2c0 [12588.180630] ? smp_apic_timer_interrupt+0xa6/0x270 [12588.182773] ? apic_timer_interrupt+0xf/0x20 [12588.185177] [12588.186368] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12588.189110] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12588.192004] do_raw_spin_lock+0x177/0x190 [12588.194994] _raw_spin_lock+0x21/0x30 [12588.197391] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12588.200701] process_one_work+0x2c8/0x700 [12588.202772] worker_thread+0x296/0x6e0 [12588.204879] ? rescuer_thread+0x570/0x570 [12588.207095] kthread+0x1d1/0x200 [12588.208533] ? set_kthread_struct+0x70/0x70 [12588.210705] ret_from_fork+0x1f/0x30 [12593.925827] rcu: INFO: rcu_sched self-detected stall on CPU [12593.934805] rcu: 2-...!: (6061733 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [12593.943020] (t=10680236 jiffies g=258701 q=4567) [12593.947466] rcu: rcu_sched kthread starved for 10677031 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12593.957608] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12593.967598] rcu: RCU grace-period kthread stack dump: [12593.969986] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12593.985532] Call Trace: [12593.987984] __schedule+0x351/0xcb0 [12593.991538] schedule+0xc0/0x180 [12593.994926] schedule_timeout+0xb4/0x190 [12593.998660] ? do_raw_spin_unlock+0x75/0x190 [12594.001474] ? __next_timer_interrupt+0x160/0x160 [12594.006366] ? prepare_to_swait_event+0xf3/0x140 [12594.008803] rcu_gp_kthread+0x8dd/0x15b0 [12594.011029] ? force_qs_rnp+0x2c0/0x2c0 [12594.012890] kthread+0x1d1/0x200 [12594.014641] ? set_kthread_struct+0x70/0x70 [12594.016502] ret_from_fork+0x1f/0x30 [12594.017906] rcu: Stack dump where RCU GP kthread last ran: [12594.020531] Sending NMI from CPU 2 to CPUs 0: [12594.026294] NMI backtrace for cpu 0 [12594.026301] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12594.026304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12594.026305] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12594.026307] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12594.026310] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [12594.026313] RAX: 0000000000002f50 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12594.026314] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12594.026316] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12594.026318] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12594.026320] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12594.026321] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12594.026626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12594.026628] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12594.026630] Call Trace: [12594.026672] [12594.026673] ? show_regs.cold.9+0x22/0x2f [12594.026674] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12594.026676] ? nmi_cpu_backtrace_handler+0x15/0x40 [12594.026677] ? nmi_handle+0xbd/0x210 [12594.026678] ? default_do_nmi+0x4b/0x1b0 [12594.026679] ? do_nmi+0x280/0x380 [12594.026680] ? end_repeat_nmi+0x16/0x1f [12594.026682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12594.026683] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12594.026684] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12594.026685] [12594.026686] do_raw_spin_lock+0x177/0x190 [12594.026688] _raw_spin_lock+0x21/0x30 [12594.026689] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12594.026690] ? do_raw_spin_unlock+0x75/0x190 [12594.026691] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12594.026692] ? do_wait_intr+0xf0/0xf0 [12594.026693] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12594.026694] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12594.026695] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12594.026697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.026698] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12594.026699] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12594.026700] mdc_intent_lock+0x353/0x7e0 [mdc] [12594.026701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.026703] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12594.026704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12594.026706] lmv_intent_open+0x338/0x1150 [lmv] [12594.026707] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.026708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.026709] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12594.026711] ll_intent_lock+0xc5/0xaa0 [lustre] [12594.026712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.026714] ll_lookup_it+0xe0e/0x2270 [lustre] [12594.026715] ? do_raw_spin_unlock+0x75/0x190 [12594.026716] ? _raw_spin_unlock+0x12/0x30 [12594.026717] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12594.026718] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12594.026720] ll_atomic_open+0x36a/0xd70 [lustre] [12594.026721] lookup_open+0x356/0xb40 [12594.026722] do_last+0x313/0xfc0 [12594.026723] ? nd_jump_root+0xe5/0x160 [12594.026724] ? path_init+0x437/0x520 [12594.026725] path_openat+0xf7/0x500 [12594.026726] ? do_raw_spin_unlock+0x75/0x190 [12594.026728] do_filp_open+0x99/0x140 [12594.026729] ? getname_flags+0x6e/0x330 [12594.026730] ? __check_object_size+0xff/0x256 [12594.026731] ? do_raw_spin_unlock+0x75/0x190 [12594.026733] ? _raw_spin_unlock+0x12/0x30 [12594.026733] do_sys_openat2+0x2b4/0x410 [12594.026734] do_sys_open+0x73/0xa0 [12594.026735] __x64_sys_openat+0x24/0x30 [12594.026736] do_syscall_64+0xc1/0x440 [12594.026737] entry_SYSCALL_64_after_hwframe+0x49/0xae [12594.026738] RIP: 0033:0x7fa749552332 [12594.026740] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12594.026742] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12594.026744] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12594.026745] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12594.026747] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12594.026748] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12594.026749] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12594.027519] NMI backtrace for cpu 2 [12594.249226] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12594.254533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12594.260894] Call Trace: [12594.262378] [12594.263634] ? dump_stack+0xbb/0x10e [12594.265562] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12594.268625] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12594.273520] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12594.278026] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12594.280453] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12594.284123] ? rcu_dump_cpu_stacks+0x11b/0x171 [12594.289996] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12594.293623] ? update_process_times+0x6d/0xc0 [12594.300433] ? tick_sched_handle.isra.18+0x27/0x90 [12594.305772] ? tick_sched_timer+0x53/0xb0 [12594.308029] ? __hrtimer_run_queues+0x1c2/0x4d0 [12594.317619] ? tick_sched_handle.isra.18+0x90/0x90 [12594.322481] ? hrtimer_interrupt+0x128/0x2c0 [12594.328998] ? smp_apic_timer_interrupt+0xa6/0x270 [12594.333962] ? apic_timer_interrupt+0xf/0x20 [12594.336003] [12594.337259] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12594.340572] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12594.343713] ? do_raw_spin_lock+0x177/0x190 [12594.349354] ? _raw_spin_lock+0x21/0x30 [12594.353355] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12594.358218] ? obd_mod_free+0x46/0x140 [ptlrpc] [12594.364496] ? mdc_commit_open+0x91/0xb0 [mdc] [12594.371492] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12594.378375] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12594.383114] ? after_reply+0xe2c/0x1780 [ptlrpc] [12594.388383] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12594.397488] ? __next_timer_interrupt+0x160/0x160 [12594.401384] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12594.404441] ? do_wait_intr+0xf0/0xf0 [12594.406596] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12594.413262] ? kthread+0x1d1/0x200 [12594.416023] ? set_kthread_struct+0x70/0x70 [12594.420663] ? ret_from_fork+0x1f/0x30 [12600.059021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12600.063479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12600.111707] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12600.122107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12600.127970] RIP: 0010:native_safe_halt+0x2e/0x50 [12600.137365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12600.146301] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12600.149526] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12600.152890] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12600.156232] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12600.165774] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12600.168248] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12600.171555] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12600.176275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12600.178774] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12600.181379] Call Trace: [12600.182474] [12600.183140] ? show_regs.cold.9+0x22/0x2f [12600.186113] ? watchdog_timer_fn.cold.6+0x62/0x10a [12600.189929] ? __hrtimer_run_queues+0x1c2/0x4d0 [12600.191805] ? watchdog+0x40/0x40 [12600.193914] ? hrtimer_interrupt+0x128/0x2c0 [12600.195974] ? smp_apic_timer_interrupt+0xa6/0x270 [12600.197789] ? apic_timer_interrupt+0xf/0x20 [12600.199337] [12600.201246] ? native_safe_halt+0x2e/0x50 [12600.202882] kvm_wait+0xcd/0xe0 [12600.203900] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12600.207317] do_raw_spin_lock+0x177/0x190 [12600.208413] _raw_spin_lock+0x21/0x30 [12600.210896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12600.213665] ? do_raw_spin_unlock+0x75/0x190 [12600.215740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12600.218208] ? do_wait_intr+0xf0/0xf0 [12600.219858] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12600.224015] mdc_close+0x53b/0xd70 [mdc] [12600.225712] lmv_close+0x18d/0x490 [lmv] [12600.227772] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12600.230909] ll_md_real_close+0xa5/0x2a0 [lustre] [12600.234764] ll_md_close+0x274/0x800 [lustre] [12600.237271] ll_file_release+0x128/0xc50 [lustre] [12600.239585] __fput+0x10d/0x440 [12600.243902] ____fput+0x12/0x20 [12600.247490] task_work_run+0xc6/0x110 [12600.249262] exit_to_usermode_loop+0x1dd/0x1f0 [12600.251920] do_syscall_64+0x3d6/0x440 [12600.253708] entry_SYSCALL_64_after_hwframe+0x49/0xae [12600.255414] RIP: 0033:0x7fa749552cd8 [12600.260765] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12600.273402] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12600.276752] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12600.281763] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12600.286332] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12600.291552] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12600.293908] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12612.590485] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10695s! [12612.599922] Showing busy workqueues and worker pools: [12612.603432] workqueue events: flags=0x0 [12612.609475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12612.618387] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12612.632711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12612.645703] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12612.660962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.672341] pending: kfree_rcu_monitor [12612.673797] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12612.681555] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12612.699939] workqueue events_long: flags=0x0 [12612.704910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.714100] pending: lu_objects_flush [obdclass] [12612.720299] workqueue events_unbound: flags=0x2 [12612.723317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12612.732833] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12612.742134] workqueue events_power_efficient: flags=0x80 [12612.746295] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.752403] pending: check_lifetime [12612.755728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.759642] pending: neigh_periodic_work [12612.763676] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.768931] pending: neigh_periodic_work [12612.772208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.781892] pending: do_cache_clean [12612.788323] workqueue mm_percpu_wq: flags=0x8 [12612.793156] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.800739] pending: vmstat_update [12612.802125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.810274] pending: vmstat_update [12612.813030] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.821283] pending: vmstat_update [12612.823775] workqueue writeback: flags=0x4a [12612.829903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12612.842376] pending: wb_workfn, wb_workfn [12612.849918] workqueue xprtiod: flags=0xa [12612.852392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12612.856265] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12612.864081] workqueue ipv6_addrconf: flags=0x40008 [12612.865760] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12612.872616] pending: addrconf_verify_work [12612.875824] workqueue knbd0-recv: flags=0x1a [12612.878698] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12612.882265] in-flight: 651:recv_work [12612.892078] workqueue ptlrpc_pinger: flags=0x2 [12612.895261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12612.899531] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12612.907535] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10695s workers=4 idle: 37 31846 28897 [12612.918632] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12602s workers=2 idle: 653 [12616.057029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12616.064498] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12616.093049] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12616.098660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12616.103043] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12616.106393] RIP: 0010:native_safe_halt+0x2e/0x50 [12616.108442] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12616.116744] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12616.120389] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12616.123872] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12616.127204] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12616.130871] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12616.134392] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12616.137806] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12616.141723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12616.144131] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12616.147429] Call Trace: [12616.148818] [12616.149881] ? show_regs.cold.9+0x22/0x2f [12616.151271] ? watchdog_timer_fn.cold.6+0x62/0x10a [12616.153704] ? __hrtimer_run_queues+0x1c2/0x4d0 [12616.155874] ? watchdog+0x40/0x40 [12616.157451] ? hrtimer_interrupt+0x128/0x2c0 [12616.159428] ? smp_apic_timer_interrupt+0xa6/0x270 [12616.161849] ? apic_timer_interrupt+0xf/0x20 [12616.163882] [12616.164904] ? native_safe_halt+0x2e/0x50 [12616.168917] kvm_wait+0xcd/0xe0 [12616.175774] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12616.178140] do_raw_spin_lock+0x177/0x190 [12616.181118] _raw_spin_lock+0x21/0x30 [12616.183414] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12616.187276] process_one_work+0x2c8/0x700 [12616.194768] worker_thread+0x296/0x6e0 [12616.197244] ? rescuer_thread+0x570/0x570 [12616.202058] kthread+0x1d1/0x200 [12616.204051] ? set_kthread_struct+0x70/0x70 [12616.208175] ret_from_fork+0x1f/0x30 [12620.058192] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12620.063465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12620.089316] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12620.093989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12620.097730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12620.100130] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12620.109550] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12620.113285] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12620.116690] RDX: 0000000000007238 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12620.120288] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12620.123665] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12620.127344] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12620.130960] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12620.134997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12620.137963] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12620.141265] Call Trace: [12620.142039] [12620.142574] ? show_regs.cold.9+0x22/0x2f [12620.144425] ? watchdog_timer_fn.cold.6+0x62/0x10a [12620.147453] ? __hrtimer_run_queues+0x1c2/0x4d0 [12620.150578] ? watchdog+0x40/0x40 [12620.152520] ? hrtimer_interrupt+0x128/0x2c0 [12620.155555] ? smp_apic_timer_interrupt+0xa6/0x270 [12620.158008] ? apic_timer_interrupt+0xf/0x20 [12620.160134] [12620.161657] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12620.164104] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12620.167418] do_raw_spin_lock+0x177/0x190 [12620.169781] _raw_spin_lock+0x21/0x30 [12620.171125] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12620.173921] obd_mod_free+0x46/0x140 [ptlrpc] [12620.176546] mdc_commit_open+0x91/0xb0 [mdc] [12620.178432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12620.180781] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12620.183280] after_reply+0xe2c/0x1780 [ptlrpc] [12620.187405] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12620.190685] ? __next_timer_interrupt+0x160/0x160 [12620.193492] ptlrpcd+0xa82/0xe40 [ptlrpc] [12620.197452] ? do_wait_intr+0xf0/0xf0 [12620.201658] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12620.204086] kthread+0x1d1/0x200 [12620.208370] ? set_kthread_struct+0x70/0x70 [12620.209849] ret_from_fork+0x1f/0x30 [12628.059169] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12628.064584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12628.097342] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12628.103378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12628.109210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12628.112094] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12628.121332] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12628.125434] RAX: 0000000000000785 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12628.129022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12628.133318] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12628.137412] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12628.142104] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12628.145407] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12628.149222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12628.153041] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12628.157850] Call Trace: [12628.158942] [12628.159933] ? show_regs.cold.9+0x22/0x2f [12628.162158] ? watchdog_timer_fn.cold.6+0x62/0x10a [12628.163566] ? __hrtimer_run_queues+0x1c2/0x4d0 [12628.165687] ? watchdog+0x40/0x40 [12628.167394] ? hrtimer_interrupt+0x128/0x2c0 [12628.170912] ? smp_apic_timer_interrupt+0xa6/0x270 [12628.173024] ? apic_timer_interrupt+0xf/0x20 [12628.178388] [12628.180006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12628.186171] do_raw_spin_lock+0x177/0x190 [12628.188371] _raw_spin_lock+0x21/0x30 [12628.190389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12628.195981] ? do_raw_spin_unlock+0x75/0x190 [12628.200260] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12628.211999] ? do_wait_intr+0xf0/0xf0 [12628.213093] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12628.214602] mdc_close+0x53b/0xd70 [mdc] [12628.221358] lmv_close+0x18d/0x490 [lmv] [12628.224982] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12628.229704] ll_md_real_close+0xa5/0x2a0 [lustre] [12628.232944] ll_md_close+0x274/0x800 [lustre] [12628.236896] ll_file_release+0x128/0xc50 [lustre] [12628.240172] __fput+0x10d/0x440 [12628.242952] ____fput+0x12/0x20 [12628.245180] task_work_run+0xc6/0x110 [12628.247143] exit_to_usermode_loop+0x1dd/0x1f0 [12628.250669] do_syscall_64+0x3d6/0x440 [12628.252552] entry_SYSCALL_64_after_hwframe+0x49/0xae [12628.254933] RIP: 0033:0x7fa749552cd8 [12628.256707] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12628.266783] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12628.273070] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12628.277227] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12628.282230] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12628.288579] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12628.292917] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12640.057555] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12640.061096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12640.099936] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12640.108373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12640.114119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12640.119924] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12640.137385] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12640.145961] RAX: 0000000000005c1e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12640.159708] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12640.168745] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12640.177041] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12640.179387] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12640.189710] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12640.205317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12640.217455] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12640.224009] Call Trace: [12640.226202] [12640.227684] ? show_regs.cold.9+0x22/0x2f [12640.229922] ? watchdog_timer_fn.cold.6+0x62/0x10a [12640.235002] ? __hrtimer_run_queues+0x1c2/0x4d0 [12640.241677] ? watchdog+0x40/0x40 [12640.245727] ? hrtimer_interrupt+0x128/0x2c0 [12640.251097] ? smp_apic_timer_interrupt+0xa6/0x270 [12640.260491] ? apic_timer_interrupt+0xf/0x20 [12640.262109] [12640.262889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12640.276472] do_raw_spin_lock+0x177/0x190 [12640.281153] _raw_spin_lock+0x21/0x30 [12640.292129] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12640.294108] ? do_raw_spin_unlock+0x75/0x190 [12640.305763] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12640.312220] ? do_wait_intr+0xf0/0xf0 [12640.315774] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12640.321181] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12640.325136] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12640.328995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12640.332054] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12640.334866] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12640.337075] mdc_intent_lock+0x353/0x7e0 [mdc] [12640.342693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12640.345128] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12640.353218] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12640.357398] lmv_intent_open+0x338/0x1150 [lmv] [12640.359034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12640.362731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12640.364857] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12640.367691] ll_intent_lock+0xc5/0xaa0 [lustre] [12640.373122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12640.376645] ll_lookup_it+0xe0e/0x2270 [lustre] [12640.379024] ? do_raw_spin_unlock+0x75/0x190 [12640.381176] ? _raw_spin_unlock+0x12/0x30 [12640.383123] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12640.385842] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12640.388825] ll_atomic_open+0x36a/0xd70 [lustre] [12640.394142] lookup_open+0x356/0xb40 [12640.397810] do_last+0x313/0xfc0 [12640.401091] ? nd_jump_root+0xe5/0x160 [12640.403563] ? path_init+0x437/0x520 [12640.404827] path_openat+0xf7/0x500 [12640.406133] ? do_raw_spin_unlock+0x75/0x190 [12640.408108] do_filp_open+0x99/0x140 [12640.409692] ? getname_flags+0x6e/0x330 [12640.411001] ? __check_object_size+0xff/0x256 [12640.412813] ? do_raw_spin_unlock+0x75/0x190 [12640.414240] ? _raw_spin_unlock+0x12/0x30 [12640.416031] do_sys_openat2+0x2b4/0x410 [12640.417488] do_sys_open+0x73/0xa0 [12640.418914] __x64_sys_openat+0x24/0x30 [12640.420423] do_syscall_64+0xc1/0x440 [12640.422441] entry_SYSCALL_64_after_hwframe+0x49/0xae [12640.424860] RIP: 0033:0x7fa749552332 [12640.426451] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12640.434350] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12640.437619] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12640.440363] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12640.443326] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12640.446269] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12640.449623] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12643.298030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10726s! [12643.301995] Showing busy workqueues and worker pools: [12643.306390] workqueue events: flags=0x0 [12643.311131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12643.317155] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12643.336451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12643.345057] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12643.355967] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.366448] pending: kfree_rcu_monitor [12643.369925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12643.379502] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12643.389569] workqueue events_long: flags=0x0 [12643.394024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.400370] pending: lu_objects_flush [obdclass] [12643.406295] workqueue events_unbound: flags=0x2 [12643.411790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12643.413979] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12643.426763] workqueue events_power_efficient: flags=0x80 [12643.431018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.440788] pending: check_lifetime [12643.447722] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.456125] pending: neigh_periodic_work [12643.462010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.470879] pending: neigh_periodic_work [12643.472359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.479131] pending: do_cache_clean [12643.493385] workqueue mm_percpu_wq: flags=0x8 [12643.494836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.503832] pending: vmstat_update [12643.507695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.522168] pending: vmstat_update [12643.526956] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.534835] pending: vmstat_update [12643.536201] workqueue writeback: flags=0x4a [12643.546570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12643.553380] pending: wb_workfn, wb_workfn [12643.554722] workqueue xprtiod: flags=0xa [12643.556668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12643.565580] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12643.584140] workqueue ipv6_addrconf: flags=0x40008 [12643.591725] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12643.597048] pending: addrconf_verify_work [12643.608485] workqueue knbd0-recv: flags=0x1a [12643.613167] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12643.626249] in-flight: 651:recv_work [12643.629019] workqueue ptlrpc_pinger: flags=0x2 [12643.640043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12643.641878] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12643.653099] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10726s workers=4 idle: 37 31846 28897 [12643.666238] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12633s workers=2 idle: 653 [12644.059642] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12644.062931] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12644.092300] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12644.098094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12644.103143] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12644.106318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12644.109766] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12644.119709] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12644.125459] RAX: 0000000000007068 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12644.128608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12644.132250] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12644.136441] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12644.140528] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12644.144648] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12644.150029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12644.152802] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12644.160811] Call Trace: [12644.163740] [12644.165443] ? show_regs.cold.9+0x22/0x2f [12644.167587] ? watchdog_timer_fn.cold.6+0x62/0x10a [12644.169861] ? __hrtimer_run_queues+0x1c2/0x4d0 [12644.172063] ? watchdog+0x40/0x40 [12644.179011] ? hrtimer_interrupt+0x128/0x2c0 [12644.180392] ? smp_apic_timer_interrupt+0xa6/0x270 [12644.187960] ? apic_timer_interrupt+0xf/0x20 [12644.192289] [12644.195078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12644.200647] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12644.205201] do_raw_spin_lock+0x177/0x190 [12644.210003] _raw_spin_lock+0x21/0x30 [12644.213947] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12644.218417] process_one_work+0x2c8/0x700 [12644.224133] worker_thread+0x296/0x6e0 [12644.228044] ? rescuer_thread+0x570/0x570 [12644.232410] kthread+0x1d1/0x200 [12644.235793] ? set_kthread_struct+0x70/0x70 [12644.237177] ret_from_fork+0x1f/0x30 [12648.060245] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12648.069613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12648.123188] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12648.134559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12648.138034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12648.144177] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12648.159148] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12648.166345] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12648.171191] RDX: 0000000000007f02 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12648.174356] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12648.177469] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12648.181211] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12648.189846] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12648.197073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12648.200647] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12648.207696] Call Trace: [12648.208721] [12648.209766] ? show_regs.cold.9+0x22/0x2f [12648.213316] ? watchdog_timer_fn.cold.6+0x62/0x10a [12648.216350] ? __hrtimer_run_queues+0x1c2/0x4d0 [12648.218393] ? watchdog+0x40/0x40 [12648.220884] ? hrtimer_interrupt+0x128/0x2c0 [12648.225933] ? smp_apic_timer_interrupt+0xa6/0x270 [12648.234965] ? apic_timer_interrupt+0xf/0x20 [12648.237812] [12648.240266] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12648.246348] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12648.251966] do_raw_spin_lock+0x177/0x190 [12648.255001] _raw_spin_lock+0x21/0x30 [12648.256663] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12648.261036] obd_mod_free+0x46/0x140 [ptlrpc] [12648.266071] mdc_commit_open+0x91/0xb0 [mdc] [12648.269542] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12648.275683] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12648.280502] after_reply+0xe2c/0x1780 [ptlrpc] [12648.284363] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12648.293672] ? __next_timer_interrupt+0x160/0x160 [12648.297519] ptlrpcd+0xa82/0xe40 [ptlrpc] [12648.302799] ? do_wait_intr+0xf0/0xf0 [12648.308715] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12648.314815] kthread+0x1d1/0x200 [12648.318035] ? set_kthread_struct+0x70/0x70 [12648.324593] ret_from_fork+0x1f/0x30 [12656.068264] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12656.081851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12656.143066] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12656.155932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12656.171251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12656.178611] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12656.193167] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12656.199503] RAX: 0000000000007806 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12656.207456] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12656.212161] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12656.215265] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12656.219542] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12656.223572] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12656.232472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12656.240759] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12656.245120] Call Trace: [12656.246209] [12656.247120] ? show_regs.cold.9+0x22/0x2f [12656.250296] ? watchdog_timer_fn.cold.6+0x62/0x10a [12656.257752] ? __hrtimer_run_queues+0x1c2/0x4d0 [12656.261556] ? watchdog+0x40/0x40 [12656.263529] ? hrtimer_interrupt+0x128/0x2c0 [12656.266117] ? smp_apic_timer_interrupt+0xa6/0x270 [12656.270869] ? apic_timer_interrupt+0xf/0x20 [12656.274498] [12656.275515] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12656.281946] do_raw_spin_lock+0x177/0x190 [12656.285866] _raw_spin_lock+0x21/0x30 [12656.288095] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12656.291098] ? do_raw_spin_unlock+0x75/0x190 [12656.293563] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12656.300897] ? do_wait_intr+0xf0/0xf0 [12656.302259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12656.305092] mdc_close+0x53b/0xd70 [mdc] [12656.307574] lmv_close+0x18d/0x490 [lmv] [12656.308978] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12656.311979] ll_md_real_close+0xa5/0x2a0 [lustre] [12656.314412] ll_md_close+0x274/0x800 [lustre] [12656.316609] ll_file_release+0x128/0xc50 [lustre] [12656.321057] __fput+0x10d/0x440 [12656.322773] ____fput+0x12/0x20 [12656.324607] task_work_run+0xc6/0x110 [12656.326711] exit_to_usermode_loop+0x1dd/0x1f0 [12656.328559] do_syscall_64+0x3d6/0x440 [12656.330201] entry_SYSCALL_64_after_hwframe+0x49/0xae [12656.335472] RIP: 0033:0x7fa749552cd8 [12656.336985] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12656.350274] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12656.354890] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12656.358630] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12656.362607] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12656.375584] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12656.381806] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12668.054201] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [12668.059839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12668.087704] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12668.093438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12668.101613] RIP: 0010:native_safe_halt+0x2e/0x50 [12668.104460] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12668.119351] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12668.123114] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12668.126721] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [12668.129616] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12668.132535] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12668.135329] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12668.138094] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12668.141551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12668.143984] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12668.146844] Call Trace: [12668.147929] [12668.148865] ? show_regs.cold.9+0x22/0x2f [12668.150295] ? watchdog_timer_fn.cold.6+0x62/0x10a [12668.153087] ? __hrtimer_run_queues+0x1c2/0x4d0 [12668.154851] ? watchdog+0x40/0x40 [12668.156683] ? hrtimer_interrupt+0x128/0x2c0 [12668.158320] ? smp_apic_timer_interrupt+0xa6/0x270 [12668.160344] ? apic_timer_interrupt+0xf/0x20 [12668.162163] [12668.163008] ? native_safe_halt+0x2e/0x50 [12668.164527] kvm_wait+0xcd/0xe0 [12668.165676] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12668.167731] do_raw_spin_lock+0x177/0x190 [12668.169362] _raw_spin_lock+0x21/0x30 [12668.170877] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12668.173481] ? do_raw_spin_unlock+0x75/0x190 [12668.175185] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12668.177453] ? do_wait_intr+0xf0/0xf0 [12668.179008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12668.181182] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12668.183385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12668.186053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12668.188324] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12668.190580] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12668.192556] mdc_intent_lock+0x353/0x7e0 [mdc] [12668.194311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12668.196584] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12668.198691] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12668.200833] lmv_intent_open+0x338/0x1150 [lmv] [12668.202531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12668.204458] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12668.206570] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12668.208206] ll_intent_lock+0xc5/0xaa0 [lustre] [12668.210145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12668.212270] ll_lookup_it+0xe0e/0x2270 [lustre] [12668.214183] ? do_raw_spin_unlock+0x75/0x190 [12668.215863] ? _raw_spin_unlock+0x12/0x30 [12668.217287] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12668.219271] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12668.221139] ll_atomic_open+0x36a/0xd70 [lustre] [12668.223014] lookup_open+0x356/0xb40 [12668.224348] do_last+0x313/0xfc0 [12668.225531] ? nd_jump_root+0xe5/0x160 [12668.226937] ? path_init+0x437/0x520 [12668.228229] path_openat+0xf7/0x500 [12668.229592] ? do_raw_spin_unlock+0x75/0x190 [12668.231159] do_filp_open+0x99/0x140 [12668.232318] ? getname_flags+0x6e/0x330 [12668.234063] ? __check_object_size+0xff/0x256 [12668.235833] ? do_raw_spin_unlock+0x75/0x190 [12668.237351] ? _raw_spin_unlock+0x12/0x30 [12668.238873] do_sys_openat2+0x2b4/0x410 [12668.240253] do_sys_open+0x73/0xa0 [12668.241546] __x64_sys_openat+0x24/0x30 [12668.243120] do_syscall_64+0xc1/0x440 [12668.244683] entry_SYSCALL_64_after_hwframe+0x49/0xae [12668.246670] RIP: 0033:0x7fa749552332 [12668.248196] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12668.255547] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12668.258966] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12668.261942] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12668.264781] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12668.267316] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12668.270330] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12672.063810] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12672.070810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12672.132225] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12672.140148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12672.154647] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12672.158339] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12672.171065] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12672.192672] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12672.201200] RAX: 0000000000006d90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12672.204922] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12672.209382] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12672.215012] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12672.220407] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12672.224756] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12672.229149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12672.232304] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12672.235104] Call Trace: [12672.236329] [12672.237248] ? show_regs.cold.9+0x22/0x2f [12672.239059] ? watchdog_timer_fn.cold.6+0x62/0x10a [12672.241363] ? __hrtimer_run_queues+0x1c2/0x4d0 [12672.243770] ? watchdog+0x40/0x40 [12672.246349] ? hrtimer_interrupt+0x128/0x2c0 [12672.251152] ? smp_apic_timer_interrupt+0xa6/0x270 [12672.258061] ? apic_timer_interrupt+0xf/0x20 [12672.263323] [12672.267308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12672.272615] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12672.280442] do_raw_spin_lock+0x177/0x190 [12672.285404] _raw_spin_lock+0x21/0x30 [12672.288155] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12672.296809] process_one_work+0x2c8/0x700 [12672.303896] worker_thread+0x296/0x6e0 [12672.307983] ? rescuer_thread+0x570/0x570 [12672.316660] kthread+0x1d1/0x200 [12672.324968] ? set_kthread_struct+0x70/0x70 [12672.327623] ret_from_fork+0x1f/0x30 [12674.018029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10756s! [12674.025062] Showing busy workqueues and worker pools: [12674.029432] workqueue events: flags=0x0 [12674.031631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12674.036875] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12674.045382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12674.049957] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12674.055604] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.059816] pending: kfree_rcu_monitor [12674.062430] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12674.066824] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12674.072694] workqueue events_long: flags=0x0 [12674.075019] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.079455] pending: lu_objects_flush [obdclass] [12674.082607] workqueue events_unbound: flags=0x2 [12674.085667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12674.090343] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12674.095946] workqueue events_power_efficient: flags=0x80 [12674.100076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.104388] pending: check_lifetime [12674.106798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.111571] pending: neigh_periodic_work [12674.115044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.119841] pending: neigh_periodic_work [12674.122563] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.127348] pending: do_cache_clean [12674.130560] workqueue mm_percpu_wq: flags=0x8 [12674.136940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.140048] pending: vmstat_update [12674.142048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.146252] pending: vmstat_update [12674.148626] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12674.151936] pending: vmstat_update [12674.153851] workqueue writeback: flags=0x4a [12674.156467] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12674.160044] pending: wb_workfn, wb_workfn [12674.163411] workqueue xprtiod: flags=0xa [12674.165623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12674.169476] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12674.174811] workqueue ipv6_addrconf: flags=0x40008 [12674.177603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12674.181555] pending: addrconf_verify_work [12674.185815] workqueue knbd0-recv: flags=0x1a [12674.188315] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12674.192378] in-flight: 651:recv_work [12674.194501] workqueue ptlrpc_pinger: flags=0x2 [12674.196918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12674.199931] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12674.203435] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10757s workers=4 idle: 37 31846 28897 [12674.208966] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12664s workers=2 idle: 653 [12676.065551] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12676.068630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12676.095536] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12676.106067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12676.114584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12676.119341] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12676.137528] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12676.141003] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12676.147960] RDX: 00000000000013bd RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12676.151810] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12676.155288] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12676.159877] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12676.167681] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12676.170307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12676.177147] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12676.181754] Call Trace: [12676.183115] [12676.184644] ? show_regs.cold.9+0x22/0x2f [12676.187852] ? watchdog_timer_fn.cold.6+0x62/0x10a [12676.191980] ? __hrtimer_run_queues+0x1c2/0x4d0 [12676.193581] ? watchdog+0x40/0x40 [12676.196126] ? hrtimer_interrupt+0x128/0x2c0 [12676.198170] ? smp_apic_timer_interrupt+0xa6/0x270 [12676.199941] ? apic_timer_interrupt+0xf/0x20 [12676.201364] [12676.201988] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12676.203861] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12676.205692] do_raw_spin_lock+0x177/0x190 [12676.207143] _raw_spin_lock+0x21/0x30 [12676.212176] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12676.214476] obd_mod_free+0x46/0x140 [ptlrpc] [12676.219312] mdc_commit_open+0x91/0xb0 [mdc] [12676.221267] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12676.225291] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12676.228336] after_reply+0xe2c/0x1780 [ptlrpc] [12676.230582] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12676.233521] ? __next_timer_interrupt+0x160/0x160 [12676.238417] ptlrpcd+0xa82/0xe40 [ptlrpc] [12676.241244] ? do_wait_intr+0xf0/0xf0 [12676.242842] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12676.248536] kthread+0x1d1/0x200 [12676.250258] ? set_kthread_struct+0x70/0x70 [12676.251622] ret_from_fork+0x1f/0x30 [12684.059110] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12684.063810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12684.097894] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12684.106958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12684.111766] RIP: 0010:native_safe_halt+0x2e/0x50 [12684.115550] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12684.131457] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12684.138429] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12684.145434] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12684.152336] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12684.159211] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12684.169175] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12684.174098] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12684.181772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12684.186983] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12684.191047] Call Trace: [12684.192425] [12684.193858] ? show_regs.cold.9+0x22/0x2f [12684.196902] ? watchdog_timer_fn.cold.6+0x62/0x10a [12684.199978] ? __hrtimer_run_queues+0x1c2/0x4d0 [12684.202025] ? watchdog+0x40/0x40 [12684.203574] ? hrtimer_interrupt+0x128/0x2c0 [12684.205116] ? smp_apic_timer_interrupt+0xa6/0x270 [12684.211675] ? apic_timer_interrupt+0xf/0x20 [12684.214462] [12684.215979] ? native_safe_halt+0x2e/0x50 [12684.219830] kvm_wait+0xcd/0xe0 [12684.224437] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12684.227955] do_raw_spin_lock+0x177/0x190 [12684.230823] _raw_spin_lock+0x21/0x30 [12684.233850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12684.239199] ? do_raw_spin_unlock+0x75/0x190 [12684.242079] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12684.246094] ? do_wait_intr+0xf0/0xf0 [12684.247422] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12684.252209] mdc_close+0x53b/0xd70 [mdc] [12684.256205] lmv_close+0x18d/0x490 [lmv] [12684.265333] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12684.281806] ll_md_real_close+0xa5/0x2a0 [lustre] [12684.289008] ll_md_close+0x274/0x800 [lustre] [12684.292014] ll_file_release+0x128/0xc50 [lustre] [12684.296873] __fput+0x10d/0x440 [12684.298911] ____fput+0x12/0x20 [12684.304641] task_work_run+0xc6/0x110 [12684.306441] exit_to_usermode_loop+0x1dd/0x1f0 [12684.309610] do_syscall_64+0x3d6/0x440 [12684.311873] entry_SYSCALL_64_after_hwframe+0x49/0xae [12684.315863] RIP: 0033:0x7fa749552cd8 [12684.317690] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12684.330734] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12684.336867] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12684.344791] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12684.353195] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12684.359234] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12684.372330] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12700.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12700.057032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12700.058813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12700.058885] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12700.058892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12700.058896] RIP: 0010:native_safe_halt+0x2e/0x50 [12700.058909] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12700.058913] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12700.058917] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12700.058919] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [12700.058921] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12700.058923] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12700.058926] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12700.058930] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12700.058933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12700.058935] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12700.058943] Call Trace: [12700.058948] [12700.058954] ? show_regs.cold.9+0x22/0x2f [12700.058964] ? watchdog_timer_fn.cold.6+0x62/0x10a [12700.058970] ? __hrtimer_run_queues+0x1c2/0x4d0 [12700.058974] ? watchdog+0x40/0x40 [12700.058978] ? hrtimer_interrupt+0x128/0x2c0 [12700.058982] ? smp_apic_timer_interrupt+0xa6/0x270 [12700.058989] ? apic_timer_interrupt+0xf/0x20 [12700.058993] [12700.058995] ? native_safe_halt+0x2e/0x50 [12700.058999] kvm_wait+0xcd/0xe0 [12700.059007] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12700.059018] do_raw_spin_lock+0x177/0x190 [12700.059023] _raw_spin_lock+0x21/0x30 [12700.059028] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12700.059452] ? do_raw_spin_unlock+0x75/0x190 [12700.059458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12700.059853] ? do_wait_intr+0xf0/0xf0 [12700.059862] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12700.060093] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12700.060395] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12700.060779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12700.060886] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12700.060920] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12700.068108] Modules linked in: [12700.086865] mdc_intent_lock+0x353/0x7e0 [mdc] [12700.086918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12700.087033] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12700.087359] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12700.087389] lmv_intent_open+0x338/0x1150 [lmv] [12700.087417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12700.095158] lustre(O) [12700.098493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12700.098612] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12700.098646] ll_intent_lock+0xc5/0xaa0 [lustre] [12700.098732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12700.098813] ll_lookup_it+0xe0e/0x2270 [lustre] [12700.098892] ? do_raw_spin_unlock+0x75/0x190 [12700.098900] ? _raw_spin_unlock+0x12/0x30 [12700.098907] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12700.098933] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12700.098952] ll_atomic_open+0x36a/0xd70 [lustre] [12700.099037] lookup_open+0x356/0xb40 [12700.099045] do_last+0x313/0xfc0 [12700.099050] ? nd_jump_root+0xe5/0x160 [12700.099054] ? path_init+0x437/0x520 [12700.099059] path_openat+0xf7/0x500 [12700.099064] ? do_raw_spin_unlock+0x75/0x190 [12700.099069] do_filp_open+0x99/0x140 [12700.099075] ? getname_flags+0x6e/0x330 [12700.099079] ? __check_object_size+0xff/0x256 [12700.099087] ? do_raw_spin_unlock+0x75/0x190 [12700.099091] ? _raw_spin_unlock+0x12/0x30 [12700.099095] do_sys_openat2+0x2b4/0x410 [12700.099099] do_sys_open+0x73/0xa0 [12700.099103] __x64_sys_openat+0x24/0x30 [12700.099107] do_syscall_64+0xc1/0x440 [12700.099114] entry_SYSCALL_64_after_hwframe+0x49/0xae [12700.099121] RIP: 0033:0x7fa749552332 [12700.099128] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12700.099132] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12700.099138] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12700.099140] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12700.099142] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12700.099145] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12700.099147] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12700.404042] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12700.432654] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12700.438973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12700.444068] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12700.451250] RIP: 0010:native_safe_halt+0x2e/0x50 [12700.459764] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12700.471047] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12700.474895] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12700.478748] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12700.481480] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12700.484831] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12700.488349] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12700.491147] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12700.494598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12700.497013] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12700.499715] Call Trace: [12700.500624] [12700.501740] ? show_regs.cold.9+0x22/0x2f [12700.503273] ? watchdog_timer_fn.cold.6+0x62/0x10a [12700.504969] ? __hrtimer_run_queues+0x1c2/0x4d0 [12700.506631] ? watchdog+0x40/0x40 [12700.508499] ? hrtimer_interrupt+0x128/0x2c0 [12700.510310] ? smp_apic_timer_interrupt+0xa6/0x270 [12700.512408] ? apic_timer_interrupt+0xf/0x20 [12700.514024] [12700.514860] ? native_safe_halt+0x2e/0x50 [12700.516913] kvm_wait+0xcd/0xe0 [12700.518236] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12700.524537] do_raw_spin_lock+0x177/0x190 [12700.528099] _raw_spin_lock+0x21/0x30 [12700.532152] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12700.538976] process_one_work+0x2c8/0x700 [12700.543221] worker_thread+0x296/0x6e0 [12700.546513] ? rescuer_thread+0x570/0x570 [12700.548600] kthread+0x1d1/0x200 [12700.552605] ? set_kthread_struct+0x70/0x70 [12700.556577] ret_from_fork+0x1f/0x30 [12704.058751] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12704.062837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12704.084260] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12704.087625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12704.090921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12704.093717] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12704.100228] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12704.104466] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12704.108316] RDX: 00000000000053c2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12704.111255] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12704.113792] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12704.116270] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12704.120834] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12704.124511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12704.126862] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12704.129992] Call Trace: [12704.130792] [12704.135039] ? show_regs.cold.9+0x22/0x2f [12704.137048] ? watchdog_timer_fn.cold.6+0x62/0x10a [12704.141989] ? __hrtimer_run_queues+0x1c2/0x4d0 [12704.142968] ? watchdog+0x40/0x40 [12704.143780] ? hrtimer_interrupt+0x128/0x2c0 [12704.144724] ? smp_apic_timer_interrupt+0xa6/0x270 [12704.147568] ? apic_timer_interrupt+0xf/0x20 [12704.150034] [12704.151038] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12704.155515] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12704.157758] do_raw_spin_lock+0x177/0x190 [12704.159721] _raw_spin_lock+0x21/0x30 [12704.161212] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12704.164827] obd_mod_free+0x46/0x140 [ptlrpc] [12704.166978] mdc_commit_open+0x91/0xb0 [mdc] [12704.169008] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12704.175475] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12704.177821] after_reply+0xe2c/0x1780 [ptlrpc] [12704.180747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12704.184589] ? __next_timer_interrupt+0x160/0x160 [12704.187094] ptlrpcd+0xa82/0xe40 [ptlrpc] [12704.189759] ? do_wait_intr+0xf0/0xf0 [12704.191004] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12704.194846] kthread+0x1d1/0x200 [12704.197061] ? set_kthread_struct+0x70/0x70 [12704.199802] ret_from_fork+0x1f/0x30 [12704.228861] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10787s! [12704.235021] Showing busy workqueues and worker pools: [12704.237117] workqueue events: flags=0x0 [12704.238442] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12704.243002] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12704.247970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12704.250310] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12704.254047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.257616] pending: kfree_rcu_monitor [12704.260331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12704.262431] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12704.266329] workqueue events_long: flags=0x0 [12704.267683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.271670] pending: lu_objects_flush [obdclass] [12704.274205] workqueue events_unbound: flags=0x2 [12704.275424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12704.278475] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12704.282282] workqueue events_power_efficient: flags=0x80 [12704.283691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.286759] pending: check_lifetime [12704.288020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.291320] pending: neigh_periodic_work [12704.300930] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.303573] pending: neigh_periodic_work [12704.304996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.308057] pending: do_cache_clean [12704.309291] workqueue mm_percpu_wq: flags=0x8 [12704.310860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.318012] pending: vmstat_update [12704.320919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.326871] pending: vmstat_update [12704.331016] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.333980] pending: vmstat_update [12704.338028] workqueue writeback: flags=0x4a [12704.340282] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12704.344252] pending: wb_workfn, wb_workfn [12704.345886] workqueue xprtiod: flags=0xa [12704.346989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12704.349865] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12704.353803] workqueue ipv6_addrconf: flags=0x40008 [12704.360104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12704.363200] pending: addrconf_verify_work [12704.364912] workqueue knbd0-recv: flags=0x1a [12704.370015] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12704.371759] in-flight: 651:recv_work [12704.373536] workqueue ptlrpc_pinger: flags=0x2 [12704.375013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12704.377198] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12704.386016] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10787s workers=4 idle: 37 31846 28897 [12704.388839] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12694s workers=2 idle: 653 [12712.059131] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12712.062428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12712.082118] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12712.087547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12712.091215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12712.093849] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12712.101431] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12712.104005] RAX: 00000000000056fe RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12712.107183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12712.110411] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12712.113008] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12712.115899] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12712.118910] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12712.121917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12712.123996] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12712.127175] Call Trace: [12712.128367] [12712.129197] ? show_regs.cold.9+0x22/0x2f [12712.130989] ? watchdog_timer_fn.cold.6+0x62/0x10a [12712.132460] ? __hrtimer_run_queues+0x1c2/0x4d0 [12712.134063] ? watchdog+0x40/0x40 [12712.135314] ? hrtimer_interrupt+0x128/0x2c0 [12712.137409] ? smp_apic_timer_interrupt+0xa6/0x270 [12712.138907] ? apic_timer_interrupt+0xf/0x20 [12712.140581] [12712.141416] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12712.143586] do_raw_spin_lock+0x177/0x190 [12712.145270] _raw_spin_lock+0x21/0x30 [12712.146556] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12712.148820] ? do_raw_spin_unlock+0x75/0x190 [12712.150330] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12712.153028] ? do_wait_intr+0xf0/0xf0 [12712.154271] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12712.156523] mdc_close+0x53b/0xd70 [mdc] [12712.158011] lmv_close+0x18d/0x490 [lmv] [12712.159537] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12712.161969] ll_md_real_close+0xa5/0x2a0 [lustre] [12712.164681] ll_md_close+0x274/0x800 [lustre] [12712.166317] ll_file_release+0x128/0xc50 [lustre] [12712.168303] __fput+0x10d/0x440 [12712.169247] ____fput+0x12/0x20 [12712.170437] task_work_run+0xc6/0x110 [12712.171901] exit_to_usermode_loop+0x1dd/0x1f0 [12712.173421] do_syscall_64+0x3d6/0x440 [12712.174793] entry_SYSCALL_64_after_hwframe+0x49/0xae [12712.176518] RIP: 0033:0x7fa749552cd8 [12712.177864] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12712.185520] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12712.188932] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12712.191903] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12712.194842] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12712.197906] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12712.201156] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12728.057370] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12728.062506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12728.154216] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12728.165792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12728.179952] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12728.190764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12728.199666] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12728.233768] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12728.245469] RAX: 0000000000005186 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12728.251252] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12728.255246] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12728.261788] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12728.269835] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12728.280531] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12728.286502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12728.290027] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12728.292903] Call Trace: [12728.293823] [12728.295662] ? show_regs.cold.9+0x22/0x2f [12728.298526] ? watchdog_timer_fn.cold.6+0x62/0x10a [12728.300484] ? __hrtimer_run_queues+0x1c2/0x4d0 [12728.303470] ? watchdog+0x40/0x40 [12728.304991] ? hrtimer_interrupt+0x128/0x2c0 [12728.307283] ? smp_apic_timer_interrupt+0xa6/0x270 [12728.309309] ? apic_timer_interrupt+0xf/0x20 [12728.311216] [12728.311901] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12728.314283] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12728.316531] do_raw_spin_lock+0x177/0x190 [12728.318544] _raw_spin_lock+0x21/0x30 [12728.320257] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12728.322802] process_one_work+0x2c8/0x700 [12728.324677] worker_thread+0x296/0x6e0 [12728.326871] ? rescuer_thread+0x570/0x570 [12728.328635] kthread+0x1d1/0x200 [12728.330021] ? set_kthread_struct+0x70/0x70 [12728.331896] ret_from_fork+0x1f/0x30 [12732.057797] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12732.060577] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12732.073613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12732.074465] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12732.074473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12732.074476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12732.074489] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12732.074494] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12732.074498] RAX: 0000000000007785 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12732.074500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12732.074502] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12732.074504] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12732.074506] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12732.074509] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12732.074511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12732.074514] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12732.074534] Call Trace: [12732.074540] [12732.074545] ? show_regs.cold.9+0x22/0x2f [12732.074553] ? watchdog_timer_fn.cold.6+0x62/0x10a [12732.074559] ? __hrtimer_run_queues+0x1c2/0x4d0 [12732.074563] ? watchdog+0x40/0x40 [12732.074567] ? hrtimer_interrupt+0x128/0x2c0 [12732.074571] ? smp_apic_timer_interrupt+0xa6/0x270 [12732.074579] ? apic_timer_interrupt+0xf/0x20 [12732.074583] [12732.074586] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12732.074593] do_raw_spin_lock+0x177/0x190 [12732.074598] _raw_spin_lock+0x21/0x30 [12732.074603] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12732.086265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12732.086345] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12732.086353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12732.086357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12732.086372] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12732.086376] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12732.086381] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12732.086390] RDX: 0000000000007e1a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12732.086393] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12732.086396] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12732.086398] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12732.086402] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12732.086405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12732.086407] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12732.086416] Call Trace: [12732.086422] [12732.086428] ? show_regs.cold.9+0x22/0x2f [12732.086438] ? watchdog_timer_fn.cold.6+0x62/0x10a [12732.086446] ? __hrtimer_run_queues+0x1c2/0x4d0 [12732.086450] ? watchdog+0x40/0x40 [12732.086455] ? hrtimer_interrupt+0x128/0x2c0 [12732.086460] ? smp_apic_timer_interrupt+0xa6/0x270 [12732.086469] ? apic_timer_interrupt+0xf/0x20 [12732.086474] [12732.086477] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12732.086482] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12732.086487] do_raw_spin_lock+0x177/0x190 [12732.086492] _raw_spin_lock+0x21/0x30 [12732.086511] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12732.087095] obd_mod_free+0x46/0x140 [ptlrpc] [12732.087300] mdc_commit_open+0x91/0xb0 [mdc] [12732.087345] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12732.088208] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12732.089237] after_reply+0xe2c/0x1780 [ptlrpc] [12732.089439] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12732.091097] ? __next_timer_interrupt+0x160/0x160 [12732.091109] ptlrpcd+0xa82/0xe40 [ptlrpc] [12732.091344] ? do_wait_intr+0xf0/0xf0 [12732.091354] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12732.091560] kthread+0x1d1/0x200 [12732.091567] ? set_kthread_struct+0x70/0x70 [12732.091571] ret_from_fork+0x1f/0x30 [12732.892409] ? do_raw_spin_unlock+0x75/0x190 [12732.898969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12732.912827] ? do_wait_intr+0xf0/0xf0 [12732.918596] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12732.923255] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12732.931994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12732.946839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12732.953109] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12732.965705] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12732.976046] mdc_intent_lock+0x353/0x7e0 [mdc] [12732.989307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12732.995844] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12733.009108] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12733.027586] lmv_intent_open+0x338/0x1150 [lmv] [12733.029113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12733.042035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12733.052687] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12733.058578] ll_intent_lock+0xc5/0xaa0 [lustre] [12733.071684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12733.086124] ll_lookup_it+0xe0e/0x2270 [lustre] [12733.094147] ? do_raw_spin_unlock+0x75/0x190 [12733.109594] ? _raw_spin_unlock+0x12/0x30 [12733.120986] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12733.125355] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12733.133339] ll_atomic_open+0x36a/0xd70 [lustre] [12733.139030] lookup_open+0x356/0xb40 [12733.141440] do_last+0x313/0xfc0 [12733.144107] ? nd_jump_root+0xe5/0x160 [12733.157712] ? path_init+0x437/0x520 [12733.160343] path_openat+0xf7/0x500 [12733.167046] ? do_raw_spin_unlock+0x75/0x190 [12733.171046] do_filp_open+0x99/0x140 [12733.175171] ? getname_flags+0x6e/0x330 [12733.179845] ? __check_object_size+0xff/0x256 [12733.181843] ? do_raw_spin_unlock+0x75/0x190 [12733.183827] ? _raw_spin_unlock+0x12/0x30 [12733.185397] do_sys_openat2+0x2b4/0x410 [12733.187735] do_sys_open+0x73/0xa0 [12733.194507] __x64_sys_openat+0x24/0x30 [12733.196884] do_syscall_64+0xc1/0x440 [12733.201305] entry_SYSCALL_64_after_hwframe+0x49/0xae [12733.205865] RIP: 0033:0x7fa749552332 [12733.209146] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12733.230270] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12733.239532] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12733.253120] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12733.258065] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12733.262590] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12733.266363] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12734.436123] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10817s! [12734.456465] Showing busy workqueues and worker pools: [12734.459293] workqueue events: flags=0x0 [12734.461718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12734.464819] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12734.470881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12734.473837] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12734.478169] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.481302] pending: kfree_rcu_monitor [12734.483253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12734.486025] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12734.490308] workqueue events_long: flags=0x0 [12734.492306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.495566] pending: lu_objects_flush [obdclass] [12734.497852] workqueue events_unbound: flags=0x2 [12734.499746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12734.502795] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12734.508259] workqueue events_power_efficient: flags=0x80 [12734.511358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.514367] pending: check_lifetime [12734.516215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.519518] pending: neigh_periodic_work [12734.521744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.524456] pending: neigh_periodic_work [12734.526478] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.529593] pending: do_cache_clean [12734.531425] workqueue mm_percpu_wq: flags=0x8 [12734.533281] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.536463] pending: vmstat_update [12734.538522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.541922] pending: vmstat_update [12734.544178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.547448] pending: vmstat_update [12734.549218] workqueue writeback: flags=0x4a [12734.551355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12734.554390] pending: wb_workfn, wb_workfn [12734.556649] workqueue xprtiod: flags=0xa [12734.558580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12734.561888] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12734.565480] workqueue ipv6_addrconf: flags=0x40008 [12734.567538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12734.570172] pending: addrconf_verify_work [12734.571491] workqueue knbd0-recv: flags=0x1a [12734.572743] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12734.575175] in-flight: 651:recv_work [12734.576383] workqueue ptlrpc_pinger: flags=0x2 [12734.577896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12734.579987] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12734.582013] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10817s workers=4 idle: 37 31846 28897 [12734.585261] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12724s workers=2 idle: 653 [12740.059154] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12740.062694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12740.086885] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12740.092594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12740.096817] RIP: 0010:native_safe_halt+0x2e/0x50 [12740.099369] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12740.108013] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12740.110462] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12740.113450] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12740.116825] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12740.118932] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12740.121991] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12740.125224] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12740.128080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12740.130127] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12740.133184] Call Trace: [12740.134216] [12740.134998] ? show_regs.cold.9+0x22/0x2f [12740.136435] ? watchdog_timer_fn.cold.6+0x62/0x10a [12740.138288] ? __hrtimer_run_queues+0x1c2/0x4d0 [12740.140094] ? watchdog+0x40/0x40 [12740.141176] ? hrtimer_interrupt+0x128/0x2c0 [12740.142388] ? smp_apic_timer_interrupt+0xa6/0x270 [12740.143706] ? apic_timer_interrupt+0xf/0x20 [12740.145045] [12740.145844] ? native_safe_halt+0x2e/0x50 [12740.147499] kvm_wait+0xcd/0xe0 [12740.148760] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12740.150345] do_raw_spin_lock+0x177/0x190 [12740.152086] _raw_spin_lock+0x21/0x30 [12740.153697] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12740.156474] ? do_raw_spin_unlock+0x75/0x190 [12740.158458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12740.160704] ? do_wait_intr+0xf0/0xf0 [12740.162198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12740.164418] mdc_close+0x53b/0xd70 [mdc] [12740.166251] lmv_close+0x18d/0x490 [lmv] [12740.168011] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12740.170725] ll_md_real_close+0xa5/0x2a0 [lustre] [12740.173373] ll_md_close+0x274/0x800 [lustre] [12740.175661] ll_file_release+0x128/0xc50 [lustre] [12740.178220] __fput+0x10d/0x440 [12740.179644] ____fput+0x12/0x20 [12740.181119] task_work_run+0xc6/0x110 [12740.183053] exit_to_usermode_loop+0x1dd/0x1f0 [12740.185472] do_syscall_64+0x3d6/0x440 [12740.187218] entry_SYSCALL_64_after_hwframe+0x49/0xae [12740.189688] RIP: 0033:0x7fa749552cd8 [12740.191214] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12740.199442] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12740.202568] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12740.205119] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12740.208497] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12740.211957] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12740.215559] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12756.057877] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12756.068148] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12756.115385] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12756.129396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12756.136274] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12756.140350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12756.142350] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12756.159355] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12756.172151] RAX: 0000000000003892 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12756.183329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12756.191571] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12756.199685] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12756.207418] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12756.214869] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12756.225366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12756.228465] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12756.237968] Call Trace: [12756.239394] [12756.242585] ? show_regs.cold.9+0x22/0x2f [12756.249680] ? watchdog_timer_fn.cold.6+0x62/0x10a [12756.251981] ? __hrtimer_run_queues+0x1c2/0x4d0 [12756.256887] ? watchdog+0x40/0x40 [12756.258132] ? hrtimer_interrupt+0x128/0x2c0 [12756.261021] ? smp_apic_timer_interrupt+0xa6/0x270 [12756.263392] ? apic_timer_interrupt+0xf/0x20 [12756.268646] [12756.270262] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12756.279959] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12756.288397] do_raw_spin_lock+0x177/0x190 [12756.291998] _raw_spin_lock+0x21/0x30 [12756.296289] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12756.299661] process_one_work+0x2c8/0x700 [12756.306224] worker_thread+0x296/0x6e0 [12756.312952] ? rescuer_thread+0x570/0x570 [12756.314917] kthread+0x1d1/0x200 [12756.321824] ? set_kthread_struct+0x70/0x70 [12756.323289] ret_from_fork+0x1f/0x30 [12760.054335] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12760.058527] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12760.061709] Modules linked in: [12760.065951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12760.066029] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12760.066037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12760.066039] RIP: 0010:native_safe_halt+0x2e/0x50 [12760.066053] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12760.066057] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12760.066061] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12760.066064] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12760.066066] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12760.066069] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12760.066072] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12760.066075] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12760.066078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12760.066080] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12760.066090] Call Trace: [12760.066095] [12760.066100] ? show_regs.cold.9+0x22/0x2f [12760.066109] ? watchdog_timer_fn.cold.6+0x62/0x10a [12760.066115] ? __hrtimer_run_queues+0x1c2/0x4d0 [12760.066120] ? watchdog+0x40/0x40 [12760.066124] ? hrtimer_interrupt+0x128/0x2c0 [12760.066127] ? smp_apic_timer_interrupt+0xa6/0x270 [12760.066199] ? apic_timer_interrupt+0xf/0x20 [12760.066204] [12760.066208] ? native_safe_halt+0x2e/0x50 [12760.066212] kvm_wait+0xcd/0xe0 [12760.066219] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12760.066227] do_raw_spin_lock+0x177/0x190 [12760.066231] _raw_spin_lock+0x21/0x30 [12760.066236] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12760.067837] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12760.067905] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12760.067912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12760.067915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12760.067927] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12760.067931] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12760.067936] RAX: 0000000000002cc7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12760.067939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12760.067941] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12760.067943] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12760.067945] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12760.067948] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12760.067951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12760.067953] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12760.067962] Call Trace: [12760.067967] [12760.067971] ? show_regs.cold.9+0x22/0x2f [12760.067981] ? watchdog_timer_fn.cold.6+0x62/0x10a [12760.067987] ? __hrtimer_run_queues+0x1c2/0x4d0 [12760.067992] ? watchdog+0x40/0x40 [12760.067997] ? hrtimer_interrupt+0x128/0x2c0 [12760.068001] ? smp_apic_timer_interrupt+0xa6/0x270 [12760.068009] ? apic_timer_interrupt+0xf/0x20 [12760.068013] [12760.068016] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12760.068022] do_raw_spin_lock+0x177/0x190 [12760.068027] _raw_spin_lock+0x21/0x30 [12760.068031] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12760.092258] obd_mod_free+0x46/0x140 [ptlrpc] [12760.092663] mdc_commit_open+0x91/0xb0 [mdc] [12760.092713] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12760.093161] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12760.093389] after_reply+0xe2c/0x1780 [ptlrpc] [12760.093623] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12760.093822] ? __next_timer_interrupt+0x160/0x160 [12760.093832] ptlrpcd+0xa82/0xe40 [ptlrpc] [12760.105564] ? do_raw_spin_unlock+0x75/0x190 [12760.111302] ? do_wait_intr+0xf0/0xf0 [12760.111322] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12760.111658] kthread+0x1d1/0x200 [12760.111664] ? set_kthread_struct+0x70/0x70 [12760.111668] ret_from_fork+0x1f/0x30 [12760.359321] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12760.363788] ? do_wait_intr+0xf0/0xf0 [12760.366409] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12760.369336] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12760.372052] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12760.376997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.380813] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12760.388956] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12760.394657] mdc_intent_lock+0x353/0x7e0 [mdc] [12760.399158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.406099] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12760.408806] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12760.411112] lmv_intent_open+0x338/0x1150 [lmv] [12760.415082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.417705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.420643] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12760.426989] ll_intent_lock+0xc5/0xaa0 [lustre] [12760.429873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.431850] ll_lookup_it+0xe0e/0x2270 [lustre] [12760.433830] ? do_raw_spin_unlock+0x75/0x190 [12760.435288] ? _raw_spin_unlock+0x12/0x30 [12760.436649] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12760.439032] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12760.441906] ll_atomic_open+0x36a/0xd70 [lustre] [12760.444433] lookup_open+0x356/0xb40 [12760.446102] do_last+0x313/0xfc0 [12760.449478] ? nd_jump_root+0xe5/0x160 [12760.451005] ? path_init+0x437/0x520 [12760.454002] path_openat+0xf7/0x500 [12760.455879] ? do_raw_spin_unlock+0x75/0x190 [12760.458873] do_filp_open+0x99/0x140 [12760.460328] ? getname_flags+0x6e/0x330 [12760.461819] ? __check_object_size+0xff/0x256 [12760.463320] ? do_raw_spin_unlock+0x75/0x190 [12760.464739] ? _raw_spin_unlock+0x12/0x30 [12760.465784] do_sys_openat2+0x2b4/0x410 [12760.466884] do_sys_open+0x73/0xa0 [12760.470274] __x64_sys_openat+0x24/0x30 [12760.473359] do_syscall_64+0xc1/0x440 [12760.475071] entry_SYSCALL_64_after_hwframe+0x49/0xae [12760.479356] RIP: 0033:0x7fa749552332 [12760.482031] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12760.490666] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12760.494019] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12760.500716] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12760.504123] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12760.507008] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12760.511533] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12764.645580] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10847s! [12764.651056] Showing busy workqueues and worker pools: [12764.653350] workqueue events: flags=0x0 [12764.655305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12764.661671] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12764.666596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12764.672031] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12764.680725] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.687719] pending: kfree_rcu_monitor [12764.691397] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12764.696039] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12764.705448] workqueue events_long: flags=0x0 [12764.709203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.714529] pending: lu_objects_flush [obdclass] [12764.721048] workqueue events_unbound: flags=0x2 [12764.726404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12764.732506] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12764.740563] workqueue events_power_efficient: flags=0x80 [12764.747679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.750744] pending: check_lifetime [12764.754987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.763187] pending: neigh_periodic_work [12764.767944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.776114] pending: neigh_periodic_work [12764.779613] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.785862] pending: do_cache_clean [12764.787735] workqueue mm_percpu_wq: flags=0x8 [12764.793355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.801790] pending: vmstat_update [12764.803604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.809313] pending: vmstat_update [12764.814449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.822596] pending: vmstat_update [12764.825679] workqueue writeback: flags=0x4a [12764.831630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12764.843090] pending: wb_workfn, wb_workfn [12764.847303] workqueue xprtiod: flags=0xa [12764.851903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12764.857943] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12764.865543] workqueue ipv6_addrconf: flags=0x40008 [12764.872247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12764.879329] pending: addrconf_verify_work [12764.882511] workqueue knbd0-recv: flags=0x1a [12764.885042] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12764.890619] in-flight: 651:recv_work [12764.896635] workqueue ptlrpc_pinger: flags=0x2 [12764.901092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12764.905739] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12764.910232] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10847s workers=4 idle: 37 31846 28897 [12764.919558] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12754s workers=2 idle: 653 [12768.059722] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12768.066804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12768.106190] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12768.118030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12768.123675] RIP: 0010:native_safe_halt+0x2e/0x50 [12768.127398] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12768.142765] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12768.150701] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12768.156932] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12768.162483] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12768.166208] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12768.171905] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12768.181291] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12768.186500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12768.189654] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12768.193124] Call Trace: [12768.193924] [12768.197870] ? show_regs.cold.9+0x22/0x2f [12768.200180] ? watchdog_timer_fn.cold.6+0x62/0x10a [12768.204443] ? __hrtimer_run_queues+0x1c2/0x4d0 [12768.205971] ? watchdog+0x40/0x40 [12768.207003] ? hrtimer_interrupt+0x128/0x2c0 [12768.217331] ? smp_apic_timer_interrupt+0xa6/0x270 [12768.225137] ? apic_timer_interrupt+0xf/0x20 [12768.229338] [12768.231239] ? native_safe_halt+0x2e/0x50 [12768.233204] kvm_wait+0xcd/0xe0 [12768.234731] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12768.237409] do_raw_spin_lock+0x177/0x190 [12768.239549] _raw_spin_lock+0x21/0x30 [12768.243175] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12768.252756] ? do_raw_spin_unlock+0x75/0x190 [12768.258257] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12768.267241] ? do_wait_intr+0xf0/0xf0 [12768.274718] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12768.279660] mdc_close+0x53b/0xd70 [mdc] [12768.286692] lmv_close+0x18d/0x490 [lmv] [12768.289534] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12768.297857] ll_md_real_close+0xa5/0x2a0 [lustre] [12768.301710] ll_md_close+0x274/0x800 [lustre] [12768.304050] ll_file_release+0x128/0xc50 [lustre] [12768.306483] __fput+0x10d/0x440 [12768.308197] ____fput+0x12/0x20 [12768.309957] task_work_run+0xc6/0x110 [12768.311471] exit_to_usermode_loop+0x1dd/0x1f0 [12768.317512] do_syscall_64+0x3d6/0x440 [12768.319318] entry_SYSCALL_64_after_hwframe+0x49/0xae [12768.322271] RIP: 0033:0x7fa749552cd8 [12768.326985] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12768.348230] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12768.354658] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12768.364722] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12768.368792] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12768.373178] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12768.378532] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12773.928060] rcu: INFO: rcu_sched self-detected stall on CPU [12773.932684] rcu: 2-...!: (6175192 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [12773.940444] (t=10860239 jiffies g=258701 q=4599) [12773.942600] rcu: rcu_sched kthread starved for 10857034 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12773.948221] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12773.953867] rcu: RCU grace-period kthread stack dump: [12773.957499] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12773.963806] Call Trace: [12773.965154] __schedule+0x351/0xcb0 [12773.968646] schedule+0xc0/0x180 [12773.970608] schedule_timeout+0xb4/0x190 [12773.972374] ? do_raw_spin_unlock+0x75/0x190 [12773.974147] ? __next_timer_interrupt+0x160/0x160 [12773.976131] ? prepare_to_swait_event+0xf3/0x140 [12773.977825] rcu_gp_kthread+0x8dd/0x15b0 [12773.979532] ? force_qs_rnp+0x2c0/0x2c0 [12773.981243] kthread+0x1d1/0x200 [12773.982662] ? set_kthread_struct+0x70/0x70 [12773.984667] ret_from_fork+0x1f/0x30 [12773.986365] rcu: Stack dump where RCU GP kthread last ran: [12773.988515] Sending NMI from CPU 2 to CPUs 0: [12773.993856] NMI backtrace for cpu 0 [12773.993864] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12773.993866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12773.993868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12773.993870] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12773.993873] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 [12773.993875] RAX: 00000000000047df RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12773.993877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12773.993879] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12773.993880] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12773.993882] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12773.993884] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12773.993886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12773.993887] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12773.993888] Call Trace: [12773.993889] [12773.993890] ? show_regs.cold.9+0x22/0x2f [12773.993891] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12773.993893] ? nmi_cpu_backtrace_handler+0x15/0x40 [12773.993894] ? nmi_handle+0xbd/0x210 [12773.993895] ? default_do_nmi+0x4b/0x1b0 [12773.993896] ? do_nmi+0x280/0x380 [12773.993898] ? end_repeat_nmi+0x16/0x1f [12773.993899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12773.993900] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12773.993902] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12773.993903] [12773.993904] do_raw_spin_lock+0x177/0x190 [12773.993905] _raw_spin_lock+0x21/0x30 [12773.993906] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12773.993908] ? do_raw_spin_unlock+0x75/0x190 [12773.993909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12773.993911] ? do_wait_intr+0xf0/0xf0 [12773.993912] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12773.993913] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12773.993914] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12773.993916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12773.993917] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12773.993918] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12773.993920] mdc_intent_lock+0x353/0x7e0 [mdc] [12773.993922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12773.993923] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12773.993924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12773.993925] lmv_intent_open+0x338/0x1150 [lmv] [12773.993926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12773.993927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12773.993929] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12773.993930] ll_intent_lock+0xc5/0xaa0 [lustre] [12773.993931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12773.993932] ll_lookup_it+0xe0e/0x2270 [lustre] [12773.993940] ? do_raw_spin_unlock+0x75/0x190 [12773.993942] ? _raw_spin_unlock+0x12/0x30 [12773.993943] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12773.993944] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12773.993945] ll_atomic_open+0x36a/0xd70 [lustre] [12773.993946] lookup_open+0x356/0xb40 [12773.993947] do_last+0x313/0xfc0 [12773.993949] ? nd_jump_root+0xe5/0x160 [12773.993950] ? path_init+0x437/0x520 [12773.993951] path_openat+0xf7/0x500 [12773.993952] ? do_raw_spin_unlock+0x75/0x190 [12773.993953] do_filp_open+0x99/0x140 [12773.993954] ? getname_flags+0x6e/0x330 [12773.993955] ? __check_object_size+0xff/0x256 [12773.993956] ? do_raw_spin_unlock+0x75/0x190 [12773.993957] ? _raw_spin_unlock+0x12/0x30 [12773.993958] do_sys_openat2+0x2b4/0x410 [12773.993959] do_sys_open+0x73/0xa0 [12773.993961] __x64_sys_openat+0x24/0x30 [12773.993962] do_syscall_64+0xc1/0x440 [12773.993963] entry_SYSCALL_64_after_hwframe+0x49/0xae [12773.993964] RIP: 0033:0x7fa749552332 [12773.993966] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12773.993968] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12773.993971] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12773.993972] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12773.993974] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12773.993975] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12773.993976] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12773.994161] NMI backtrace for cpu 2 [12774.355513] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12774.366470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12774.371026] Call Trace: [12774.371833] [12774.381614] ? dump_stack+0xbb/0x10e [12774.386439] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12774.395613] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12774.405567] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12774.416614] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12774.420748] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12774.425163] ? rcu_dump_cpu_stacks+0x11b/0x171 [12774.427766] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12774.431138] ? update_process_times+0x6d/0xc0 [12774.434902] ? tick_sched_handle.isra.18+0x27/0x90 [12774.437157] ? tick_sched_timer+0x53/0xb0 [12774.439661] ? __hrtimer_run_queues+0x1c2/0x4d0 [12774.443091] ? tick_sched_handle.isra.18+0x90/0x90 [12774.446798] ? hrtimer_interrupt+0x128/0x2c0 [12774.451873] ? smp_apic_timer_interrupt+0xa6/0x270 [12774.456689] ? apic_timer_interrupt+0xf/0x20 [12774.460661] [12774.461691] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12774.464558] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12774.467571] ? do_raw_spin_lock+0x177/0x190 [12774.469682] ? _raw_spin_lock+0x21/0x30 [12774.471646] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12774.474573] ? obd_mod_free+0x46/0x140 [ptlrpc] [12774.477456] ? mdc_commit_open+0x91/0xb0 [mdc] [12774.480042] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12774.483464] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12774.487041] ? after_reply+0xe2c/0x1780 [ptlrpc] [12774.489614] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12774.493507] ? __next_timer_interrupt+0x160/0x160 [12774.495296] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12774.497027] ? do_wait_intr+0xf0/0xf0 [12774.498236] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12774.501386] ? kthread+0x1d1/0x200 [12774.504052] ? set_kthread_struct+0x70/0x70 [12774.506590] ? ret_from_fork+0x1f/0x30 [12784.060321] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12784.063960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12784.092853] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12784.097262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12784.100882] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12784.102671] RIP: 0010:native_safe_halt+0x2e/0x50 [12784.104249] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12784.109465] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12784.112470] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12784.114984] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12784.117322] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12784.119978] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12784.121790] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12784.124031] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12784.125968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12784.134359] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12784.136857] Call Trace: [12784.137567] [12784.141718] ? show_regs.cold.9+0x22/0x2f [12784.142978] ? watchdog_timer_fn.cold.6+0x62/0x10a [12784.145439] ? __hrtimer_run_queues+0x1c2/0x4d0 [12784.150838] ? watchdog+0x40/0x40 [12784.154153] ? hrtimer_interrupt+0x128/0x2c0 [12784.158629] ? smp_apic_timer_interrupt+0xa6/0x270 [12784.164210] ? apic_timer_interrupt+0xf/0x20 [12784.167333] [12784.168077] ? native_safe_halt+0x2e/0x50 [12784.171458] kvm_wait+0xcd/0xe0 [12784.176110] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12784.177580] do_raw_spin_lock+0x177/0x190 [12784.178784] _raw_spin_lock+0x21/0x30 [12784.182698] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12784.186852] process_one_work+0x2c8/0x700 [12784.188401] worker_thread+0x296/0x6e0 [12784.190199] ? rescuer_thread+0x570/0x570 [12784.191358] kthread+0x1d1/0x200 [12784.192371] ? set_kthread_struct+0x70/0x70 [12784.194023] ret_from_fork+0x1f/0x30 [12795.361052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10878s! [12795.364612] Showing busy workqueues and worker pools: [12795.367378] workqueue events: flags=0x0 [12795.373582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12795.378009] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12795.395340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12795.405867] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12795.413878] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.418994] pending: kfree_rcu_monitor [12795.422179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12795.425337] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12795.434350] workqueue events_long: flags=0x0 [12795.437338] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.441732] pending: lu_objects_flush [obdclass] [12795.444943] workqueue events_unbound: flags=0x2 [12795.450307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12795.454270] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12795.459571] workqueue events_power_efficient: flags=0x80 [12795.466513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.470864] pending: check_lifetime [12795.472385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.476219] pending: neigh_periodic_work [12795.478921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.485398] pending: neigh_periodic_work [12795.487616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.492702] pending: do_cache_clean [12795.494747] workqueue mm_percpu_wq: flags=0x8 [12795.497710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.502014] pending: vmstat_update [12795.506619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.511926] pending: vmstat_update [12795.515024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.518857] pending: vmstat_update [12795.526025] workqueue writeback: flags=0x4a [12795.528931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12795.532687] pending: wb_workfn, wb_workfn [12795.534314] workqueue xprtiod: flags=0xa [12795.537319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12795.546177] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12795.554322] workqueue ipv6_addrconf: flags=0x40008 [12795.559984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12795.568899] pending: addrconf_verify_work [12795.574312] workqueue knbd0-recv: flags=0x1a [12795.576927] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12795.583983] in-flight: 651:recv_work [12795.586620] workqueue ptlrpc_pinger: flags=0x2 [12795.588051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12795.596278] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12795.601811] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10878s workers=4 idle: 37 31846 28897 [12795.611265] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12785s workers=2 idle: 653 [12796.059017] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12796.063633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12796.078370] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12796.082365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12796.085601] RIP: 0010:native_safe_halt+0x2e/0x50 [12796.092775] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12796.104029] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12796.111223] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12796.118747] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12796.124888] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12796.128527] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12796.139495] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12796.147554] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12796.153337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12796.156099] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12796.161302] Call Trace: [12796.162115] [12796.162568] ? show_regs.cold.9+0x22/0x2f [12796.169461] ? watchdog_timer_fn.cold.6+0x62/0x10a [12796.173679] ? __hrtimer_run_queues+0x1c2/0x4d0 [12796.179048] ? watchdog+0x40/0x40 [12796.180162] ? hrtimer_interrupt+0x128/0x2c0 [12796.183451] ? smp_apic_timer_interrupt+0xa6/0x270 [12796.189575] ? apic_timer_interrupt+0xf/0x20 [12796.195011] [12796.197325] ? native_safe_halt+0x2e/0x50 [12796.199010] kvm_wait+0xcd/0xe0 [12796.202412] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12796.207125] do_raw_spin_lock+0x177/0x190 [12796.209327] _raw_spin_lock+0x21/0x30 [12796.210634] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12796.212644] ? do_raw_spin_unlock+0x75/0x190 [12796.215140] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12796.222353] ? do_wait_intr+0xf0/0xf0 [12796.226420] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12796.228898] mdc_close+0x53b/0xd70 [mdc] [12796.233325] lmv_close+0x18d/0x490 [lmv] [12796.235611] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12796.241692] ll_md_real_close+0xa5/0x2a0 [lustre] [12796.246129] ll_md_close+0x274/0x800 [lustre] [12796.249356] ll_file_release+0x128/0xc50 [lustre] [12796.251439] __fput+0x10d/0x440 [12796.256017] ____fput+0x12/0x20 [12796.257218] task_work_run+0xc6/0x110 [12796.262407] exit_to_usermode_loop+0x1dd/0x1f0 [12796.264113] do_syscall_64+0x3d6/0x440 [12796.265231] entry_SYSCALL_64_after_hwframe+0x49/0xae [12796.267175] RIP: 0033:0x7fa749552cd8 [12796.268462] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12796.285481] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12796.293278] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12796.305892] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12796.311078] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12796.315286] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12796.324628] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12800.058009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12800.060249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12800.073737] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12800.077396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12800.080206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12800.081990] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12800.088952] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12800.091345] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12800.094348] RDX: 0000000000002092 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12800.096459] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12800.098765] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12800.101678] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12800.104338] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12800.107402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12800.109562] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12800.112102] Call Trace: [12800.112828] [12800.113651] ? show_regs.cold.9+0x22/0x2f [12800.114746] ? watchdog_timer_fn.cold.6+0x62/0x10a [12800.116643] ? __hrtimer_run_queues+0x1c2/0x4d0 [12800.118233] ? watchdog+0x40/0x40 [12800.119482] ? hrtimer_interrupt+0x128/0x2c0 [12800.120860] ? smp_apic_timer_interrupt+0xa6/0x270 [12800.122372] ? apic_timer_interrupt+0xf/0x20 [12800.123651] [12800.124444] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12800.125950] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12800.127659] do_raw_spin_lock+0x177/0x190 [12800.128786] _raw_spin_lock+0x21/0x30 [12800.129954] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12800.131845] obd_mod_free+0x46/0x140 [ptlrpc] [12800.133343] mdc_commit_open+0x91/0xb0 [mdc] [12800.134654] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12800.136231] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12800.138263] after_reply+0xe2c/0x1780 [ptlrpc] [12800.139878] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12800.141600] ? __next_timer_interrupt+0x160/0x160 [12800.143223] ptlrpcd+0xa82/0xe40 [ptlrpc] [12800.144775] ? do_wait_intr+0xf0/0xf0 [12800.146087] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12800.148213] kthread+0x1d1/0x200 [12800.149567] ? set_kthread_struct+0x70/0x70 [12800.151219] ret_from_fork+0x1f/0x30 [12812.057631] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12812.063339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12812.082825] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12812.087964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12812.091372] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12812.094561] RIP: 0010:native_safe_halt+0x2e/0x50 [12812.096354] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12812.103006] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12812.105974] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12812.108852] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12812.111103] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12812.113419] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12812.115414] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12812.118001] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12812.121215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12812.122990] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12812.125606] Call Trace: [12812.126592] [12812.127361] ? show_regs.cold.9+0x22/0x2f [12812.128609] ? watchdog_timer_fn.cold.6+0x62/0x10a [12812.130701] ? __hrtimer_run_queues+0x1c2/0x4d0 [12812.133381] ? watchdog+0x40/0x40 [12812.134627] ? hrtimer_interrupt+0x128/0x2c0 [12812.137209] ? smp_apic_timer_interrupt+0xa6/0x270 [12812.139216] ? apic_timer_interrupt+0xf/0x20 [12812.142066] [12812.142865] ? native_safe_halt+0x2e/0x50 [12812.144539] kvm_wait+0xcd/0xe0 [12812.146244] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12812.148528] do_raw_spin_lock+0x177/0x190 [12812.150521] _raw_spin_lock+0x21/0x30 [12812.152456] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12812.154765] process_one_work+0x2c8/0x700 [12812.157392] worker_thread+0x296/0x6e0 [12812.160254] ? rescuer_thread+0x570/0x570 [12812.162153] kthread+0x1d1/0x200 [12812.166524] ? set_kthread_struct+0x70/0x70 [12812.169165] ret_from_fork+0x1f/0x30 [12820.054091] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12820.060442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12820.095705] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12820.101282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12820.111349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12820.113476] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12820.140873] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12820.145712] RAX: 0000000000007e90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12820.150005] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12820.158607] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12820.164036] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12820.172689] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12820.181988] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12820.193573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12820.195320] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12820.212483] Call Trace: [12820.214385] [12820.215659] ? show_regs.cold.9+0x22/0x2f [12820.217158] ? watchdog_timer_fn.cold.6+0x62/0x10a [12820.224942] ? __hrtimer_run_queues+0x1c2/0x4d0 [12820.230141] ? watchdog+0x40/0x40 [12820.233311] ? hrtimer_interrupt+0x128/0x2c0 [12820.237298] ? smp_apic_timer_interrupt+0xa6/0x270 [12820.239110] ? apic_timer_interrupt+0xf/0x20 [12820.246988] [12820.251008] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12820.255618] do_raw_spin_lock+0x177/0x190 [12820.260434] _raw_spin_lock+0x21/0x30 [12820.264003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12820.266095] ? do_raw_spin_unlock+0x75/0x190 [12820.274375] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12820.278493] ? do_wait_intr+0xf0/0xf0 [12820.282055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12820.285627] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12820.292545] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12820.298267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12820.303530] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12820.307491] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12820.310907] mdc_intent_lock+0x353/0x7e0 [mdc] [12820.313592] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12820.321504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12820.330496] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12820.334513] lmv_intent_open+0x338/0x1150 [lmv] [12820.336489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12820.345756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12820.352060] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12820.357734] ll_intent_lock+0xc5/0xaa0 [lustre] [12820.362313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12820.366280] ll_lookup_it+0xe0e/0x2270 [lustre] [12820.375530] ? do_raw_spin_unlock+0x75/0x190 [12820.382002] ? _raw_spin_unlock+0x12/0x30 [12820.384618] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12820.392148] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12820.398934] ll_atomic_open+0x36a/0xd70 [lustre] [12820.404022] lookup_open+0x356/0xb40 [12820.405849] do_last+0x313/0xfc0 [12820.407092] ? nd_jump_root+0xe5/0x160 [12820.410412] ? path_init+0x437/0x520 [12820.413904] path_openat+0xf7/0x500 [12820.420778] ? do_raw_spin_unlock+0x75/0x190 [12820.424716] do_filp_open+0x99/0x140 [12820.425992] ? getname_flags+0x6e/0x330 [12820.427377] ? __check_object_size+0xff/0x256 [12820.437271] ? do_raw_spin_unlock+0x75/0x190 [12820.442374] ? _raw_spin_unlock+0x12/0x30 [12820.448047] do_sys_openat2+0x2b4/0x410 [12820.451420] do_sys_open+0x73/0xa0 [12820.453446] __x64_sys_openat+0x24/0x30 [12820.454779] do_syscall_64+0xc1/0x440 [12820.456056] entry_SYSCALL_64_after_hwframe+0x49/0xae [12820.461931] RIP: 0033:0x7fa749552332 [12820.463853] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12820.474945] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12820.479187] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12820.483833] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12820.486944] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12820.489324] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12820.492840] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12824.060803] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [12824.077908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12824.148494] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12824.161601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12824.176676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12824.184814] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12824.202579] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12824.207746] RAX: 0000000000001ec6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12824.215956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12824.230193] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12824.242925] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12824.260715] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12824.265257] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12824.275975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12824.281448] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12824.293896] Call Trace: [12824.299199] [12824.299971] ? show_regs.cold.9+0x22/0x2f [12824.301416] ? watchdog_timer_fn.cold.6+0x62/0x10a [12824.312254] ? __hrtimer_run_queues+0x1c2/0x4d0 [12824.318638] ? watchdog+0x40/0x40 [12824.324654] ? hrtimer_interrupt+0x128/0x2c0 [12824.331653] ? smp_apic_timer_interrupt+0xa6/0x270 [12824.334842] ? apic_timer_interrupt+0xf/0x20 [12824.344944] [12824.346892] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12824.354247] do_raw_spin_lock+0x177/0x190 [12824.358290] _raw_spin_lock+0x21/0x30 [12824.363281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12824.368063] ? do_raw_spin_unlock+0x75/0x190 [12824.370966] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12824.382673] ? do_wait_intr+0xf0/0xf0 [12824.387861] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12824.394145] mdc_close+0x53b/0xd70 [mdc] [12824.396328] lmv_close+0x18d/0x490 [lmv] [12824.409276] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12824.418926] ll_md_real_close+0xa5/0x2a0 [lustre] [12824.425402] ll_md_close+0x274/0x800 [lustre] [12824.429521] ll_file_release+0x128/0xc50 [lustre] [12824.437972] __fput+0x10d/0x440 [12824.440170] ____fput+0x12/0x20 [12824.441347] task_work_run+0xc6/0x110 [12824.447260] exit_to_usermode_loop+0x1dd/0x1f0 [12824.449594] do_syscall_64+0x3d6/0x440 [12824.450837] entry_SYSCALL_64_after_hwframe+0x49/0xae [12824.453821] RIP: 0033:0x7fa749552cd8 [12824.456839] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12824.485946] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12824.488286] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12824.503951] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12824.512207] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12824.515647] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12824.520738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12826.084189] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10908s! [12826.088655] Showing busy workqueues and worker pools: [12826.093530] workqueue events: flags=0x0 [12826.096023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12826.101843] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12826.119578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12826.132482] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12826.141600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.157906] pending: kfree_rcu_monitor [12826.163336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12826.171403] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12826.176466] workqueue events_long: flags=0x0 [12826.179752] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.185823] pending: lu_objects_flush [obdclass] [12826.189795] workqueue events_unbound: flags=0x2 [12826.194724] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12826.197846] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12826.209779] workqueue events_power_efficient: flags=0x80 [12826.222685] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.228531] pending: check_lifetime [12826.238544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.243669] pending: neigh_periodic_work [12826.248529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.258539] pending: neigh_periodic_work [12826.261640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.268348] pending: do_cache_clean [12826.270247] workqueue mm_percpu_wq: flags=0x8 [12826.273677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.281586] pending: vmstat_update [12826.283093] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.288687] pending: vmstat_update [12826.290064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.297383] pending: vmstat_update [12826.300884] workqueue writeback: flags=0x4a [12826.307360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12826.313532] pending: wb_workfn, wb_workfn [12826.315077] workqueue xprtiod: flags=0xa [12826.319326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12826.327944] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12826.337165] workqueue ipv6_addrconf: flags=0x40008 [12826.340857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12826.352838] pending: addrconf_verify_work [12826.355330] workqueue knbd0-recv: flags=0x1a [12826.358598] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12826.365925] in-flight: 651:recv_work [12826.367655] workqueue ptlrpc_pinger: flags=0x2 [12826.369564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12826.374962] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12826.381673] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10909s workers=4 idle: 37 31846 28897 [12826.387951] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12816s workers=2 idle: 653 [12828.062118] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12828.077285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12828.126725] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12828.135579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12828.142356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12828.152289] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12828.177867] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12828.194450] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12828.200593] RDX: 0000000000001dc3 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12828.203898] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12828.217075] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12828.223214] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12828.239888] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12828.250534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12828.252572] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12828.263305] Call Trace: [12828.270600] [12828.271478] ? show_regs.cold.9+0x22/0x2f [12828.281492] ? watchdog_timer_fn.cold.6+0x62/0x10a [12828.285726] ? __hrtimer_run_queues+0x1c2/0x4d0 [12828.288833] ? watchdog+0x40/0x40 [12828.293900] ? hrtimer_interrupt+0x128/0x2c0 [12828.299288] ? smp_apic_timer_interrupt+0xa6/0x270 [12828.309061] ? apic_timer_interrupt+0xf/0x20 [12828.314155] [12828.314801] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12828.318655] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12828.323482] do_raw_spin_lock+0x177/0x190 [12828.328243] _raw_spin_lock+0x21/0x30 [12828.333311] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12828.343009] obd_mod_free+0x46/0x140 [ptlrpc] [12828.349322] mdc_commit_open+0x91/0xb0 [mdc] [12828.352658] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12828.359131] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12828.369363] after_reply+0xe2c/0x1780 [ptlrpc] [12828.378715] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12828.384788] ? __next_timer_interrupt+0x160/0x160 [12828.390614] ptlrpcd+0xa82/0xe40 [ptlrpc] [12828.393336] ? do_wait_intr+0xf0/0xf0 [12828.399195] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12828.405290] kthread+0x1d1/0x200 [12828.406431] ? set_kthread_struct+0x70/0x70 [12828.407792] ret_from_fork+0x1f/0x30 [12840.068347] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12840.083877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12840.157190] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12840.171413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12840.179031] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12840.192793] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12840.209110] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12840.240144] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12840.248229] RAX: 0000000000003484 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12840.255129] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12840.260242] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12840.264289] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12840.269144] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12840.272136] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12840.275486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12840.278048] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12840.280685] Call Trace: [12840.282009] [12840.282853] ? show_regs.cold.9+0x22/0x2f [12840.284460] ? watchdog_timer_fn.cold.6+0x62/0x10a [12840.287108] ? __hrtimer_run_queues+0x1c2/0x4d0 [12840.289103] ? watchdog+0x40/0x40 [12840.290534] ? hrtimer_interrupt+0x128/0x2c0 [12840.293095] ? smp_apic_timer_interrupt+0xa6/0x270 [12840.295466] ? apic_timer_interrupt+0xf/0x20 [12840.297403] [12840.298318] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12840.301202] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12840.304717] do_raw_spin_lock+0x177/0x190 [12840.306428] _raw_spin_lock+0x21/0x30 [12840.308050] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12840.312096] process_one_work+0x2c8/0x700 [12840.314318] worker_thread+0x296/0x6e0 [12840.316561] ? rescuer_thread+0x570/0x570 [12840.319289] kthread+0x1d1/0x200 [12840.321228] ? set_kthread_struct+0x70/0x70 [12840.323182] ret_from_fork+0x1f/0x30 [12852.054398] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [12852.057601] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12852.059013] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12852.086826] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12852.092017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12852.093913] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12852.093921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12852.093924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12852.093937] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12852.093941] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12852.093946] RAX: 0000000000004297 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12852.093948] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12852.093951] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12852.093953] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12852.093955] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12852.093959] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12852.093962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12852.093964] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12852.093976] Call Trace: [12852.093982] [12852.093988] ? show_regs.cold.9+0x22/0x2f [12852.093999] ? watchdog_timer_fn.cold.6+0x62/0x10a [12852.094005] ? __hrtimer_run_queues+0x1c2/0x4d0 [12852.094010] ? watchdog+0x40/0x40 [12852.094015] ? hrtimer_interrupt+0x128/0x2c0 [12852.094019] ? smp_apic_timer_interrupt+0xa6/0x270 [12852.094029] ? apic_timer_interrupt+0xf/0x20 [12852.094035] [12852.094038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12852.094046] do_raw_spin_lock+0x177/0x190 [12852.094051] _raw_spin_lock+0x21/0x30 [12852.094509] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12852.094853] ? do_raw_spin_unlock+0x75/0x190 [12852.094860] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12852.095116] ? do_wait_intr+0xf0/0xf0 [12852.095129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12852.099901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12852.099914] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12852.099928] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12852.099933] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 [12852.131402] mdc_close+0x53b/0xd70 [mdc] [12852.131454] lmv_close+0x18d/0x490 [lmv] [12852.145769] ORIG_RAX: ffffffffffffff13 [12852.145781] RAX: 0000000000002c40 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12852.145783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12852.145786] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12852.145788] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12852.145790] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12852.145793] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12852.145797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12852.145799] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12852.145819] Call Trace: [12852.145827] [12852.145833] ? show_regs.cold.9+0x22/0x2f [12852.145847] ? watchdog_timer_fn.cold.6+0x62/0x10a [12852.145853] ? __hrtimer_run_queues+0x1c2/0x4d0 [12852.145857] ? watchdog+0x40/0x40 [12852.145861] ? hrtimer_interrupt+0x128/0x2c0 [12852.145864] ? smp_apic_timer_interrupt+0xa6/0x270 [12852.145872] ? apic_timer_interrupt+0xf/0x20 [12852.145876] [12852.145879] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12852.145889] do_raw_spin_lock+0x177/0x190 [12852.145893] _raw_spin_lock+0x21/0x30 [12852.145898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12852.153182] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12852.153328] ll_md_real_close+0xa5/0x2a0 [lustre] [12852.153712] ll_md_close+0x274/0x800 [lustre] [12852.153798] ll_file_release+0x128/0xc50 [lustre] [12852.153888] __fput+0x10d/0x440 [12852.153898] ____fput+0x12/0x20 [12852.153902] task_work_run+0xc6/0x110 [12852.153912] exit_to_usermode_loop+0x1dd/0x1f0 [12852.153919] do_syscall_64+0x3d6/0x440 [12852.153923] entry_SYSCALL_64_after_hwframe+0x49/0xae [12852.153933] RIP: 0033:0x7fa749552cd8 [12852.153940] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12852.153945] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12852.153950] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12852.153952] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12852.153954] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12852.153957] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12852.153959] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12852.455407] ? do_raw_spin_unlock+0x75/0x190 [12852.456738] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12852.462354] ? do_wait_intr+0xf0/0xf0 [12852.467163] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12852.474573] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12852.480757] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12852.482425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12852.487563] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12852.489592] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12852.494407] mdc_intent_lock+0x353/0x7e0 [mdc] [12852.497095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12852.499434] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12852.501370] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12852.505912] lmv_intent_open+0x338/0x1150 [lmv] [12852.507462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12852.511229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12852.516333] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12852.522607] ll_intent_lock+0xc5/0xaa0 [lustre] [12852.525610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12852.527704] ll_lookup_it+0xe0e/0x2270 [lustre] [12852.529894] ? do_raw_spin_unlock+0x75/0x190 [12852.530997] ? _raw_spin_unlock+0x12/0x30 [12852.536931] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12852.538449] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12852.554199] ll_atomic_open+0x36a/0xd70 [lustre] [12852.555619] lookup_open+0x356/0xb40 [12852.556596] do_last+0x313/0xfc0 [12852.557448] ? nd_jump_root+0xe5/0x160 [12852.559666] ? path_init+0x437/0x520 [12852.560749] path_openat+0xf7/0x500 [12852.563112] ? do_raw_spin_unlock+0x75/0x190 [12852.570296] do_filp_open+0x99/0x140 [12852.572898] ? getname_flags+0x6e/0x330 [12852.574990] ? __check_object_size+0xff/0x256 [12852.582880] ? do_raw_spin_unlock+0x75/0x190 [12852.587636] ? _raw_spin_unlock+0x12/0x30 [12852.588812] do_sys_openat2+0x2b4/0x410 [12852.591128] do_sys_open+0x73/0xa0 [12852.592086] __x64_sys_openat+0x24/0x30 [12852.593038] do_syscall_64+0xc1/0x440 [12852.599080] entry_SYSCALL_64_after_hwframe+0x49/0xae [12852.602473] RIP: 0033:0x7fa749552332 [12852.603765] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12852.622837] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12852.630168] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12852.632699] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12852.637198] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12852.641965] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12852.646630] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12856.058016] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12856.060891] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12856.098852] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12856.108909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12856.112351] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12856.114987] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12856.128476] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12856.138334] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12856.151754] RDX: 00000000000013c8 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12856.157756] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12856.163424] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12856.170719] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12856.175786] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12856.179261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12856.184650] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12856.189682] Call Trace: [12856.190752] [12856.192827] ? show_regs.cold.9+0x22/0x2f [12856.194675] ? watchdog_timer_fn.cold.6+0x62/0x10a [12856.197024] ? __hrtimer_run_queues+0x1c2/0x4d0 [12856.198788] ? watchdog+0x40/0x40 [12856.201786] ? hrtimer_interrupt+0x128/0x2c0 [12856.203196] ? smp_apic_timer_interrupt+0xa6/0x270 [12856.207136] ? apic_timer_interrupt+0xf/0x20 [12856.209813] [12856.210490] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12856.212907] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12856.216609] do_raw_spin_lock+0x177/0x190 [12856.218808] _raw_spin_lock+0x21/0x30 [12856.222907] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12856.227081] obd_mod_free+0x46/0x140 [ptlrpc] [12856.230160] mdc_commit_open+0x91/0xb0 [mdc] [12856.231455] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12856.233516] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12856.235406] after_reply+0xe2c/0x1780 [ptlrpc] [12856.239734] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12856.242339] ? __next_timer_interrupt+0x160/0x160 [12856.246775] ptlrpcd+0xa82/0xe40 [ptlrpc] [12856.249223] ? do_wait_intr+0xf0/0xf0 [12856.250929] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12856.253773] kthread+0x1d1/0x200 [12856.255170] ? set_kthread_struct+0x70/0x70 [12856.261182] ret_from_fork+0x1f/0x30 [12856.801677] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10939s! [12856.804817] Showing busy workqueues and worker pools: [12856.808966] workqueue events: flags=0x0 [12856.811421] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12856.816670] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12856.822721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12856.829625] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12856.834139] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.837112] pending: kfree_rcu_monitor [12856.838766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12856.842784] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12856.847316] workqueue events_long: flags=0x0 [12856.853194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.858888] pending: lu_objects_flush [obdclass] [12856.864326] workqueue events_unbound: flags=0x2 [12856.866391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12856.872532] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12856.885110] workqueue events_power_efficient: flags=0x80 [12856.886616] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.888498] pending: check_lifetime [12856.897988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.900278] pending: neigh_periodic_work [12856.902859] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.906322] pending: neigh_periodic_work [12856.908127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.913380] pending: do_cache_clean [12856.915134] workqueue mm_percpu_wq: flags=0x8 [12856.916838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.919066] pending: vmstat_update [12856.920611] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.922940] pending: vmstat_update [12856.924290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.927205] pending: vmstat_update [12856.929175] workqueue writeback: flags=0x4a [12856.931639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12856.940731] pending: wb_workfn, wb_workfn [12856.942090] workqueue xprtiod: flags=0xa [12856.947936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12856.950372] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12856.955857] workqueue ipv6_addrconf: flags=0x40008 [12856.962346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12856.965580] pending: addrconf_verify_work [12856.969692] workqueue knbd0-recv: flags=0x1a [12856.976137] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12856.984307] in-flight: 651:recv_work [12856.989323] workqueue ptlrpc_pinger: flags=0x2 [12856.990989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12856.993567] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12857.001375] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10939s workers=4 idle: 37 31846 28897 [12857.008405] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12846s workers=2 idle: 653 [12868.058963] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [12868.062954] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12868.086568] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12868.091119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12868.095167] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12868.099026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12868.102250] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12868.111862] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12868.117129] RAX: 00000000000040f9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12868.123640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12868.126654] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12868.130345] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12868.134245] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12868.137137] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12868.141559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12868.144474] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12868.147967] Call Trace: [12868.149779] [12868.150869] ? show_regs.cold.9+0x22/0x2f [12868.153168] ? watchdog_timer_fn.cold.6+0x62/0x10a [12868.155143] ? __hrtimer_run_queues+0x1c2/0x4d0 [12868.156961] ? watchdog+0x40/0x40 [12868.158544] ? hrtimer_interrupt+0x128/0x2c0 [12868.160292] ? smp_apic_timer_interrupt+0xa6/0x270 [12868.162381] ? apic_timer_interrupt+0xf/0x20 [12868.164363] [12868.165504] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12868.168419] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12868.170268] do_raw_spin_lock+0x177/0x190 [12868.172063] _raw_spin_lock+0x21/0x30 [12868.173784] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12868.175897] process_one_work+0x2c8/0x700 [12868.177847] worker_thread+0x296/0x6e0 [12868.179345] ? rescuer_thread+0x570/0x570 [12868.180914] kthread+0x1d1/0x200 [12868.182568] ? set_kthread_struct+0x70/0x70 [12868.184299] ret_from_fork+0x1f/0x30 [12880.059143] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12880.072494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12880.101286] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12880.107310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12880.112700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12880.117097] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12880.127465] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12880.132145] RAX: 0000000000000587 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12880.136815] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12880.140886] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12880.144899] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12880.148805] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12880.152727] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12880.155867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12880.158666] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12880.161879] Call Trace: [12880.162938] [12880.163819] ? show_regs.cold.9+0x22/0x2f [12880.165882] ? watchdog_timer_fn.cold.6+0x62/0x10a [12880.168113] ? __hrtimer_run_queues+0x1c2/0x4d0 [12880.170450] ? watchdog+0x40/0x40 [12880.172248] ? hrtimer_interrupt+0x128/0x2c0 [12880.174714] ? smp_apic_timer_interrupt+0xa6/0x270 [12880.176733] ? apic_timer_interrupt+0xf/0x20 [12880.178270] [12880.179268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12880.181279] do_raw_spin_lock+0x177/0x190 [12880.182877] _raw_spin_lock+0x21/0x30 [12880.184461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12880.187266] ? do_raw_spin_unlock+0x75/0x190 [12880.189795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12880.193182] ? do_wait_intr+0xf0/0xf0 [12880.195027] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12880.198900] mdc_close+0x53b/0xd70 [mdc] [12880.200816] lmv_close+0x18d/0x490 [lmv] [12880.203169] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12880.206300] ll_md_real_close+0xa5/0x2a0 [lustre] [12880.209071] ll_md_close+0x274/0x800 [lustre] [12880.211531] ll_file_release+0x128/0xc50 [lustre] [12880.213986] __fput+0x10d/0x440 [12880.215732] ____fput+0x12/0x20 [12880.217636] task_work_run+0xc6/0x110 [12880.219365] exit_to_usermode_loop+0x1dd/0x1f0 [12880.221842] do_syscall_64+0x3d6/0x440 [12880.227303] entry_SYSCALL_64_after_hwframe+0x49/0xae [12880.229756] RIP: 0033:0x7fa749552cd8 [12880.231961] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12880.246168] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12880.252795] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12880.256772] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12880.262121] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12880.267396] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12880.271594] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12884.054059] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [12884.058043] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12884.059291] Modules linked in: [12884.062710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [12884.064552] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12884.064685] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12884.064693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12884.064696] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12884.064711] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12884.064715] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12884.064720] RAX: 000000000000769a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12884.064722] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12884.064725] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12884.064727] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12884.064729] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12884.064733] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12884.064736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12884.064738] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12884.064747] Call Trace: [12884.064753] [12884.064758] ? show_regs.cold.9+0x22/0x2f [12884.064767] ? watchdog_timer_fn.cold.6+0x62/0x10a [12884.064773] ? __hrtimer_run_queues+0x1c2/0x4d0 [12884.064778] ? watchdog+0x40/0x40 [12884.064782] ? hrtimer_interrupt+0x128/0x2c0 [12884.064787] ? smp_apic_timer_interrupt+0xa6/0x270 [12884.064795] ? apic_timer_interrupt+0xf/0x20 [12884.064800] [12884.064803] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12884.064810] do_raw_spin_lock+0x177/0x190 [12884.064814] _raw_spin_lock+0x21/0x30 [12884.065046] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12884.065324] ? do_raw_spin_unlock+0x75/0x190 [12884.065330] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12884.065729] ? do_wait_intr+0xf0/0xf0 [12884.065738] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12884.066105] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12884.066334] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12884.066704] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.066813] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12884.071314] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12884.071382] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12884.071389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12884.071391] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12884.071405] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12884.071410] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12884.071414] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12884.071416] RDX: 00000000000071a0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12884.071418] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12884.071420] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12884.071422] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12884.071425] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12884.071428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12884.071430] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12884.071438] Call Trace: [12884.071444] [12884.071533] ? show_regs.cold.9+0x22/0x2f [12884.071544] ? watchdog_timer_fn.cold.6+0x62/0x10a [12884.071549] ? __hrtimer_run_queues+0x1c2/0x4d0 [12884.071553] ? watchdog+0x40/0x40 [12884.071558] ? hrtimer_interrupt+0x128/0x2c0 [12884.071562] ? smp_apic_timer_interrupt+0xa6/0x270 [12884.071570] ? apic_timer_interrupt+0xf/0x20 [12884.071575] [12884.071578] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12884.071582] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12884.071587] do_raw_spin_lock+0x177/0x190 [12884.071591] _raw_spin_lock+0x21/0x30 [12884.071597] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12884.072226] obd_mod_free+0x46/0x140 [ptlrpc] [12884.072841] mdc_commit_open+0x91/0xb0 [mdc] [12884.073082] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12884.101999] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12884.102061] mdc_intent_lock+0x353/0x7e0 [mdc] [12884.102092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.102206] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12884.109337] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12884.109729] after_reply+0xe2c/0x1780 [ptlrpc] [12884.110182] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12884.110408] ? __next_timer_interrupt+0x160/0x160 [12884.110422] ptlrpcd+0xa82/0xe40 [ptlrpc] [12884.116760] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12884.116804] lmv_intent_open+0x338/0x1150 [lmv] [12884.117107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.117257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.117348] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12884.117374] ll_intent_lock+0xc5/0xaa0 [lustre] [12884.117482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.117577] ll_lookup_it+0xe0e/0x2270 [lustre] [12884.117664] ? do_raw_spin_unlock+0x75/0x190 [12884.117673] ? _raw_spin_unlock+0x12/0x30 [12884.117679] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12884.117714] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12884.117734] ll_atomic_open+0x36a/0xd70 [lustre] [12884.117930] lookup_open+0x356/0xb40 [12884.117939] do_last+0x313/0xfc0 [12884.117945] ? nd_jump_root+0xe5/0x160 [12884.118079] ? path_init+0x437/0x520 [12884.118085] path_openat+0xf7/0x500 [12884.118090] ? do_raw_spin_unlock+0x75/0x190 [12884.118094] do_filp_open+0x99/0x140 [12884.118099] ? getname_flags+0x6e/0x330 [12884.118104] ? __check_object_size+0xff/0x256 [12884.118111] ? do_raw_spin_unlock+0x75/0x190 [12884.118115] ? _raw_spin_unlock+0x12/0x30 [12884.118120] do_sys_openat2+0x2b4/0x410 [12884.118125] do_sys_open+0x73/0xa0 [12884.118129] __x64_sys_openat+0x24/0x30 [12884.118138] do_syscall_64+0xc1/0x440 [12884.118149] entry_SYSCALL_64_after_hwframe+0x49/0xae [12884.118158] RIP: 0033:0x7fa749552332 [12884.118166] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12884.118170] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12884.118175] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12884.118177] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12884.118179] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12884.118181] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12884.118184] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12884.633623] ? do_wait_intr+0xf0/0xf0 [12884.641792] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12884.645795] kthread+0x1d1/0x200 [12884.647529] ? set_kthread_struct+0x70/0x70 [12884.650721] ret_from_fork+0x1f/0x30 [12887.524576] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10970s! [12887.539561] Showing busy workqueues and worker pools: [12887.541551] workqueue events: flags=0x0 [12887.547136] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12887.555627] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12887.566783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12887.569417] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12887.578170] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.582517] pending: kfree_rcu_monitor [12887.585317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12887.592046] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12887.603379] workqueue events_long: flags=0x0 [12887.608070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.614183] pending: lu_objects_flush [obdclass] [12887.625236] workqueue events_unbound: flags=0x2 [12887.633161] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12887.642276] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12887.650960] workqueue events_power_efficient: flags=0x80 [12887.653988] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.659824] pending: check_lifetime [12887.663026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.673286] pending: neigh_periodic_work [12887.678975] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.684209] pending: neigh_periodic_work [12887.686714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.690693] pending: do_cache_clean [12887.692587] workqueue mm_percpu_wq: flags=0x8 [12887.694047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.698728] pending: vmstat_update [12887.701596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.704706] pending: vmstat_update [12887.708206] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.712012] pending: vmstat_update [12887.714850] workqueue writeback: flags=0x4a [12887.717715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12887.721404] pending: wb_workfn, wb_workfn [12887.723737] workqueue xprtiod: flags=0xa [12887.726054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12887.729031] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12887.733779] workqueue ipv6_addrconf: flags=0x40008 [12887.736848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12887.741300] pending: addrconf_verify_work [12887.744276] workqueue knbd0-recv: flags=0x1a [12887.749719] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12887.752032] in-flight: 651:recv_work [12887.753776] workqueue ptlrpc_pinger: flags=0x2 [12887.756165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12887.759029] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12887.761694] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10970s workers=4 idle: 37 31846 28897 [12887.765202] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12877s workers=2 idle: 653 [12896.061097] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12896.074170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12896.100501] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12896.105351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12896.109077] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12896.112079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12896.119187] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12896.140224] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12896.145738] RAX: 0000000000007f27 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12896.147852] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12896.156306] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12896.164503] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12896.168988] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12896.180106] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12896.185171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12896.188237] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12896.192057] Call Trace: [12896.193277] [12896.194176] ? show_regs.cold.9+0x22/0x2f [12896.196275] ? watchdog_timer_fn.cold.6+0x62/0x10a [12896.199227] ? __hrtimer_run_queues+0x1c2/0x4d0 [12896.208010] ? watchdog+0x40/0x40 [12896.213273] ? hrtimer_interrupt+0x128/0x2c0 [12896.220706] ? smp_apic_timer_interrupt+0xa6/0x270 [12896.229671] ? apic_timer_interrupt+0xf/0x20 [12896.234512] [12896.235245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12896.241810] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12896.243549] do_raw_spin_lock+0x177/0x190 [12896.250441] _raw_spin_lock+0x21/0x30 [12896.252657] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12896.257280] process_one_work+0x2c8/0x700 [12896.262021] worker_thread+0x296/0x6e0 [12896.265233] ? rescuer_thread+0x570/0x570 [12896.270060] kthread+0x1d1/0x200 [12896.271231] ? set_kthread_struct+0x70/0x70 [12896.272714] ret_from_fork+0x1f/0x30 [12908.059008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12908.062920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12908.091343] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12908.096126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12908.099901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12908.102767] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12908.113344] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12908.117398] RAX: 00000000000035ab RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12908.120535] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12908.122869] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12908.126043] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12908.128710] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12908.132015] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12908.135291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12908.137815] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12908.140416] Call Trace: [12908.141066] [12908.141548] ? show_regs.cold.9+0x22/0x2f [12908.142812] ? watchdog_timer_fn.cold.6+0x62/0x10a [12908.144598] ? __hrtimer_run_queues+0x1c2/0x4d0 [12908.146514] ? watchdog+0x40/0x40 [12908.147767] ? hrtimer_interrupt+0x128/0x2c0 [12908.149265] ? smp_apic_timer_interrupt+0xa6/0x270 [12908.151320] ? apic_timer_interrupt+0xf/0x20 [12908.153123] [12908.154120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12908.156762] do_raw_spin_lock+0x177/0x190 [12908.158493] _raw_spin_lock+0x21/0x30 [12908.160223] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12908.163221] ? do_raw_spin_unlock+0x75/0x190 [12908.165176] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12908.167978] ? do_wait_intr+0xf0/0xf0 [12908.173182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12908.180000] mdc_close+0x53b/0xd70 [mdc] [12908.189052] lmv_close+0x18d/0x490 [lmv] [12908.190461] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12908.202353] ll_md_real_close+0xa5/0x2a0 [lustre] [12908.207697] ll_md_close+0x274/0x800 [lustre] [12908.211049] ll_file_release+0x128/0xc50 [lustre] [12908.214467] __fput+0x10d/0x440 [12908.216177] ____fput+0x12/0x20 [12908.217422] task_work_run+0xc6/0x110 [12908.221982] exit_to_usermode_loop+0x1dd/0x1f0 [12908.226339] do_syscall_64+0x3d6/0x440 [12908.228848] entry_SYSCALL_64_after_hwframe+0x49/0xae [12908.235078] RIP: 0033:0x7fa749552cd8 [12908.237337] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12908.253744] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12908.258269] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12908.265696] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12908.269374] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12908.273407] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12908.280434] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12912.057651] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [12912.069912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12912.070496] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [12912.101492] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12912.101506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12912.101509] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12912.101524] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12912.101528] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12912.101534] RAX: 0000000000006295 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12912.101536] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12912.101539] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12912.101541] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12912.101544] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12912.101547] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12912.101550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12912.101553] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12912.101561] Call Trace: [12912.101567] [12912.101573] ? show_regs.cold.9+0x22/0x2f [12912.101583] ? watchdog_timer_fn.cold.6+0x62/0x10a [12912.101589] ? __hrtimer_run_queues+0x1c2/0x4d0 [12912.101593] ? watchdog+0x40/0x40 [12912.101597] ? hrtimer_interrupt+0x128/0x2c0 [12912.101601] ? smp_apic_timer_interrupt+0xa6/0x270 [12912.101609] ? apic_timer_interrupt+0xf/0x20 [12912.101614] [12912.101616] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12912.101625] do_raw_spin_lock+0x177/0x190 [12912.101629] _raw_spin_lock+0x21/0x30 [12912.101634] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12912.102067] ? do_raw_spin_unlock+0x75/0x190 [12912.102073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12912.102426] ? do_wait_intr+0xf0/0xf0 [12912.102434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12912.102716] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12912.102943] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12912.107055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12912.107389] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12912.107397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12912.107401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12912.107423] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12912.107430] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12912.107435] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12912.107438] RDX: 0000000000006797 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12912.107440] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12912.107443] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12912.107445] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12912.107448] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12912.107451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12912.107458] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12912.107466] Call Trace: [12912.107581] [12912.107592] ? show_regs.cold.9+0x22/0x2f [12912.107602] ? watchdog_timer_fn.cold.6+0x62/0x10a [12912.107609] ? __hrtimer_run_queues+0x1c2/0x4d0 [12912.107618] ? watchdog+0x40/0x40 [12912.107623] ? hrtimer_interrupt+0x128/0x2c0 [12912.107628] ? smp_apic_timer_interrupt+0xa6/0x270 [12912.107637] ? apic_timer_interrupt+0xf/0x20 [12912.107641] [12912.107644] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12912.107649] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12912.107654] do_raw_spin_lock+0x177/0x190 [12912.107659] _raw_spin_lock+0x21/0x30 [12912.107666] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12912.108794] obd_mod_free+0x46/0x140 [ptlrpc] [12912.109048] mdc_commit_open+0x91/0xb0 [mdc] [12912.109097] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12912.109302] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12912.109646] after_reply+0xe2c/0x1780 [ptlrpc] [12912.109861] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12912.110060] ? __next_timer_interrupt+0x160/0x160 [12912.110070] ptlrpcd+0xa82/0xe40 [ptlrpc] [12912.110387] ? do_wait_intr+0xf0/0xf0 [12912.110398] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12912.110609] kthread+0x1d1/0x200 [12912.110616] ? set_kthread_struct+0x70/0x70 [12912.110620] ret_from_fork+0x1f/0x30 [12912.340187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.342701] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12912.344826] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12912.346688] mdc_intent_lock+0x353/0x7e0 [mdc] [12912.348618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.350128] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12912.352535] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12912.356254] lmv_intent_open+0x338/0x1150 [lmv] [12912.358939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.360979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.362936] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12912.364603] ll_intent_lock+0xc5/0xaa0 [lustre] [12912.366465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.368283] ll_lookup_it+0xe0e/0x2270 [lustre] [12912.369982] ? do_raw_spin_unlock+0x75/0x190 [12912.371062] ? _raw_spin_unlock+0x12/0x30 [12912.372327] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12912.373984] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12912.375611] ll_atomic_open+0x36a/0xd70 [lustre] [12912.376960] lookup_open+0x356/0xb40 [12912.378128] do_last+0x313/0xfc0 [12912.379088] ? nd_jump_root+0xe5/0x160 [12912.380083] ? path_init+0x437/0x520 [12912.381044] path_openat+0xf7/0x500 [12912.382209] ? do_raw_spin_unlock+0x75/0x190 [12912.383856] do_filp_open+0x99/0x140 [12912.385858] ? getname_flags+0x6e/0x330 [12912.386947] ? __check_object_size+0xff/0x256 [12912.388197] ? do_raw_spin_unlock+0x75/0x190 [12912.392147] ? _raw_spin_unlock+0x12/0x30 [12912.396644] do_sys_openat2+0x2b4/0x410 [12912.397646] do_sys_open+0x73/0xa0 [12912.399039] __x64_sys_openat+0x24/0x30 [12912.401954] do_syscall_64+0xc1/0x440 [12912.403542] entry_SYSCALL_64_after_hwframe+0x49/0xae [12912.405772] RIP: 0033:0x7fa749552332 [12912.407413] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12912.414777] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12912.427938] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12912.432156] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12912.438676] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12912.443718] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12912.451004] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12918.242078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11001s! [12918.253129] Showing busy workqueues and worker pools: [12918.256668] workqueue events: flags=0x0 [12918.260710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12918.264876] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12918.276315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12918.282943] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12918.292855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.299320] pending: kfree_rcu_monitor [12918.302329] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12918.305008] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12918.311858] workqueue events_long: flags=0x0 [12918.314954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.318303] pending: lu_objects_flush [obdclass] [12918.321371] workqueue events_unbound: flags=0x2 [12918.323862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12918.328771] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12918.332795] workqueue events_power_efficient: flags=0x80 [12918.335111] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.339939] pending: check_lifetime [12918.342209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.345462] pending: neigh_periodic_work [12918.347893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.351031] pending: neigh_periodic_work [12918.354125] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.359505] pending: do_cache_clean [12918.360994] workqueue mm_percpu_wq: flags=0x8 [12918.368822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.374348] pending: vmstat_update [12918.378357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.386914] pending: vmstat_update [12918.389310] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12918.396986] pending: vmstat_update [12918.399799] workqueue writeback: flags=0x4a [12918.405953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12918.411298] pending: wb_workfn, wb_workfn [12918.421292] workqueue xprtiod: flags=0xa [12918.429172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12918.436424] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12918.448892] workqueue ipv6_addrconf: flags=0x40008 [12918.452940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12918.462899] pending: addrconf_verify_work [12918.470356] workqueue knbd0-recv: flags=0x1a [12918.480920] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12918.487385] in-flight: 651:recv_work [12918.495316] workqueue ptlrpc_pinger: flags=0x2 [12918.496988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12918.504352] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12918.511079] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11001s workers=4 idle: 37 31846 28897 [12918.520807] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12908s workers=2 idle: 653 [12924.058467] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12924.062941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12924.102520] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12924.107871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12924.115217] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12924.118825] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12924.120365] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12924.127020] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12924.129480] RAX: 0000000000007b19 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12924.131946] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12924.134383] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12924.137974] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12924.140455] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12924.143099] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12924.146126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12924.148099] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12924.150695] Call Trace: [12924.151399] [12924.152512] ? show_regs.cold.9+0x22/0x2f [12924.153940] ? watchdog_timer_fn.cold.6+0x62/0x10a [12924.156006] ? __hrtimer_run_queues+0x1c2/0x4d0 [12924.159076] ? watchdog+0x40/0x40 [12924.161187] ? hrtimer_interrupt+0x128/0x2c0 [12924.166064] ? smp_apic_timer_interrupt+0xa6/0x270 [12924.170160] ? apic_timer_interrupt+0xf/0x20 [12924.177190] [12924.180329] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12924.187909] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12924.189905] do_raw_spin_lock+0x177/0x190 [12924.194107] _raw_spin_lock+0x21/0x30 [12924.196162] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12924.200515] process_one_work+0x2c8/0x700 [12924.203134] worker_thread+0x296/0x6e0 [12924.207145] ? rescuer_thread+0x570/0x570 [12924.209441] kthread+0x1d1/0x200 [12924.212865] ? set_kthread_struct+0x70/0x70 [12924.218206] ret_from_fork+0x1f/0x30 [12936.059095] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12936.062840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12936.083186] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12936.088009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12936.091919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12936.094695] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12936.104728] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12936.109710] RAX: 0000000000007122 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12936.113236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12936.116970] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12936.120316] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12936.123961] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12936.128578] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12936.133237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12936.136281] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12936.140460] Call Trace: [12936.141956] [12936.143200] ? show_regs.cold.9+0x22/0x2f [12936.145004] ? watchdog_timer_fn.cold.6+0x62/0x10a [12936.147779] ? __hrtimer_run_queues+0x1c2/0x4d0 [12936.150398] ? watchdog+0x40/0x40 [12936.152304] ? hrtimer_interrupt+0x128/0x2c0 [12936.154590] ? smp_apic_timer_interrupt+0xa6/0x270 [12936.157488] ? apic_timer_interrupt+0xf/0x20 [12936.159031] [12936.160329] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12936.163945] do_raw_spin_lock+0x177/0x190 [12936.165916] _raw_spin_lock+0x21/0x30 [12936.167776] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12936.170388] ? do_raw_spin_unlock+0x75/0x190 [12936.172998] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12936.175801] ? do_wait_intr+0xf0/0xf0 [12936.177674] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12936.180112] mdc_close+0x53b/0xd70 [mdc] [12936.182360] lmv_close+0x18d/0x490 [lmv] [12936.184615] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12936.187838] ll_md_real_close+0xa5/0x2a0 [lustre] [12936.190831] ll_md_close+0x274/0x800 [lustre] [12936.194477] ll_file_release+0x128/0xc50 [lustre] [12936.197377] __fput+0x10d/0x440 [12936.201850] ____fput+0x12/0x20 [12936.203595] task_work_run+0xc6/0x110 [12936.208885] exit_to_usermode_loop+0x1dd/0x1f0 [12936.211615] do_syscall_64+0x3d6/0x440 [12936.213963] entry_SYSCALL_64_after_hwframe+0x49/0xae [12936.218631] RIP: 0033:0x7fa749552cd8 [12936.222284] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12936.233312] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12936.245277] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12936.249204] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12936.253608] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12936.256526] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12936.260873] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12940.058154] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12940.061028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12940.077730] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12940.081504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12940.085684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12940.087970] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12940.094711] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12940.096914] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12940.099797] RDX: 0000000000002673 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12940.102139] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12940.104866] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12940.107469] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12940.110118] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12940.113248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12940.115494] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12940.118461] Call Trace: [12940.119492] [12940.120219] ? show_regs.cold.9+0x22/0x2f [12940.121593] ? watchdog_timer_fn.cold.6+0x62/0x10a [12940.123629] ? __hrtimer_run_queues+0x1c2/0x4d0 [12940.125301] ? watchdog+0x40/0x40 [12940.126742] ? hrtimer_interrupt+0x128/0x2c0 [12940.128408] ? smp_apic_timer_interrupt+0xa6/0x270 [12940.130291] ? apic_timer_interrupt+0xf/0x20 [12940.132213] [12940.133035] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12940.136284] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12940.138622] do_raw_spin_lock+0x177/0x190 [12940.140044] _raw_spin_lock+0x21/0x30 [12940.143659] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12940.145780] obd_mod_free+0x46/0x140 [ptlrpc] [12940.154078] mdc_commit_open+0x91/0xb0 [mdc] [12940.157433] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12940.159632] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12940.163586] after_reply+0xe2c/0x1780 [ptlrpc] [12940.167698] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12940.174392] ? __next_timer_interrupt+0x160/0x160 [12940.179348] ptlrpcd+0xa82/0xe40 [ptlrpc] [12940.181630] ? do_wait_intr+0xf0/0xf0 [12940.187234] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12940.189382] kthread+0x1d1/0x200 [12940.192696] ? set_kthread_struct+0x70/0x70 [12940.194199] ret_from_fork+0x1f/0x30 [12944.054254] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [12944.058897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12944.103715] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12944.118480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12944.127895] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12944.138637] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12944.163293] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12944.173382] RAX: 0000000000004d84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12944.184372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12944.194679] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12944.201418] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12944.207380] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12944.214868] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12944.219424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12944.226413] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12944.234840] Call Trace: [12944.238521] [12944.239581] ? show_regs.cold.9+0x22/0x2f [12944.243337] ? watchdog_timer_fn.cold.6+0x62/0x10a [12944.246634] ? __hrtimer_run_queues+0x1c2/0x4d0 [12944.249533] ? watchdog+0x40/0x40 [12944.252718] ? hrtimer_interrupt+0x128/0x2c0 [12944.256372] ? smp_apic_timer_interrupt+0xa6/0x270 [12944.261049] ? apic_timer_interrupt+0xf/0x20 [12944.269631] [12944.272841] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12944.277542] do_raw_spin_lock+0x177/0x190 [12944.279820] _raw_spin_lock+0x21/0x30 [12944.283296] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12944.287752] ? do_raw_spin_unlock+0x75/0x190 [12944.292170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12944.295020] ? do_wait_intr+0xf0/0xf0 [12944.296883] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12944.299825] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12944.302998] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12944.305763] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12944.308594] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12944.312040] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12944.314649] mdc_intent_lock+0x353/0x7e0 [mdc] [12944.317323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12944.320589] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12944.324269] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12944.327900] lmv_intent_open+0x338/0x1150 [lmv] [12944.330863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12944.335525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12944.341511] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12944.347054] ll_intent_lock+0xc5/0xaa0 [lustre] [12944.351688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12944.355962] ll_lookup_it+0xe0e/0x2270 [lustre] [12944.361632] ? do_raw_spin_unlock+0x75/0x190 [12944.370714] ? _raw_spin_unlock+0x12/0x30 [12944.386110] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12944.388010] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12944.399903] ll_atomic_open+0x36a/0xd70 [lustre] [12944.407623] lookup_open+0x356/0xb40 [12944.411801] do_last+0x313/0xfc0 [12944.416612] ? nd_jump_root+0xe5/0x160 [12944.424170] ? path_init+0x437/0x520 [12944.425495] path_openat+0xf7/0x500 [12944.426733] ? do_raw_spin_unlock+0x75/0x190 [12944.440825] do_filp_open+0x99/0x140 [12944.450588] ? getname_flags+0x6e/0x330 [12944.458778] ? __check_object_size+0xff/0x256 [12944.460336] ? do_raw_spin_unlock+0x75/0x190 [12944.469807] ? _raw_spin_unlock+0x12/0x30 [12944.471981] do_sys_openat2+0x2b4/0x410 [12944.476632] do_sys_open+0x73/0xa0 [12944.480536] __x64_sys_openat+0x24/0x30 [12944.481922] do_syscall_64+0xc1/0x440 [12944.485369] entry_SYSCALL_64_after_hwframe+0x49/0xae [12944.489389] RIP: 0033:0x7fa749552332 [12944.490748] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12944.500823] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12944.506840] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12944.513088] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12944.520362] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12944.525573] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12944.533417] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12948.963201] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11031s! [12948.975532] Showing busy workqueues and worker pools: [12948.979647] workqueue events: flags=0x0 [12948.986635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12948.991258] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12949.000854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12949.005253] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12949.014243] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.020511] pending: kfree_rcu_monitor [12949.026161] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12949.035292] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12949.042024] workqueue events_long: flags=0x0 [12949.048941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.058499] pending: lu_objects_flush [obdclass] [12949.063803] workqueue events_unbound: flags=0x2 [12949.067600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12949.077783] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12949.103487] workqueue events_power_efficient: flags=0x80 [12949.111983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.124353] pending: check_lifetime [12949.131033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.136694] pending: neigh_periodic_work [12949.139086] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.147232] pending: neigh_periodic_work [12949.152596] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.159265] pending: do_cache_clean [12949.161969] workqueue mm_percpu_wq: flags=0x8 [12949.172067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.176145] pending: vmstat_update [12949.179314] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.185835] pending: vmstat_update [12949.187213] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12949.196663] pending: vmstat_update [12949.201121] workqueue writeback: flags=0x4a [12949.205015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12949.211207] pending: wb_workfn, wb_workfn [12949.218320] workqueue xprtiod: flags=0xa [12949.219842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12949.225781] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12949.230817] workqueue ipv6_addrconf: flags=0x40008 [12949.233535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12949.245436] pending: addrconf_verify_work [12949.249657] workqueue knbd0-recv: flags=0x1a [12949.253751] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12949.257249] in-flight: 651:recv_work [12949.260625] workqueue ptlrpc_pinger: flags=0x2 [12949.263213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12949.267389] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12949.273011] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11032s workers=4 idle: 37 31846 28897 [12949.277155] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12939s workers=2 idle: 653 [12952.057614] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12952.069077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12952.133466] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12952.150988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12952.171694] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12952.191267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12952.199538] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12952.226320] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12952.242413] RAX: 0000000000004a64 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12952.256009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12952.272664] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12952.283816] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12952.291255] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12952.301330] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12952.315534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12952.321645] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12952.331455] Call Trace: [12952.337341] [12952.341621] ? show_regs.cold.9+0x22/0x2f [12952.345262] ? watchdog_timer_fn.cold.6+0x62/0x10a [12952.351744] ? __hrtimer_run_queues+0x1c2/0x4d0 [12952.360753] ? watchdog+0x40/0x40 [12952.361862] ? hrtimer_interrupt+0x128/0x2c0 [12952.366524] ? smp_apic_timer_interrupt+0xa6/0x270 [12952.369986] ? apic_timer_interrupt+0xf/0x20 [12952.374076] [12952.375016] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12952.377704] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12952.392684] do_raw_spin_lock+0x177/0x190 [12952.393662] _raw_spin_lock+0x21/0x30 [12952.395431] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12952.399841] process_one_work+0x2c8/0x700 [12952.401908] worker_thread+0x296/0x6e0 [12952.403605] ? rescuer_thread+0x570/0x570 [12952.405618] kthread+0x1d1/0x200 [12952.407447] ? set_kthread_struct+0x70/0x70 [12952.409370] ret_from_fork+0x1f/0x30 [12953.934651] rcu: INFO: rcu_sched self-detected stall on CPU [12953.949565] rcu: 2-...!: (6287070 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [12953.952409] (t=11040245 jiffies g=258701 q=4643) [12953.954281] rcu: rcu_sched kthread starved for 11037040 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12953.958138] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12953.961946] rcu: RCU grace-period kthread stack dump: [12953.963916] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12953.967637] Call Trace: [12953.968648] __schedule+0x351/0xcb0 [12953.972598] schedule+0xc0/0x180 [12953.974470] schedule_timeout+0xb4/0x190 [12953.976816] ? do_raw_spin_unlock+0x75/0x190 [12953.978658] ? __next_timer_interrupt+0x160/0x160 [12953.980347] ? prepare_to_swait_event+0xf3/0x140 [12953.982412] rcu_gp_kthread+0x8dd/0x15b0 [12953.983992] ? force_qs_rnp+0x2c0/0x2c0 [12953.985387] kthread+0x1d1/0x200 [12953.986633] ? set_kthread_struct+0x70/0x70 [12953.988270] ret_from_fork+0x1f/0x30 [12953.989558] rcu: Stack dump where RCU GP kthread last ran: [12953.991547] Sending NMI from CPU 2 to CPUs 0: [12953.995842] NMI backtrace for cpu 0 [12953.995848] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12953.995850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12953.995852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12953.995853] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12953.995855] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [12953.995857] RAX: 000000000000580c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12953.995858] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [12953.995860] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [12953.995861] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [12953.995862] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [12953.995864] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [12953.995865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12953.995867] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [12953.995868] Call Trace: [12953.995869] [12953.995870] ? show_regs.cold.9+0x22/0x2f [12953.995871] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12953.995872] ? nmi_cpu_backtrace_handler+0x15/0x40 [12953.995873] ? nmi_handle+0xbd/0x210 [12953.995874] ? default_do_nmi+0x4b/0x1b0 [12953.995875] ? do_nmi+0x280/0x380 [12953.995876] ? end_repeat_nmi+0x16/0x1f [12953.995877] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12953.995879] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12953.995880] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12953.995881] [12953.995882] do_raw_spin_lock+0x177/0x190 [12953.995883] _raw_spin_lock+0x21/0x30 [12953.995884] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12953.995885] ? do_raw_spin_unlock+0x75/0x190 [12953.995886] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12953.995887] ? do_wait_intr+0xf0/0xf0 [12953.995888] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12953.995890] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12953.995891] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12953.995892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12953.995893] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12953.995894] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12953.995895] mdc_intent_lock+0x353/0x7e0 [mdc] [12953.995897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12953.996056] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12953.996057] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12953.996058] lmv_intent_open+0x338/0x1150 [lmv] [12953.996058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12953.996059] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12953.996060] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12953.996061] ll_intent_lock+0xc5/0xaa0 [lustre] [12953.996062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12953.996062] ll_lookup_it+0xe0e/0x2270 [lustre] [12953.996063] ? do_raw_spin_unlock+0x75/0x190 [12953.996064] ? _raw_spin_unlock+0x12/0x30 [12953.996065] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12953.996066] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12953.996066] ll_atomic_open+0x36a/0xd70 [lustre] [12953.996067] lookup_open+0x356/0xb40 [12953.996068] do_last+0x313/0xfc0 [12953.996068] ? nd_jump_root+0xe5/0x160 [12953.996069] ? path_init+0x437/0x520 [12953.996070] path_openat+0xf7/0x500 [12953.996070] ? do_raw_spin_unlock+0x75/0x190 [12953.996071] do_filp_open+0x99/0x140 [12953.996071] ? getname_flags+0x6e/0x330 [12953.996072] ? __check_object_size+0xff/0x256 [12953.996073] ? do_raw_spin_unlock+0x75/0x190 [12953.996073] ? _raw_spin_unlock+0x12/0x30 [12953.996074] do_sys_openat2+0x2b4/0x410 [12953.996074] do_sys_open+0x73/0xa0 [12953.996075] __x64_sys_openat+0x24/0x30 [12953.996075] do_syscall_64+0xc1/0x440 [12953.996076] entry_SYSCALL_64_after_hwframe+0x49/0xae [12953.996076] RIP: 0033:0x7fa749552332 [12953.996077] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12953.996079] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12953.996081] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [12953.996081] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [12953.996082] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [12953.996083] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [12953.996084] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [12953.996186] NMI backtrace for cpu 2 [12954.309487] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12954.328109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12954.339509] Call Trace: [12954.342453] [12954.343583] ? dump_stack+0xbb/0x10e [12954.346198] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12954.348508] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12954.351604] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12954.354988] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12954.358549] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12954.361551] ? rcu_dump_cpu_stacks+0x11b/0x171 [12954.363698] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12954.366589] ? update_process_times+0x6d/0xc0 [12954.372551] ? tick_sched_handle.isra.18+0x27/0x90 [12954.376779] ? tick_sched_timer+0x53/0xb0 [12954.381692] ? __hrtimer_run_queues+0x1c2/0x4d0 [12954.389456] ? tick_sched_handle.isra.18+0x90/0x90 [12954.399275] ? hrtimer_interrupt+0x128/0x2c0 [12954.402027] ? smp_apic_timer_interrupt+0xa6/0x270 [12954.409359] ? apic_timer_interrupt+0xf/0x20 [12954.412384] [12954.413533] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12954.417133] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12954.419577] ? do_raw_spin_lock+0x177/0x190 [12954.421498] ? _raw_spin_lock+0x21/0x30 [12954.423287] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12954.425707] ? obd_mod_free+0x46/0x140 [ptlrpc] [12954.427970] ? mdc_commit_open+0x91/0xb0 [mdc] [12954.430080] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12954.432725] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12954.435998] ? after_reply+0xe2c/0x1780 [ptlrpc] [12954.438231] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12954.441280] ? __next_timer_interrupt+0x160/0x160 [12954.443430] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12954.445619] ? do_wait_intr+0xf0/0xf0 [12954.447308] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12954.449962] ? kthread+0x1d1/0x200 [12954.451554] ? set_kthread_struct+0x70/0x70 [12954.453588] ? ret_from_fork+0x1f/0x30 [12964.059087] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12964.074579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12964.147492] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12964.153944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12964.158019] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12964.167973] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12964.194122] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12964.203943] RAX: 0000000000004009 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12964.209384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12964.215777] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12964.221353] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12964.226151] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12964.236789] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12964.248119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12964.258074] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12964.272960] Call Trace: [12964.274221] [12964.276774] ? show_regs.cold.9+0x22/0x2f [12964.280935] ? watchdog_timer_fn.cold.6+0x62/0x10a [12964.287132] ? __hrtimer_run_queues+0x1c2/0x4d0 [12964.290879] ? watchdog+0x40/0x40 [12964.294337] ? hrtimer_interrupt+0x128/0x2c0 [12964.303119] ? smp_apic_timer_interrupt+0xa6/0x270 [12964.306346] ? apic_timer_interrupt+0xf/0x20 [12964.307797] [12964.310067] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12964.313990] do_raw_spin_lock+0x177/0x190 [12964.316883] _raw_spin_lock+0x21/0x30 [12964.321632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12964.327133] ? do_raw_spin_unlock+0x75/0x190 [12964.334432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12964.343060] ? do_wait_intr+0xf0/0xf0 [12964.348918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12964.358464] mdc_close+0x53b/0xd70 [mdc] [12964.366714] lmv_close+0x18d/0x490 [lmv] [12964.371762] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12964.379932] ll_md_real_close+0xa5/0x2a0 [lustre] [12964.385320] ll_md_close+0x274/0x800 [lustre] [12964.388285] ll_file_release+0x128/0xc50 [lustre] [12964.393746] __fput+0x10d/0x440 [12964.394835] ____fput+0x12/0x20 [12964.399493] task_work_run+0xc6/0x110 [12964.400930] exit_to_usermode_loop+0x1dd/0x1f0 [12964.406237] do_syscall_64+0x3d6/0x440 [12964.414576] entry_SYSCALL_64_after_hwframe+0x49/0xae [12964.417871] RIP: 0033:0x7fa749552cd8 [12964.419228] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12964.447802] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12964.458987] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12964.468177] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12964.475149] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12964.484511] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12964.495628] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12979.680290] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11062s! [12979.685357] Showing busy workqueues and worker pools: [12979.687108] workqueue events: flags=0x0 [12979.694289] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12979.707615] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [12979.712427] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12979.715296] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12979.719633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.722661] pending: kfree_rcu_monitor [12979.724760] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12979.728391] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [12979.733973] workqueue events_long: flags=0x0 [12979.736202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.741344] pending: lu_objects_flush [obdclass] [12979.745320] workqueue events_unbound: flags=0x2 [12979.748749] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [12979.752789] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12979.758409] workqueue events_power_efficient: flags=0x80 [12979.761665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.766019] pending: check_lifetime [12979.767882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.771443] pending: neigh_periodic_work [12979.774660] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.779516] pending: neigh_periodic_work [12979.781919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.786149] pending: do_cache_clean [12979.788373] workqueue mm_percpu_wq: flags=0x8 [12979.791693] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.794722] pending: vmstat_update [12979.796704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.800114] pending: vmstat_update [12979.801491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12979.804027] pending: vmstat_update [12979.805640] workqueue writeback: flags=0x4a [12979.807328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12979.810107] pending: wb_workfn, wb_workfn [12979.812058] workqueue xprtiod: flags=0xa [12979.813755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12979.816570] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12979.819928] workqueue ipv6_addrconf: flags=0x40008 [12979.822030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12979.824869] pending: addrconf_verify_work [12979.826926] workqueue knbd0-recv: flags=0x1a [12979.829045] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12979.832443] in-flight: 651:recv_work [12979.834277] workqueue ptlrpc_pinger: flags=0x2 [12979.836205] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12979.839015] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [12979.841716] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11062s workers=4 idle: 37 31846 28897 [12979.845235] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12969s workers=2 idle: 653 [12980.058682] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [12980.063794] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [12980.064540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12980.064626] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12980.064631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12980.064637] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12980.065162] RIP: 0010:native_safe_halt+0x2e/0x50 [12980.065172] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12980.065176] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12980.065180] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12980.065182] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [12980.065185] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [12980.065187] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [12980.065189] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [12980.065192] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [12980.065195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12980.065197] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [12980.065206] Call Trace: [12980.065212] [12980.065217] ? show_regs.cold.9+0x22/0x2f [12980.065226] ? watchdog_timer_fn.cold.6+0x62/0x10a [12980.065231] ? __hrtimer_run_queues+0x1c2/0x4d0 [12980.065235] ? watchdog+0x40/0x40 [12980.065239] ? hrtimer_interrupt+0x128/0x2c0 [12980.065243] ? smp_apic_timer_interrupt+0xa6/0x270 [12980.065248] ? apic_timer_interrupt+0xf/0x20 [12980.065253] [12980.065256] ? native_safe_halt+0x2e/0x50 [12980.065259] kvm_wait+0xcd/0xe0 [12980.065267] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12980.065274] do_raw_spin_lock+0x177/0x190 [12980.065278] _raw_spin_lock+0x21/0x30 [12980.065360] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12980.066076] process_one_work+0x2c8/0x700 [12980.066084] worker_thread+0x296/0x6e0 [12980.066089] ? rescuer_thread+0x570/0x570 [12980.066093] kthread+0x1d1/0x200 [12980.066098] ? set_kthread_struct+0x70/0x70 [12980.066102] ret_from_fork+0x1f/0x30 [12980.283133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12980.325862] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12980.341492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12980.353898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12980.359029] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12980.384735] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12980.393517] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [12980.397998] RDX: 000000000000009f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [12980.405514] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [12980.413271] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [12980.422155] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12980.429185] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [12980.437373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12980.445143] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [12980.453437] Call Trace: [12980.456711] [12980.457688] ? show_regs.cold.9+0x22/0x2f [12980.463367] ? watchdog_timer_fn.cold.6+0x62/0x10a [12980.466449] ? __hrtimer_run_queues+0x1c2/0x4d0 [12980.471072] ? watchdog+0x40/0x40 [12980.478100] ? hrtimer_interrupt+0x128/0x2c0 [12980.481637] ? smp_apic_timer_interrupt+0xa6/0x270 [12980.487392] ? apic_timer_interrupt+0xf/0x20 [12980.489682] [12980.491011] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12980.496524] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12980.501221] do_raw_spin_lock+0x177/0x190 [12980.503347] _raw_spin_lock+0x21/0x30 [12980.506158] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12980.509266] obd_mod_free+0x46/0x140 [ptlrpc] [12980.512887] mdc_commit_open+0x91/0xb0 [mdc] [12980.516193] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12980.520269] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12980.527392] after_reply+0xe2c/0x1780 [ptlrpc] [12980.532720] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12980.538925] ? __next_timer_interrupt+0x160/0x160 [12980.549737] ptlrpcd+0xa82/0xe40 [ptlrpc] [12980.558472] ? do_wait_intr+0xf0/0xf0 [12980.564226] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12980.574649] kthread+0x1d1/0x200 [12980.578892] ? set_kthread_struct+0x70/0x70 [12980.580744] ret_from_fork+0x1f/0x30 [12992.059320] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [12992.067740] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12992.105059] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12992.114997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12992.122100] RIP: 0010:native_safe_halt+0x2e/0x50 [12992.126662] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12992.141614] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12992.147240] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [12992.153247] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [12992.157745] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [12992.163781] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [12992.170228] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [12992.177293] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [12992.185289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12992.190529] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [12992.195202] Call Trace: [12992.196692] [12992.199169] ? show_regs.cold.9+0x22/0x2f [12992.201783] ? watchdog_timer_fn.cold.6+0x62/0x10a [12992.204925] ? __hrtimer_run_queues+0x1c2/0x4d0 [12992.207803] ? watchdog+0x40/0x40 [12992.209747] ? hrtimer_interrupt+0x128/0x2c0 [12992.212682] ? smp_apic_timer_interrupt+0xa6/0x270 [12992.216794] ? apic_timer_interrupt+0xf/0x20 [12992.219921] [12992.225211] ? native_safe_halt+0x2e/0x50 [12992.228028] kvm_wait+0xcd/0xe0 [12992.231133] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12992.235390] do_raw_spin_lock+0x177/0x190 [12992.237252] _raw_spin_lock+0x21/0x30 [12992.239686] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12992.245957] ? do_raw_spin_unlock+0x75/0x190 [12992.248848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12992.252270] ? do_wait_intr+0xf0/0xf0 [12992.256130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12992.260081] mdc_close+0x53b/0xd70 [mdc] [12992.262296] lmv_close+0x18d/0x490 [lmv] [12992.264710] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12992.269055] ll_md_real_close+0xa5/0x2a0 [lustre] [12992.273890] ll_md_close+0x274/0x800 [lustre] [12992.277928] ll_file_release+0x128/0xc50 [lustre] [12992.284731] __fput+0x10d/0x440 [12992.286689] ____fput+0x12/0x20 [12992.289429] task_work_run+0xc6/0x110 [12992.292208] exit_to_usermode_loop+0x1dd/0x1f0 [12992.297941] do_syscall_64+0x3d6/0x440 [12992.299995] entry_SYSCALL_64_after_hwframe+0x49/0xae [12992.303388] RIP: 0033:0x7fa749552cd8 [12992.306027] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12992.317766] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12992.323864] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [12992.327975] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [12992.332936] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [12992.336509] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [12992.340308] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13004.054046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13004.058755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13004.079348] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13004.083517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13004.094083] RIP: 0010:native_safe_halt+0x2e/0x50 [13004.097174] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13004.108428] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13004.121694] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13004.134978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13004.144321] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13004.148556] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13004.156579] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13004.165461] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13004.173721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13004.177064] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13004.180978] Call Trace: [13004.182857] [13004.184974] ? show_regs.cold.9+0x22/0x2f [13004.187616] ? watchdog_timer_fn.cold.6+0x62/0x10a [13004.191031] ? __hrtimer_run_queues+0x1c2/0x4d0 [13004.196591] ? watchdog+0x40/0x40 [13004.198345] ? hrtimer_interrupt+0x128/0x2c0 [13004.204634] ? smp_apic_timer_interrupt+0xa6/0x270 [13004.211310] ? apic_timer_interrupt+0xf/0x20 [13004.214173] [13004.215286] ? native_safe_halt+0x2e/0x50 [13004.217508] kvm_wait+0xcd/0xe0 [13004.220803] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13004.228426] do_raw_spin_lock+0x177/0x190 [13004.232511] _raw_spin_lock+0x21/0x30 [13004.233809] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13004.240326] ? do_raw_spin_unlock+0x75/0x190 [13004.242947] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13004.249319] ? do_wait_intr+0xf0/0xf0 [13004.251672] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13004.255373] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13004.258689] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13004.269722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13004.272358] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13004.275622] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13004.278259] mdc_intent_lock+0x353/0x7e0 [mdc] [13004.280356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13004.285207] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13004.288966] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13004.293018] lmv_intent_open+0x338/0x1150 [lmv] [13004.302161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13004.306114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13004.308834] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13004.312321] ll_intent_lock+0xc5/0xaa0 [lustre] [13004.314441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13004.317944] ll_lookup_it+0xe0e/0x2270 [lustre] [13004.323660] ? do_raw_spin_unlock+0x75/0x190 [13004.328173] ? _raw_spin_unlock+0x12/0x30 [13004.333306] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13004.336289] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13004.340256] ll_atomic_open+0x36a/0xd70 [lustre] [13004.342944] lookup_open+0x356/0xb40 [13004.345125] do_last+0x313/0xfc0 [13004.346832] ? nd_jump_root+0xe5/0x160 [13004.354910] ? path_init+0x437/0x520 [13004.357971] path_openat+0xf7/0x500 [13004.359849] ? do_raw_spin_unlock+0x75/0x190 [13004.362225] do_filp_open+0x99/0x140 [13004.364590] ? getname_flags+0x6e/0x330 [13004.366323] ? __check_object_size+0xff/0x256 [13004.369182] ? do_raw_spin_unlock+0x75/0x190 [13004.371627] ? _raw_spin_unlock+0x12/0x30 [13004.374052] do_sys_openat2+0x2b4/0x410 [13004.378193] do_sys_open+0x73/0xa0 [13004.380367] __x64_sys_openat+0x24/0x30 [13004.381845] do_syscall_64+0xc1/0x440 [13004.385523] entry_SYSCALL_64_after_hwframe+0x49/0xae [13004.387637] RIP: 0033:0x7fa749552332 [13004.392259] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13004.403824] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13004.408386] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13004.412741] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13004.415894] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13004.417888] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13004.424771] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13008.057340] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13008.058112] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13008.062185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13008.062266] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13008.062273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13008.062279] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13008.062557] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13008.062569] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13008.062572] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13008.062577] RAX: 0000000000002902 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13008.062579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13008.062581] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13008.062583] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13008.062585] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13008.062588] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13008.062592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13008.062595] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13008.062603] Call Trace: [13008.062608] [13008.062613] ? show_regs.cold.9+0x22/0x2f [13008.062628] ? watchdog_timer_fn.cold.6+0x62/0x10a [13008.062634] ? __hrtimer_run_queues+0x1c2/0x4d0 [13008.062638] ? watchdog+0x40/0x40 [13008.062642] ? hrtimer_interrupt+0x128/0x2c0 [13008.062646] ? smp_apic_timer_interrupt+0xa6/0x270 [13008.062654] ? apic_timer_interrupt+0xf/0x20 [13008.062659] [13008.062662] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13008.062666] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13008.062671] do_raw_spin_lock+0x177/0x190 [13008.062676] _raw_spin_lock+0x21/0x30 [13008.062681] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13008.062897] process_one_work+0x2c8/0x700 [13008.062911] worker_thread+0x296/0x6e0 [13008.063050] ? rescuer_thread+0x570/0x570 [13008.063055] kthread+0x1d1/0x200 [13008.063060] ? set_kthread_struct+0x70/0x70 [13008.063063] ret_from_fork+0x1f/0x30 [13008.282781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13008.305213] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13008.309948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13008.313699] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13008.316160] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13008.327337] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13008.332005] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13008.335688] RDX: 0000000000003057 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13008.339864] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13008.343195] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13008.346240] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13008.350653] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13008.354013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13008.355953] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13008.358893] Call Trace: [13008.359651] [13008.360564] ? show_regs.cold.9+0x22/0x2f [13008.362413] ? watchdog_timer_fn.cold.6+0x62/0x10a [13008.364354] ? __hrtimer_run_queues+0x1c2/0x4d0 [13008.366116] ? watchdog+0x40/0x40 [13008.367579] ? hrtimer_interrupt+0x128/0x2c0 [13008.369537] ? smp_apic_timer_interrupt+0xa6/0x270 [13008.372141] ? apic_timer_interrupt+0xf/0x20 [13008.374151] [13008.375151] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13008.377391] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13008.379389] do_raw_spin_lock+0x177/0x190 [13008.381183] _raw_spin_lock+0x21/0x30 [13008.382625] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13008.384909] obd_mod_free+0x46/0x140 [ptlrpc] [13008.387045] mdc_commit_open+0x91/0xb0 [mdc] [13008.391113] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13008.396588] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13008.399212] after_reply+0xe2c/0x1780 [ptlrpc] [13008.401417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13008.404055] ? __next_timer_interrupt+0x160/0x160 [13008.406106] ptlrpcd+0xa82/0xe40 [ptlrpc] [13008.408168] ? do_wait_intr+0xf0/0xf0 [13008.409816] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13008.411788] kthread+0x1d1/0x200 [13008.412807] ? set_kthread_struct+0x70/0x70 [13008.414907] ret_from_fork+0x1f/0x30 [13009.889089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11092s! [13009.898618] Showing busy workqueues and worker pools: [13009.905265] workqueue events: flags=0x0 [13009.908020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13009.911803] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13009.921096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13009.925097] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13009.930858] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.934550] pending: kfree_rcu_monitor [13009.936199] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13009.943426] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13009.948390] workqueue events_long: flags=0x0 [13009.950846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.953097] pending: lu_objects_flush [obdclass] [13009.954979] workqueue events_unbound: flags=0x2 [13009.956493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13009.960246] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13009.964556] workqueue events_power_efficient: flags=0x80 [13009.967269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.970883] pending: check_lifetime [13009.972749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.976289] pending: neigh_periodic_work [13009.978561] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.985550] pending: neigh_periodic_work [13009.988322] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.991522] pending: do_cache_clean [13009.993405] workqueue mm_percpu_wq: flags=0x8 [13009.994841] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.996544] pending: vmstat_update [13009.998392] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13010.000848] pending: vmstat_update [13010.002027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13010.004403] pending: vmstat_update [13010.005496] workqueue writeback: flags=0x4a [13010.006873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13010.009389] pending: wb_workfn, wb_workfn [13010.010729] workqueue xprtiod: flags=0xa [13010.011901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13010.014299] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13010.018554] workqueue ipv6_addrconf: flags=0x40008 [13010.022430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13010.026541] pending: addrconf_verify_work [13010.028888] workqueue knbd0-recv: flags=0x1a [13010.030849] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13010.033830] in-flight: 651:recv_work [13010.035570] workqueue ptlrpc_pinger: flags=0x2 [13010.037567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13010.040339] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13010.043303] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11092s workers=4 idle: 37 31846 28897 [13010.046497] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12999s workers=2 idle: 653 [13020.059208] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13020.063495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13020.102939] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13020.111489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13020.132247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13020.136472] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13020.160607] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13020.170320] RAX: 0000000000002432 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13020.182389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13020.187201] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13020.195663] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13020.201730] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13020.206006] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13020.214808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13020.220972] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13020.226436] Call Trace: [13020.228922] [13020.231080] ? show_regs.cold.9+0x22/0x2f [13020.235236] ? watchdog_timer_fn.cold.6+0x62/0x10a [13020.239788] ? __hrtimer_run_queues+0x1c2/0x4d0 [13020.246207] ? watchdog+0x40/0x40 [13020.247885] ? hrtimer_interrupt+0x128/0x2c0 [13020.251862] ? smp_apic_timer_interrupt+0xa6/0x270 [13020.254899] ? apic_timer_interrupt+0xf/0x20 [13020.259341] [13020.260367] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13020.265926] do_raw_spin_lock+0x177/0x190 [13020.272111] _raw_spin_lock+0x21/0x30 [13020.273970] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13020.282714] ? do_raw_spin_unlock+0x75/0x190 [13020.286327] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13020.297091] ? do_wait_intr+0xf0/0xf0 [13020.301082] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13020.304219] mdc_close+0x53b/0xd70 [mdc] [13020.306136] lmv_close+0x18d/0x490 [lmv] [13020.309998] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13020.315065] ll_md_real_close+0xa5/0x2a0 [lustre] [13020.319031] ll_md_close+0x274/0x800 [lustre] [13020.323068] ll_file_release+0x128/0xc50 [lustre] [13020.326352] __fput+0x10d/0x440 [13020.328232] ____fput+0x12/0x20 [13020.330750] task_work_run+0xc6/0x110 [13020.338933] exit_to_usermode_loop+0x1dd/0x1f0 [13020.341116] do_syscall_64+0x3d6/0x440 [13020.344537] entry_SYSCALL_64_after_hwframe+0x49/0xae [13020.347617] RIP: 0033:0x7fa749552cd8 [13020.351491] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13020.372130] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13020.378207] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13020.385787] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13020.389558] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13020.394656] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13020.402859] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13036.055512] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13036.060171] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13036.061290] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13036.061418] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13036.061425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13036.061428] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13036.061442] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13036.061446] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13036.061451] RAX: 0000000000007e0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13036.061453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13036.061455] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13036.061457] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13036.061460] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13036.061463] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13036.061466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13036.061468] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13036.061475] Call Trace: [13036.061480] [13036.061485] ? show_regs.cold.9+0x22/0x2f [13036.061493] ? watchdog_timer_fn.cold.6+0x62/0x10a [13036.061499] ? __hrtimer_run_queues+0x1c2/0x4d0 [13036.061502] ? watchdog+0x40/0x40 [13036.061506] ? hrtimer_interrupt+0x128/0x2c0 [13036.061510] ? smp_apic_timer_interrupt+0xa6/0x270 [13036.061518] ? apic_timer_interrupt+0xf/0x20 [13036.061522] [13036.061524] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13036.061530] do_raw_spin_lock+0x177/0x190 [13036.061534] _raw_spin_lock+0x21/0x30 [13036.061538] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13036.061863] ? do_raw_spin_unlock+0x75/0x190 [13036.061870] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13036.062066] ? do_wait_intr+0xf0/0xf0 [13036.062075] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13036.062266] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13036.062474] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13036.062697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.062819] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13036.064972] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13036.064991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13036.065073] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13036.065080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13036.065543] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13036.065602] mdc_intent_lock+0x353/0x7e0 [mdc] [13036.065631] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.065089] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13036.066740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13036.066758] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13036.066764] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13036.066768] RAX: 0000000000002860 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13036.066771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13036.066773] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13036.066775] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13036.066778] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13036.066781] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13036.066785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13036.066787] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13036.066797] Call Trace: [13036.066804] [13036.066809] ? show_regs.cold.9+0x22/0x2f [13036.066820] ? watchdog_timer_fn.cold.6+0x62/0x10a [13036.066827] ? __hrtimer_run_queues+0x1c2/0x4d0 [13036.066832] ? watchdog+0x40/0x40 [13036.066836] ? hrtimer_interrupt+0x128/0x2c0 [13036.066841] ? smp_apic_timer_interrupt+0xa6/0x270 [13036.066866] ? apic_timer_interrupt+0xf/0x20 [13036.066870] [13036.066873] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13036.066878] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13036.066884] do_raw_spin_lock+0x177/0x190 [13036.066889] _raw_spin_lock+0x21/0x30 [13036.066895] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13036.067181] process_one_work+0x2c8/0x700 [13036.067189] worker_thread+0x296/0x6e0 [13036.067193] ? rescuer_thread+0x570/0x570 [13036.067197] kthread+0x1d1/0x200 [13036.067202] ? set_kthread_struct+0x70/0x70 [13036.067206] ret_from_fork+0x1f/0x30 [13036.067390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13036.074387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13036.074479] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13036.074489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13036.074492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13036.074511] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13036.074516] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13036.074521] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13036.074523] RDX: 0000000000001283 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13036.074526] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13036.074529] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13036.074532] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13036.074535] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13036.074538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13036.074541] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13036.074550] Call Trace: [13036.074557] [13036.074566] ? show_regs.cold.9+0x22/0x2f [13036.074578] ? watchdog_timer_fn.cold.6+0x62/0x10a [13036.074586] ? __hrtimer_run_queues+0x1c2/0x4d0 [13036.079628] ? watchdog+0x40/0x40 [13036.079637] ? hrtimer_interrupt+0x128/0x2c0 [13036.079642] ? smp_apic_timer_interrupt+0xa6/0x270 [13036.079652] ? apic_timer_interrupt+0xf/0x20 [13036.079656] [13036.079662] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13036.121179] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13036.121236] lmv_intent_open+0x338/0x1150 [lmv] [13036.121271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.121444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.121530] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13036.121559] ll_intent_lock+0xc5/0xaa0 [lustre] [13036.121856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.121951] ll_lookup_it+0xe0e/0x2270 [lustre] [13036.122462] ? do_raw_spin_unlock+0x75/0x190 [13036.122471] ? _raw_spin_unlock+0x12/0x30 [13036.122479] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13036.122850] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13036.122874] ll_atomic_open+0x36a/0xd70 [lustre] [13036.122962] lookup_open+0x356/0xb40 [13036.122972] do_last+0x313/0xfc0 [13036.122977] ? nd_jump_root+0xe5/0x160 [13036.122982] ? path_init+0x437/0x520 [13036.122986] path_openat+0xf7/0x500 [13036.122991] ? do_raw_spin_unlock+0x75/0x190 [13036.122995] do_filp_open+0x99/0x140 [13036.123002] ? getname_flags+0x6e/0x330 [13036.123007] ? __check_object_size+0xff/0x256 [13036.123015] ? do_raw_spin_unlock+0x75/0x190 [13036.123019] ? _raw_spin_unlock+0x12/0x30 [13036.123024] do_sys_openat2+0x2b4/0x410 [13036.123029] do_sys_open+0x73/0xa0 [13036.123034] __x64_sys_openat+0x24/0x30 [13036.123038] do_syscall_64+0xc1/0x440 [13036.123045] entry_SYSCALL_64_after_hwframe+0x49/0xae [13036.123053] RIP: 0033:0x7fa749552332 [13036.123110] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13036.123116] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13036.123121] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13036.123124] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13036.123126] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13036.123128] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13036.123130] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13036.855789] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13036.857491] do_raw_spin_lock+0x177/0x190 [13036.863049] _raw_spin_lock+0x21/0x30 [13036.865030] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13036.869744] obd_mod_free+0x46/0x140 [ptlrpc] [13036.873916] mdc_commit_open+0x91/0xb0 [mdc] [13036.877065] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13036.884034] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13036.889979] after_reply+0xe2c/0x1780 [ptlrpc] [13036.896971] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13036.901700] ? __next_timer_interrupt+0x160/0x160 [13036.905260] ptlrpcd+0xa82/0xe40 [ptlrpc] [13036.907177] ? do_wait_intr+0xf0/0xf0 [13036.909183] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13036.912169] kthread+0x1d1/0x200 [13036.914790] ? set_kthread_struct+0x70/0x70 [13036.916727] ret_from_fork+0x1f/0x30 [13040.097403] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11122s! [13040.103508] Showing busy workqueues and worker pools: [13040.106564] workqueue events: flags=0x0 [13040.108222] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13040.111613] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13040.115863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13040.118559] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13040.122208] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.124476] pending: kfree_rcu_monitor [13040.126482] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13040.129283] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13040.133238] workqueue events_long: flags=0x0 [13040.134667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.137437] pending: lu_objects_flush [obdclass] [13040.139455] workqueue events_unbound: flags=0x2 [13040.141174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13040.143832] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13040.148101] workqueue events_power_efficient: flags=0x80 [13040.150582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.153580] pending: check_lifetime [13040.155389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.158119] pending: neigh_periodic_work [13040.160044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.162384] pending: neigh_periodic_work [13040.163736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.166148] pending: do_cache_clean [13040.167442] workqueue mm_percpu_wq: flags=0x8 [13040.168888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.171218] pending: vmstat_update [13040.172691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.175633] pending: vmstat_update [13040.177607] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13040.180807] pending: vmstat_update [13040.182758] workqueue writeback: flags=0x4a [13040.185379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13040.188707] pending: wb_workfn, wb_workfn [13040.190848] workqueue xprtiod: flags=0xa [13040.192856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13040.195858] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13040.199629] workqueue ipv6_addrconf: flags=0x40008 [13040.202011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13040.205066] pending: addrconf_verify_work [13040.207176] workqueue knbd0-recv: flags=0x1a [13040.209460] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13040.212180] in-flight: 651:recv_work [13040.213392] workqueue ptlrpc_pinger: flags=0x2 [13040.215274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13040.217932] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13040.220345] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11123s workers=4 idle: 37 31846 28897 [13040.223761] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13030s workers=2 idle: 653 [13048.065506] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13048.075547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13048.107735] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13048.118344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13048.124227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13048.138923] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13048.154564] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13048.159336] RAX: 0000000000004189 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13048.165733] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13048.174342] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13048.177757] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13048.188417] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13048.199263] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13048.203703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13048.207876] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13048.214866] Call Trace: [13048.215924] [13048.224298] ? show_regs.cold.9+0x22/0x2f [13048.226161] ? watchdog_timer_fn.cold.6+0x62/0x10a [13048.230083] ? __hrtimer_run_queues+0x1c2/0x4d0 [13048.234631] ? watchdog+0x40/0x40 [13048.238226] ? hrtimer_interrupt+0x128/0x2c0 [13048.242202] ? smp_apic_timer_interrupt+0xa6/0x270 [13048.246010] ? apic_timer_interrupt+0xf/0x20 [13048.248141] [13048.249035] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13048.252728] do_raw_spin_lock+0x177/0x190 [13048.256802] _raw_spin_lock+0x21/0x30 [13048.258646] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13048.266058] ? do_raw_spin_unlock+0x75/0x190 [13048.270250] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13048.272855] ? do_wait_intr+0xf0/0xf0 [13048.274811] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13048.277869] mdc_close+0x53b/0xd70 [mdc] [13048.281529] lmv_close+0x18d/0x490 [lmv] [13048.286752] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13048.289804] ll_md_real_close+0xa5/0x2a0 [lustre] [13048.292793] ll_md_close+0x274/0x800 [lustre] [13048.306471] ll_file_release+0x128/0xc50 [lustre] [13048.311694] __fput+0x10d/0x440 [13048.316780] ____fput+0x12/0x20 [13048.318886] task_work_run+0xc6/0x110 [13048.321484] exit_to_usermode_loop+0x1dd/0x1f0 [13048.325886] do_syscall_64+0x3d6/0x440 [13048.328772] entry_SYSCALL_64_after_hwframe+0x49/0xae [13048.339963] RIP: 0033:0x7fa749552cd8 [13048.346781] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13048.366859] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13048.376370] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13048.379961] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13048.383761] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13048.390950] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13048.398072] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13064.058756] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13064.058896] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13064.067786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13064.067871] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13064.067879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13064.067882] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13064.067898] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13064.067903] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13064.067907] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13064.067910] RDX: 0000000000006bf8 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13064.067912] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13064.067915] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13064.067917] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13064.067920] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13064.067923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13064.067926] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13064.067934] Call Trace: [13064.067939] [13064.067945] ? show_regs.cold.9+0x22/0x2f [13064.067953] ? watchdog_timer_fn.cold.6+0x62/0x10a [13064.067960] ? __hrtimer_run_queues+0x1c2/0x4d0 [13064.067964] ? watchdog+0x40/0x40 [13064.067968] ? hrtimer_interrupt+0x128/0x2c0 [13064.067972] ? smp_apic_timer_interrupt+0xa6/0x270 [13064.067981] ? apic_timer_interrupt+0xf/0x20 [13064.067985] [13064.067988] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13064.067992] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13064.067996] do_raw_spin_lock+0x177/0x190 [13064.068001] _raw_spin_lock+0x21/0x30 [13064.068007] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13064.068628] obd_mod_free+0x46/0x140 [ptlrpc] [13064.068857] mdc_commit_open+0x91/0xb0 [mdc] [13064.068893] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13064.069089] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13064.069300] after_reply+0xe2c/0x1780 [ptlrpc] [13064.069499] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13064.069694] ? __next_timer_interrupt+0x160/0x160 [13064.069704] ptlrpcd+0xa82/0xe40 [ptlrpc] [13064.069909] ? do_wait_intr+0xf0/0xf0 [13064.069919] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13064.070135] kthread+0x1d1/0x200 [13064.070142] ? set_kthread_struct+0x70/0x70 [13064.070146] ret_from_fork+0x1f/0x30 [13064.290305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13064.327154] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13064.335064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13064.340039] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13064.347274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13064.350556] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13064.365044] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13064.374688] RAX: 0000000000001f8c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13064.378738] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13064.381724] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13064.385981] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13064.391781] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13064.394823] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13064.400630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13064.405618] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13064.416360] Call Trace: [13064.418740] [13064.420431] ? show_regs.cold.9+0x22/0x2f [13064.425086] ? watchdog_timer_fn.cold.6+0x62/0x10a [13064.427310] ? __hrtimer_run_queues+0x1c2/0x4d0 [13064.432476] ? watchdog+0x40/0x40 [13064.435428] ? hrtimer_interrupt+0x128/0x2c0 [13064.436885] ? smp_apic_timer_interrupt+0xa6/0x270 [13064.438945] ? apic_timer_interrupt+0xf/0x20 [13064.440908] [13064.442020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13064.445355] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13064.450546] do_raw_spin_lock+0x177/0x190 [13064.452784] _raw_spin_lock+0x21/0x30 [13064.461483] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13064.469961] process_one_work+0x2c8/0x700 [13064.475015] worker_thread+0x296/0x6e0 [13064.482972] ? rescuer_thread+0x570/0x570 [13064.487409] kthread+0x1d1/0x200 [13064.494422] ? set_kthread_struct+0x70/0x70 [13064.499651] ret_from_fork+0x1f/0x30 [13068.054049] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13068.056872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13068.113164] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13068.121722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13068.132718] RIP: 0010:native_safe_halt+0x2e/0x50 [13068.140278] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13068.151155] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13068.155716] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13068.167088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [13068.170370] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13068.180226] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13068.195290] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13068.205439] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13068.212847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13068.223508] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13068.231722] Call Trace: [13068.234182] [13068.234899] ? show_regs.cold.9+0x22/0x2f [13068.240713] ? watchdog_timer_fn.cold.6+0x62/0x10a [13068.245796] ? __hrtimer_run_queues+0x1c2/0x4d0 [13068.247246] ? watchdog+0x40/0x40 [13068.248097] ? hrtimer_interrupt+0x128/0x2c0 [13068.254358] ? smp_apic_timer_interrupt+0xa6/0x270 [13068.255759] ? apic_timer_interrupt+0xf/0x20 [13068.256969] [13068.260666] ? native_safe_halt+0x2e/0x50 [13068.261964] kvm_wait+0xcd/0xe0 [13068.271212] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13068.278147] do_raw_spin_lock+0x177/0x190 [13068.290429] _raw_spin_lock+0x21/0x30 [13068.298702] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13068.303968] ? do_raw_spin_unlock+0x75/0x190 [13068.309721] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13068.317861] ? do_wait_intr+0xf0/0xf0 [13068.321336] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13068.328461] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13068.330522] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13068.336550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13068.340616] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13068.345432] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13068.348112] mdc_intent_lock+0x353/0x7e0 [mdc] [13068.350198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13068.351896] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13068.355469] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13068.362594] lmv_intent_open+0x338/0x1150 [lmv] [13068.365331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13068.367953] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13068.372004] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13068.373834] ll_intent_lock+0xc5/0xaa0 [lustre] [13068.376899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13068.379103] ll_lookup_it+0xe0e/0x2270 [lustre] [13068.381442] ? do_raw_spin_unlock+0x75/0x190 [13068.384149] ? _raw_spin_unlock+0x12/0x30 [13068.387927] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13068.395598] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13068.405099] ll_atomic_open+0x36a/0xd70 [lustre] [13068.407048] lookup_open+0x356/0xb40 [13068.413573] do_last+0x313/0xfc0 [13068.417548] ? nd_jump_root+0xe5/0x160 [13068.426517] ? path_init+0x437/0x520 [13068.432139] path_openat+0xf7/0x500 [13068.435909] ? do_raw_spin_unlock+0x75/0x190 [13068.440108] do_filp_open+0x99/0x140 [13068.444538] ? getname_flags+0x6e/0x330 [13068.446456] ? __check_object_size+0xff/0x256 [13068.447914] ? do_raw_spin_unlock+0x75/0x190 [13068.457551] ? _raw_spin_unlock+0x12/0x30 [13068.462391] do_sys_openat2+0x2b4/0x410 [13068.466334] do_sys_open+0x73/0xa0 [13068.468654] __x64_sys_openat+0x24/0x30 [13068.478079] do_syscall_64+0xc1/0x440 [13068.480857] entry_SYSCALL_64_after_hwframe+0x49/0xae [13068.488892] RIP: 0033:0x7fa749552332 [13068.491991] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13068.507600] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13068.518762] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13068.527604] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13068.538077] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13068.545444] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13068.554548] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13070.316637] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11153s! [13070.322044] Showing busy workqueues and worker pools: [13070.330522] workqueue events: flags=0x0 [13070.332739] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13070.340370] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13070.348045] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13070.351208] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13070.365095] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.374241] pending: kfree_rcu_monitor [13070.381216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13070.392134] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13070.399887] workqueue events_long: flags=0x0 [13070.402132] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.406695] pending: lu_objects_flush [obdclass] [13070.414691] workqueue events_unbound: flags=0x2 [13070.417194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13070.421600] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13070.427242] workqueue events_power_efficient: flags=0x80 [13070.431363] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.448813] pending: check_lifetime [13070.458742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.472390] pending: neigh_periodic_work [13070.475043] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.485440] pending: neigh_periodic_work [13070.493943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.499413] pending: do_cache_clean [13070.503057] workqueue mm_percpu_wq: flags=0x8 [13070.510884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.523961] pending: vmstat_update [13070.528475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.535066] pending: vmstat_update [13070.540271] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13070.544552] pending: vmstat_update [13070.548461] workqueue writeback: flags=0x4a [13070.554165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13070.563208] pending: wb_workfn, wb_workfn [13070.568666] workqueue xprtiod: flags=0xa [13070.584683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13070.588335] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13070.597813] workqueue ipv6_addrconf: flags=0x40008 [13070.604025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13070.608632] pending: addrconf_verify_work [13070.610891] workqueue knbd0-recv: flags=0x1a [13070.612487] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13070.617981] in-flight: 651:recv_work [13070.621039] workqueue ptlrpc_pinger: flags=0x2 [13070.622553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13070.625444] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13070.632527] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11153s workers=4 idle: 37 31846 28897 [13070.638049] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13060s workers=2 idle: 653 [13076.059057] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13076.062266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13076.085435] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13076.091421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13076.096589] RIP: 0010:native_safe_halt+0x2e/0x50 [13076.099740] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13076.109934] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13076.113817] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13076.117468] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13076.121023] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13076.124291] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13076.127550] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13076.130823] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13076.134880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13076.137679] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13076.141231] Call Trace: [13076.142253] [13076.143251] ? show_regs.cold.9+0x22/0x2f [13076.145749] ? watchdog_timer_fn.cold.6+0x62/0x10a [13076.148368] ? __hrtimer_run_queues+0x1c2/0x4d0 [13076.150584] ? watchdog+0x40/0x40 [13076.152421] ? hrtimer_interrupt+0x128/0x2c0 [13076.154677] ? smp_apic_timer_interrupt+0xa6/0x270 [13076.156876] ? apic_timer_interrupt+0xf/0x20 [13076.159044] [13076.160202] ? native_safe_halt+0x2e/0x50 [13076.162029] kvm_wait+0xcd/0xe0 [13076.163353] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13076.165745] do_raw_spin_lock+0x177/0x190 [13076.167659] _raw_spin_lock+0x21/0x30 [13076.169227] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13076.171879] ? do_raw_spin_unlock+0x75/0x190 [13076.173848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13076.176272] ? do_wait_intr+0xf0/0xf0 [13076.177853] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13076.180592] mdc_close+0x53b/0xd70 [mdc] [13076.182371] lmv_close+0x18d/0x490 [lmv] [13076.184577] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13076.187223] ll_md_real_close+0xa5/0x2a0 [lustre] [13076.189392] ll_md_close+0x274/0x800 [lustre] [13076.191960] ll_file_release+0x128/0xc50 [lustre] [13076.194278] __fput+0x10d/0x440 [13076.195929] ____fput+0x12/0x20 [13076.197129] task_work_run+0xc6/0x110 [13076.199168] exit_to_usermode_loop+0x1dd/0x1f0 [13076.200976] do_syscall_64+0x3d6/0x440 [13076.202805] entry_SYSCALL_64_after_hwframe+0x49/0xae [13076.204851] RIP: 0033:0x7fa749552cd8 [13076.206696] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13076.214833] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13076.218322] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13076.221562] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13076.224753] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13076.228259] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13076.231656] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13092.057017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13092.059623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13092.072825] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13092.115315] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13092.115634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13092.115641] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13092.115913] RIP: 0010:native_safe_halt+0x2e/0x50 [13092.116045] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13092.116049] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13092.116053] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13092.116056] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13092.116058] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13092.116060] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13092.116063] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13092.116066] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13092.116069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13092.116071] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13092.116079] Call Trace: [13092.116084] [13092.116089] ? show_regs.cold.9+0x22/0x2f [13092.116099] ? watchdog_timer_fn.cold.6+0x62/0x10a [13092.116106] ? __hrtimer_run_queues+0x1c2/0x4d0 [13092.116111] ? watchdog+0x40/0x40 [13092.116116] ? hrtimer_interrupt+0x128/0x2c0 [13092.116120] ? smp_apic_timer_interrupt+0xa6/0x270 [13092.116127] ? apic_timer_interrupt+0xf/0x20 [13092.116131] [13092.116134] ? native_safe_halt+0x2e/0x50 [13092.116138] kvm_wait+0xcd/0xe0 [13092.116146] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13092.116156] do_raw_spin_lock+0x177/0x190 [13092.116160] _raw_spin_lock+0x21/0x30 [13092.116165] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13092.116710] process_one_work+0x2c8/0x700 [13092.116718] worker_thread+0x296/0x6e0 [13092.116722] ? rescuer_thread+0x570/0x570 [13092.116781] kthread+0x1d1/0x200 [13092.116786] ? set_kthread_struct+0x70/0x70 [13092.116790] ret_from_fork+0x1f/0x30 [13092.316786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13092.346616] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13092.352097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13092.361058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13092.364399] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13092.386722] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13092.389287] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13092.393430] RDX: 00000000000078a8 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13092.396247] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13092.402290] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13092.408279] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13092.418782] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13092.431063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13092.433845] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13092.442563] Call Trace: [13092.444842] [13092.445898] ? show_regs.cold.9+0x22/0x2f [13092.452405] ? watchdog_timer_fn.cold.6+0x62/0x10a [13092.457837] ? __hrtimer_run_queues+0x1c2/0x4d0 [13092.466783] ? watchdog+0x40/0x40 [13092.467961] ? hrtimer_interrupt+0x128/0x2c0 [13092.476221] ? smp_apic_timer_interrupt+0xa6/0x270 [13092.478594] ? apic_timer_interrupt+0xf/0x20 [13092.485568] [13092.486740] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13092.491741] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13092.500812] do_raw_spin_lock+0x177/0x190 [13092.514679] _raw_spin_lock+0x21/0x30 [13092.516892] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13092.529816] obd_mod_free+0x46/0x140 [ptlrpc] [13092.535455] mdc_commit_open+0x91/0xb0 [mdc] [13092.540211] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13092.543070] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13092.552832] after_reply+0xe2c/0x1780 [ptlrpc] [13092.557230] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13092.566630] ? __next_timer_interrupt+0x160/0x160 [13092.573125] ptlrpcd+0xa82/0xe40 [ptlrpc] [13092.577918] ? do_wait_intr+0xf0/0xf0 [13092.581796] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13092.584833] kthread+0x1d1/0x200 [13092.586840] ? set_kthread_struct+0x70/0x70 [13092.589215] ret_from_fork+0x1f/0x30 [13096.054360] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13096.060898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13096.126719] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13096.136238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13096.143606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13096.150791] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13096.165144] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13096.169558] RAX: 0000000000007e06 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13096.179612] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13096.184898] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13096.191759] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13096.198646] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13096.205676] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13096.209454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13096.211411] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13096.215418] Call Trace: [13096.216299] [13096.218662] ? show_regs.cold.9+0x22/0x2f [13096.220738] ? watchdog_timer_fn.cold.6+0x62/0x10a [13096.223788] ? __hrtimer_run_queues+0x1c2/0x4d0 [13096.227078] ? watchdog+0x40/0x40 [13096.228729] ? hrtimer_interrupt+0x128/0x2c0 [13096.234567] ? smp_apic_timer_interrupt+0xa6/0x270 [13096.239820] ? apic_timer_interrupt+0xf/0x20 [13096.241269] [13096.244177] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13096.248778] do_raw_spin_lock+0x177/0x190 [13096.251036] _raw_spin_lock+0x21/0x30 [13096.257653] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13096.261340] ? do_raw_spin_unlock+0x75/0x190 [13096.263508] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13096.266734] ? do_wait_intr+0xf0/0xf0 [13096.274839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13096.282767] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13096.286990] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13096.290803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13096.294935] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13096.296918] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13096.300422] mdc_intent_lock+0x353/0x7e0 [mdc] [13096.302425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13096.310463] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13096.317928] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13096.319930] lmv_intent_open+0x338/0x1150 [lmv] [13096.321527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13096.329230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13096.332515] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13096.336865] ll_intent_lock+0xc5/0xaa0 [lustre] [13096.341812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13096.345426] ll_lookup_it+0xe0e/0x2270 [lustre] [13096.348581] ? do_raw_spin_unlock+0x75/0x190 [13096.350463] ? _raw_spin_unlock+0x12/0x30 [13096.356717] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13096.358272] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13096.370711] ll_atomic_open+0x36a/0xd70 [lustre] [13096.380604] lookup_open+0x356/0xb40 [13096.384832] do_last+0x313/0xfc0 [13096.390101] ? nd_jump_root+0xe5/0x160 [13096.392392] ? path_init+0x437/0x520 [13096.395544] path_openat+0xf7/0x500 [13096.398559] ? do_raw_spin_unlock+0x75/0x190 [13096.402708] do_filp_open+0x99/0x140 [13096.405641] ? getname_flags+0x6e/0x330 [13096.412596] ? __check_object_size+0xff/0x256 [13096.419654] ? do_raw_spin_unlock+0x75/0x190 [13096.423729] ? _raw_spin_unlock+0x12/0x30 [13096.425135] do_sys_openat2+0x2b4/0x410 [13096.426456] do_sys_open+0x73/0xa0 [13096.436218] __x64_sys_openat+0x24/0x30 [13096.443390] do_syscall_64+0xc1/0x440 [13096.445624] entry_SYSCALL_64_after_hwframe+0x49/0xae [13096.449667] RIP: 0033:0x7fa749552332 [13096.455315] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13096.474647] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13096.481545] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13096.486098] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13096.493180] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13096.497829] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13096.506833] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13101.029781] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11183s! [13101.034885] Showing busy workqueues and worker pools: [13101.037604] workqueue events: flags=0x0 [13101.040013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13101.044145] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13101.053559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13101.061698] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13101.066626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.070540] pending: kfree_rcu_monitor [13101.072733] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13101.076407] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13101.080804] workqueue events_long: flags=0x0 [13101.083181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.087064] pending: lu_objects_flush [obdclass] [13101.089587] workqueue events_unbound: flags=0x2 [13101.091925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13101.095374] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13101.101016] workqueue events_power_efficient: flags=0x80 [13101.103704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.107331] pending: check_lifetime [13101.109892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.113107] pending: neigh_periodic_work [13101.115171] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.118160] pending: neigh_periodic_work [13101.120316] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.123378] pending: do_cache_clean [13101.125088] workqueue mm_percpu_wq: flags=0x8 [13101.127068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.130330] pending: vmstat_update [13101.132047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.135118] pending: vmstat_update [13101.137079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13101.140350] pending: vmstat_update [13101.142083] workqueue writeback: flags=0x4a [13101.144122] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13101.147086] pending: wb_workfn, wb_workfn [13101.149097] workqueue xprtiod: flags=0xa [13101.150819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13101.153795] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13101.157815] workqueue ipv6_addrconf: flags=0x40008 [13101.160040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13101.174334] pending: addrconf_verify_work [13101.176862] workqueue knbd0-recv: flags=0x1a [13101.181300] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13101.184603] in-flight: 651:recv_work [13101.188300] workqueue ptlrpc_pinger: flags=0x2 [13101.191761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13101.196408] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13101.200103] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11184s workers=4 idle: 37 31846 28897 [13101.207538] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13091s workers=2 idle: 653 [13104.059081] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13104.063686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13104.090899] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13104.096523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13104.102193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13104.105580] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13104.124889] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13104.129752] RAX: 0000000000007783 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13104.134893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13104.138541] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13104.141814] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13104.146098] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13104.150155] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13104.153625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13104.155263] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13104.157666] Call Trace: [13104.158849] [13104.159820] ? show_regs.cold.9+0x22/0x2f [13104.161582] ? watchdog_timer_fn.cold.6+0x62/0x10a [13104.164247] ? __hrtimer_run_queues+0x1c2/0x4d0 [13104.167992] ? watchdog+0x40/0x40 [13104.170878] ? hrtimer_interrupt+0x128/0x2c0 [13104.173054] ? smp_apic_timer_interrupt+0xa6/0x270 [13104.176453] ? apic_timer_interrupt+0xf/0x20 [13104.180433] [13104.182103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13104.187978] do_raw_spin_lock+0x177/0x190 [13104.190599] _raw_spin_lock+0x21/0x30 [13104.193104] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13104.196157] ? do_raw_spin_unlock+0x75/0x190 [13104.198734] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13104.203800] ? do_wait_intr+0xf0/0xf0 [13104.205910] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13104.208832] mdc_close+0x53b/0xd70 [mdc] [13104.211032] lmv_close+0x18d/0x490 [lmv] [13104.213019] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13104.222487] ll_md_real_close+0xa5/0x2a0 [lustre] [13104.230096] ll_md_close+0x274/0x800 [lustre] [13104.235983] ll_file_release+0x128/0xc50 [lustre] [13104.241005] __fput+0x10d/0x440 [13104.243084] ____fput+0x12/0x20 [13104.245800] task_work_run+0xc6/0x110 [13104.251969] exit_to_usermode_loop+0x1dd/0x1f0 [13104.254991] do_syscall_64+0x3d6/0x440 [13104.257507] entry_SYSCALL_64_after_hwframe+0x49/0xae [13104.264075] RIP: 0033:0x7fa749552cd8 [13104.266484] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13104.277150] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13104.281053] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13104.284867] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13104.288756] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13104.292568] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13104.302553] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13120.057061] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13120.058437] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13120.062306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13120.062379] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13120.062386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13120.062391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13120.071557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13120.071651] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13120.071662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13120.071665] RIP: 0010:native_safe_halt+0x2e/0x50 [13120.071683] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13120.071687] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13120.071692] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13120.071694] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13120.071697] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13120.071702] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13120.071705] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13120.071708] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13120.071711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13120.071714] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13120.071723] Call Trace: [13120.071732] [13120.071738] ? show_regs.cold.9+0x22/0x2f [13120.071751] ? watchdog_timer_fn.cold.6+0x62/0x10a [13120.071757] ? __hrtimer_run_queues+0x1c2/0x4d0 [13120.071765] ? watchdog+0x40/0x40 [13120.071770] ? hrtimer_interrupt+0x128/0x2c0 [13120.071774] ? smp_apic_timer_interrupt+0xa6/0x270 [13120.071784] ? apic_timer_interrupt+0xf/0x20 [13120.071789] [13120.071795] ? native_safe_halt+0x2e/0x50 [13120.071799] kvm_wait+0xcd/0xe0 [13120.071809] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13120.071817] do_raw_spin_lock+0x177/0x190 [13120.071825] _raw_spin_lock+0x21/0x30 [13120.071831] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13120.072429] obd_mod_free+0x46/0x140 [ptlrpc] [13120.072745] mdc_commit_open+0x91/0xb0 [mdc] [13120.115809] [13120.115818] RIP: 0010:native_safe_halt+0x2e/0x50 [13120.127756] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13120.128094] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13120.128347] after_reply+0xe2c/0x1780 [ptlrpc] [13120.128918] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13120.143823] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13120.144267] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13120.144273] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13120.144276] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13120.144278] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13120.144281] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13120.144453] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13120.144456] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13120.144459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13120.144593] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13120.144607] Call Trace: [13120.144612] [13120.144617] ? show_regs.cold.9+0x22/0x2f [13120.144630] ? watchdog_timer_fn.cold.6+0x62/0x10a [13120.144637] ? __hrtimer_run_queues+0x1c2/0x4d0 [13120.144641] ? watchdog+0x40/0x40 [13120.144645] ? hrtimer_interrupt+0x128/0x2c0 [13120.144649] ? smp_apic_timer_interrupt+0xa6/0x270 [13120.146732] ? apic_timer_interrupt+0xf/0x20 [13120.146736] [13120.146740] ? native_safe_halt+0x2e/0x50 [13120.146744] kvm_wait+0xcd/0xe0 [13120.146757] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13120.146767] do_raw_spin_lock+0x177/0x190 [13120.146771] _raw_spin_lock+0x21/0x30 [13120.146775] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13120.147082] process_one_work+0x2c8/0x700 [13120.147088] worker_thread+0x296/0x6e0 [13120.147092] ? rescuer_thread+0x570/0x570 [13120.147095] kthread+0x1d1/0x200 [13120.147100] ? set_kthread_struct+0x70/0x70 [13120.147103] ret_from_fork+0x1f/0x30 [13120.397703] ? __next_timer_interrupt+0x160/0x160 [13120.400379] ptlrpcd+0xa82/0xe40 [ptlrpc] [13120.403271] ? do_wait_intr+0xf0/0xf0 [13120.405728] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13120.409009] kthread+0x1d1/0x200 [13120.410262] ? set_kthread_struct+0x70/0x70 [13120.412582] ret_from_fork+0x1f/0x30 [13128.054032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13128.055924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13128.082952] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13128.093231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13128.098634] RIP: 0010:native_safe_halt+0x2e/0x50 [13128.102016] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13128.121912] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13128.125928] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13128.138546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [13128.147248] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13128.161279] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13128.167626] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13128.174325] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13128.181903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13128.184854] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13128.187963] Call Trace: [13128.191185] [13128.192733] ? show_regs.cold.9+0x22/0x2f [13128.196183] ? watchdog_timer_fn.cold.6+0x62/0x10a [13128.198104] ? __hrtimer_run_queues+0x1c2/0x4d0 [13128.201227] ? watchdog+0x40/0x40 [13128.205671] ? hrtimer_interrupt+0x128/0x2c0 [13128.210290] ? smp_apic_timer_interrupt+0xa6/0x270 [13128.213968] ? apic_timer_interrupt+0xf/0x20 [13128.216469] [13128.217233] ? native_safe_halt+0x2e/0x50 [13128.219237] kvm_wait+0xcd/0xe0 [13128.224007] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13128.231616] do_raw_spin_lock+0x177/0x190 [13128.248708] _raw_spin_lock+0x21/0x30 [13128.252141] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13128.257647] ? do_raw_spin_unlock+0x75/0x190 [13128.260291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13128.270664] ? do_wait_intr+0xf0/0xf0 [13128.274260] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13128.281303] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13128.283368] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13128.293614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13128.302875] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13128.313178] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13128.318641] mdc_intent_lock+0x353/0x7e0 [mdc] [13128.320303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13128.330224] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13128.337332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13128.341249] lmv_intent_open+0x338/0x1150 [lmv] [13128.342778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13128.347233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13128.357899] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13128.361102] ll_intent_lock+0xc5/0xaa0 [lustre] [13128.372777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13128.383078] ll_lookup_it+0xe0e/0x2270 [lustre] [13128.391022] ? do_raw_spin_unlock+0x75/0x190 [13128.401490] ? _raw_spin_unlock+0x12/0x30 [13128.406109] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13128.412605] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13128.426288] ll_atomic_open+0x36a/0xd70 [lustre] [13128.428159] lookup_open+0x356/0xb40 [13128.432846] do_last+0x313/0xfc0 [13128.438239] ? nd_jump_root+0xe5/0x160 [13128.440525] ? path_init+0x437/0x520 [13128.443247] path_openat+0xf7/0x500 [13128.449541] ? do_raw_spin_unlock+0x75/0x190 [13128.453449] do_filp_open+0x99/0x140 [13128.456197] ? getname_flags+0x6e/0x330 [13128.460008] ? __check_object_size+0xff/0x256 [13128.464178] ? do_raw_spin_unlock+0x75/0x190 [13128.467493] ? _raw_spin_unlock+0x12/0x30 [13128.471583] do_sys_openat2+0x2b4/0x410 [13128.474519] do_sys_open+0x73/0xa0 [13128.477407] __x64_sys_openat+0x24/0x30 [13128.480457] do_syscall_64+0xc1/0x440 [13128.484108] entry_SYSCALL_64_after_hwframe+0x49/0xae [13128.486859] RIP: 0033:0x7fa749552332 [13128.488533] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13128.496677] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13128.505298] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13128.509238] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13128.513342] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13128.517517] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13128.521910] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13131.232028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11214s! [13131.236065] Showing busy workqueues and worker pools: [13131.243582] workqueue events: flags=0x0 [13131.253418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13131.260690] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13131.278944] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13131.288508] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13131.300563] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.307070] pending: kfree_rcu_monitor [13131.310612] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13131.318986] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13131.334646] workqueue events_long: flags=0x0 [13131.336844] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.342504] pending: lu_objects_flush [obdclass] [13131.347515] workqueue events_unbound: flags=0x2 [13131.348992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13131.352535] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13131.361410] workqueue events_power_efficient: flags=0x80 [13131.364488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.368258] pending: check_lifetime [13131.370655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.378608] pending: neigh_periodic_work [13131.384469] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.392529] pending: neigh_periodic_work [13131.400531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.408907] pending: do_cache_clean [13131.411590] workqueue mm_percpu_wq: flags=0x8 [13131.418472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.424459] pending: vmstat_update [13131.429456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.434647] pending: vmstat_update [13131.441460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.449258] pending: vmstat_update [13131.452557] workqueue writeback: flags=0x4a [13131.457108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13131.460124] pending: wb_workfn, wb_workfn [13131.461383] workqueue xprtiod: flags=0xa [13131.463395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13131.468694] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13131.472841] workqueue ipv6_addrconf: flags=0x40008 [13131.474349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13131.484522] pending: addrconf_verify_work [13131.490479] workqueue knbd0-recv: flags=0x1a [13131.493343] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13131.498118] in-flight: 651:recv_work [13131.501766] workqueue ptlrpc_pinger: flags=0x2 [13131.505992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13131.508770] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13131.512103] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11214s workers=4 idle: 37 31846 28897 [13131.517051] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13121s workers=2 idle: 653 [13132.061629] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13132.068013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13132.133399] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13132.147342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13132.157570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13132.164942] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13132.179051] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13132.183541] RAX: 0000000000005986 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13132.186891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13132.190447] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13132.195088] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13132.198480] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13132.201786] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13132.205873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13132.209478] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13132.213750] Call Trace: [13132.215665] [13132.216664] ? show_regs.cold.9+0x22/0x2f [13132.219894] ? watchdog_timer_fn.cold.6+0x62/0x10a [13132.231732] ? __hrtimer_run_queues+0x1c2/0x4d0 [13132.237580] ? watchdog+0x40/0x40 [13132.241595] ? hrtimer_interrupt+0x128/0x2c0 [13132.246095] ? smp_apic_timer_interrupt+0xa6/0x270 [13132.251886] ? apic_timer_interrupt+0xf/0x20 [13132.286370] [13132.289879] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13132.293111] do_raw_spin_lock+0x177/0x190 [13132.296013] _raw_spin_lock+0x21/0x30 [13132.298164] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13132.300211] ? do_raw_spin_unlock+0x75/0x190 [13132.301723] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13132.304195] ? do_wait_intr+0xf0/0xf0 [13132.306084] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13132.309010] mdc_close+0x53b/0xd70 [mdc] [13132.310941] lmv_close+0x18d/0x490 [lmv] [13132.312588] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13132.315146] ll_md_real_close+0xa5/0x2a0 [lustre] [13132.317192] ll_md_close+0x274/0x800 [lustre] [13132.319116] ll_file_release+0x128/0xc50 [lustre] [13132.321133] __fput+0x10d/0x440 [13132.322455] ____fput+0x12/0x20 [13132.323748] task_work_run+0xc6/0x110 [13132.325338] exit_to_usermode_loop+0x1dd/0x1f0 [13132.327098] do_syscall_64+0x3d6/0x440 [13132.328690] entry_SYSCALL_64_after_hwframe+0x49/0xae [13132.330565] RIP: 0033:0x7fa749552cd8 [13132.332055] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13132.339878] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13132.343218] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13132.346224] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13132.349196] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13132.352224] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13132.354991] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13133.937013] rcu: INFO: rcu_sched self-detected stall on CPU [13133.939237] rcu: 2-...!: (6399049 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [13133.943311] (t=11220248 jiffies g=258701 q=4689) [13133.945351] rcu: rcu_sched kthread starved for 11217043 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [13133.949297] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13133.952650] rcu: RCU grace-period kthread stack dump: [13133.954771] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13133.959009] Call Trace: [13133.959828] __schedule+0x351/0xcb0 [13133.961132] schedule+0xc0/0x180 [13133.962662] schedule_timeout+0xb4/0x190 [13133.963957] ? do_raw_spin_unlock+0x75/0x190 [13133.965844] ? __next_timer_interrupt+0x160/0x160 [13133.967948] ? prepare_to_swait_event+0xf3/0x140 [13133.969913] rcu_gp_kthread+0x8dd/0x15b0 [13133.971822] ? force_qs_rnp+0x2c0/0x2c0 [13133.973336] kthread+0x1d1/0x200 [13133.974809] ? set_kthread_struct+0x70/0x70 [13133.976397] ret_from_fork+0x1f/0x30 [13133.977363] rcu: Stack dump where RCU GP kthread last ran: [13133.979122] Sending NMI from CPU 2 to CPUs 0: [13133.982330] NMI backtrace for cpu 0 [13133.982338] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13133.982340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13133.982341] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13133.982343] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13133.982345] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 [13133.982348] RAX: 0000000000006828 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13133.982350] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13133.982351] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13133.982353] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13133.982354] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13133.982355] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13133.982357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13133.982358] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13133.982360] Call Trace: [13133.982361] [13133.982362] ? show_regs.cold.9+0x22/0x2f [13133.982362] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13133.982364] ? nmi_cpu_backtrace_handler+0x15/0x40 [13133.982365] ? nmi_handle+0xbd/0x210 [13133.982366] ? default_do_nmi+0x4b/0x1b0 [13133.982367] ? do_nmi+0x280/0x380 [13133.982367] ? end_repeat_nmi+0x16/0x1f [13133.982368] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13133.982370] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13133.982371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13133.982372] [13133.982373] do_raw_spin_lock+0x177/0x190 [13133.982374] _raw_spin_lock+0x21/0x30 [13133.982375] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13133.982376] ? do_raw_spin_unlock+0x75/0x190 [13133.982378] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13133.982379] ? do_wait_intr+0xf0/0xf0 [13133.982380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13133.982381] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13133.982382] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13133.982384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13133.982385] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13133.982386] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13133.982388] mdc_intent_lock+0x353/0x7e0 [mdc] [13133.982389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13133.982390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13133.982392] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13133.982393] lmv_intent_open+0x338/0x1150 [lmv] [13133.982394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13133.982396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13133.982397] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13133.982398] ll_intent_lock+0xc5/0xaa0 [lustre] [13133.982400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13133.982401] ll_lookup_it+0xe0e/0x2270 [lustre] [13133.982402] ? do_raw_spin_unlock+0x75/0x190 [13133.982403] ? _raw_spin_unlock+0x12/0x30 [13133.982404] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13133.982406] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13133.982407] ll_atomic_open+0x36a/0xd70 [lustre] [13133.982408] lookup_open+0x356/0xb40 [13133.982409] do_last+0x313/0xfc0 [13133.982410] ? nd_jump_root+0xe5/0x160 [13133.982411] ? path_init+0x437/0x520 [13133.982412] path_openat+0xf7/0x500 [13133.982413] ? do_raw_spin_unlock+0x75/0x190 [13133.982414] do_filp_open+0x99/0x140 [13133.982415] ? getname_flags+0x6e/0x330 [13133.982417] ? __check_object_size+0xff/0x256 [13133.982418] ? do_raw_spin_unlock+0x75/0x190 [13133.982419] ? _raw_spin_unlock+0x12/0x30 [13133.982420] do_sys_openat2+0x2b4/0x410 [13133.982421] do_sys_open+0x73/0xa0 [13133.982422] __x64_sys_openat+0x24/0x30 [13133.982423] do_syscall_64+0xc1/0x440 [13133.982424] entry_SYSCALL_64_after_hwframe+0x49/0xae [13133.982426] RIP: 0033:0x7fa749552332 [13133.982428] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13133.982430] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13133.982433] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13133.982434] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13133.982436] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13133.982438] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13133.982439] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13133.983905] NMI backtrace for cpu 2 [13134.232322] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13134.236754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13134.240393] Call Trace: [13134.241548] [13134.242625] ? dump_stack+0xbb/0x10e [13134.243994] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13134.246205] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13134.248971] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13134.251229] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13134.253156] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13134.254821] ? rcu_dump_cpu_stacks+0x11b/0x171 [13134.256356] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13134.258164] ? update_process_times+0x6d/0xc0 [13134.260262] ? tick_sched_handle.isra.18+0x27/0x90 [13134.263090] ? tick_sched_timer+0x53/0xb0 [13134.264563] ? __hrtimer_run_queues+0x1c2/0x4d0 [13134.266253] ? tick_sched_handle.isra.18+0x90/0x90 [13134.267834] ? hrtimer_interrupt+0x128/0x2c0 [13134.271518] ? smp_apic_timer_interrupt+0xa6/0x270 [13134.273194] ? apic_timer_interrupt+0xf/0x20 [13134.274731] [13134.275551] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13134.277523] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13134.279291] ? do_raw_spin_lock+0x177/0x190 [13134.280916] ? _raw_spin_lock+0x21/0x30 [13134.282364] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13134.284771] ? obd_mod_free+0x46/0x140 [ptlrpc] [13134.286631] ? mdc_commit_open+0x91/0xb0 [mdc] [13134.288272] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13134.290113] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13134.292283] ? after_reply+0xe2c/0x1780 [ptlrpc] [13134.294134] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13134.296402] ? __next_timer_interrupt+0x160/0x160 [13134.297956] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13134.299701] ? do_wait_intr+0xf0/0xf0 [13134.300974] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13134.302782] ? kthread+0x1d1/0x200 [13134.303977] ? set_kthread_struct+0x70/0x70 [13134.305509] ? ret_from_fork+0x1f/0x30 [13148.059602] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13148.064287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13148.103898] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13148.110822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13148.117900] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13148.122867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13148.130861] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13148.146060] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13148.154216] RAX: 000000000000130a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13148.158945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13148.162945] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13148.170160] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13148.174181] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13148.181221] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13148.189898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13148.193288] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13148.198053] Call Trace: [13148.199283] [13148.200846] ? show_regs.cold.9+0x22/0x2f [13148.203307] ? watchdog_timer_fn.cold.6+0x62/0x10a [13148.208351] ? __hrtimer_run_queues+0x1c2/0x4d0 [13148.210971] ? watchdog+0x40/0x40 [13148.215665] ? hrtimer_interrupt+0x128/0x2c0 [13148.218166] ? smp_apic_timer_interrupt+0xa6/0x270 [13148.220899] ? apic_timer_interrupt+0xf/0x20 [13148.223139] [13148.224203] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13148.227283] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13148.230999] do_raw_spin_lock+0x177/0x190 [13148.236650] _raw_spin_lock+0x21/0x30 [13148.239131] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13148.250394] process_one_work+0x2c8/0x700 [13148.255672] worker_thread+0x296/0x6e0 [13148.259973] ? rescuer_thread+0x570/0x570 [13148.267301] kthread+0x1d1/0x200 [13148.269646] ? set_kthread_struct+0x70/0x70 [13148.272377] ret_from_fork+0x1f/0x30 [13160.054009] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13160.057657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13160.062030] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13160.065123] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13160.065199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13160.065304] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13160.065310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13160.065312] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13160.065326] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13160.065331] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13160.065334] RAX: 0000000000000687 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13160.065337] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13160.065339] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13160.065341] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13160.065343] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13160.065346] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13160.065348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13160.065351] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13160.065360] Call Trace: [13160.065365] [13160.065377] ? show_regs.cold.9+0x22/0x2f [13160.065386] ? watchdog_timer_fn.cold.6+0x62/0x10a [13160.065397] ? __hrtimer_run_queues+0x1c2/0x4d0 [13160.065401] ? watchdog+0x40/0x40 [13160.065405] ? hrtimer_interrupt+0x128/0x2c0 [13160.065410] ? smp_apic_timer_interrupt+0xa6/0x270 [13160.065427] ? apic_timer_interrupt+0xf/0x20 [13160.065432] [13160.065435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13160.065442] do_raw_spin_lock+0x177/0x190 [13160.065447] _raw_spin_lock+0x21/0x30 [13160.065452] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13160.065716] ? do_raw_spin_unlock+0x75/0x190 [13160.065723] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13160.065944] ? do_wait_intr+0xf0/0xf0 [13160.065952] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13160.066365] mdc_close+0x53b/0xd70 [mdc] [13160.067437] lmv_close+0x18d/0x490 [lmv] [13160.067467] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13160.067959] ll_md_real_close+0xa5/0x2a0 [lustre] [13160.068044] ll_md_close+0x274/0x800 [lustre] [13160.068127] ll_file_release+0x128/0xc50 [lustre] [13160.068642] __fput+0x10d/0x440 [13160.068653] ____fput+0x12/0x20 [13160.068658] task_work_run+0xc6/0x110 [13160.068668] exit_to_usermode_loop+0x1dd/0x1f0 [13160.068676] do_syscall_64+0x3d6/0x440 [13160.068681] entry_SYSCALL_64_after_hwframe+0x49/0xae [13160.068691] RIP: 0033:0x7fa749552cd8 [13160.068700] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13160.068703] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13160.068709] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13160.068711] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13160.068713] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13160.068715] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13160.068718] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13160.080587] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13160.080602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13160.080605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13160.080619] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13160.080622] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13160.080628] RAX: 00000000000001d7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13160.080630] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13160.080632] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13160.080634] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13160.080636] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13160.080639] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13160.080641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13160.080644] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13160.080652] Call Trace: [13160.080658] [13160.080664] ? show_regs.cold.9+0x22/0x2f [13160.080675] ? watchdog_timer_fn.cold.6+0x62/0x10a [13160.080681] ? __hrtimer_run_queues+0x1c2/0x4d0 [13160.080686] ? watchdog+0x40/0x40 [13160.080690] ? hrtimer_interrupt+0x128/0x2c0 [13160.080694] ? smp_apic_timer_interrupt+0xa6/0x270 [13160.080703] ? apic_timer_interrupt+0xf/0x20 [13160.080707] [13160.080710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13160.080717] do_raw_spin_lock+0x177/0x190 [13160.080723] _raw_spin_lock+0x21/0x30 [13160.080727] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13160.081015] ? do_raw_spin_unlock+0x75/0x190 [13160.081022] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13160.081287] ? do_wait_intr+0xf0/0xf0 [13160.081297] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13160.081510] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13160.081757] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13160.082050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.082311] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13160.082346] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13160.096732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13160.096810] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13160.096818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13160.096821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13160.096836] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13160.096840] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13160.096844] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13160.096846] RDX: 00000000000034f4 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13160.096848] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13160.096850] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13160.096853] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13160.096856] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13160.096858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13160.096861] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13160.096869] Call Trace: [13160.096875] [13160.096881] ? show_regs.cold.9+0x22/0x2f [13160.096890] ? watchdog_timer_fn.cold.6+0x62/0x10a [13160.096896] ? __hrtimer_run_queues+0x1c2/0x4d0 [13160.096900] ? watchdog+0x40/0x40 [13160.096904] ? hrtimer_interrupt+0x128/0x2c0 [13160.096908] ? smp_apic_timer_interrupt+0xa6/0x270 [13160.096916] ? apic_timer_interrupt+0xf/0x20 [13160.096921] [13160.096924] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13160.096928] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13160.096933] do_raw_spin_lock+0x177/0x190 [13160.096938] _raw_spin_lock+0x21/0x30 [13160.096944] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13160.099878] mdc_intent_lock+0x353/0x7e0 [mdc] [13160.099937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.100065] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13160.100547] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13160.100578] lmv_intent_open+0x338/0x1150 [lmv] [13160.100609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.100701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.100783] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13160.100805] ll_intent_lock+0xc5/0xaa0 [lustre] [13160.100886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.100964] ll_lookup_it+0xe0e/0x2270 [lustre] [13160.101048] ? do_raw_spin_unlock+0x75/0x190 [13160.101056] ? _raw_spin_unlock+0x12/0x30 [13160.101063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13160.101091] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13160.101109] ll_atomic_open+0x36a/0xd70 [lustre] [13160.101265] lookup_open+0x356/0xb40 [13160.101275] do_last+0x313/0xfc0 [13160.101280] ? nd_jump_root+0xe5/0x160 [13160.101285] ? path_init+0x437/0x520 [13160.101289] path_openat+0xf7/0x500 [13160.101295] ? do_raw_spin_unlock+0x75/0x190 [13160.101300] do_filp_open+0x99/0x140 [13160.101306] ? getname_flags+0x6e/0x330 [13160.101311] ? __check_object_size+0xff/0x256 [13160.101318] ? do_raw_spin_unlock+0x75/0x190 [13160.101322] ? _raw_spin_unlock+0x12/0x30 [13160.101326] do_sys_openat2+0x2b4/0x410 [13160.101331] do_sys_open+0x73/0xa0 [13160.101336] __x64_sys_openat+0x24/0x30 [13160.101340] do_syscall_64+0xc1/0x440 [13160.101347] entry_SYSCALL_64_after_hwframe+0x49/0xae [13160.101353] RIP: 0033:0x7fa749552332 [13160.101361] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13160.101365] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13160.101370] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13160.101372] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13160.101374] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13160.101376] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13160.101379] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13160.551744] obd_mod_free+0x46/0x140 [ptlrpc] [13160.553847] mdc_commit_open+0x91/0xb0 [mdc] [13160.560448] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13160.567002] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13160.572976] after_reply+0xe2c/0x1780 [ptlrpc] [13160.576972] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13160.581878] ? __next_timer_interrupt+0x160/0x160 [13160.590198] ptlrpcd+0xa82/0xe40 [ptlrpc] [13160.596448] ? do_wait_intr+0xf0/0xf0 [13160.598895] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13160.603766] kthread+0x1d1/0x200 [13160.606029] ? set_kthread_struct+0x70/0x70 [13160.608494] ret_from_fork+0x1f/0x30 [13161.952931] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11244s! [13161.957118] Showing busy workqueues and worker pools: [13161.959923] workqueue events: flags=0x0 [13161.961807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13161.964577] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13161.969572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13161.972202] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13161.976042] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13161.978707] pending: kfree_rcu_monitor [13161.980176] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13161.982883] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13161.986517] workqueue events_long: flags=0x0 [13161.988528] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13161.991408] pending: lu_objects_flush [obdclass] [13161.993348] workqueue events_unbound: flags=0x2 [13161.994737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13161.996821] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13162.000423] workqueue events_power_efficient: flags=0x80 [13162.002345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13162.005014] pending: check_lifetime [13162.006479] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13162.009071] pending: neigh_periodic_work [13162.010930] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13162.013832] pending: neigh_periodic_work [13162.014983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13162.016911] pending: do_cache_clean [13162.018172] workqueue mm_percpu_wq: flags=0x8 [13162.019822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13162.022626] pending: vmstat_update [13162.024543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13162.027448] pending: vmstat_update [13162.029196] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13162.032054] pending: vmstat_update [13162.033587] workqueue writeback: flags=0x4a [13162.035291] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13162.037840] pending: wb_workfn, wb_workfn [13162.039320] workqueue xprtiod: flags=0xa [13162.040934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13162.043162] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13162.045982] workqueue ipv6_addrconf: flags=0x40008 [13162.047757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13162.050270] pending: addrconf_verify_work [13162.051851] workqueue knbd0-recv: flags=0x1a [13162.053898] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13162.056955] in-flight: 651:recv_work [13162.058596] workqueue ptlrpc_pinger: flags=0x2 [13162.060557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13162.063083] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13162.066013] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11244s workers=4 idle: 37 31846 28897 [13162.069199] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13151s workers=2 idle: 653 [13176.057060] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13176.062153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13176.084670] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13176.090437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13176.094609] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13176.098023] RIP: 0010:native_safe_halt+0x2e/0x50 [13176.100644] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13176.107968] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13176.112329] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13176.120625] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13176.126809] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13176.129547] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13176.132703] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13176.139488] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13176.148791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13176.154067] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13176.160805] Call Trace: [13176.162628] [13176.163441] ? show_regs.cold.9+0x22/0x2f [13176.165032] ? watchdog_timer_fn.cold.6+0x62/0x10a [13176.169150] ? __hrtimer_run_queues+0x1c2/0x4d0 [13176.170401] ? watchdog+0x40/0x40 [13176.174338] ? hrtimer_interrupt+0x128/0x2c0 [13176.176973] ? smp_apic_timer_interrupt+0xa6/0x270 [13176.182795] ? apic_timer_interrupt+0xf/0x20 [13176.184624] [13176.186542] ? native_safe_halt+0x2e/0x50 [13176.188204] kvm_wait+0xcd/0xe0 [13176.189183] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13176.191697] do_raw_spin_lock+0x177/0x190 [13176.194042] _raw_spin_lock+0x21/0x30 [13176.198633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13176.203172] process_one_work+0x2c8/0x700 [13176.204574] worker_thread+0x296/0x6e0 [13176.206620] ? rescuer_thread+0x570/0x570 [13176.208603] kthread+0x1d1/0x200 [13176.212832] ? set_kthread_struct+0x70/0x70 [13176.216672] ret_from_fork+0x1f/0x30 [13188.054066] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13188.058024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13188.061252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [13188.061934] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13188.061940] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13188.062008] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13188.062014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13188.062017] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13188.062032] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13188.062037] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13188.062041] RAX: 00000000000053eb RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13188.062044] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13188.062046] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13188.062048] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13188.062051] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13188.062054] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13188.062058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13188.062060] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13188.062070] Call Trace: [13188.062076] [13188.062081] ? show_regs.cold.9+0x22/0x2f [13188.062091] ? watchdog_timer_fn.cold.6+0x62/0x10a [13188.062098] ? __hrtimer_run_queues+0x1c2/0x4d0 [13188.062103] ? watchdog+0x40/0x40 [13188.062107] ? hrtimer_interrupt+0x128/0x2c0 [13188.062112] ? smp_apic_timer_interrupt+0xa6/0x270 [13188.062120] ? apic_timer_interrupt+0xf/0x20 [13188.062124] [13188.062127] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13188.062133] do_raw_spin_lock+0x177/0x190 [13188.062138] _raw_spin_lock+0x21/0x30 [13188.062144] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13188.063079] ? do_raw_spin_unlock+0x75/0x190 [13188.063086] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13188.063917] ? do_wait_intr+0xf0/0xf0 [13188.064027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [13188.063936] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13188.064049] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13188.064094] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13188.064099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13188.064150] mdc_close+0x53b/0xd70 [mdc] [13188.064191] lmv_close+0x18d/0x490 [lmv] [13188.064326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13188.064336] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13188.064340] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13188.064344] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13188.064347] RDX: 0000000000000230 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13188.064349] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13188.064352] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13188.064354] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13188.064357] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13188.064359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13188.064362] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13188.064369] Call Trace: [13188.064371] [13188.064374] ? show_regs.cold.9+0x22/0x2f [13188.064381] ? watchdog_timer_fn.cold.6+0x62/0x10a [13188.064387] ? __hrtimer_run_queues+0x1c2/0x4d0 [13188.064390] ? watchdog+0x40/0x40 [13188.064394] ? hrtimer_interrupt+0x128/0x2c0 [13188.064398] ? smp_apic_timer_interrupt+0xa6/0x270 [13188.064406] ? apic_timer_interrupt+0xf/0x20 [13188.064411] [13188.064413] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13188.064418] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13188.064421] do_raw_spin_lock+0x177/0x190 [13188.064425] _raw_spin_lock+0x21/0x30 [13188.064430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13188.064221] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13188.064811] obd_mod_free+0x46/0x140 [ptlrpc] [13188.065056] ll_md_real_close+0xa5/0x2a0 [lustre] [13188.065168] ll_md_close+0x274/0x800 [lustre] [13188.065431] mdc_commit_open+0x91/0xb0 [mdc] [13188.065474] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13188.065835] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13188.065248] ll_file_release+0x128/0xc50 [lustre] [13188.066112] __fput+0x10d/0x440 [13188.066122] ____fput+0x12/0x20 [13188.066126] task_work_run+0xc6/0x110 [13188.066133] exit_to_usermode_loop+0x1dd/0x1f0 [13188.066140] do_syscall_64+0x3d6/0x440 [13188.066145] entry_SYSCALL_64_after_hwframe+0x49/0xae [13188.066153] RIP: 0033:0x7fa749552cd8 [13188.066160] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13188.066164] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13188.066169] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13188.066172] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13188.066174] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13188.066229] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13188.066231] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13188.066292] after_reply+0xe2c/0x1780 [ptlrpc] [13188.066491] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13188.066862] ? __next_timer_interrupt+0x160/0x160 [13188.066870] ptlrpcd+0xa82/0xe40 [ptlrpc] [13188.074009] ec(O) fid(O) [13188.077291] ? do_wait_intr+0xf0/0xf0 [13188.102615] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13188.102669] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13188.102677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13188.102679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13188.102696] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13188.102701] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13188.102705] RAX: 0000000000003b59 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13188.106798] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13188.110328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13188.112991] kthread+0x1d1/0x200 [13188.120077] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13188.120092] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13188.120095] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13188.120099] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13188.120102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13188.120105] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13188.120114] Call Trace: [13188.120120] [13188.120126] ? show_regs.cold.9+0x22/0x2f [13188.120139] ? watchdog_timer_fn.cold.6+0x62/0x10a [13188.120146] ? __hrtimer_run_queues+0x1c2/0x4d0 [13188.120151] ? watchdog+0x40/0x40 [13188.120155] ? hrtimer_interrupt+0x128/0x2c0 [13188.120160] ? smp_apic_timer_interrupt+0xa6/0x270 [13188.120169] ? apic_timer_interrupt+0xf/0x20 [13188.123872] ? set_kthread_struct+0x70/0x70 [13188.123892] ret_from_fork+0x1f/0x30 [13188.126628] [13188.126641] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13188.126658] do_raw_spin_lock+0x177/0x190 [13188.126664] _raw_spin_lock+0x21/0x30 [13188.126672] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13188.405848] ? do_raw_spin_unlock+0x75/0x190 [13188.407526] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13188.409798] ? do_wait_intr+0xf0/0xf0 [13188.411060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13188.413318] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13188.415549] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13188.418029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.420064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13188.422077] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13188.423900] mdc_intent_lock+0x353/0x7e0 [mdc] [13188.425887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.427913] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13188.430811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13188.432710] lmv_intent_open+0x338/0x1150 [lmv] [13188.434707] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.436724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.438952] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13188.440660] ll_intent_lock+0xc5/0xaa0 [lustre] [13188.442748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.444758] ll_lookup_it+0xe0e/0x2270 [lustre] [13188.446376] ? do_raw_spin_unlock+0x75/0x190 [13188.448012] ? _raw_spin_unlock+0x12/0x30 [13188.449338] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13188.451221] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13188.452906] ll_atomic_open+0x36a/0xd70 [lustre] [13188.454741] lookup_open+0x356/0xb40 [13188.456063] do_last+0x313/0xfc0 [13188.457079] ? nd_jump_root+0xe5/0x160 [13188.458615] ? path_init+0x437/0x520 [13188.460099] path_openat+0xf7/0x500 [13188.461513] ? do_raw_spin_unlock+0x75/0x190 [13188.463155] do_filp_open+0x99/0x140 [13188.464686] ? getname_flags+0x6e/0x330 [13188.466274] ? __check_object_size+0xff/0x256 [13188.467821] ? do_raw_spin_unlock+0x75/0x190 [13188.469387] ? _raw_spin_unlock+0x12/0x30 [13188.470981] do_sys_openat2+0x2b4/0x410 [13188.472547] do_sys_open+0x73/0xa0 [13188.473798] __x64_sys_openat+0x24/0x30 [13188.475285] do_syscall_64+0xc1/0x440 [13188.476586] entry_SYSCALL_64_after_hwframe+0x49/0xae [13188.478840] RIP: 0033:0x7fa749552332 [13188.480115] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13188.487147] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13188.490038] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13188.492743] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13188.495725] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13188.498445] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13188.501206] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13192.161054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11275s! [13192.164198] Showing busy workqueues and worker pools: [13192.166167] workqueue events: flags=0x0 [13192.167355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13192.170400] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13192.175128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13192.177906] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13192.181707] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.184160] pending: kfree_rcu_monitor [13192.185444] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13192.188035] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13192.191951] workqueue events_long: flags=0x0 [13192.193430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.196513] pending: lu_objects_flush [obdclass] [13192.198378] workqueue events_unbound: flags=0x2 [13192.200283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13192.202478] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13192.205952] workqueue events_power_efficient: flags=0x80 [13192.214642] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.222886] pending: check_lifetime [13192.224330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.231197] pending: neigh_periodic_work [13192.232786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.235641] pending: neigh_periodic_work [13192.237714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.240684] pending: do_cache_clean [13192.242164] workqueue mm_percpu_wq: flags=0x8 [13192.243670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.246170] pending: vmstat_update [13192.247761] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.250403] pending: vmstat_update [13192.252090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.254332] pending: vmstat_update [13192.259637] workqueue writeback: flags=0x4a [13192.273063] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13192.277400] pending: wb_workfn, wb_workfn [13192.283907] workqueue xprtiod: flags=0xa [13192.287165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13192.291892] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13192.302996] workqueue ipv6_addrconf: flags=0x40008 [13192.307791] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13192.314788] pending: addrconf_verify_work [13192.319257] workqueue knbd0-recv: flags=0x1a [13192.322102] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13192.330607] in-flight: 651:recv_work [13192.336442] workqueue ptlrpc_pinger: flags=0x2 [13192.340534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13192.353370] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13192.358391] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11275s workers=4 idle: 37 31846 28897 [13192.363115] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13182s workers=2 idle: 653 [13204.057155] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13204.060554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13204.082347] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13204.085991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13204.089252] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13204.091694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13204.094771] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13204.101786] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13204.105191] RAX: 0000000000005706 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13204.108698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13204.113304] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13204.120485] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13204.126692] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13204.130974] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13204.139690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13204.148158] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13204.159287] Call Trace: [13204.162794] [13204.163804] ? show_regs.cold.9+0x22/0x2f [13204.170600] ? watchdog_timer_fn.cold.6+0x62/0x10a [13204.174728] ? __hrtimer_run_queues+0x1c2/0x4d0 [13204.177942] ? watchdog+0x40/0x40 [13204.182963] ? hrtimer_interrupt+0x128/0x2c0 [13204.186136] ? smp_apic_timer_interrupt+0xa6/0x270 [13204.193898] ? apic_timer_interrupt+0xf/0x20 [13204.195427] [13204.196237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13204.204000] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13204.207340] do_raw_spin_lock+0x177/0x190 [13204.209491] _raw_spin_lock+0x21/0x30 [13204.214856] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13204.222119] process_one_work+0x2c8/0x700 [13204.223742] worker_thread+0x296/0x6e0 [13204.227292] ? rescuer_thread+0x570/0x570 [13204.235753] kthread+0x1d1/0x200 [13204.236911] ? set_kthread_struct+0x70/0x70 [13204.249558] ret_from_fork+0x1f/0x30 [13216.058033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13216.061379] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13216.061750] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13216.078776] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13216.078790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13216.078793] RIP: 0010:native_safe_halt+0x2e/0x50 [13216.078807] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13216.078811] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13216.078816] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13216.078818] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13216.078820] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13216.078822] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13216.078825] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13216.078828] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13216.078831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13216.078833] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13216.078841] Call Trace: [13216.078846] [13216.078852] ? show_regs.cold.9+0x22/0x2f [13216.078861] ? watchdog_timer_fn.cold.6+0x62/0x10a [13216.078867] ? __hrtimer_run_queues+0x1c2/0x4d0 [13216.078872] ? watchdog+0x40/0x40 [13216.078876] ? hrtimer_interrupt+0x128/0x2c0 [13216.078880] ? smp_apic_timer_interrupt+0xa6/0x270 [13216.078886] ? apic_timer_interrupt+0xf/0x20 [13216.078890] [13216.078893] ? native_safe_halt+0x2e/0x50 [13216.078896] kvm_wait+0xcd/0xe0 [13216.078904] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13216.078911] do_raw_spin_lock+0x177/0x190 [13216.078916] _raw_spin_lock+0x21/0x30 [13216.078923] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13216.079254] obd_mod_free+0x46/0x140 [ptlrpc] [13216.079455] mdc_commit_open+0x91/0xb0 [mdc] [13216.079491] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13216.079687] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13216.079877] after_reply+0xe2c/0x1780 [ptlrpc] [13216.083468] Modules linked in: [13216.087939] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13216.096336] lustre(O) [13216.098578] ? __next_timer_interrupt+0x160/0x160 [13216.107873] osp(O) [13216.110722] ptlrpcd+0xa82/0xe40 [ptlrpc] [13216.113369] ofd(O) [13216.116375] ? do_wait_intr+0xf0/0xf0 [13216.119272] lod(O) [13216.121998] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13216.124982] mdt(O) [13216.128239] kthread+0x1d1/0x200 [13216.130569] mdd(O) [13216.133376] ? set_kthread_struct+0x70/0x70 [13216.137142] mgs(O) [13216.138009] ret_from_fork+0x1f/0x30 [13216.233093] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13216.266469] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13216.271908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13216.276605] RIP: 0010:native_safe_halt+0x2e/0x50 [13216.279106] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13216.295932] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13216.298639] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13216.305904] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13216.309654] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13216.312814] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13216.322621] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13216.325902] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13216.334619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13216.341026] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13216.346681] Call Trace: [13216.348513] [13216.351353] ? show_regs.cold.9+0x22/0x2f [13216.354914] ? watchdog_timer_fn.cold.6+0x62/0x10a [13216.358903] ? __hrtimer_run_queues+0x1c2/0x4d0 [13216.365638] ? watchdog+0x40/0x40 [13216.369131] ? hrtimer_interrupt+0x128/0x2c0 [13216.378606] ? smp_apic_timer_interrupt+0xa6/0x270 [13216.383391] ? apic_timer_interrupt+0xf/0x20 [13216.388923] [13216.389833] ? native_safe_halt+0x2e/0x50 [13216.396137] kvm_wait+0xcd/0xe0 [13216.402312] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13216.403884] do_raw_spin_lock+0x177/0x190 [13216.405103] _raw_spin_lock+0x21/0x30 [13216.414418] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13216.416346] ? do_raw_spin_unlock+0x75/0x190 [13216.420973] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13216.425972] ? do_wait_intr+0xf0/0xf0 [13216.428694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13216.432117] mdc_close+0x53b/0xd70 [mdc] [13216.440001] lmv_close+0x18d/0x490 [lmv] [13216.445061] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13216.454610] ll_md_real_close+0xa5/0x2a0 [lustre] [13216.462388] ll_md_close+0x274/0x800 [lustre] [13216.463839] ll_file_release+0x128/0xc50 [lustre] [13216.465303] __fput+0x10d/0x440 [13216.474510] ____fput+0x12/0x20 [13216.478533] task_work_run+0xc6/0x110 [13216.485403] exit_to_usermode_loop+0x1dd/0x1f0 [13216.491744] do_syscall_64+0x3d6/0x440 [13216.495253] entry_SYSCALL_64_after_hwframe+0x49/0xae [13216.496638] RIP: 0033:0x7fa749552cd8 [13216.497584] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13216.512566] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13216.516371] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13216.520114] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13216.525846] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13216.534276] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13216.539551] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13220.054063] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13220.057068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13220.076011] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13220.082343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13220.085750] RIP: 0010:native_safe_halt+0x2e/0x50 [13220.088072] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13220.101253] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13220.105664] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13220.108599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [13220.111784] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13220.116410] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13220.118864] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13220.121508] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13220.125129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13220.128076] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13220.130261] Call Trace: [13220.132258] [13220.132934] ? show_regs.cold.9+0x22/0x2f [13220.134214] ? watchdog_timer_fn.cold.6+0x62/0x10a [13220.136964] ? __hrtimer_run_queues+0x1c2/0x4d0 [13220.138412] ? watchdog+0x40/0x40 [13220.140777] ? hrtimer_interrupt+0x128/0x2c0 [13220.142054] ? smp_apic_timer_interrupt+0xa6/0x270 [13220.144714] ? apic_timer_interrupt+0xf/0x20 [13220.146616] [13220.147116] ? native_safe_halt+0x2e/0x50 [13220.148182] kvm_wait+0xcd/0xe0 [13220.151091] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13220.152789] do_raw_spin_lock+0x177/0x190 [13220.153963] _raw_spin_lock+0x21/0x30 [13220.155192] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13220.157398] ? do_raw_spin_unlock+0x75/0x190 [13220.159635] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13220.161908] ? do_wait_intr+0xf0/0xf0 [13220.163301] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13220.165641] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13220.168177] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13220.170741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13220.173191] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13220.176227] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13220.178617] mdc_intent_lock+0x353/0x7e0 [mdc] [13220.180360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13220.183830] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13220.187136] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13220.190233] lmv_intent_open+0x338/0x1150 [lmv] [13220.191996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13220.194297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13220.196703] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13220.198990] ll_intent_lock+0xc5/0xaa0 [lustre] [13220.200865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13220.203499] ll_lookup_it+0xe0e/0x2270 [lustre] [13220.205474] ? do_raw_spin_unlock+0x75/0x190 [13220.207939] ? _raw_spin_unlock+0x12/0x30 [13220.211736] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13220.213337] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13220.215104] ll_atomic_open+0x36a/0xd70 [lustre] [13220.216968] lookup_open+0x356/0xb40 [13220.218282] do_last+0x313/0xfc0 [13220.219274] ? nd_jump_root+0xe5/0x160 [13220.220498] ? path_init+0x437/0x520 [13220.221900] path_openat+0xf7/0x500 [13220.223050] ? do_raw_spin_unlock+0x75/0x190 [13220.224410] do_filp_open+0x99/0x140 [13220.225678] ? getname_flags+0x6e/0x330 [13220.227018] ? __check_object_size+0xff/0x256 [13220.228355] ? do_raw_spin_unlock+0x75/0x190 [13220.229716] ? _raw_spin_unlock+0x12/0x30 [13220.231156] do_sys_openat2+0x2b4/0x410 [13220.232275] do_sys_open+0x73/0xa0 [13220.233325] __x64_sys_openat+0x24/0x30 [13220.234775] do_syscall_64+0xc1/0x440 [13220.235870] entry_SYSCALL_64_after_hwframe+0x49/0xae [13220.237409] RIP: 0033:0x7fa749552332 [13220.238756] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13220.244524] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13220.246902] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13220.249130] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13220.251512] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13220.253763] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13220.255910] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13222.880353] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11305s! [13222.883923] Showing busy workqueues and worker pools: [13222.886538] workqueue events: flags=0x0 [13222.888365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13222.891953] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13222.897339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13222.900308] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13222.903418] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.905776] pending: kfree_rcu_monitor [13222.907097] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13222.909530] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13222.912934] workqueue events_long: flags=0x0 [13222.914649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.917189] pending: lu_objects_flush [obdclass] [13222.918946] workqueue events_unbound: flags=0x2 [13222.920536] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13222.922556] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13222.926175] workqueue events_power_efficient: flags=0x80 [13222.928231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.930377] pending: check_lifetime [13222.931867] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.934130] pending: neigh_periodic_work [13222.935543] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.938269] pending: neigh_periodic_work [13222.940189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.943263] pending: do_cache_clean [13222.945024] workqueue mm_percpu_wq: flags=0x8 [13222.946947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.950393] pending: vmstat_update [13222.952288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.957362] pending: vmstat_update [13222.959458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.964401] pending: vmstat_update [13222.974155] workqueue writeback: flags=0x4a [13222.977528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13222.984180] pending: wb_workfn, wb_workfn [13222.990385] workqueue xprtiod: flags=0xa [13222.993357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13223.000294] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13223.008879] workqueue ipv6_addrconf: flags=0x40008 [13223.011923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13223.016456] pending: addrconf_verify_work [13223.019014] workqueue knbd0-recv: flags=0x1a [13223.021651] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13223.029135] in-flight: 651:recv_work [13223.032747] workqueue ptlrpc_pinger: flags=0x2 [13223.035784] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13223.046509] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13223.053091] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11305s workers=4 idle: 37 31846 28897 [13223.060280] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13212s workers=2 idle: 653 [13232.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13232.059791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13232.116077] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13232.125170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13232.133724] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13232.136984] RIP: 0010:native_safe_halt+0x2e/0x50 [13232.138821] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13232.154127] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13232.166488] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13232.174819] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13232.178114] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13232.185995] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13232.191552] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13232.197140] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13232.203589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13232.208798] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13232.213146] Call Trace: [13232.214232] [13232.215096] ? show_regs.cold.9+0x22/0x2f [13232.218014] ? watchdog_timer_fn.cold.6+0x62/0x10a [13232.220784] ? __hrtimer_run_queues+0x1c2/0x4d0 [13232.224991] ? watchdog+0x40/0x40 [13232.229033] ? hrtimer_interrupt+0x128/0x2c0 [13232.233026] ? smp_apic_timer_interrupt+0xa6/0x270 [13232.235408] ? apic_timer_interrupt+0xf/0x20 [13232.237750] [13232.241490] ? native_safe_halt+0x2e/0x50 [13232.245780] kvm_wait+0xcd/0xe0 [13232.247473] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13232.253490] do_raw_spin_lock+0x177/0x190 [13232.255773] _raw_spin_lock+0x21/0x30 [13232.267229] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13232.273790] process_one_work+0x2c8/0x700 [13232.275283] worker_thread+0x296/0x6e0 [13232.283322] ? rescuer_thread+0x570/0x570 [13232.287950] kthread+0x1d1/0x200 [13232.290039] ? set_kthread_struct+0x70/0x70 [13232.291569] ret_from_fork+0x1f/0x30 [13244.058030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13244.059793] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13244.062064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13244.062479] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13244.062487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13244.062490] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13244.062504] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13244.062509] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13244.062513] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13244.062516] RDX: 0000000000004d16 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13244.062518] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13244.062520] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13244.062522] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13244.062525] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13244.062528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13244.062530] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13244.062539] Call Trace: [13244.062544] [13244.062549] ? show_regs.cold.9+0x22/0x2f [13244.062558] ? watchdog_timer_fn.cold.6+0x62/0x10a [13244.062564] ? __hrtimer_run_queues+0x1c2/0x4d0 [13244.062569] ? watchdog+0x40/0x40 [13244.062573] ? hrtimer_interrupt+0x128/0x2c0 [13244.062577] ? smp_apic_timer_interrupt+0xa6/0x270 [13244.062585] ? apic_timer_interrupt+0xf/0x20 [13244.062589] [13244.062592] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13244.062597] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13244.062602] do_raw_spin_lock+0x177/0x190 [13244.062606] _raw_spin_lock+0x21/0x30 [13244.062612] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13244.063426] obd_mod_free+0x46/0x140 [ptlrpc] [13244.063627] mdc_commit_open+0x91/0xb0 [mdc] [13244.063847] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13244.064057] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13244.064254] after_reply+0xe2c/0x1780 [ptlrpc] [13244.064513] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13244.073728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13244.073805] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13244.073814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13244.073817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13244.073831] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13244.073835] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13244.073839] RAX: 0000000000006b0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13244.073842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13244.073844] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13244.073846] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13244.073848] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13244.073851] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13244.073853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13244.073855] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13244.073863] Call Trace: [13244.073869] [13244.073875] ? show_regs.cold.9+0x22/0x2f [13244.073885] ? watchdog_timer_fn.cold.6+0x62/0x10a [13244.073891] ? __hrtimer_run_queues+0x1c2/0x4d0 [13244.073896] ? watchdog+0x40/0x40 [13244.073900] ? hrtimer_interrupt+0x128/0x2c0 [13244.073903] ? smp_apic_timer_interrupt+0xa6/0x270 [13244.073912] ? apic_timer_interrupt+0xf/0x20 [13244.073917] [13244.073919] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13244.073926] do_raw_spin_lock+0x177/0x190 [13244.073931] _raw_spin_lock+0x21/0x30 [13244.073936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13244.074246] ? do_raw_spin_unlock+0x75/0x190 [13244.074251] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13244.074553] ? do_wait_intr+0xf0/0xf0 [13244.074561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13244.074943] mdc_close+0x53b/0xd70 [mdc] [13244.074995] lmv_close+0x18d/0x490 [lmv] [13244.075035] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13244.075212] ll_md_real_close+0xa5/0x2a0 [lustre] [13244.075482] ll_md_close+0x274/0x800 [lustre] [13244.075563] ll_file_release+0x128/0xc50 [lustre] [13244.075643] __fput+0x10d/0x440 [13244.075651] ____fput+0x12/0x20 [13244.075656] task_work_run+0xc6/0x110 [13244.075784] exit_to_usermode_loop+0x1dd/0x1f0 [13244.075791] do_syscall_64+0x3d6/0x440 [13244.075797] entry_SYSCALL_64_after_hwframe+0x49/0xae [13244.075804] RIP: 0033:0x7fa749552cd8 [13244.075811] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13244.075815] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13244.075819] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13244.075821] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13244.075823] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13244.075825] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13244.075827] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13244.623112] ? __next_timer_interrupt+0x160/0x160 [13244.627258] ptlrpcd+0xa82/0xe40 [ptlrpc] [13244.635104] ? do_wait_intr+0xf0/0xf0 [13244.640571] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13244.644398] kthread+0x1d1/0x200 [13244.647950] ? set_kthread_struct+0x70/0x70 [13244.651967] ret_from_fork+0x1f/0x30 [13248.054523] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13248.059720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13248.113535] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13248.132226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13248.139797] RIP: 0010:native_safe_halt+0x2e/0x50 [13248.142083] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13248.163046] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13248.175642] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13248.182494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13248.198244] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13248.203854] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13248.211743] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13248.218084] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13248.221140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13248.230049] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13248.236627] Call Trace: [13248.238542] [13248.242819] ? show_regs.cold.9+0x22/0x2f [13248.250854] ? watchdog_timer_fn.cold.6+0x62/0x10a [13248.257039] ? __hrtimer_run_queues+0x1c2/0x4d0 [13248.268526] ? watchdog+0x40/0x40 [13248.270930] ? hrtimer_interrupt+0x128/0x2c0 [13248.273799] ? smp_apic_timer_interrupt+0xa6/0x270 [13248.275412] ? apic_timer_interrupt+0xf/0x20 [13248.285428] [13248.286784] ? native_safe_halt+0x2e/0x50 [13248.291819] kvm_wait+0xcd/0xe0 [13248.293847] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13248.298646] do_raw_spin_lock+0x177/0x190 [13248.301975] _raw_spin_lock+0x21/0x30 [13248.303229] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13248.307073] ? do_raw_spin_unlock+0x75/0x190 [13248.309263] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13248.313071] ? do_wait_intr+0xf0/0xf0 [13248.315513] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13248.318127] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13248.320810] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13248.324517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13248.331243] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13248.335020] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13248.336129] mdc_intent_lock+0x353/0x7e0 [mdc] [13248.339675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13248.342848] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13248.346697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13248.349433] lmv_intent_open+0x338/0x1150 [lmv] [13248.351789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13248.354171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13248.356655] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13248.358470] ll_intent_lock+0xc5/0xaa0 [lustre] [13248.360521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13248.362836] ll_lookup_it+0xe0e/0x2270 [lustre] [13248.365460] ? do_raw_spin_unlock+0x75/0x190 [13248.369475] ? _raw_spin_unlock+0x12/0x30 [13248.372528] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13248.376945] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13248.380053] ll_atomic_open+0x36a/0xd70 [lustre] [13248.386755] lookup_open+0x356/0xb40 [13248.388693] do_last+0x313/0xfc0 [13248.391995] ? nd_jump_root+0xe5/0x160 [13248.395897] ? path_init+0x437/0x520 [13248.397688] path_openat+0xf7/0x500 [13248.400571] ? do_raw_spin_unlock+0x75/0x190 [13248.403685] do_filp_open+0x99/0x140 [13248.405050] ? getname_flags+0x6e/0x330 [13248.406439] ? __check_object_size+0xff/0x256 [13248.417697] ? do_raw_spin_unlock+0x75/0x190 [13248.421267] ? _raw_spin_unlock+0x12/0x30 [13248.425836] do_sys_openat2+0x2b4/0x410 [13248.429475] do_sys_open+0x73/0xa0 [13248.432201] __x64_sys_openat+0x24/0x30 [13248.435056] do_syscall_64+0xc1/0x440 [13248.438544] entry_SYSCALL_64_after_hwframe+0x49/0xae [13248.441242] RIP: 0033:0x7fa749552332 [13248.443429] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13248.459210] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13248.469682] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13248.472165] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13248.481390] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13248.489594] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13248.495893] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13253.090519] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11335s! [13253.100582] Showing busy workqueues and worker pools: [13253.104941] workqueue events: flags=0x0 [13253.109601] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13253.113784] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13253.128118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13253.134398] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13253.139176] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.142845] pending: kfree_rcu_monitor [13253.147367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13253.154444] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13253.164080] workqueue events_long: flags=0x0 [13253.170219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.175682] pending: lu_objects_flush [obdclass] [13253.178088] workqueue events_unbound: flags=0x2 [13253.180288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13253.183869] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13253.188432] workqueue events_power_efficient: flags=0x80 [13253.192429] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.199628] pending: check_lifetime [13253.201707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.209073] pending: neigh_periodic_work [13253.213629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.217291] pending: neigh_periodic_work [13253.221478] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.225054] pending: do_cache_clean [13253.230890] workqueue mm_percpu_wq: flags=0x8 [13253.234334] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.237344] pending: vmstat_update [13253.239034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.242447] pending: vmstat_update [13253.244535] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.249299] pending: vmstat_update [13253.251208] workqueue writeback: flags=0x4a [13253.253016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13253.255176] pending: wb_workfn, wb_workfn [13253.257200] workqueue xprtiod: flags=0xa [13253.258567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13253.261537] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13253.265853] workqueue ipv6_addrconf: flags=0x40008 [13253.268318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13253.271959] pending: addrconf_verify_work [13253.275737] workqueue knbd0-recv: flags=0x1a [13253.278321] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13253.281460] in-flight: 651:recv_work [13253.284687] workqueue ptlrpc_pinger: flags=0x2 [13253.287352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13253.290763] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13253.294111] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11336s workers=4 idle: 37 31846 28897 [13253.298764] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13243s workers=2 idle: 653 [13260.068658] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13260.078992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13260.118099] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13260.128940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13260.134917] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13260.141123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13260.143982] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13260.162612] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13260.169524] RAX: 0000000000006c12 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13260.173201] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13260.176361] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13260.188619] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13260.195203] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13260.198692] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13260.201379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13260.204480] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13260.208254] Call Trace: [13260.209488] [13260.210523] ? show_regs.cold.9+0x22/0x2f [13260.212488] ? watchdog_timer_fn.cold.6+0x62/0x10a [13260.215237] ? __hrtimer_run_queues+0x1c2/0x4d0 [13260.217732] ? watchdog+0x40/0x40 [13260.219561] ? hrtimer_interrupt+0x128/0x2c0 [13260.221718] ? smp_apic_timer_interrupt+0xa6/0x270 [13260.223661] ? apic_timer_interrupt+0xf/0x20 [13260.225313] [13260.226271] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13260.231322] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13260.234169] do_raw_spin_lock+0x177/0x190 [13260.239808] _raw_spin_lock+0x21/0x30 [13260.242847] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13260.246999] process_one_work+0x2c8/0x700 [13260.249214] worker_thread+0x296/0x6e0 [13260.251057] ? rescuer_thread+0x570/0x570 [13260.254971] kthread+0x1d1/0x200 [13260.258874] ? set_kthread_struct+0x70/0x70 [13260.261869] ret_from_fork+0x1f/0x30 [13272.058101] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13272.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13272.061515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13272.061592] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13272.061600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13272.061603] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13272.061617] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13272.061622] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13272.061627] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13272.061629] RDX: 0000000000001fd2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13272.061631] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13272.061634] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13272.061636] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13272.061639] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13272.061642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13272.061644] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13272.061654] Call Trace: [13272.061660] [13272.061665] ? show_regs.cold.9+0x22/0x2f [13272.061673] ? watchdog_timer_fn.cold.6+0x62/0x10a [13272.061681] ? __hrtimer_run_queues+0x1c2/0x4d0 [13272.061686] ? watchdog+0x40/0x40 [13272.061690] ? hrtimer_interrupt+0x128/0x2c0 [13272.061695] ? smp_apic_timer_interrupt+0xa6/0x270 [13272.061703] ? apic_timer_interrupt+0xf/0x20 [13272.061707] [13272.061710] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13272.061714] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13272.061718] do_raw_spin_lock+0x177/0x190 [13272.061723] _raw_spin_lock+0x21/0x30 [13272.061729] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13272.064886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13272.064957] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13272.064965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13272.064968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13272.064981] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13272.064985] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13272.064989] RAX: 0000000000003ce9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13272.064992] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13272.064994] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13272.064996] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13272.064998] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13272.065001] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13272.065004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13272.065006] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13272.065015] Call Trace: [13272.065020] [13272.065025] ? show_regs.cold.9+0x22/0x2f [13272.065034] ? watchdog_timer_fn.cold.6+0x62/0x10a [13272.065040] ? __hrtimer_run_queues+0x1c2/0x4d0 [13272.065044] ? watchdog+0x40/0x40 [13272.065048] ? hrtimer_interrupt+0x128/0x2c0 [13272.065052] ? smp_apic_timer_interrupt+0xa6/0x270 [13272.065060] ? apic_timer_interrupt+0xf/0x20 [13272.065064] [13272.065068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13272.065075] do_raw_spin_lock+0x177/0x190 [13272.065085] _raw_spin_lock+0x21/0x30 [13272.065091] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13272.086517] obd_mod_free+0x46/0x140 [ptlrpc] [13272.090253] ? do_raw_spin_unlock+0x75/0x190 [13272.093826] mdc_commit_open+0x91/0xb0 [mdc] [13272.093872] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13272.094122] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13272.094402] after_reply+0xe2c/0x1780 [ptlrpc] [13272.094604] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13272.094806] ? __next_timer_interrupt+0x160/0x160 [13272.094817] ptlrpcd+0xa82/0xe40 [ptlrpc] [13272.095029] ? do_wait_intr+0xf0/0xf0 [13272.095039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13272.095403] kthread+0x1d1/0x200 [13272.095409] ? set_kthread_struct+0x70/0x70 [13272.095413] ret_from_fork+0x1f/0x30 [13272.532135] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13272.537950] ? do_wait_intr+0xf0/0xf0 [13272.539682] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13272.548069] mdc_close+0x53b/0xd70 [mdc] [13272.553536] lmv_close+0x18d/0x490 [lmv] [13272.562202] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13272.570435] ll_md_real_close+0xa5/0x2a0 [lustre] [13272.577414] ll_md_close+0x274/0x800 [lustre] [13272.582484] ll_file_release+0x128/0xc50 [lustre] [13272.584848] __fput+0x10d/0x440 [13272.587867] ____fput+0x12/0x20 [13272.591313] task_work_run+0xc6/0x110 [13272.594402] exit_to_usermode_loop+0x1dd/0x1f0 [13272.596658] do_syscall_64+0x3d6/0x440 [13272.600620] entry_SYSCALL_64_after_hwframe+0x49/0xae [13272.605203] RIP: 0033:0x7fa749552cd8 [13272.606827] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13272.622706] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13272.627944] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13272.633793] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13272.636979] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13272.640351] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13272.645610] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13280.054334] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13280.067665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13280.100801] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13280.122011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13280.141970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13280.144371] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13280.154368] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13280.164593] RAX: 0000000000001e19 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13280.171335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13280.179826] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13280.190519] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13280.201043] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13280.207994] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13280.217546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13280.219817] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13280.231166] Call Trace: [13280.232135] [13280.252275] ? show_regs.cold.9+0x22/0x2f [13280.253797] ? watchdog_timer_fn.cold.6+0x62/0x10a [13280.261453] ? __hrtimer_run_queues+0x1c2/0x4d0 [13280.265821] ? watchdog+0x40/0x40 [13280.275536] ? hrtimer_interrupt+0x128/0x2c0 [13280.277650] ? smp_apic_timer_interrupt+0xa6/0x270 [13280.289788] ? apic_timer_interrupt+0xf/0x20 [13280.293031] [13280.294426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13280.296604] do_raw_spin_lock+0x177/0x190 [13280.307620] _raw_spin_lock+0x21/0x30 [13280.314979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13280.327463] ? do_raw_spin_unlock+0x75/0x190 [13280.335887] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13280.337908] ? do_wait_intr+0xf0/0xf0 [13280.346215] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13280.351611] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13280.356961] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13280.375925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13280.385565] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13280.393753] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13280.402897] mdc_intent_lock+0x353/0x7e0 [mdc] [13280.404682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13280.414454] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13280.418063] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13280.424813] lmv_intent_open+0x338/0x1150 [lmv] [13280.427917] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13280.430675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13280.433112] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13280.435064] ll_intent_lock+0xc5/0xaa0 [lustre] [13280.436768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13280.445781] ll_lookup_it+0xe0e/0x2270 [lustre] [13280.449254] ? do_raw_spin_unlock+0x75/0x190 [13280.455733] ? _raw_spin_unlock+0x12/0x30 [13280.458293] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13280.461010] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13280.465837] ll_atomic_open+0x36a/0xd70 [lustre] [13280.472565] lookup_open+0x356/0xb40 [13280.480263] do_last+0x313/0xfc0 [13280.482738] ? nd_jump_root+0xe5/0x160 [13280.485722] ? path_init+0x437/0x520 [13280.488496] path_openat+0xf7/0x500 [13280.494754] ? do_raw_spin_unlock+0x75/0x190 [13280.497425] do_filp_open+0x99/0x140 [13280.499425] ? getname_flags+0x6e/0x330 [13280.501423] ? __check_object_size+0xff/0x256 [13280.503744] ? do_raw_spin_unlock+0x75/0x190 [13280.505829] ? _raw_spin_unlock+0x12/0x30 [13280.510848] do_sys_openat2+0x2b4/0x410 [13280.513251] do_sys_open+0x73/0xa0 [13280.519911] __x64_sys_openat+0x24/0x30 [13280.525660] do_syscall_64+0xc1/0x440 [13280.530326] entry_SYSCALL_64_after_hwframe+0x49/0xae [13280.536408] RIP: 0033:0x7fa749552332 [13280.538404] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13280.551823] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13280.560535] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13280.565605] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13280.568443] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13280.573373] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13280.576151] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13283.809655] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11366s! [13283.818439] Showing busy workqueues and worker pools: [13283.821635] workqueue events: flags=0x0 [13283.825166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13283.830078] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13283.839386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13283.846456] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13283.854600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.859786] pending: kfree_rcu_monitor [13283.862174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13283.867750] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13283.877124] workqueue events_long: flags=0x0 [13283.880262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.887326] pending: lu_objects_flush [obdclass] [13283.899836] workqueue events_unbound: flags=0x2 [13283.908368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13283.911060] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13283.921312] workqueue events_power_efficient: flags=0x80 [13283.924656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.932056] pending: check_lifetime [13283.936085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.940156] pending: neigh_periodic_work [13283.946141] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.950481] pending: neigh_periodic_work [13283.952516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.956426] pending: do_cache_clean [13283.960758] workqueue mm_percpu_wq: flags=0x8 [13283.962963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.966483] pending: vmstat_update [13283.969389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.972492] pending: vmstat_update [13283.973841] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.992570] pending: vmstat_update [13283.995222] workqueue writeback: flags=0x4a [13283.998226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13284.004124] pending: wb_workfn, wb_workfn [13284.007353] workqueue xprtiod: flags=0xa [13284.010232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13284.013701] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13284.017376] workqueue ipv6_addrconf: flags=0x40008 [13284.021342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13284.023582] pending: addrconf_verify_work [13284.028601] workqueue knbd0-recv: flags=0x1a [13284.031792] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13284.040636] in-flight: 651:recv_work [13284.043389] workqueue ptlrpc_pinger: flags=0x2 [13284.046235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13284.048923] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13284.053097] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11366s workers=4 idle: 37 31846 28897 [13284.056317] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13273s workers=2 idle: 653 [13288.057307] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13288.066864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13288.126533] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13288.145284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13288.152049] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13288.159541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13288.163359] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13288.171797] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13288.184594] RAX: 0000000000007b94 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13288.194893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13288.206592] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13288.212199] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13288.228637] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13288.240862] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13288.242975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13288.252545] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13288.256986] Call Trace: [13288.258720] [13288.260124] ? show_regs.cold.9+0x22/0x2f [13288.262296] ? watchdog_timer_fn.cold.6+0x62/0x10a [13288.269415] ? __hrtimer_run_queues+0x1c2/0x4d0 [13288.276810] ? watchdog+0x40/0x40 [13288.280876] ? hrtimer_interrupt+0x128/0x2c0 [13288.288038] ? smp_apic_timer_interrupt+0xa6/0x270 [13288.294002] ? apic_timer_interrupt+0xf/0x20 [13288.296122] [13288.297191] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13288.306877] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13288.308997] do_raw_spin_lock+0x177/0x190 [13288.312009] _raw_spin_lock+0x21/0x30 [13288.313778] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13288.317058] process_one_work+0x2c8/0x700 [13288.319306] worker_thread+0x296/0x6e0 [13288.330603] ? rescuer_thread+0x570/0x570 [13288.338025] kthread+0x1d1/0x200 [13288.339095] ? set_kthread_struct+0x70/0x70 [13288.347698] ret_from_fork+0x1f/0x30 [13300.059241] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13300.070447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13300.071749] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13300.123242] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13300.123257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13300.123259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13300.123273] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13300.123277] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13300.123283] RAX: 0000000000004ec7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13300.123285] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13300.123287] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13300.123289] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13300.123291] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13300.123294] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13300.123297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13300.123299] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13300.123307] Call Trace: [13300.123312] [13300.123317] ? show_regs.cold.9+0x22/0x2f [13300.123327] ? watchdog_timer_fn.cold.6+0x62/0x10a [13300.123333] ? __hrtimer_run_queues+0x1c2/0x4d0 [13300.123337] ? watchdog+0x40/0x40 [13300.123341] ? hrtimer_interrupt+0x128/0x2c0 [13300.123346] ? smp_apic_timer_interrupt+0xa6/0x270 [13300.123354] ? apic_timer_interrupt+0xf/0x20 [13300.123359] [13300.123361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13300.123369] do_raw_spin_lock+0x177/0x190 [13300.123375] _raw_spin_lock+0x21/0x30 [13300.123380] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13300.124108] ? do_raw_spin_unlock+0x75/0x190 [13300.124116] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13300.124503] ? do_wait_intr+0xf0/0xf0 [13300.124512] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13300.124928] mdc_close+0x53b/0xd70 [mdc] [13300.124977] lmv_close+0x18d/0x490 [lmv] [13300.125319] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13300.125461] ll_md_real_close+0xa5/0x2a0 [lustre] [13300.125557] ll_md_close+0x274/0x800 [lustre] [13300.125655] ll_file_release+0x128/0xc50 [lustre] [13300.125922] __fput+0x10d/0x440 [13300.125932] ____fput+0x12/0x20 [13300.125936] task_work_run+0xc6/0x110 [13300.125944] exit_to_usermode_loop+0x1dd/0x1f0 [13300.125952] do_syscall_64+0x3d6/0x440 [13300.125956] entry_SYSCALL_64_after_hwframe+0x49/0xae [13300.125963] RIP: 0033:0x7fa749552cd8 [13300.125983] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13300.125987] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13300.125992] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13300.125994] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13300.125996] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13300.125999] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13300.126001] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13300.345837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13300.367640] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13300.373121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13300.377042] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13300.380013] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13300.391626] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13300.398252] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13300.402092] RDX: 0000000000005254 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13300.407489] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13300.411353] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13300.415311] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13300.419128] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13300.423989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13300.427155] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13300.431257] Call Trace: [13300.432939] [13300.433836] ? show_regs.cold.9+0x22/0x2f [13300.435545] ? watchdog_timer_fn.cold.6+0x62/0x10a [13300.438264] ? __hrtimer_run_queues+0x1c2/0x4d0 [13300.442307] ? watchdog+0x40/0x40 [13300.444082] ? hrtimer_interrupt+0x128/0x2c0 [13300.448046] ? smp_apic_timer_interrupt+0xa6/0x270 [13300.451223] ? apic_timer_interrupt+0xf/0x20 [13300.453523] [13300.459341] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13300.462451] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13300.476403] do_raw_spin_lock+0x177/0x190 [13300.479173] _raw_spin_lock+0x21/0x30 [13300.484597] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13300.494994] obd_mod_free+0x46/0x140 [ptlrpc] [13300.502213] mdc_commit_open+0x91/0xb0 [mdc] [13300.508156] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13300.512463] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13300.523317] after_reply+0xe2c/0x1780 [ptlrpc] [13300.526652] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13300.535178] ? __next_timer_interrupt+0x160/0x160 [13300.538522] ptlrpcd+0xa82/0xe40 [ptlrpc] [13300.548181] ? do_wait_intr+0xf0/0xf0 [13300.554445] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13300.558031] kthread+0x1d1/0x200 [13300.565245] ? set_kthread_struct+0x70/0x70 [13300.568122] ret_from_fork+0x1f/0x30 [13312.054506] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13312.062310] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13312.107304] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13312.123307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13312.133763] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13312.136900] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13312.160221] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13312.167112] RAX: 000000000000572a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13312.177473] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13312.183721] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13312.190076] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13312.195633] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13312.199708] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13312.208152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13312.213114] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13312.218423] Call Trace: [13312.221056] [13312.223062] ? show_regs.cold.9+0x22/0x2f [13312.231249] ? watchdog_timer_fn.cold.6+0x62/0x10a [13312.237713] ? __hrtimer_run_queues+0x1c2/0x4d0 [13312.240440] ? watchdog+0x40/0x40 [13312.248881] ? hrtimer_interrupt+0x128/0x2c0 [13312.250439] ? smp_apic_timer_interrupt+0xa6/0x270 [13312.259977] ? apic_timer_interrupt+0xf/0x20 [13312.265513] [13312.269327] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13312.277953] do_raw_spin_lock+0x177/0x190 [13312.285777] _raw_spin_lock+0x21/0x30 [13312.286980] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13312.297731] ? do_raw_spin_unlock+0x75/0x190 [13312.299492] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13312.307086] ? do_wait_intr+0xf0/0xf0 [13312.311791] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13312.321338] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13312.327795] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13312.339186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.349087] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13312.351041] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13312.361082] mdc_intent_lock+0x353/0x7e0 [mdc] [13312.369377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.378391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13312.387384] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13312.400626] lmv_intent_open+0x338/0x1150 [lmv] [13312.402377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.417994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.429119] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13312.432099] ll_intent_lock+0xc5/0xaa0 [lustre] [13312.440095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.448197] ll_lookup_it+0xe0e/0x2270 [lustre] [13312.455335] ? do_raw_spin_unlock+0x75/0x190 [13312.464257] ? _raw_spin_unlock+0x12/0x30 [13312.469142] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13312.476369] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13312.486824] ll_atomic_open+0x36a/0xd70 [lustre] [13312.490816] lookup_open+0x356/0xb40 [13312.496775] do_last+0x313/0xfc0 [13312.502324] ? nd_jump_root+0xe5/0x160 [13312.511060] ? path_init+0x437/0x520 [13312.512298] path_openat+0xf7/0x500 [13312.517043] ? do_raw_spin_unlock+0x75/0x190 [13312.520399] do_filp_open+0x99/0x140 [13312.531378] ? getname_flags+0x6e/0x330 [13312.532732] ? __check_object_size+0xff/0x256 [13312.534077] ? do_raw_spin_unlock+0x75/0x190 [13312.539850] ? _raw_spin_unlock+0x12/0x30 [13312.543781] do_sys_openat2+0x2b4/0x410 [13312.548285] do_sys_open+0x73/0xa0 [13312.551123] __x64_sys_openat+0x24/0x30 [13312.560139] do_syscall_64+0xc1/0x440 [13312.564060] entry_SYSCALL_64_after_hwframe+0x49/0xae [13312.571112] RIP: 0033:0x7fa749552332 [13312.573366] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13312.599210] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13312.607320] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13312.613513] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13312.629068] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13312.641487] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13312.643237] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13313.940019] rcu: INFO: rcu_sched self-detected stall on CPU [13313.945896] rcu: 2-...!: (6494674 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [13313.956847] (t=11400251 jiffies g=258701 q=4740) [13313.973253] rcu: rcu_sched kthread starved for 11397046 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [13313.986752] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13313.998234] rcu: RCU grace-period kthread stack dump: [13313.999905] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13314.017587] Call Trace: [13314.028227] __schedule+0x351/0xcb0 [13314.030407] schedule+0xc0/0x180 [13314.035906] schedule_timeout+0xb4/0x190 [13314.037855] ? do_raw_spin_unlock+0x75/0x190 [13314.044676] ? __next_timer_interrupt+0x160/0x160 [13314.048969] ? prepare_to_swait_event+0xf3/0x140 [13314.054630] rcu_gp_kthread+0x8dd/0x15b0 [13314.055933] ? force_qs_rnp+0x2c0/0x2c0 [13314.060867] kthread+0x1d1/0x200 [13314.061927] ? set_kthread_struct+0x70/0x70 [13314.063262] ret_from_fork+0x1f/0x30 [13314.067412] rcu: Stack dump where RCU GP kthread last ran: [13314.069814] NMI backtrace for cpu 2 [13314.071235] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13314.078943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13314.085645] Call Trace: [13314.087266] [13314.088475] ? dump_stack+0xbb/0x10e [13314.089494] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13314.098325] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13314.102119] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13314.106590] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13314.112189] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13314.117258] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [13314.123222] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [13314.124588] ? update_process_times+0x6d/0xc0 [13314.131372] ? tick_sched_handle.isra.18+0x27/0x90 [13314.138088] ? tick_sched_timer+0x53/0xb0 [13314.145255] ? __hrtimer_run_queues+0x1c2/0x4d0 [13314.152413] ? tick_sched_handle.isra.18+0x90/0x90 [13314.160283] ? hrtimer_interrupt+0x128/0x2c0 [13314.161900] ? smp_apic_timer_interrupt+0xa6/0x270 [13314.164746] ? apic_timer_interrupt+0xf/0x20 [13314.166197] [13314.176970] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13314.185037] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13314.192944] ? do_raw_spin_lock+0x177/0x190 [13314.196637] ? _raw_spin_lock+0x21/0x30 [13314.202994] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13314.211853] ? obd_mod_free+0x46/0x140 [ptlrpc] [13314.213503] ? mdc_commit_open+0x91/0xb0 [mdc] [13314.222241] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13314.224109] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13314.233160] ? after_reply+0xe2c/0x1780 [ptlrpc] [13314.237761] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13314.247131] ? __next_timer_interrupt+0x160/0x160 [13314.262542] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13314.264062] ? do_wait_intr+0xf0/0xf0 [13314.273581] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13314.281782] ? kthread+0x1d1/0x200 [13314.287207] ? set_kthread_struct+0x70/0x70 [13314.291142] ? ret_from_fork+0x1f/0x30 [13314.305543] NMI backtrace for cpu 2 [13314.306557] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13314.326422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13314.339371] Call Trace: [13314.343366] [13314.350010] ? dump_stack+0xbb/0x10e [13314.359936] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13314.368119] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13314.372347] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13314.378831] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13314.393039] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13314.394777] ? rcu_dump_cpu_stacks+0x11b/0x171 [13314.404178] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13314.407172] ? update_process_times+0x6d/0xc0 [13314.412226] ? tick_sched_handle.isra.18+0x27/0x90 [13314.421875] ? tick_sched_timer+0x53/0xb0 [13314.429709] ? __hrtimer_run_queues+0x1c2/0x4d0 [13314.437917] ? tick_sched_handle.isra.18+0x90/0x90 [13314.442287] ? hrtimer_interrupt+0x128/0x2c0 [13314.448568] ? smp_apic_timer_interrupt+0xa6/0x270 [13314.453418] ? apic_timer_interrupt+0xf/0x20 [13314.459609] [13314.460293] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13314.468550] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13314.470172] ? do_raw_spin_lock+0x177/0x190 [13314.471574] ? _raw_spin_lock+0x21/0x30 [13314.479911] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13314.482023] ? obd_mod_free+0x46/0x140 [ptlrpc] [13314.495021] ? mdc_commit_open+0x91/0xb0 [mdc] [13314.496630] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13314.507269] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13314.512477] ? after_reply+0xe2c/0x1780 [ptlrpc] [13314.514313] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13314.523053] ? __next_timer_interrupt+0x160/0x160 [13314.530159] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13314.534868] ? do_wait_intr+0xf0/0xf0 [13314.542986] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13314.544838] ? kthread+0x1d1/0x200 [13314.552961] ? set_kthread_struct+0x70/0x70 [13314.554532] ? ret_from_fork+0x1f/0x30 [13314.562288] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11397s! [13314.572905] Showing busy workqueues and worker pools: [13314.578127] workqueue events: flags=0x0 [13314.586239] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13314.596777] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13314.607959] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13314.615263] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13314.625781] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.635599] pending: kfree_rcu_monitor [13314.636948] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13314.651900] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13314.667520] workqueue events_long: flags=0x0 [13314.668942] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.677761] pending: lu_objects_flush [obdclass] [13314.685248] workqueue events_unbound: flags=0x2 [13314.686824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13314.706981] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13314.723799] workqueue events_power_efficient: flags=0x80 [13314.728983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.735370] pending: check_lifetime [13314.736840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.744538] pending: neigh_periodic_work [13314.747640] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.754394] pending: neigh_periodic_work [13314.761168] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.773963] pending: do_cache_clean [13314.776258] workqueue mm_percpu_wq: flags=0x8 [13314.781499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.784826] pending: vmstat_update [13314.786977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.789961] pending: vmstat_update [13314.791563] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.795684] pending: vmstat_update [13314.803460] workqueue writeback: flags=0x4a [13314.805010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13314.812697] pending: wb_workfn, wb_workfn [13314.816784] workqueue xprtiod: flags=0xa [13314.819849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13314.823869] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13314.828019] workqueue ipv6_addrconf: flags=0x40008 [13314.829967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13314.832954] pending: addrconf_verify_work [13314.836316] workqueue knbd0-recv: flags=0x1a [13314.843252] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13314.850402] in-flight: 651:recv_work [13314.855088] workqueue ptlrpc_pinger: flags=0x2 [13314.862625] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13314.867322] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13314.877758] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11397s workers=4 idle: 37 31846 28897 [13314.889738] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13304s workers=2 idle: 653 [13316.066549] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13316.072670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13316.116282] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13316.127731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13316.142312] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13316.147309] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13316.149714] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13316.158742] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13316.162248] RAX: 0000000000006783 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13316.165146] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13316.167960] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13316.170702] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13316.173444] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13316.175994] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13316.180433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13316.182785] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13316.185671] Call Trace: [13316.186751] [13316.187594] ? show_regs.cold.9+0x22/0x2f [13316.189273] ? watchdog_timer_fn.cold.6+0x62/0x10a [13316.191421] ? __hrtimer_run_queues+0x1c2/0x4d0 [13316.193199] ? watchdog+0x40/0x40 [13316.194461] ? hrtimer_interrupt+0x128/0x2c0 [13316.196390] ? smp_apic_timer_interrupt+0xa6/0x270 [13316.198266] ? apic_timer_interrupt+0xf/0x20 [13316.200234] [13316.201331] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13316.203380] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13316.205712] do_raw_spin_lock+0x177/0x190 [13316.207095] _raw_spin_lock+0x21/0x30 [13316.209904] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13316.212189] process_one_work+0x2c8/0x700 [13316.215964] worker_thread+0x296/0x6e0 [13316.217386] ? rescuer_thread+0x570/0x570 [13316.220185] kthread+0x1d1/0x200 [13316.221307] ? set_kthread_struct+0x70/0x70 [13316.223182] ret_from_fork+0x1f/0x30 [13328.059029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13328.061965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13328.079570] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13328.084043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13328.087232] RIP: 0010:native_safe_halt+0x2e/0x50 [13328.088882] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13328.095836] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13328.098650] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13328.101770] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13328.104547] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13328.106608] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13328.109181] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13328.111815] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13328.114669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13328.116590] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13328.119881] Call Trace: [13328.120882] [13328.121811] ? show_regs.cold.9+0x22/0x2f [13328.123398] ? watchdog_timer_fn.cold.6+0x62/0x10a [13328.125281] ? __hrtimer_run_queues+0x1c2/0x4d0 [13328.126915] ? watchdog+0x40/0x40 [13328.128009] ? hrtimer_interrupt+0x128/0x2c0 [13328.129664] ? smp_apic_timer_interrupt+0xa6/0x270 [13328.131989] ? apic_timer_interrupt+0xf/0x20 [13328.133587] [13328.134918] ? native_safe_halt+0x2e/0x50 [13328.136396] kvm_wait+0xcd/0xe0 [13328.137548] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13328.139467] do_raw_spin_lock+0x177/0x190 [13328.141010] _raw_spin_lock+0x21/0x30 [13328.142551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13328.145653] ? do_raw_spin_unlock+0x75/0x190 [13328.147437] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13328.149165] ? do_wait_intr+0xf0/0xf0 [13328.151034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13328.153448] mdc_close+0x53b/0xd70 [mdc] [13328.154899] lmv_close+0x18d/0x490 [lmv] [13328.156694] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13328.158773] ll_md_real_close+0xa5/0x2a0 [lustre] [13328.160969] ll_md_close+0x274/0x800 [lustre] [13328.162888] ll_file_release+0x128/0xc50 [lustre] [13328.164796] __fput+0x10d/0x440 [13328.168854] ____fput+0x12/0x20 [13328.170688] task_work_run+0xc6/0x110 [13328.172027] exit_to_usermode_loop+0x1dd/0x1f0 [13328.173795] do_syscall_64+0x3d6/0x440 [13328.175635] entry_SYSCALL_64_after_hwframe+0x49/0xae [13328.177691] RIP: 0033:0x7fa749552cd8 [13328.179163] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13328.186626] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13328.189737] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13328.192615] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13328.195698] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13328.198684] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13328.201704] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13340.057904] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13340.058214] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13340.066498] Modules linked in: [13340.068499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror [13340.071500] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13340.071573] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13340.071581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13340.071584] RIP: 0010:native_safe_halt+0x2e/0x50 [13340.071599] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13340.071603] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13340.071607] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13340.071627] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [13340.071630] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13340.071632] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13340.071634] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13340.071637] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13340.071640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13340.071642] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13340.071650] Call Trace: [13340.071655] [13340.131233] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13340.131251] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13340.131258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13340.131261] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13340.131304] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13340.131309] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13340.131313] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13340.131316] RDX: 00000000000058fb RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13340.131318] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13340.131319] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13340.131321] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13340.131325] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13340.131327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13340.131330] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13340.131338] Call Trace: [13340.131344] [13340.131349] ? show_regs.cold.9+0x22/0x2f [13340.131359] ? watchdog_timer_fn.cold.6+0x62/0x10a [13340.131365] ? __hrtimer_run_queues+0x1c2/0x4d0 [13340.131369] ? watchdog+0x40/0x40 [13340.131373] ? hrtimer_interrupt+0x128/0x2c0 [13340.131377] ? smp_apic_timer_interrupt+0xa6/0x270 [13340.131387] ? apic_timer_interrupt+0xf/0x20 [13340.131391] [13340.131394] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13340.131399] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13340.131403] do_raw_spin_lock+0x177/0x190 [13340.131408] _raw_spin_lock+0x21/0x30 [13340.131415] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13340.132096] obd_mod_free+0x46/0x140 [ptlrpc] [13340.132324] mdc_commit_open+0x91/0xb0 [mdc] [13340.132359] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13340.132561] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13340.132917] after_reply+0xe2c/0x1780 [ptlrpc] [13340.133182] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13340.133390] ? __next_timer_interrupt+0x160/0x160 [13340.133401] ptlrpcd+0xa82/0xe40 [ptlrpc] [13340.134165] ? do_wait_intr+0xf0/0xf0 [13340.134175] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13340.137815] kthread+0x1d1/0x200 [13340.137825] ? set_kthread_struct+0x70/0x70 [13340.137830] ret_from_fork+0x1f/0x30 [13340.611825] ? show_regs.cold.9+0x22/0x2f [13340.622273] ? watchdog_timer_fn.cold.6+0x62/0x10a [13340.624062] ? __hrtimer_run_queues+0x1c2/0x4d0 [13340.636132] ? watchdog+0x40/0x40 [13340.638710] ? hrtimer_interrupt+0x128/0x2c0 [13340.650945] ? smp_apic_timer_interrupt+0xa6/0x270 [13340.652648] ? apic_timer_interrupt+0xf/0x20 [13340.660576] [13340.661375] ? native_safe_halt+0x2e/0x50 [13340.662874] kvm_wait+0xcd/0xe0 [13340.670057] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13340.671944] do_raw_spin_lock+0x177/0x190 [13340.675865] _raw_spin_lock+0x21/0x30 [13340.677180] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13340.682563] ? do_raw_spin_unlock+0x75/0x190 [13340.684832] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13340.687316] ? do_wait_intr+0xf0/0xf0 [13340.690355] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13340.696871] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13340.699330] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13340.706929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.709343] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13340.711780] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13340.713438] mdc_intent_lock+0x353/0x7e0 [mdc] [13340.719034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.723704] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13340.727371] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13340.730971] lmv_intent_open+0x338/0x1150 [lmv] [13340.732954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.736232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.738251] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13340.741267] ll_intent_lock+0xc5/0xaa0 [lustre] [13340.744730] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.748384] ll_lookup_it+0xe0e/0x2270 [lustre] [13340.750169] ? do_raw_spin_unlock+0x75/0x190 [13340.755462] ? _raw_spin_unlock+0x12/0x30 [13340.760351] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13340.765529] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13340.768538] ll_atomic_open+0x36a/0xd70 [lustre] [13340.770862] lookup_open+0x356/0xb40 [13340.772450] do_last+0x313/0xfc0 [13340.773819] ? nd_jump_root+0xe5/0x160 [13340.775628] ? path_init+0x437/0x520 [13340.779356] path_openat+0xf7/0x500 [13340.781330] ? do_raw_spin_unlock+0x75/0x190 [13340.784176] do_filp_open+0x99/0x140 [13340.787786] ? getname_flags+0x6e/0x330 [13340.794514] ? __check_object_size+0xff/0x256 [13340.801636] ? do_raw_spin_unlock+0x75/0x190 [13340.803131] ? _raw_spin_unlock+0x12/0x30 [13340.813622] do_sys_openat2+0x2b4/0x410 [13340.815163] do_sys_open+0x73/0xa0 [13340.818473] __x64_sys_openat+0x24/0x30 [13340.819909] do_syscall_64+0xc1/0x440 [13340.822109] entry_SYSCALL_64_after_hwframe+0x49/0xae [13340.824300] RIP: 0033:0x7fa749552332 [13340.826657] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13340.843651] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13340.852395] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13340.855379] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13340.862385] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13340.872810] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13340.882115] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13344.061049] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13344.068280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13344.115891] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13344.121778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13344.126296] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13344.136123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13344.137783] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13344.158342] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13344.162379] RAX: 0000000000007113 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13344.166450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13344.173100] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13344.178270] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13344.183761] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13344.195649] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13344.202626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13344.208489] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13344.215126] Call Trace: [13344.216065] [13344.217395] ? show_regs.cold.9+0x22/0x2f [13344.221521] ? watchdog_timer_fn.cold.6+0x62/0x10a [13344.224573] ? __hrtimer_run_queues+0x1c2/0x4d0 [13344.233216] ? watchdog+0x40/0x40 [13344.235236] ? hrtimer_interrupt+0x128/0x2c0 [13344.240571] ? smp_apic_timer_interrupt+0xa6/0x270 [13344.244543] ? apic_timer_interrupt+0xf/0x20 [13344.252192] [13344.254743] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13344.258499] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13344.260739] do_raw_spin_lock+0x177/0x190 [13344.265158] _raw_spin_lock+0x21/0x30 [13344.268477] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13344.274043] process_one_work+0x2c8/0x700 [13344.275368] worker_thread+0x296/0x6e0 [13344.276472] ? rescuer_thread+0x570/0x570 [13344.284607] kthread+0x1d1/0x200 [13344.286364] ? set_kthread_struct+0x70/0x70 [13344.287901] ret_from_fork+0x1f/0x30 [13345.249466] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11428s! [13345.256905] Showing busy workqueues and worker pools: [13345.258575] workqueue events: flags=0x0 [13345.259578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13345.267435] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13345.272486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13345.275591] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13345.279778] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.288760] pending: kfree_rcu_monitor [13345.295067] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13345.299478] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13345.309134] workqueue events_long: flags=0x0 [13345.313486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.326216] pending: lu_objects_flush [obdclass] [13345.335256] workqueue events_unbound: flags=0x2 [13345.342389] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13345.350376] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13345.365136] workqueue events_power_efficient: flags=0x80 [13345.369503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.377434] pending: check_lifetime [13345.381954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.386111] pending: neigh_periodic_work [13345.393128] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.403675] pending: neigh_periodic_work [13345.407223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.413470] pending: do_cache_clean [13345.420471] workqueue mm_percpu_wq: flags=0x8 [13345.423014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.428373] pending: vmstat_update [13345.446046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.448309] pending: vmstat_update [13345.459702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13345.466259] pending: vmstat_update [13345.467606] workqueue writeback: flags=0x4a [13345.473027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13345.476903] pending: wb_workfn, wb_workfn [13345.478701] workqueue xprtiod: flags=0xa [13345.489742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13345.492076] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13345.513109] workqueue ipv6_addrconf: flags=0x40008 [13345.517453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13345.525110] pending: addrconf_verify_work [13345.532348] workqueue knbd0-recv: flags=0x1a [13345.536818] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13345.542937] in-flight: 651:recv_work [13345.554085] workqueue ptlrpc_pinger: flags=0x2 [13345.559670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13345.567166] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13345.575216] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11428s workers=4 idle: 37 31846 28897 [13345.582898] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13335s workers=2 idle: 653 [13356.059032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13356.063698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13356.082017] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13356.086596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13356.091830] RIP: 0010:native_safe_halt+0x2e/0x50 [13356.093536] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13356.100724] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13356.104554] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13356.107228] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13356.110182] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13356.112786] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13356.115240] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13356.118415] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13356.121690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13356.124364] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13356.127001] Call Trace: [13356.128059] [13356.128816] ? show_regs.cold.9+0x22/0x2f [13356.130510] ? watchdog_timer_fn.cold.6+0x62/0x10a [13356.132423] ? __hrtimer_run_queues+0x1c2/0x4d0 [13356.134304] ? watchdog+0x40/0x40 [13356.136178] ? hrtimer_interrupt+0x128/0x2c0 [13356.137852] ? smp_apic_timer_interrupt+0xa6/0x270 [13356.139660] ? apic_timer_interrupt+0xf/0x20 [13356.141581] [13356.142421] ? native_safe_halt+0x2e/0x50 [13356.144374] kvm_wait+0xcd/0xe0 [13356.145822] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13356.148164] do_raw_spin_lock+0x177/0x190 [13356.149683] _raw_spin_lock+0x21/0x30 [13356.151423] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13356.153925] ? do_raw_spin_unlock+0x75/0x190 [13356.155413] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13356.157579] ? do_wait_intr+0xf0/0xf0 [13356.158938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13356.161317] mdc_close+0x53b/0xd70 [mdc] [13356.162957] lmv_close+0x18d/0x490 [lmv] [13356.164830] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13356.167558] ll_md_real_close+0xa5/0x2a0 [lustre] [13356.169507] ll_md_close+0x274/0x800 [lustre] [13356.171352] ll_file_release+0x128/0xc50 [lustre] [13356.173360] __fput+0x10d/0x440 [13356.174732] ____fput+0x12/0x20 [13356.176093] task_work_run+0xc6/0x110 [13356.177469] exit_to_usermode_loop+0x1dd/0x1f0 [13356.179433] do_syscall_64+0x3d6/0x440 [13356.180758] entry_SYSCALL_64_after_hwframe+0x49/0xae [13356.182917] RIP: 0033:0x7fa749552cd8 [13356.184308] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13356.191958] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13356.194619] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13356.197426] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13356.200365] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13356.203304] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13356.206172] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13368.058017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13368.061165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13368.087763] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13368.092296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13368.097637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13368.100252] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13368.118403] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13368.121651] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13368.125112] RDX: 0000000000006266 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13368.127981] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13368.130640] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13368.133226] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13368.135814] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13368.138749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13368.140659] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13368.144249] Call Trace: [13368.148069] [13368.148749] ? show_regs.cold.9+0x22/0x2f [13368.150271] ? watchdog_timer_fn.cold.6+0x62/0x10a [13368.152213] ? __hrtimer_run_queues+0x1c2/0x4d0 [13368.153744] ? watchdog+0x40/0x40 [13368.155062] ? hrtimer_interrupt+0x128/0x2c0 [13368.157619] ? smp_apic_timer_interrupt+0xa6/0x270 [13368.160231] ? apic_timer_interrupt+0xf/0x20 [13368.162217] [13368.163020] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13368.166790] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13368.168281] do_raw_spin_lock+0x177/0x190 [13368.169945] _raw_spin_lock+0x21/0x30 [13368.171167] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13368.173262] obd_mod_free+0x46/0x140 [ptlrpc] [13368.175358] mdc_commit_open+0x91/0xb0 [mdc] [13368.180018] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13368.183471] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13368.196655] after_reply+0xe2c/0x1780 [ptlrpc] [13368.202850] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13368.208600] ? __next_timer_interrupt+0x160/0x160 [13368.211164] ptlrpcd+0xa82/0xe40 [ptlrpc] [13368.222080] ? do_wait_intr+0xf0/0xf0 [13368.225288] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13368.233552] kthread+0x1d1/0x200 [13368.237182] ? set_kthread_struct+0x70/0x70 [13368.239841] ret_from_fork+0x1f/0x30 [13372.054702] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13372.057053] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13372.065388] Modules linked in: [13372.069311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13372.069394] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13372.069400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13372.069407] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13372.069694] RIP: 0010:native_safe_halt+0x2e/0x50 [13372.069705] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13372.069718] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13372.069723] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13372.069725] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13372.069726] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13372.069728] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13372.069731] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13372.069733] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13372.069736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13372.069738] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13372.069747] Call Trace: [13372.069754] [13372.069760] ? show_regs.cold.9+0x22/0x2f [13372.069772] ? watchdog_timer_fn.cold.6+0x62/0x10a [13372.069778] ? __hrtimer_run_queues+0x1c2/0x4d0 [13372.069783] ? watchdog+0x40/0x40 [13372.069787] ? hrtimer_interrupt+0x128/0x2c0 [13372.069791] ? smp_apic_timer_interrupt+0xa6/0x270 [13372.069797] ? apic_timer_interrupt+0xf/0x20 [13372.069803] [13372.069806] ? native_safe_halt+0x2e/0x50 [13372.069810] kvm_wait+0xcd/0xe0 [13372.069818] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13372.069826] do_raw_spin_lock+0x177/0x190 [13372.069830] _raw_spin_lock+0x21/0x30 [13372.069834] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13372.070042] process_one_work+0x2c8/0x700 [13372.070049] worker_thread+0x296/0x6e0 [13372.070054] ? rescuer_thread+0x570/0x570 [13372.070057] kthread+0x1d1/0x200 [13372.070062] ? set_kthread_struct+0x70/0x70 [13372.070066] ret_from_fork+0x1f/0x30 [13372.303171] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13372.352116] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13372.365973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13372.377836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13372.388491] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13372.403717] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13372.412078] RAX: 000000000000018e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13372.420819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13372.425737] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13372.435300] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13372.441999] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13372.447909] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13372.461148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13372.469461] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13372.476171] Call Trace: [13372.479100] [13372.480011] ? show_regs.cold.9+0x22/0x2f [13372.482747] ? watchdog_timer_fn.cold.6+0x62/0x10a [13372.489725] ? __hrtimer_run_queues+0x1c2/0x4d0 [13372.493928] ? watchdog+0x40/0x40 [13372.496030] ? hrtimer_interrupt+0x128/0x2c0 [13372.498320] ? smp_apic_timer_interrupt+0xa6/0x270 [13372.503491] ? apic_timer_interrupt+0xf/0x20 [13372.506275] [13372.507385] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13372.510371] do_raw_spin_lock+0x177/0x190 [13372.515052] _raw_spin_lock+0x21/0x30 [13372.518859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13372.523954] ? do_raw_spin_unlock+0x75/0x190 [13372.527445] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13372.534396] ? do_wait_intr+0xf0/0xf0 [13372.539232] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13372.542938] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13372.551347] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13372.559293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13372.565294] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13372.569077] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13372.572062] mdc_intent_lock+0x353/0x7e0 [mdc] [13372.574154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13372.577005] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13372.579287] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13372.581767] lmv_intent_open+0x338/0x1150 [lmv] [13372.583836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13372.585845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13372.588077] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13372.589843] ll_intent_lock+0xc5/0xaa0 [lustre] [13372.591722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13372.593810] ll_lookup_it+0xe0e/0x2270 [lustre] [13372.595577] ? do_raw_spin_unlock+0x75/0x190 [13372.597283] ? _raw_spin_unlock+0x12/0x30 [13372.599056] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13372.600778] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13372.602625] ll_atomic_open+0x36a/0xd70 [lustre] [13372.604197] lookup_open+0x356/0xb40 [13372.605470] do_last+0x313/0xfc0 [13372.607254] ? nd_jump_root+0xe5/0x160 [13372.612513] ? path_init+0x437/0x520 [13372.614176] path_openat+0xf7/0x500 [13372.615053] ? do_raw_spin_unlock+0x75/0x190 [13372.616181] do_filp_open+0x99/0x140 [13372.619622] ? getname_flags+0x6e/0x330 [13372.621330] ? __check_object_size+0xff/0x256 [13372.623356] ? do_raw_spin_unlock+0x75/0x190 [13372.625685] ? _raw_spin_unlock+0x12/0x30 [13372.627864] do_sys_openat2+0x2b4/0x410 [13372.629880] do_sys_open+0x73/0xa0 [13372.631466] __x64_sys_openat+0x24/0x30 [13372.633072] do_syscall_64+0xc1/0x440 [13372.634900] entry_SYSCALL_64_after_hwframe+0x49/0xae [13372.640890] RIP: 0033:0x7fa749552332 [13372.642049] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13372.675777] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13372.682608] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13372.693754] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13372.704384] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13372.714231] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13372.721992] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13375.986083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11458s! [13375.989613] Showing busy workqueues and worker pools: [13376.004028] workqueue events: flags=0x0 [13376.006081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13376.016717] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13376.033660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13376.046260] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13376.061379] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.068017] pending: kfree_rcu_monitor [13376.074130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13376.085600] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13376.098741] workqueue events_long: flags=0x0 [13376.103066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.110105] pending: lu_objects_flush [obdclass] [13376.113268] workqueue events_unbound: flags=0x2 [13376.115125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13376.117837] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13376.127695] workqueue events_power_efficient: flags=0x80 [13376.135330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.139391] pending: check_lifetime [13376.141419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.144524] pending: neigh_periodic_work [13376.146408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.151131] pending: neigh_periodic_work [13376.154100] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.163673] pending: do_cache_clean [13376.172321] workqueue mm_percpu_wq: flags=0x8 [13376.177440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.185849] pending: vmstat_update [13376.192032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.198217] pending: vmstat_update [13376.205450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13376.217906] pending: vmstat_update [13376.220380] workqueue writeback: flags=0x4a [13376.226084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13376.231335] pending: wb_workfn, wb_workfn [13376.236955] workqueue xprtiod: flags=0xa [13376.238164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13376.243460] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13376.251220] workqueue ipv6_addrconf: flags=0x40008 [13376.256613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13376.260246] pending: addrconf_verify_work [13376.265484] workqueue knbd0-recv: flags=0x1a [13376.273950] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13376.282716] in-flight: 651:recv_work [13376.292444] workqueue ptlrpc_pinger: flags=0x2 [13376.299110] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13376.302685] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13376.315034] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11459s workers=4 idle: 37 31846 28897 [13376.322088] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13366s workers=2 idle: 653 [13384.059593] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13384.069711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13384.121734] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13384.135469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13384.146438] RIP: 0010:native_safe_halt+0x2e/0x50 [13384.151823] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13384.168719] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13384.175575] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13384.181756] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13384.192718] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13384.201009] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13384.214965] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13384.217417] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13384.225311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13384.227844] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13384.233066] Call Trace: [13384.236519] [13384.237562] ? show_regs.cold.9+0x22/0x2f [13384.240008] ? watchdog_timer_fn.cold.6+0x62/0x10a [13384.247594] ? __hrtimer_run_queues+0x1c2/0x4d0 [13384.263518] ? watchdog+0x40/0x40 [13384.269325] ? hrtimer_interrupt+0x128/0x2c0 [13384.278765] ? smp_apic_timer_interrupt+0xa6/0x270 [13384.280543] ? apic_timer_interrupt+0xf/0x20 [13384.287844] [13384.289665] ? native_safe_halt+0x2e/0x50 [13384.294598] kvm_wait+0xcd/0xe0 [13384.299352] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13384.303939] do_raw_spin_lock+0x177/0x190 [13384.305620] _raw_spin_lock+0x21/0x30 [13384.308983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13384.311942] ? do_raw_spin_unlock+0x75/0x190 [13384.316088] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13384.319427] ? do_wait_intr+0xf0/0xf0 [13384.326621] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13384.332751] mdc_close+0x53b/0xd70 [mdc] [13384.335649] lmv_close+0x18d/0x490 [lmv] [13384.337571] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13384.340436] ll_md_real_close+0xa5/0x2a0 [lustre] [13384.344687] ll_md_close+0x274/0x800 [lustre] [13384.348505] ll_file_release+0x128/0xc50 [lustre] [13384.354973] __fput+0x10d/0x440 [13384.356742] ____fput+0x12/0x20 [13384.362428] task_work_run+0xc6/0x110 [13384.366441] exit_to_usermode_loop+0x1dd/0x1f0 [13384.372931] do_syscall_64+0x3d6/0x440 [13384.375358] entry_SYSCALL_64_after_hwframe+0x49/0xae [13384.382710] RIP: 0033:0x7fa749552cd8 [13384.389784] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13384.414222] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13384.419187] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13384.424748] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13384.427248] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13384.433164] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13384.437286] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13396.060364] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13396.065608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13396.098007] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13396.102940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13396.107636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13396.110383] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13396.118707] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13396.122716] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13396.125886] RDX: 00000000000021af RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13396.129155] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13396.132479] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13396.135825] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13396.139122] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13396.142643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13396.145565] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13396.149342] Call Trace: [13396.150524] [13396.151492] ? show_regs.cold.9+0x22/0x2f [13396.152897] ? watchdog_timer_fn.cold.6+0x62/0x10a [13396.154764] ? __hrtimer_run_queues+0x1c2/0x4d0 [13396.156866] ? watchdog+0x40/0x40 [13396.158344] ? hrtimer_interrupt+0x128/0x2c0 [13396.160302] ? smp_apic_timer_interrupt+0xa6/0x270 [13396.162530] ? apic_timer_interrupt+0xf/0x20 [13396.164463] [13396.165423] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13396.167824] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13396.170332] do_raw_spin_lock+0x177/0x190 [13396.172239] _raw_spin_lock+0x21/0x30 [13396.173841] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13396.176415] obd_mod_free+0x46/0x140 [ptlrpc] [13396.178878] mdc_commit_open+0x91/0xb0 [mdc] [13396.180758] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13396.183308] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13396.185989] after_reply+0xe2c/0x1780 [ptlrpc] [13396.188496] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13396.191155] ? __next_timer_interrupt+0x160/0x160 [13396.193337] ptlrpcd+0xa82/0xe40 [ptlrpc] [13396.195605] ? do_wait_intr+0xf0/0xf0 [13396.197155] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13396.200167] kthread+0x1d1/0x200 [13396.201477] ? set_kthread_struct+0x70/0x70 [13396.203249] ret_from_fork+0x1f/0x30 [13400.062167] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13400.069363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13400.116292] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13400.131909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13400.147619] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13400.160806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13400.173229] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13400.188485] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13400.195728] RAX: 0000000000004790 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13400.200156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13400.204444] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13400.208447] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13400.213716] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13400.222543] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13400.225069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13400.227692] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13400.234142] Call Trace: [13400.235799] [13400.236780] ? show_regs.cold.9+0x22/0x2f [13400.239210] ? watchdog_timer_fn.cold.6+0x62/0x10a [13400.241639] ? __hrtimer_run_queues+0x1c2/0x4d0 [13400.245350] ? watchdog+0x40/0x40 [13400.248075] ? hrtimer_interrupt+0x128/0x2c0 [13400.251880] ? smp_apic_timer_interrupt+0xa6/0x270 [13400.254470] ? apic_timer_interrupt+0xf/0x20 [13400.256787] [13400.258183] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13400.261503] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13400.264807] do_raw_spin_lock+0x177/0x190 [13400.266586] _raw_spin_lock+0x21/0x30 [13400.269273] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13400.272793] process_one_work+0x2c8/0x700 [13400.275293] worker_thread+0x296/0x6e0 [13400.277274] ? rescuer_thread+0x570/0x570 [13400.279851] kthread+0x1d1/0x200 [13400.281806] ? set_kthread_struct+0x70/0x70 [13400.285876] ret_from_fork+0x1f/0x30 [13404.054728] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13404.065284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13404.110117] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13404.123794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13404.136084] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13404.140909] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13404.156320] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13404.164593] RAX: 00000000000004c7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13404.169729] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13404.174613] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13404.177777] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13404.182555] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13404.187829] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13404.191406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13404.193295] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13404.205491] Call Trace: [13404.207802] [13404.210279] ? show_regs.cold.9+0x22/0x2f [13404.215184] ? watchdog_timer_fn.cold.6+0x62/0x10a [13404.218793] ? __hrtimer_run_queues+0x1c2/0x4d0 [13404.226586] ? watchdog+0x40/0x40 [13404.229716] ? hrtimer_interrupt+0x128/0x2c0 [13404.234198] ? smp_apic_timer_interrupt+0xa6/0x270 [13404.238321] ? apic_timer_interrupt+0xf/0x20 [13404.242473] [13404.243655] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13404.248166] do_raw_spin_lock+0x177/0x190 [13404.250200] _raw_spin_lock+0x21/0x30 [13404.256618] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13404.260325] ? do_raw_spin_unlock+0x75/0x190 [13404.261906] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13404.282264] ? do_wait_intr+0xf0/0xf0 [13404.288492] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13404.294696] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13404.297689] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13404.301845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13404.307762] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13404.313707] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13404.318660] mdc_intent_lock+0x353/0x7e0 [mdc] [13404.322484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13404.324512] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13404.330766] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13404.335725] lmv_intent_open+0x338/0x1150 [lmv] [13404.341536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13404.346167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13404.350518] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13404.356385] ll_intent_lock+0xc5/0xaa0 [lustre] [13404.357843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13404.364568] ll_lookup_it+0xe0e/0x2270 [lustre] [13404.373029] ? do_raw_spin_unlock+0x75/0x190 [13404.377481] ? _raw_spin_unlock+0x12/0x30 [13404.383317] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13404.386523] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13404.394942] ll_atomic_open+0x36a/0xd70 [lustre] [13404.396834] lookup_open+0x356/0xb40 [13404.400745] do_last+0x313/0xfc0 [13404.402199] ? nd_jump_root+0xe5/0x160 [13404.403785] ? path_init+0x437/0x520 [13404.410691] path_openat+0xf7/0x500 [13404.415269] ? do_raw_spin_unlock+0x75/0x190 [13404.418526] do_filp_open+0x99/0x140 [13404.421770] ? getname_flags+0x6e/0x330 [13404.424287] ? __check_object_size+0xff/0x256 [13404.431431] ? do_raw_spin_unlock+0x75/0x190 [13404.437101] ? _raw_spin_unlock+0x12/0x30 [13404.440655] do_sys_openat2+0x2b4/0x410 [13404.444828] do_sys_open+0x73/0xa0 [13404.446574] __x64_sys_openat+0x24/0x30 [13404.449575] do_syscall_64+0xc1/0x440 [13404.455455] entry_SYSCALL_64_after_hwframe+0x49/0xae [13404.458714] RIP: 0033:0x7fa749552332 [13404.462546] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13404.483654] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13404.489591] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13404.495710] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13404.501840] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13404.512387] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13404.516555] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13406.695035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11489s! [13406.700482] Showing busy workqueues and worker pools: [13406.704239] workqueue events: flags=0x0 [13406.711494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13406.719750] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13406.733411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13406.740034] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13406.754180] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.760630] pending: kfree_rcu_monitor [13406.766483] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13406.771241] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13406.785562] workqueue events_long: flags=0x0 [13406.787273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.802500] pending: lu_objects_flush [obdclass] [13406.805237] workqueue events_unbound: flags=0x2 [13406.811429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13406.820495] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13406.833427] workqueue events_power_efficient: flags=0x80 [13406.843719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.845387] pending: check_lifetime [13406.853194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.856557] pending: neigh_periodic_work [13406.858077] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.866466] pending: neigh_periodic_work [13406.868043] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.872851] pending: do_cache_clean [13406.874287] workqueue mm_percpu_wq: flags=0x8 [13406.878022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.880417] pending: vmstat_update [13406.884232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.888570] pending: vmstat_update [13406.890147] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13406.893240] pending: vmstat_update [13406.897260] workqueue writeback: flags=0x4a [13406.900281] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13406.903484] pending: wb_workfn, wb_workfn [13406.905855] workqueue xprtiod: flags=0xa [13406.908292] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13406.911267] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13406.916207] workqueue ipv6_addrconf: flags=0x40008 [13406.918614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13406.924957] pending: addrconf_verify_work [13406.932684] workqueue knbd0-recv: flags=0x1a [13406.938668] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13406.943820] in-flight: 651:recv_work [13406.947954] workqueue ptlrpc_pinger: flags=0x2 [13406.951119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13406.954975] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13406.964182] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11489s workers=4 idle: 37 31846 28897 [13406.973218] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13396s workers=2 idle: 653 [13412.059023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13412.062939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13412.084920] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13412.089428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13412.093493] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13412.096480] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13412.104163] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13412.107561] RAX: 0000000000002959 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13412.112227] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13412.114567] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13412.118433] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13412.121676] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13412.125206] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13412.135177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13412.139097] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13412.143547] Call Trace: [13412.146911] [13412.148027] ? show_regs.cold.9+0x22/0x2f [13412.149826] ? watchdog_timer_fn.cold.6+0x62/0x10a [13412.155437] ? __hrtimer_run_queues+0x1c2/0x4d0 [13412.158234] ? watchdog+0x40/0x40 [13412.159900] ? hrtimer_interrupt+0x128/0x2c0 [13412.162020] ? smp_apic_timer_interrupt+0xa6/0x270 [13412.166210] ? apic_timer_interrupt+0xf/0x20 [13412.168591] [13412.169759] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13412.171814] do_raw_spin_lock+0x177/0x190 [13412.173753] _raw_spin_lock+0x21/0x30 [13412.175855] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13412.178609] ? do_raw_spin_unlock+0x75/0x190 [13412.180817] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13412.183582] ? do_wait_intr+0xf0/0xf0 [13412.185678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13412.188437] mdc_close+0x53b/0xd70 [mdc] [13412.194353] lmv_close+0x18d/0x490 [lmv] [13412.198453] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13412.203608] ll_md_real_close+0xa5/0x2a0 [lustre] [13412.208077] ll_md_close+0x274/0x800 [lustre] [13412.210739] ll_file_release+0x128/0xc50 [lustre] [13412.213147] __fput+0x10d/0x440 [13412.216740] ____fput+0x12/0x20 [13412.219955] task_work_run+0xc6/0x110 [13412.224032] exit_to_usermode_loop+0x1dd/0x1f0 [13412.230831] do_syscall_64+0x3d6/0x440 [13412.233052] entry_SYSCALL_64_after_hwframe+0x49/0xae [13412.237145] RIP: 0033:0x7fa749552cd8 [13412.238898] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13412.252430] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13412.264259] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13412.270223] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13412.277643] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13412.286186] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13412.293096] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13424.058074] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13424.061375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13424.081180] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13424.085742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13424.090308] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13424.093116] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13424.102023] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13424.105898] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13424.109492] RDX: 00000000000054a4 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13424.112444] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13424.116597] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13424.120156] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13424.123937] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13424.128107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13424.130483] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13424.134029] Call Trace: [13424.135225] [13424.136455] ? show_regs.cold.9+0x22/0x2f [13424.138713] ? watchdog_timer_fn.cold.6+0x62/0x10a [13424.141320] ? __hrtimer_run_queues+0x1c2/0x4d0 [13424.143446] ? watchdog+0x40/0x40 [13424.144883] ? hrtimer_interrupt+0x128/0x2c0 [13424.146980] ? smp_apic_timer_interrupt+0xa6/0x270 [13424.149135] ? apic_timer_interrupt+0xf/0x20 [13424.151400] [13424.152241] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13424.154844] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13424.157272] do_raw_spin_lock+0x177/0x190 [13424.159170] _raw_spin_lock+0x21/0x30 [13424.161200] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13424.164075] obd_mod_free+0x46/0x140 [ptlrpc] [13424.166952] mdc_commit_open+0x91/0xb0 [mdc] [13424.169820] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13424.172947] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13424.176311] after_reply+0xe2c/0x1780 [ptlrpc] [13424.179089] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13424.182043] ? __next_timer_interrupt+0x160/0x160 [13424.184211] ptlrpcd+0xa82/0xe40 [ptlrpc] [13424.187169] ? do_wait_intr+0xf0/0xf0 [13424.188914] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13424.191972] kthread+0x1d1/0x200 [13424.193077] ? set_kthread_struct+0x70/0x70 [13424.194878] ret_from_fork+0x1f/0x30 [13428.057935] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13428.065613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13428.130927] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13428.139955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13428.158148] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13428.170260] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13428.172246] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13428.202702] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13428.216638] RAX: 000000000000698f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13428.229597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13428.242574] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13428.252232] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13428.260589] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13428.273973] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13428.282757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13428.295220] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13428.307270] Call Trace: [13428.309862] [13428.311314] ? show_regs.cold.9+0x22/0x2f [13428.313915] ? watchdog_timer_fn.cold.6+0x62/0x10a [13428.317101] ? __hrtimer_run_queues+0x1c2/0x4d0 [13428.327147] ? watchdog+0x40/0x40 [13428.328430] ? hrtimer_interrupt+0x128/0x2c0 [13428.331024] ? smp_apic_timer_interrupt+0xa6/0x270 [13428.338338] ? apic_timer_interrupt+0xf/0x20 [13428.339923] [13428.340534] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13428.342852] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13428.347106] do_raw_spin_lock+0x177/0x190 [13428.351982] _raw_spin_lock+0x21/0x30 [13428.355974] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13428.362840] process_one_work+0x2c8/0x700 [13428.365853] worker_thread+0x296/0x6e0 [13428.369882] ? rescuer_thread+0x570/0x570 [13428.372497] kthread+0x1d1/0x200 [13428.373744] ? set_kthread_struct+0x70/0x70 [13428.379050] ret_from_fork+0x1f/0x30 [13432.054099] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13432.063182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13432.115813] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13432.130276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13432.141089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13432.148784] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13432.173567] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13432.186563] RAX: 0000000000003a8a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13432.198523] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13432.206332] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13432.212355] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13432.221146] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13432.228163] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13432.233515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13432.243510] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13432.257094] Call Trace: [13432.260305] [13432.263965] ? show_regs.cold.9+0x22/0x2f [13432.268301] ? watchdog_timer_fn.cold.6+0x62/0x10a [13432.275077] ? __hrtimer_run_queues+0x1c2/0x4d0 [13432.280709] ? watchdog+0x40/0x40 [13432.286052] ? hrtimer_interrupt+0x128/0x2c0 [13432.291105] ? smp_apic_timer_interrupt+0xa6/0x270 [13432.292595] ? apic_timer_interrupt+0xf/0x20 [13432.297239] [13432.303924] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13432.311446] do_raw_spin_lock+0x177/0x190 [13432.315336] _raw_spin_lock+0x21/0x30 [13432.319557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13432.327031] ? do_raw_spin_unlock+0x75/0x190 [13432.328393] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13432.340021] ? do_wait_intr+0xf0/0xf0 [13432.347300] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13432.350882] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13432.353571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13432.360237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13432.366267] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13432.370075] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13432.372508] mdc_intent_lock+0x353/0x7e0 [mdc] [13432.375090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13432.378179] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13432.383257] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13432.386273] lmv_intent_open+0x338/0x1150 [lmv] [13432.388002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13432.392941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13432.396353] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13432.404565] ll_intent_lock+0xc5/0xaa0 [lustre] [13432.413005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13432.416197] ll_lookup_it+0xe0e/0x2270 [lustre] [13432.417515] ? do_raw_spin_unlock+0x75/0x190 [13432.428017] ? _raw_spin_unlock+0x12/0x30 [13432.429167] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13432.434189] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13432.435891] ll_atomic_open+0x36a/0xd70 [lustre] [13432.442231] lookup_open+0x356/0xb40 [13432.447580] do_last+0x313/0xfc0 [13432.454372] ? nd_jump_root+0xe5/0x160 [13432.455794] ? path_init+0x437/0x520 [13432.467534] path_openat+0xf7/0x500 [13432.474996] ? do_raw_spin_unlock+0x75/0x190 [13432.480763] do_filp_open+0x99/0x140 [13432.482088] ? getname_flags+0x6e/0x330 [13432.483393] ? __check_object_size+0xff/0x256 [13432.493033] ? do_raw_spin_unlock+0x75/0x190 [13432.494423] ? _raw_spin_unlock+0x12/0x30 [13432.501487] do_sys_openat2+0x2b4/0x410 [13432.511270] do_sys_open+0x73/0xa0 [13432.512664] __x64_sys_openat+0x24/0x30 [13432.513963] do_syscall_64+0xc1/0x440 [13432.522451] entry_SYSCALL_64_after_hwframe+0x49/0xae [13432.524250] RIP: 0033:0x7fa749552332 [13432.531382] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13432.545199] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13432.549362] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13432.553021] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13432.562394] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13432.568608] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13432.572961] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13437.413465] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11520s! [13437.425165] Showing busy workqueues and worker pools: [13437.429346] workqueue events: flags=0x0 [13437.431895] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13437.443752] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13437.453153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13437.463346] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13437.478010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.482237] pending: kfree_rcu_monitor [13437.484168] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13437.491673] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13437.496710] workqueue events_long: flags=0x0 [13437.499105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.500728] pending: lu_objects_flush [obdclass] [13437.508136] workqueue events_unbound: flags=0x2 [13437.510987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13437.516108] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13437.527018] workqueue events_power_efficient: flags=0x80 [13437.529799] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.534553] pending: check_lifetime [13437.536675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.540040] pending: neigh_periodic_work [13437.544882] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.547383] pending: neigh_periodic_work [13437.557125] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.563202] pending: do_cache_clean [13437.564958] workqueue mm_percpu_wq: flags=0x8 [13437.570327] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.579398] pending: vmstat_update [13437.583637] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.590166] pending: vmstat_update [13437.592709] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13437.598042] pending: vmstat_update [13437.604846] workqueue writeback: flags=0x4a [13437.606463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13437.617209] pending: wb_workfn, wb_workfn [13437.620239] workqueue xprtiod: flags=0xa [13437.622728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13437.629691] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13437.638231] workqueue ipv6_addrconf: flags=0x40008 [13437.642832] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13437.654719] pending: addrconf_verify_work [13437.656249] workqueue knbd0-recv: flags=0x1a [13437.666629] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13437.670096] in-flight: 651:recv_work [13437.672697] workqueue ptlrpc_pinger: flags=0x2 [13437.676211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13437.678685] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13437.685664] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11520s workers=4 idle: 37 31846 28897 [13437.696420] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13427s workers=2 idle: 653 [13440.059275] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13440.063812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13440.084906] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13440.088749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13440.093985] RIP: 0010:native_safe_halt+0x2e/0x50 [13440.102754] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13440.113350] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13440.116916] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13440.120866] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13440.125465] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13440.129112] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13440.133595] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13440.138500] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13440.144433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13440.147727] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13440.151532] Call Trace: [13440.152902] [13440.154426] ? show_regs.cold.9+0x22/0x2f [13440.162125] ? watchdog_timer_fn.cold.6+0x62/0x10a [13440.166567] ? __hrtimer_run_queues+0x1c2/0x4d0 [13440.169693] ? watchdog+0x40/0x40 [13440.173037] ? hrtimer_interrupt+0x128/0x2c0 [13440.176658] ? smp_apic_timer_interrupt+0xa6/0x270 [13440.180383] ? apic_timer_interrupt+0xf/0x20 [13440.182053] [13440.187594] ? native_safe_halt+0x2e/0x50 [13440.190138] kvm_wait+0xcd/0xe0 [13440.197217] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13440.203191] do_raw_spin_lock+0x177/0x190 [13440.205006] _raw_spin_lock+0x21/0x30 [13440.206596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13440.209500] ? do_raw_spin_unlock+0x75/0x190 [13440.211804] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13440.215030] ? do_wait_intr+0xf0/0xf0 [13440.217602] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13440.226446] mdc_close+0x53b/0xd70 [mdc] [13440.230418] lmv_close+0x18d/0x490 [lmv] [13440.232295] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13440.237889] ll_md_real_close+0xa5/0x2a0 [lustre] [13440.244476] ll_md_close+0x274/0x800 [lustre] [13440.248291] ll_file_release+0x128/0xc50 [lustre] [13440.250520] __fput+0x10d/0x440 [13440.251981] ____fput+0x12/0x20 [13440.253553] task_work_run+0xc6/0x110 [13440.256202] exit_to_usermode_loop+0x1dd/0x1f0 [13440.258161] do_syscall_64+0x3d6/0x440 [13440.261292] entry_SYSCALL_64_after_hwframe+0x49/0xae [13440.263048] RIP: 0033:0x7fa749552cd8 [13440.264722] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13440.273474] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13440.276843] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13440.282104] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13440.288083] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13440.291862] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13440.301786] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13452.064536] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13452.073355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13452.128982] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13452.148336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13452.163509] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13452.178236] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13452.190133] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13452.199149] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13452.204571] RDX: 00000000000025ed RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13452.210985] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13452.219034] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13452.221611] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13452.236562] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13452.242536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13452.255573] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13452.262033] Call Trace: [13452.264661] [13452.265477] ? show_regs.cold.9+0x22/0x2f [13452.272737] ? watchdog_timer_fn.cold.6+0x62/0x10a [13452.278775] ? __hrtimer_run_queues+0x1c2/0x4d0 [13452.280491] ? watchdog+0x40/0x40 [13452.286129] ? hrtimer_interrupt+0x128/0x2c0 [13452.289478] ? smp_apic_timer_interrupt+0xa6/0x270 [13452.294507] ? apic_timer_interrupt+0xf/0x20 [13452.301539] [13452.303947] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13452.311324] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13452.315728] do_raw_spin_lock+0x177/0x190 [13452.320787] _raw_spin_lock+0x21/0x30 [13452.325580] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13452.334976] obd_mod_free+0x46/0x140 [ptlrpc] [13452.342298] mdc_commit_open+0x91/0xb0 [mdc] [13452.348881] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13452.356450] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13452.362001] after_reply+0xe2c/0x1780 [ptlrpc] [13452.366243] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13452.372973] ? __next_timer_interrupt+0x160/0x160 [13452.386658] ptlrpcd+0xa82/0xe40 [ptlrpc] [13452.395239] ? do_wait_intr+0xf0/0xf0 [13452.397006] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13452.404630] kthread+0x1d1/0x200 [13452.408333] ? set_kthread_struct+0x70/0x70 [13452.411866] ret_from_fork+0x1f/0x30 [13456.057817] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13456.065646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13456.119134] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13456.128408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13456.132363] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13456.136402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13456.140607] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13456.161453] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13456.165106] RAX: 0000000000006e8a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13456.169213] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13456.171812] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13456.178100] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13456.182567] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13456.188115] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13456.202305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13456.208091] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13456.213949] Call Trace: [13456.215423] [13456.216527] ? show_regs.cold.9+0x22/0x2f [13456.222399] ? watchdog_timer_fn.cold.6+0x62/0x10a [13456.225479] ? __hrtimer_run_queues+0x1c2/0x4d0 [13456.227009] ? watchdog+0x40/0x40 [13456.228830] ? hrtimer_interrupt+0x128/0x2c0 [13456.230171] ? smp_apic_timer_interrupt+0xa6/0x270 [13456.234394] ? apic_timer_interrupt+0xf/0x20 [13456.236269] [13456.237790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13456.241043] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13456.243026] do_raw_spin_lock+0x177/0x190 [13456.246400] _raw_spin_lock+0x21/0x30 [13456.248932] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13456.251942] process_one_work+0x2c8/0x700 [13456.254094] worker_thread+0x296/0x6e0 [13456.255686] ? rescuer_thread+0x570/0x570 [13456.257557] kthread+0x1d1/0x200 [13456.262194] ? set_kthread_struct+0x70/0x70 [13456.263650] ret_from_fork+0x1f/0x30 [13464.055208] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13464.069289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13464.123984] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13464.131934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13464.143558] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13464.146123] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13464.158923] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13464.162037] RAX: 0000000000002a8a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13464.168066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13464.173654] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13464.181241] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13464.186410] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13464.195567] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13464.202501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13464.209179] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13464.220057] Call Trace: [13464.224334] [13464.227047] ? show_regs.cold.9+0x22/0x2f [13464.232169] ? watchdog_timer_fn.cold.6+0x62/0x10a [13464.237535] ? __hrtimer_run_queues+0x1c2/0x4d0 [13464.244166] ? watchdog+0x40/0x40 [13464.248558] ? hrtimer_interrupt+0x128/0x2c0 [13464.253875] ? smp_apic_timer_interrupt+0xa6/0x270 [13464.257091] ? apic_timer_interrupt+0xf/0x20 [13464.259250] [13464.260267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13464.262219] do_raw_spin_lock+0x177/0x190 [13464.263654] _raw_spin_lock+0x21/0x30 [13464.265562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13464.270552] ? do_raw_spin_unlock+0x75/0x190 [13464.273372] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13464.278636] ? do_wait_intr+0xf0/0xf0 [13464.281114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13464.287646] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13464.293532] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13464.301506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13464.309120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13464.312594] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13464.315569] mdc_intent_lock+0x353/0x7e0 [mdc] [13464.318739] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13464.321881] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13464.326528] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13464.333192] lmv_intent_open+0x338/0x1150 [lmv] [13464.341208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13464.351648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13464.363175] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13464.371997] ll_intent_lock+0xc5/0xaa0 [lustre] [13464.380333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13464.385640] ll_lookup_it+0xe0e/0x2270 [lustre] [13464.391337] ? do_raw_spin_unlock+0x75/0x190 [13464.394716] ? _raw_spin_unlock+0x12/0x30 [13464.399248] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13464.405055] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13464.413476] ll_atomic_open+0x36a/0xd70 [lustre] [13464.419238] lookup_open+0x356/0xb40 [13464.427428] do_last+0x313/0xfc0 [13464.428485] ? nd_jump_root+0xe5/0x160 [13464.433476] ? path_init+0x437/0x520 [13464.440481] path_openat+0xf7/0x500 [13464.444934] ? do_raw_spin_unlock+0x75/0x190 [13464.450637] do_filp_open+0x99/0x140 [13464.458073] ? getname_flags+0x6e/0x330 [13464.460783] ? __check_object_size+0xff/0x256 [13464.463636] ? do_raw_spin_unlock+0x75/0x190 [13464.468578] ? _raw_spin_unlock+0x12/0x30 [13464.471821] do_sys_openat2+0x2b4/0x410 [13464.473138] do_sys_open+0x73/0xa0 [13464.485985] __x64_sys_openat+0x24/0x30 [13464.494651] do_syscall_64+0xc1/0x440 [13464.503161] entry_SYSCALL_64_after_hwframe+0x49/0xae [13464.505572] RIP: 0033:0x7fa749552332 [13464.517921] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13464.539168] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13464.550386] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13464.558796] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13464.571005] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13464.579575] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13464.586510] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13468.059284] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13468.069304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13468.112492] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13468.121430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13468.132276] BUG: workqueue lockup - pool [13468.135272] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [13468.135292] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [13468.135296] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13468.135301] RAX: 00000000000049a8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13468.135304] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13468.135306] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13468.135308] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13468.135310] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13468.135313] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13468.135316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13468.135318] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13468.136199] Call Trace: [13468.136205] [13468.136210] ? show_regs.cold.9+0x22/0x2f [13468.136221] ? watchdog_timer_fn.cold.6+0x62/0x10a [13468.136228] ? __hrtimer_run_queues+0x1c2/0x4d0 [13468.136232] ? watchdog+0x40/0x40 [13468.136236] ? hrtimer_interrupt+0x128/0x2c0 [13468.136240] ? smp_apic_timer_interrupt+0xa6/0x270 [13468.136249] ? apic_timer_interrupt+0xf/0x20 [13468.136253] [13468.136257] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [13468.136264] do_raw_spin_lock+0x177/0x190 [13468.136269] _raw_spin_lock+0x21/0x30 [13468.136274] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13468.140392] cpus=0-3 flags=0x4 nice=0 stuck for 11550s! [13468.140410] Showing busy workqueues and worker pools: [13468.140412] workqueue events: flags=0x0 [13468.140416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13468.140426] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13468.141216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13468.141349] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13468.141964] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.141972] pending: kfree_rcu_monitor [13468.141982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13468.141988] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13468.143245] workqueue events_long: flags=0x0 [13468.143251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.143262] pending: lu_objects_flush [obdclass] [13468.143425] workqueue events_unbound: flags=0x2 [13468.143428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13468.143435] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13468.143457] workqueue events_power_efficient: flags=0x80 [13468.143460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.143468] pending: check_lifetime [13468.143476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.143484] pending: neigh_periodic_work [13468.143492] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.143499] pending: neigh_periodic_work [13468.143504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.143513] pending: do_cache_clean [13468.143522] workqueue mm_percpu_wq: flags=0x8 [13468.143524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.143531] pending: vmstat_update [13468.143536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.143543] pending: vmstat_update [13468.143547] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13468.143554] pending: vmstat_update [13468.143560] workqueue writeback: flags=0x4a [13468.143563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13468.143569] pending: wb_workfn, wb_workfn [13468.143583] workqueue xprtiod: flags=0xa [13468.143585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13468.143590] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13468.143609] workqueue ipv6_addrconf: flags=0x40008 [13468.143612] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13468.143858] pending: addrconf_verify_work [13468.143872] workqueue knbd0-recv: flags=0x1a [13468.143874] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13468.143881] in-flight: 651:recv_work [13468.143890] workqueue ptlrpc_pinger: flags=0x2 [13468.143892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13468.143898] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13468.145337] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11550s workers=4 idle: 37 31846 28897 [13468.145355] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13458s workers=2 idle: 653 [13468.675157] ? do_raw_spin_unlock+0x75/0x190 [13468.681007] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13468.683356] ? do_wait_intr+0xf0/0xf0 [13468.684634] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13468.695170] mdc_close+0x53b/0xd70 [mdc] [13468.699450] lmv_close+0x18d/0x490 [lmv] [13468.702057] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13468.708265] ll_md_real_close+0xa5/0x2a0 [lustre] [13468.713681] ll_md_close+0x274/0x800 [lustre] [13468.715783] ll_file_release+0x128/0xc50 [lustre] [13468.722597] __fput+0x10d/0x440 [13468.724370] ____fput+0x12/0x20 [13468.726385] task_work_run+0xc6/0x110 [13468.727626] exit_to_usermode_loop+0x1dd/0x1f0 [13468.730952] do_syscall_64+0x3d6/0x440 [13468.733046] entry_SYSCALL_64_after_hwframe+0x49/0xae [13468.735981] RIP: 0033:0x7fa749552cd8 [13468.741666] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13468.757695] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13468.765088] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13468.773348] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13468.778104] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13468.782492] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13468.790533] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13480.059033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13480.063372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13480.088488] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13480.093013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13480.096980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13480.099990] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13480.109521] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13480.114080] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13480.117948] RDX: 0000000000002e0a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13480.121281] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13480.125716] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13480.128916] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13480.131536] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13480.134773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13480.137253] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13480.144212] Call Trace: [13480.145930] [13480.147010] ? show_regs.cold.9+0x22/0x2f [13480.149484] ? watchdog_timer_fn.cold.6+0x62/0x10a [13480.152504] ? __hrtimer_run_queues+0x1c2/0x4d0 [13480.158876] ? watchdog+0x40/0x40 [13480.160724] ? hrtimer_interrupt+0x128/0x2c0 [13480.163247] ? smp_apic_timer_interrupt+0xa6/0x270 [13480.166188] ? apic_timer_interrupt+0xf/0x20 [13480.168265] [13480.169347] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13480.176836] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13480.179790] do_raw_spin_lock+0x177/0x190 [13480.185690] _raw_spin_lock+0x21/0x30 [13480.187156] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13480.191031] obd_mod_free+0x46/0x140 [ptlrpc] [13480.194609] mdc_commit_open+0x91/0xb0 [mdc] [13480.197115] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13480.199891] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13480.205425] after_reply+0xe2c/0x1780 [ptlrpc] [13480.208033] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13480.211204] ? __next_timer_interrupt+0x160/0x160 [13480.214135] ptlrpcd+0xa82/0xe40 [ptlrpc] [13480.216216] ? do_wait_intr+0xf0/0xf0 [13480.217931] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13480.220411] kthread+0x1d1/0x200 [13480.224325] ? set_kthread_struct+0x70/0x70 [13480.226117] ret_from_fork+0x1f/0x30 [13484.057033] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13484.060271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13484.083025] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13484.088557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13484.095610] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13484.102609] RIP: 0010:native_safe_halt+0x2e/0x50 [13484.109443] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13484.126886] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13484.132369] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13484.135739] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13484.143406] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13484.148568] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13484.153872] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13484.159179] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13484.165506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13484.169250] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13484.175152] Call Trace: [13484.176558] [13484.182238] ? show_regs.cold.9+0x22/0x2f [13484.185215] ? watchdog_timer_fn.cold.6+0x62/0x10a [13484.192045] ? __hrtimer_run_queues+0x1c2/0x4d0 [13484.197436] ? watchdog+0x40/0x40 [13484.198949] ? hrtimer_interrupt+0x128/0x2c0 [13484.201546] ? smp_apic_timer_interrupt+0xa6/0x270 [13484.206391] ? apic_timer_interrupt+0xf/0x20 [13484.209596] [13484.215611] ? native_safe_halt+0x2e/0x50 [13484.217593] kvm_wait+0xcd/0xe0 [13484.221038] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13484.226621] do_raw_spin_lock+0x177/0x190 [13484.228887] _raw_spin_lock+0x21/0x30 [13484.231876] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13484.235805] process_one_work+0x2c8/0x700 [13484.245811] worker_thread+0x296/0x6e0 [13484.248850] ? rescuer_thread+0x570/0x570 [13484.251244] kthread+0x1d1/0x200 [13484.252936] ? set_kthread_struct+0x70/0x70 [13484.258644] ret_from_fork+0x1f/0x30 [13493.943031] rcu: INFO: rcu_sched self-detected stall on CPU [13493.945520] rcu: 2-...!: (6602273 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [13493.949833] (t=11580254 jiffies g=258701 q=4740) [13493.952233] rcu: rcu_sched kthread starved for 11577049 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [13493.957540] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13493.962330] rcu: RCU grace-period kthread stack dump: [13493.966012] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13493.974313] Call Trace: [13493.975432] __schedule+0x351/0xcb0 [13493.977200] schedule+0xc0/0x180 [13493.978906] schedule_timeout+0xb4/0x190 [13493.980713] ? do_raw_spin_unlock+0x75/0x190 [13493.982621] ? __next_timer_interrupt+0x160/0x160 [13493.984231] ? prepare_to_swait_event+0xf3/0x140 [13493.986188] rcu_gp_kthread+0x8dd/0x15b0 [13493.987957] ? force_qs_rnp+0x2c0/0x2c0 [13493.989569] kthread+0x1d1/0x200 [13493.990763] ? set_kthread_struct+0x70/0x70 [13493.992470] ret_from_fork+0x1f/0x30 [13493.994038] rcu: Stack dump where RCU GP kthread last ran: [13493.996206] NMI backtrace for cpu 2 [13493.997785] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13494.001868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13494.005518] Call Trace: [13494.006657] [13494.007533] ? dump_stack+0xbb/0x10e [13494.009043] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13494.011236] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13494.013417] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13494.015435] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13494.017276] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13494.019452] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [13494.021870] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [13494.024015] ? update_process_times+0x6d/0xc0 [13494.025473] ? tick_sched_handle.isra.18+0x27/0x90 [13494.027463] ? tick_sched_timer+0x53/0xb0 [13494.028975] ? __hrtimer_run_queues+0x1c2/0x4d0 [13494.030967] ? tick_sched_handle.isra.18+0x90/0x90 [13494.033053] ? hrtimer_interrupt+0x128/0x2c0 [13494.034789] ? smp_apic_timer_interrupt+0xa6/0x270 [13494.037079] ? apic_timer_interrupt+0xf/0x20 [13494.039005] [13494.040002] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13494.042145] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13494.044690] ? do_raw_spin_lock+0x177/0x190 [13494.047042] ? _raw_spin_lock+0x21/0x30 [13494.048822] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13494.051435] ? obd_mod_free+0x46/0x140 [ptlrpc] [13494.055542] ? mdc_commit_open+0x91/0xb0 [mdc] [13494.058016] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13494.060486] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13494.062981] ? after_reply+0xe2c/0x1780 [ptlrpc] [13494.065254] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13494.067896] ? __next_timer_interrupt+0x160/0x160 [13494.070383] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13494.072855] ? do_wait_intr+0xf0/0xf0 [13494.074225] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13494.076767] ? kthread+0x1d1/0x200 [13494.078269] ? set_kthread_struct+0x70/0x70 [13494.079560] ? ret_from_fork+0x1f/0x30 [13494.081453] NMI backtrace for cpu 2 [13494.082753] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13494.086066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13494.088507] Call Trace: [13494.089282] [13494.090744] ? dump_stack+0xbb/0x10e [13494.091744] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13494.092962] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13494.094456] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13494.096115] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13494.097661] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13494.099649] ? rcu_dump_cpu_stacks+0x11b/0x171 [13494.100798] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13494.102514] ? update_process_times+0x6d/0xc0 [13494.104083] ? tick_sched_handle.isra.18+0x27/0x90 [13494.105949] ? tick_sched_timer+0x53/0xb0 [13494.107149] ? __hrtimer_run_queues+0x1c2/0x4d0 [13494.108598] ? tick_sched_handle.isra.18+0x90/0x90 [13494.109936] ? hrtimer_interrupt+0x128/0x2c0 [13494.111164] ? smp_apic_timer_interrupt+0xa6/0x270 [13494.112272] ? apic_timer_interrupt+0xf/0x20 [13494.114416] [13494.115006] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13494.116494] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13494.117946] ? do_raw_spin_lock+0x177/0x190 [13494.119155] ? _raw_spin_lock+0x21/0x30 [13494.120080] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13494.121779] ? obd_mod_free+0x46/0x140 [ptlrpc] [13494.123163] ? mdc_commit_open+0x91/0xb0 [mdc] [13494.124454] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13494.126091] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13494.128185] ? after_reply+0xe2c/0x1780 [ptlrpc] [13494.129897] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13494.132182] ? __next_timer_interrupt+0x160/0x160 [13494.134021] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13494.135706] ? do_wait_intr+0xf0/0xf0 [13494.137157] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13494.139631] ? kthread+0x1d1/0x200 [13494.140757] ? set_kthread_struct+0x70/0x70 [13494.141741] ? ret_from_fork+0x1f/0x30 [13496.054913] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13496.058331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13496.064901] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13496.092487] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13496.092508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13496.092511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13496.092529] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13496.092534] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13496.092549] RAX: 0000000000005304 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13496.092551] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13496.092553] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13496.092555] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13496.092558] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13496.092561] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13496.092564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13496.092566] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13496.092575] Call Trace: [13496.092580] [13496.092586] ? show_regs.cold.9+0x22/0x2f [13496.092595] ? watchdog_timer_fn.cold.6+0x62/0x10a [13496.092602] ? __hrtimer_run_queues+0x1c2/0x4d0 [13496.092606] ? watchdog+0x40/0x40 [13496.092611] ? hrtimer_interrupt+0x128/0x2c0 [13496.092615] ? smp_apic_timer_interrupt+0xa6/0x270 [13496.092623] ? apic_timer_interrupt+0xf/0x20 [13496.092628] [13496.092631] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13496.092640] do_raw_spin_lock+0x177/0x190 [13496.092645] _raw_spin_lock+0x21/0x30 [13496.092650] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13496.094621] ? do_raw_spin_unlock+0x75/0x190 [13496.094631] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13496.094898] ? do_wait_intr+0xf0/0xf0 [13496.094907] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13496.095110] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13496.095499] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13496.095724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13496.095834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13496.095872] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13496.095901] mdc_intent_lock+0x353/0x7e0 [mdc] [13496.095931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13496.096016] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13496.096230] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13496.096395] lmv_intent_open+0x338/0x1150 [lmv] [13496.096427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13496.096505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13496.096597] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13496.096620] ll_intent_lock+0xc5/0xaa0 [lustre] [13496.096707] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13496.096788] ll_lookup_it+0xe0e/0x2270 [lustre] [13496.096872] ? do_raw_spin_unlock+0x75/0x190 [13496.096878] ? _raw_spin_unlock+0x12/0x30 [13496.096884] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13496.096909] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13496.096927] ll_atomic_open+0x36a/0xd70 [lustre] [13496.097008] lookup_open+0x356/0xb40 [13496.097017] do_last+0x313/0xfc0 [13496.097022] ? nd_jump_root+0xe5/0x160 [13496.097026] ? path_init+0x437/0x520 [13496.097030] path_openat+0xf7/0x500 [13496.097034] ? do_raw_spin_unlock+0x75/0x190 [13496.097038] do_filp_open+0x99/0x140 [13496.097044] ? getname_flags+0x6e/0x330 [13496.097048] ? __check_object_size+0xff/0x256 [13496.097054] ? do_raw_spin_unlock+0x75/0x190 [13496.097058] ? _raw_spin_unlock+0x12/0x30 [13496.097062] do_sys_openat2+0x2b4/0x410 [13496.097065] do_sys_open+0x73/0xa0 [13496.097069] __x64_sys_openat+0x24/0x30 [13496.097072] do_syscall_64+0xc1/0x440 [13496.097078] entry_SYSCALL_64_after_hwframe+0x49/0xae [13496.097084] RIP: 0033:0x7fa749552332 [13496.097091] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13496.097094] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13496.097099] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13496.097102] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13496.097104] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13496.097106] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13496.097108] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13496.358333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13496.376739] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13496.381182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13496.384924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13496.387198] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13496.395028] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13496.398108] RAX: 0000000000004e01 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13496.401124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13496.404208] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13496.407158] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13496.410314] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13496.413148] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13496.416153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13496.418445] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13496.421186] Call Trace: [13496.422016] [13496.422735] ? show_regs.cold.9+0x22/0x2f [13496.423990] ? watchdog_timer_fn.cold.6+0x62/0x10a [13496.425929] ? __hrtimer_run_queues+0x1c2/0x4d0 [13496.427375] ? watchdog+0x40/0x40 [13496.428604] ? hrtimer_interrupt+0x128/0x2c0 [13496.430275] ? smp_apic_timer_interrupt+0xa6/0x270 [13496.431800] ? apic_timer_interrupt+0xf/0x20 [13496.433553] [13496.434298] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13496.436027] do_raw_spin_lock+0x177/0x190 [13496.437381] _raw_spin_lock+0x21/0x30 [13496.438526] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13496.440743] ? do_raw_spin_unlock+0x75/0x190 [13496.442375] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13496.444469] ? do_wait_intr+0xf0/0xf0 [13496.445885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13496.447859] mdc_close+0x53b/0xd70 [mdc] [13496.449412] lmv_close+0x18d/0x490 [lmv] [13496.451019] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13496.453023] ll_md_real_close+0xa5/0x2a0 [lustre] [13496.455025] ll_md_close+0x274/0x800 [lustre] [13496.456985] ll_file_release+0x128/0xc50 [lustre] [13496.459259] __fput+0x10d/0x440 [13496.460435] ____fput+0x12/0x20 [13496.461690] task_work_run+0xc6/0x110 [13496.463583] exit_to_usermode_loop+0x1dd/0x1f0 [13496.465162] do_syscall_64+0x3d6/0x440 [13496.466250] entry_SYSCALL_64_after_hwframe+0x49/0xae [13496.468442] RIP: 0033:0x7fa749552cd8 [13496.470145] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13496.478648] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13496.481943] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13496.484695] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13496.490995] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13496.496148] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13496.502804] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13498.343213] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11581s! [13498.355992] Showing busy workqueues and worker pools: [13498.360950] workqueue events: flags=0x0 [13498.365022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13498.371618] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13498.398641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13498.408327] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13498.416312] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.419149] pending: kfree_rcu_monitor [13498.422353] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13498.425596] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13498.433311] workqueue events_long: flags=0x0 [13498.434750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.437421] pending: lu_objects_flush [obdclass] [13498.440431] workqueue events_unbound: flags=0x2 [13498.443059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13498.446930] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13498.452307] workqueue events_power_efficient: flags=0x80 [13498.454533] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.461786] pending: check_lifetime [13498.470097] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.478450] pending: neigh_periodic_work [13498.481022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.483970] pending: neigh_periodic_work [13498.486398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.489292] pending: do_cache_clean [13498.491575] workqueue mm_percpu_wq: flags=0x8 [13498.493505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.498869] pending: vmstat_update [13498.501905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.505874] pending: vmstat_update [13498.508181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13498.511859] pending: vmstat_update [13498.516022] workqueue writeback: flags=0x4a [13498.519148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13498.523713] pending: wb_workfn, wb_workfn [13498.525987] workqueue xprtiod: flags=0xa [13498.528056] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13498.530848] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13498.535178] workqueue ipv6_addrconf: flags=0x40008 [13498.537367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13498.540519] pending: addrconf_verify_work [13498.542465] workqueue knbd0-recv: flags=0x1a [13498.544071] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13498.546753] in-flight: 651:recv_work [13498.548346] workqueue ptlrpc_pinger: flags=0x2 [13498.550816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13498.553041] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13498.555420] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11581s workers=4 idle: 37 31846 28897 [13498.558328] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13488s workers=2 idle: 653 [13512.057014] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13512.070245] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13512.155200] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13512.175584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13512.191327] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13512.207238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13512.216336] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13512.252423] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13512.265291] RAX: 000000000000288d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13512.273746] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13512.284750] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13512.297832] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13512.306997] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13512.328677] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13512.342043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13512.343899] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13512.370084] Call Trace: [13512.382971] [13512.383742] ? show_regs.cold.9+0x22/0x2f [13512.385154] ? watchdog_timer_fn.cold.6+0x62/0x10a [13512.392688] ? __hrtimer_run_queues+0x1c2/0x4d0 [13512.403454] ? watchdog+0x40/0x40 [13512.417501] ? hrtimer_interrupt+0x128/0x2c0 [13512.424828] ? smp_apic_timer_interrupt+0xa6/0x270 [13512.432367] ? apic_timer_interrupt+0xf/0x20 [13512.449720] [13512.450445] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13512.452192] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13512.465508] do_raw_spin_lock+0x177/0x190 [13512.466848] _raw_spin_lock+0x21/0x30 [13512.481718] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13512.500056] process_one_work+0x2c8/0x700 [13512.506068] worker_thread+0x296/0x6e0 [13512.520040] ? rescuer_thread+0x570/0x570 [13512.524532] kthread+0x1d1/0x200 [13512.541058] ? set_kthread_struct+0x70/0x70 [13512.554051] ret_from_fork+0x1f/0x30 [13520.058011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13520.061625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13520.082897] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13520.088010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13520.092242] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13520.095354] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13520.104713] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13520.108095] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13520.111646] RDX: 00000000000051bd RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13520.115106] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13520.118701] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13520.122344] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13520.126315] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13520.130337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13520.133313] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13520.136986] Call Trace: [13520.138153] [13520.138963] ? show_regs.cold.9+0x22/0x2f [13520.140681] ? watchdog_timer_fn.cold.6+0x62/0x10a [13520.142991] ? __hrtimer_run_queues+0x1c2/0x4d0 [13520.145181] ? watchdog+0x40/0x40 [13520.146948] ? hrtimer_interrupt+0x128/0x2c0 [13520.149131] ? smp_apic_timer_interrupt+0xa6/0x270 [13520.151659] ? apic_timer_interrupt+0xf/0x20 [13520.153621] [13520.154680] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13520.157165] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13520.159798] do_raw_spin_lock+0x177/0x190 [13520.161705] _raw_spin_lock+0x21/0x30 [13520.163446] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13520.166257] obd_mod_free+0x46/0x140 [ptlrpc] [13520.168827] mdc_commit_open+0x91/0xb0 [mdc] [13520.171172] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13520.174358] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13520.177623] after_reply+0xe2c/0x1780 [ptlrpc] [13520.180281] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13520.183665] ? __next_timer_interrupt+0x160/0x160 [13520.185785] ptlrpcd+0xa82/0xe40 [ptlrpc] [13520.188249] ? do_wait_intr+0xf0/0xf0 [13520.189988] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13520.192748] kthread+0x1d1/0x200 [13520.194364] ? set_kthread_struct+0x70/0x70 [13520.196409] ret_from_fork+0x1f/0x30 [13524.058909] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13524.060073] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13524.066227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13524.066306] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13524.066313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13524.066317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13524.066330] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13524.066334] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13524.066339] RAX: 0000000000000a1b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13524.066341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13524.066344] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13524.066346] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13524.066348] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13524.066352] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13524.066607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13524.066610] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13524.066617] Call Trace: [13524.066623] [13524.066629] ? show_regs.cold.9+0x22/0x2f [13524.066638] ? watchdog_timer_fn.cold.6+0x62/0x10a [13524.066646] ? __hrtimer_run_queues+0x1c2/0x4d0 [13524.066650] ? watchdog+0x40/0x40 [13524.066655] ? hrtimer_interrupt+0x128/0x2c0 [13524.066659] ? smp_apic_timer_interrupt+0xa6/0x270 [13524.066667] ? apic_timer_interrupt+0xf/0x20 [13524.066672] [13524.066674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13524.066681] do_raw_spin_lock+0x177/0x190 [13524.066686] _raw_spin_lock+0x21/0x30 [13524.066692] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13524.067635] ? do_raw_spin_unlock+0x75/0x190 [13524.067642] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13524.068072] ? do_wait_intr+0xf0/0xf0 [13524.068082] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13524.075866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13524.076025] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13524.076032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13524.076036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13524.076067] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13524.076072] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13524.076078] RAX: 0000000000002784 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13524.076080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13524.076083] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13524.076086] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13524.076089] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13524.076094] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13524.076097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13524.076099] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13524.076107] Call Trace: [13524.076112] [13524.076119] ? show_regs.cold.9+0x22/0x2f [13524.076129] ? watchdog_timer_fn.cold.6+0x62/0x10a [13524.076135] ? __hrtimer_run_queues+0x1c2/0x4d0 [13524.076140] ? watchdog+0x40/0x40 [13524.076144] ? hrtimer_interrupt+0x128/0x2c0 [13524.076148] ? smp_apic_timer_interrupt+0xa6/0x270 [13524.076157] ? apic_timer_interrupt+0xf/0x20 [13524.076161] [13524.076163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13524.076169] do_raw_spin_lock+0x177/0x190 [13524.076174] _raw_spin_lock+0x21/0x30 [13524.076178] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13524.076582] ? do_raw_spin_unlock+0x75/0x190 [13524.076588] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13524.076803] ? do_wait_intr+0xf0/0xf0 [13524.076810] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13524.077522] mdc_close+0x53b/0xd70 [mdc] [13524.077573] lmv_close+0x18d/0x490 [lmv] [13524.077603] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13524.077722] ll_md_real_close+0xa5/0x2a0 [lustre] [13524.077824] ll_md_close+0x274/0x800 [lustre] [13524.079806] ll_file_release+0x128/0xc50 [lustre] [13524.080080] __fput+0x10d/0x440 [13524.080090] ____fput+0x12/0x20 [13524.080095] task_work_run+0xc6/0x110 [13524.080104] exit_to_usermode_loop+0x1dd/0x1f0 [13524.080111] do_syscall_64+0x3d6/0x440 [13524.080116] entry_SYSCALL_64_after_hwframe+0x49/0xae [13524.080123] RIP: 0033:0x7fa749552cd8 [13524.080130] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13524.080134] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13524.080139] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13524.080142] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13524.080144] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13524.080147] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13524.080149] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13524.737808] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13524.747013] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13524.754031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13524.763301] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13524.767260] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13524.772317] mdc_intent_lock+0x353/0x7e0 [mdc] [13524.773968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13524.775873] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13524.789111] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13524.792419] lmv_intent_open+0x338/0x1150 [lmv] [13524.793966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13524.807674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13524.817672] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13524.822794] ll_intent_lock+0xc5/0xaa0 [lustre] [13524.829878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13524.835046] ll_lookup_it+0xe0e/0x2270 [lustre] [13524.836773] ? do_raw_spin_unlock+0x75/0x190 [13524.843521] ? _raw_spin_unlock+0x12/0x30 [13524.848932] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13524.852210] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13524.857155] ll_atomic_open+0x36a/0xd70 [lustre] [13524.860135] lookup_open+0x356/0xb40 [13524.862202] do_last+0x313/0xfc0 [13524.865173] ? nd_jump_root+0xe5/0x160 [13524.874728] ? path_init+0x437/0x520 [13524.878855] path_openat+0xf7/0x500 [13524.882503] ? do_raw_spin_unlock+0x75/0x190 [13524.885057] do_filp_open+0x99/0x140 [13524.890666] ? getname_flags+0x6e/0x330 [13524.893082] ? __check_object_size+0xff/0x256 [13524.897181] ? do_raw_spin_unlock+0x75/0x190 [13524.901688] ? _raw_spin_unlock+0x12/0x30 [13524.904767] do_sys_openat2+0x2b4/0x410 [13524.907555] do_sys_open+0x73/0xa0 [13524.911907] __x64_sys_openat+0x24/0x30 [13524.915904] do_syscall_64+0xc1/0x440 [13524.920220] entry_SYSCALL_64_after_hwframe+0x49/0xae [13524.924519] RIP: 0033:0x7fa749552332 [13524.927860] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13524.947258] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13524.955831] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13524.965341] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13524.969874] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13524.978917] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13524.986334] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13529.067096] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11611s! [13529.075740] Showing busy workqueues and worker pools: [13529.080015] workqueue events: flags=0x0 [13529.085257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13529.093806] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13529.108655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13529.120969] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13529.135183] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.138131] pending: kfree_rcu_monitor [13529.139903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13529.149416] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13529.154905] workqueue events_long: flags=0x0 [13529.159895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.166658] pending: lu_objects_flush [obdclass] [13529.173521] workqueue events_unbound: flags=0x2 [13529.180936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13529.189713] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13529.214121] workqueue events_power_efficient: flags=0x80 [13529.215952] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.223631] pending: check_lifetime [13529.232941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.239557] pending: neigh_periodic_work [13529.245168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.249138] pending: neigh_periodic_work [13529.251803] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.257841] pending: do_cache_clean [13529.260792] workqueue mm_percpu_wq: flags=0x8 [13529.265660] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.270155] pending: vmstat_update [13529.271618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.279630] pending: vmstat_update [13529.289335] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13529.294618] pending: vmstat_update [13529.296452] workqueue writeback: flags=0x4a [13529.298642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13529.301531] pending: wb_workfn, wb_workfn [13529.305080] workqueue xprtiod: flags=0xa [13529.319014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13529.326744] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13529.333427] workqueue ipv6_addrconf: flags=0x40008 [13529.341916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13529.349248] pending: addrconf_verify_work [13529.354551] workqueue knbd0-recv: flags=0x1a [13529.358630] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13529.364500] in-flight: 651:recv_work [13529.367635] workqueue ptlrpc_pinger: flags=0x2 [13529.368725] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13529.377305] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13529.383911] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11612s workers=4 idle: 37 31846 28897 [13529.391935] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13519s workers=2 idle: 653 [13540.057017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13540.060417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13540.091079] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13540.095697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13540.101379] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13540.104403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13540.107878] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13540.120024] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13540.123166] RAX: 0000000000006f10 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13540.129296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13540.133949] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13540.142144] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13540.149361] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13540.152993] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13540.157296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13540.159990] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13540.163338] Call Trace: [13540.164459] [13540.165386] ? show_regs.cold.9+0x22/0x2f [13540.167005] ? watchdog_timer_fn.cold.6+0x62/0x10a [13540.174087] ? __hrtimer_run_queues+0x1c2/0x4d0 [13540.179135] ? watchdog+0x40/0x40 [13540.182777] ? hrtimer_interrupt+0x128/0x2c0 [13540.185348] ? smp_apic_timer_interrupt+0xa6/0x270 [13540.188358] ? apic_timer_interrupt+0xf/0x20 [13540.194536] [13540.195330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13540.200896] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13540.203756] do_raw_spin_lock+0x177/0x190 [13540.205823] _raw_spin_lock+0x21/0x30 [13540.207416] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13540.209756] process_one_work+0x2c8/0x700 [13540.210746] worker_thread+0x296/0x6e0 [13540.212282] ? rescuer_thread+0x570/0x570 [13540.213528] kthread+0x1d1/0x200 [13540.214503] ? set_kthread_struct+0x70/0x70 [13540.218737] ret_from_fork+0x1f/0x30 [13548.058029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13548.063143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13548.082039] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13548.086053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13548.089108] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13548.093024] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13548.105010] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13548.108434] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13548.112756] RDX: 00000000000067c2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13548.123229] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13548.131509] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13548.140809] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13548.157532] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13548.170571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13548.184747] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13548.200999] Call Trace: [13548.205562] [13548.208810] ? show_regs.cold.9+0x22/0x2f [13548.214770] ? watchdog_timer_fn.cold.6+0x62/0x10a [13548.223758] ? __hrtimer_run_queues+0x1c2/0x4d0 [13548.227349] ? watchdog+0x40/0x40 [13548.230778] ? hrtimer_interrupt+0x128/0x2c0 [13548.235500] ? smp_apic_timer_interrupt+0xa6/0x270 [13548.240893] ? apic_timer_interrupt+0xf/0x20 [13548.244268] [13548.246328] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13548.253392] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13548.256858] do_raw_spin_lock+0x177/0x190 [13548.259036] _raw_spin_lock+0x21/0x30 [13548.261882] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13548.270464] obd_mod_free+0x46/0x140 [ptlrpc] [13548.272521] mdc_commit_open+0x91/0xb0 [mdc] [13548.276725] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13548.278196] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13548.282025] after_reply+0xe2c/0x1780 [ptlrpc] [13548.284527] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13548.286438] ? __next_timer_interrupt+0x160/0x160 [13548.288813] ptlrpcd+0xa82/0xe40 [ptlrpc] [13548.294934] ? do_wait_intr+0xf0/0xf0 [13548.298264] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13548.301926] kthread+0x1d1/0x200 [13548.302921] ? set_kthread_struct+0x70/0x70 [13548.304988] ret_from_fork+0x1f/0x30 [13552.063867] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13552.077589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13552.122855] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13552.134260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13552.139885] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13552.142826] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13552.155154] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13552.160661] RAX: 0000000000002c0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13552.165971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13552.169670] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13552.175125] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13552.178410] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13552.182230] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13552.185635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13552.188851] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13552.194366] Call Trace: [13552.195493] [13552.196445] ? show_regs.cold.9+0x22/0x2f [13552.198937] ? watchdog_timer_fn.cold.6+0x62/0x10a [13552.202085] ? __hrtimer_run_queues+0x1c2/0x4d0 [13552.206164] ? watchdog+0x40/0x40 [13552.207781] ? hrtimer_interrupt+0x128/0x2c0 [13552.209790] ? smp_apic_timer_interrupt+0xa6/0x270 [13552.212083] ? apic_timer_interrupt+0xf/0x20 [13552.214331] [13552.215246] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13552.219819] do_raw_spin_lock+0x177/0x190 [13552.221576] _raw_spin_lock+0x21/0x30 [13552.224607] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13552.227927] ? do_raw_spin_unlock+0x75/0x190 [13552.229878] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13552.233634] ? do_wait_intr+0xf0/0xf0 [13552.235911] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13552.238566] mdc_close+0x53b/0xd70 [mdc] [13552.241677] lmv_close+0x18d/0x490 [lmv] [13552.244980] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13552.247905] ll_md_real_close+0xa5/0x2a0 [lustre] [13552.250402] ll_md_close+0x274/0x800 [lustre] [13552.252334] ll_file_release+0x128/0xc50 [lustre] [13552.254859] __fput+0x10d/0x440 [13552.256185] ____fput+0x12/0x20 [13552.257814] task_work_run+0xc6/0x110 [13552.259169] exit_to_usermode_loop+0x1dd/0x1f0 [13552.261785] do_syscall_64+0x3d6/0x440 [13552.263469] entry_SYSCALL_64_after_hwframe+0x49/0xae [13552.265958] RIP: 0033:0x7fa749552cd8 [13552.267322] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13552.280703] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13552.283968] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13552.289481] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13552.295023] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13552.297748] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13552.300589] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13556.054101] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13556.056441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13556.087138] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13556.098385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13556.101187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13556.103545] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13556.113836] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13556.119352] RAX: 0000000000006c94 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13556.123408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13556.134440] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13556.140849] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13556.146687] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13556.150893] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13556.154136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13556.156234] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13556.163916] Call Trace: [13556.165659] [13556.166333] ? show_regs.cold.9+0x22/0x2f [13556.167734] ? watchdog_timer_fn.cold.6+0x62/0x10a [13556.170963] ? __hrtimer_run_queues+0x1c2/0x4d0 [13556.175773] ? watchdog+0x40/0x40 [13556.180105] ? hrtimer_interrupt+0x128/0x2c0 [13556.181762] ? smp_apic_timer_interrupt+0xa6/0x270 [13556.188314] ? apic_timer_interrupt+0xf/0x20 [13556.191309] [13556.192578] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13556.197053] do_raw_spin_lock+0x177/0x190 [13556.199963] _raw_spin_lock+0x21/0x30 [13556.202114] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13556.205914] ? do_raw_spin_unlock+0x75/0x190 [13556.208703] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13556.212946] ? do_wait_intr+0xf0/0xf0 [13556.217396] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13556.221858] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13556.225891] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13556.229794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13556.233447] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13556.236722] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13556.240950] mdc_intent_lock+0x353/0x7e0 [mdc] [13556.246468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13556.249729] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13556.254134] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13556.261315] lmv_intent_open+0x338/0x1150 [lmv] [13556.262935] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13556.267697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13556.271962] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13556.274870] ll_intent_lock+0xc5/0xaa0 [lustre] [13556.277324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13556.284638] ll_lookup_it+0xe0e/0x2270 [lustre] [13556.287716] ? do_raw_spin_unlock+0x75/0x190 [13556.292133] ? _raw_spin_unlock+0x12/0x30 [13556.295807] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13556.299036] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13556.302214] ll_atomic_open+0x36a/0xd70 [lustre] [13556.306495] lookup_open+0x356/0xb40 [13556.308946] do_last+0x313/0xfc0 [13556.310790] ? nd_jump_root+0xe5/0x160 [13556.313062] ? path_init+0x437/0x520 [13556.315360] path_openat+0xf7/0x500 [13556.319532] ? do_raw_spin_unlock+0x75/0x190 [13556.322818] do_filp_open+0x99/0x140 [13556.325661] ? getname_flags+0x6e/0x330 [13556.328059] ? __check_object_size+0xff/0x256 [13556.332953] ? do_raw_spin_unlock+0x75/0x190 [13556.335878] ? _raw_spin_unlock+0x12/0x30 [13556.337994] do_sys_openat2+0x2b4/0x410 [13556.347556] do_sys_open+0x73/0xa0 [13556.349557] __x64_sys_openat+0x24/0x30 [13556.354010] do_syscall_64+0xc1/0x440 [13556.355912] entry_SYSCALL_64_after_hwframe+0x49/0xae [13556.358705] RIP: 0033:0x7fa749552332 [13556.365717] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13556.378010] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13556.381656] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13556.387829] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13556.391500] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13556.395076] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13556.403389] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13559.782400] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11642s! [13559.796830] Showing busy workqueues and worker pools: [13559.801773] workqueue events: flags=0x0 [13559.804330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13559.808909] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13559.817977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13559.824363] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13559.830973] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.834697] pending: kfree_rcu_monitor [13559.837317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13559.845905] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13559.857925] workqueue events_long: flags=0x0 [13559.861674] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.870658] pending: lu_objects_flush [obdclass] [13559.873855] workqueue events_unbound: flags=0x2 [13559.877806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13559.886662] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13559.892505] workqueue events_power_efficient: flags=0x80 [13559.895800] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.899435] pending: check_lifetime [13559.900815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.909508] pending: neigh_periodic_work [13559.911918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.920128] pending: neigh_periodic_work [13559.922973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.929816] pending: do_cache_clean [13559.937749] workqueue mm_percpu_wq: flags=0x8 [13559.941794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.945285] pending: vmstat_update [13559.946557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.958287] pending: vmstat_update [13559.963332] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13559.967314] pending: vmstat_update [13559.969909] workqueue writeback: flags=0x4a [13559.973046] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13559.977695] pending: wb_workfn, wb_workfn [13559.980014] workqueue xprtiod: flags=0xa [13559.984873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13559.996588] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13560.005565] workqueue ipv6_addrconf: flags=0x40008 [13560.008832] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13560.012797] pending: addrconf_verify_work [13560.015020] workqueue knbd0-recv: flags=0x1a [13560.018102] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13560.025096] in-flight: 651:recv_work [13560.027199] workqueue ptlrpc_pinger: flags=0x2 [13560.029910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13560.033045] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13560.040684] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11642s workers=4 idle: 37 31846 28897 [13560.046992] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13549s workers=2 idle: 653 [13568.065315] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13568.074742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13568.145854] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13568.158366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13568.179352] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13568.181605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13568.198393] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13568.219204] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13568.232048] RAX: 0000000000005df3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13568.241635] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13568.250503] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13568.265473] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13568.267918] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13568.278758] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13568.287469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13568.290400] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13568.294189] Call Trace: [13568.295298] [13568.296333] ? show_regs.cold.9+0x22/0x2f [13568.298123] ? watchdog_timer_fn.cold.6+0x62/0x10a [13568.300565] ? __hrtimer_run_queues+0x1c2/0x4d0 [13568.311466] ? watchdog+0x40/0x40 [13568.319201] ? hrtimer_interrupt+0x128/0x2c0 [13568.330451] ? smp_apic_timer_interrupt+0xa6/0x270 [13568.339414] ? apic_timer_interrupt+0xf/0x20 [13568.345505] [13568.356667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13568.368245] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13568.374407] do_raw_spin_lock+0x177/0x190 [13568.381523] _raw_spin_lock+0x21/0x30 [13568.391852] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13568.403340] process_one_work+0x2c8/0x700 [13568.412994] worker_thread+0x296/0x6e0 [13568.417297] ? rescuer_thread+0x570/0x570 [13568.425416] kthread+0x1d1/0x200 [13568.427488] ? set_kthread_struct+0x70/0x70 [13568.439185] ret_from_fork+0x1f/0x30 [13576.058311] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13576.062311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13576.126420] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13576.138712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13576.148093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13576.154347] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13576.184630] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13576.197227] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13576.203261] RDX: 0000000000005836 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13576.210364] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13576.220115] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13576.224405] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13576.227639] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13576.238636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13576.246550] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13576.259457] Call Trace: [13576.261028] [13576.262439] ? show_regs.cold.9+0x22/0x2f [13576.269330] ? watchdog_timer_fn.cold.6+0x62/0x10a [13576.279850] ? __hrtimer_run_queues+0x1c2/0x4d0 [13576.286012] ? watchdog+0x40/0x40 [13576.291806] ? hrtimer_interrupt+0x128/0x2c0 [13576.295179] ? smp_apic_timer_interrupt+0xa6/0x270 [13576.303054] ? apic_timer_interrupt+0xf/0x20 [13576.309347] [13576.313442] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13576.321421] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13576.328667] do_raw_spin_lock+0x177/0x190 [13576.336585] _raw_spin_lock+0x21/0x30 [13576.338473] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13576.350336] obd_mod_free+0x46/0x140 [ptlrpc] [13576.359637] mdc_commit_open+0x91/0xb0 [mdc] [13576.366520] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13576.373517] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13576.385397] after_reply+0xe2c/0x1780 [ptlrpc] [13576.390715] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13576.395642] ? __next_timer_interrupt+0x160/0x160 [13576.402160] ptlrpcd+0xa82/0xe40 [ptlrpc] [13576.405401] ? do_wait_intr+0xf0/0xf0 [13576.410663] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13576.412668] kthread+0x1d1/0x200 [13576.422608] ? set_kthread_struct+0x70/0x70 [13576.424516] ret_from_fork+0x1f/0x30 [13580.066719] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13580.074336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13580.130312] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13580.147342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13580.161147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13580.167337] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13580.197388] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13580.209043] RAX: 000000000000330d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13580.236394] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13580.248993] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13580.265439] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13580.283670] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13580.298329] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13580.307489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13580.318506] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13580.335768] Call Trace: [13580.342855] [13580.343755] ? show_regs.cold.9+0x22/0x2f [13580.358559] ? watchdog_timer_fn.cold.6+0x62/0x10a [13580.370532] ? __hrtimer_run_queues+0x1c2/0x4d0 [13580.374546] ? watchdog+0x40/0x40 [13580.381909] ? hrtimer_interrupt+0x128/0x2c0 [13580.390295] ? smp_apic_timer_interrupt+0xa6/0x270 [13580.395363] ? apic_timer_interrupt+0xf/0x20 [13580.399584] [13580.402208] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13580.404732] do_raw_spin_lock+0x177/0x190 [13580.408642] _raw_spin_lock+0x21/0x30 [13580.413326] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13580.420234] ? do_raw_spin_unlock+0x75/0x190 [13580.422699] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13580.426296] ? do_wait_intr+0xf0/0xf0 [13580.428366] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13580.432377] mdc_close+0x53b/0xd70 [mdc] [13580.434541] lmv_close+0x18d/0x490 [lmv] [13580.438052] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13580.441406] ll_md_real_close+0xa5/0x2a0 [lustre] [13580.442904] ll_md_close+0x274/0x800 [lustre] [13580.448357] ll_file_release+0x128/0xc50 [lustre] [13580.453596] __fput+0x10d/0x440 [13580.454820] ____fput+0x12/0x20 [13580.458752] task_work_run+0xc6/0x110 [13580.460506] exit_to_usermode_loop+0x1dd/0x1f0 [13580.465831] do_syscall_64+0x3d6/0x440 [13580.467608] entry_SYSCALL_64_after_hwframe+0x49/0xae [13580.473763] RIP: 0033:0x7fa749552cd8 [13580.480469] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13580.502462] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13580.506577] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13580.512392] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13580.521282] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13580.523236] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13580.540504] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13584.054055] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13584.064526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13584.093097] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13584.098821] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13584.101870] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13584.105264] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13584.112957] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13584.116136] RAX: 000000000000380b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13584.119048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13584.122428] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13584.125494] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13584.128683] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13584.131953] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13584.135655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13584.138455] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13584.141478] Call Trace: [13584.142682] [13584.143307] ? show_regs.cold.9+0x22/0x2f [13584.144743] ? watchdog_timer_fn.cold.6+0x62/0x10a [13584.146979] ? __hrtimer_run_queues+0x1c2/0x4d0 [13584.149153] ? watchdog+0x40/0x40 [13584.150620] ? hrtimer_interrupt+0x128/0x2c0 [13584.152502] ? smp_apic_timer_interrupt+0xa6/0x270 [13584.154649] ? apic_timer_interrupt+0xf/0x20 [13584.156919] [13584.158416] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13584.161581] do_raw_spin_lock+0x177/0x190 [13584.164055] _raw_spin_lock+0x21/0x30 [13584.166322] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13584.169147] ? do_raw_spin_unlock+0x75/0x190 [13584.170722] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13584.173988] ? do_wait_intr+0xf0/0xf0 [13584.176369] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13584.179368] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13584.183089] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13584.186455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13584.188401] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13584.191696] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13584.194373] mdc_intent_lock+0x353/0x7e0 [mdc] [13584.196059] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13584.198423] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13584.201172] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13584.203228] lmv_intent_open+0x338/0x1150 [lmv] [13584.205124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13584.207425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13584.209602] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13584.211560] ll_intent_lock+0xc5/0xaa0 [lustre] [13584.213400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13584.215413] ll_lookup_it+0xe0e/0x2270 [lustre] [13584.217128] ? do_raw_spin_unlock+0x75/0x190 [13584.219194] ? _raw_spin_unlock+0x12/0x30 [13584.220685] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13584.222643] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13584.224172] ll_atomic_open+0x36a/0xd70 [lustre] [13584.226356] lookup_open+0x356/0xb40 [13584.227697] do_last+0x313/0xfc0 [13584.228889] ? nd_jump_root+0xe5/0x160 [13584.230664] ? path_init+0x437/0x520 [13584.231938] path_openat+0xf7/0x500 [13584.233542] ? do_raw_spin_unlock+0x75/0x190 [13584.235103] do_filp_open+0x99/0x140 [13584.236744] ? getname_flags+0x6e/0x330 [13584.238129] ? __check_object_size+0xff/0x256 [13584.239680] ? do_raw_spin_unlock+0x75/0x190 [13584.241156] ? _raw_spin_unlock+0x12/0x30 [13584.242470] do_sys_openat2+0x2b4/0x410 [13584.244076] do_sys_open+0x73/0xa0 [13584.245182] __x64_sys_openat+0x24/0x30 [13584.246911] do_syscall_64+0xc1/0x440 [13584.248210] entry_SYSCALL_64_after_hwframe+0x49/0xae [13584.249874] RIP: 0033:0x7fa749552332 [13584.251700] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13584.259271] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13584.262492] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13584.265943] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13584.269390] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13584.272510] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13584.275054] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13590.499898] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11673s! [13590.506950] Showing busy workqueues and worker pools: [13590.511592] workqueue events: flags=0x0 [13590.513055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13590.520082] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13590.527995] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13590.533958] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13590.541275] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.548777] pending: kfree_rcu_monitor [13590.552815] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13590.559212] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13590.566205] workqueue events_long: flags=0x0 [13590.568114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.574281] pending: lu_objects_flush [obdclass] [13590.580311] workqueue events_unbound: flags=0x2 [13590.582141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13590.587759] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13590.595131] workqueue events_power_efficient: flags=0x80 [13590.601855] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.606547] pending: check_lifetime [13590.611366] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.615161] pending: neigh_periodic_work [13590.619567] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.625279] pending: neigh_periodic_work [13590.628151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.636120] pending: do_cache_clean [13590.637948] workqueue mm_percpu_wq: flags=0x8 [13590.642204] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.649423] pending: vmstat_update [13590.650939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.657043] pending: vmstat_update [13590.661827] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13590.667420] pending: vmstat_update [13590.668524] workqueue writeback: flags=0x4a [13590.669704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13590.680423] pending: wb_workfn, wb_workfn [13590.685330] workqueue xprtiod: flags=0xa [13590.688898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13590.697739] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13590.707375] workqueue ipv6_addrconf: flags=0x40008 [13590.712039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13590.718457] pending: addrconf_verify_work [13590.724009] workqueue knbd0-recv: flags=0x1a [13590.728373] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13590.735422] in-flight: 651:recv_work [13590.742637] workqueue ptlrpc_pinger: flags=0x2 [13590.745507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13590.751812] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13590.757487] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11673s workers=4 idle: 37 31846 28897 [13590.765941] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13580s workers=2 idle: 653 [13596.057049] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13596.059676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13596.082924] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13596.089021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13596.092914] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13596.095600] RIP: 0010:native_safe_halt+0x2e/0x50 [13596.098239] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13596.108253] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13596.111914] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13596.116064] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13596.119953] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13596.123729] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13596.127449] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13596.131464] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13596.135871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13596.139046] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13596.142712] Call Trace: [13596.144143] [13596.145150] ? show_regs.cold.9+0x22/0x2f [13596.147753] ? watchdog_timer_fn.cold.6+0x62/0x10a [13596.150188] ? __hrtimer_run_queues+0x1c2/0x4d0 [13596.152886] ? watchdog+0x40/0x40 [13596.154832] ? hrtimer_interrupt+0x128/0x2c0 [13596.156924] ? smp_apic_timer_interrupt+0xa6/0x270 [13596.159613] ? apic_timer_interrupt+0xf/0x20 [13596.161708] [13596.162981] ? native_safe_halt+0x2e/0x50 [13596.165155] kvm_wait+0xcd/0xe0 [13596.167192] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13596.169901] do_raw_spin_lock+0x177/0x190 [13596.172446] _raw_spin_lock+0x21/0x30 [13596.174395] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13596.177711] process_one_work+0x2c8/0x700 [13596.179890] worker_thread+0x296/0x6e0 [13596.181918] ? rescuer_thread+0x570/0x570 [13596.184142] kthread+0x1d1/0x200 [13596.185918] ? set_kthread_struct+0x70/0x70 [13596.188125] ret_from_fork+0x1f/0x30 [13604.066614] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13604.073808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13604.099832] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13604.104735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13604.109531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13604.113085] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13604.120818] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13604.124104] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13604.127082] RDX: 0000000000000454 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13604.130816] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13604.134348] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13604.140048] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13604.142910] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13604.146687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13604.149194] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13604.152483] Call Trace: [13604.153547] [13604.154312] ? show_regs.cold.9+0x22/0x2f [13604.156660] ? watchdog_timer_fn.cold.6+0x62/0x10a [13604.158849] ? __hrtimer_run_queues+0x1c2/0x4d0 [13604.162787] ? watchdog+0x40/0x40 [13604.164525] ? hrtimer_interrupt+0x128/0x2c0 [13604.166441] ? smp_apic_timer_interrupt+0xa6/0x270 [13604.167793] ? apic_timer_interrupt+0xf/0x20 [13604.169211] [13604.170796] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13604.175340] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13604.178148] do_raw_spin_lock+0x177/0x190 [13604.179905] _raw_spin_lock+0x21/0x30 [13604.181876] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13604.185683] obd_mod_free+0x46/0x140 [ptlrpc] [13604.190791] mdc_commit_open+0x91/0xb0 [mdc] [13604.192958] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13604.196114] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13604.203584] after_reply+0xe2c/0x1780 [ptlrpc] [13604.207275] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13604.209976] ? __next_timer_interrupt+0x160/0x160 [13604.212492] ptlrpcd+0xa82/0xe40 [ptlrpc] [13604.216463] ? do_wait_intr+0xf0/0xf0 [13604.218695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13604.229347] kthread+0x1d1/0x200 [13604.230694] ? set_kthread_struct+0x70/0x70 [13604.233856] ret_from_fork+0x1f/0x30 [13608.060879] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13608.067900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13608.133929] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13608.151344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13608.163776] RIP: 0010:native_safe_halt+0x2e/0x50 [13608.166469] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13608.178807] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13608.185304] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13608.193727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13608.205572] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13608.213610] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13608.222222] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13608.235946] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13608.250049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13608.264715] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13608.271131] Call Trace: [13608.275838] [13608.280707] ? show_regs.cold.9+0x22/0x2f [13608.287031] ? watchdog_timer_fn.cold.6+0x62/0x10a [13608.289047] ? __hrtimer_run_queues+0x1c2/0x4d0 [13608.299485] ? watchdog+0x40/0x40 [13608.300640] ? hrtimer_interrupt+0x128/0x2c0 [13608.301944] ? smp_apic_timer_interrupt+0xa6/0x270 [13608.312629] ? apic_timer_interrupt+0xf/0x20 [13608.317049] [13608.320211] ? native_safe_halt+0x2e/0x50 [13608.326136] kvm_wait+0xcd/0xe0 [13608.328804] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13608.335951] do_raw_spin_lock+0x177/0x190 [13608.341046] _raw_spin_lock+0x21/0x30 [13608.346402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13608.351281] ? do_raw_spin_unlock+0x75/0x190 [13608.353086] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13608.355690] ? do_wait_intr+0xf0/0xf0 [13608.357513] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13608.360028] mdc_close+0x53b/0xd70 [mdc] [13608.362213] lmv_close+0x18d/0x490 [lmv] [13608.364201] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13608.366865] ll_md_real_close+0xa5/0x2a0 [lustre] [13608.369165] ll_md_close+0x274/0x800 [lustre] [13608.371221] ll_file_release+0x128/0xc50 [lustre] [13608.373391] __fput+0x10d/0x440 [13608.375159] ____fput+0x12/0x20 [13608.376631] task_work_run+0xc6/0x110 [13608.378217] exit_to_usermode_loop+0x1dd/0x1f0 [13608.384384] do_syscall_64+0x3d6/0x440 [13608.389024] entry_SYSCALL_64_after_hwframe+0x49/0xae [13608.395726] RIP: 0033:0x7fa749552cd8 [13608.398181] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13608.419643] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13608.427911] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13608.435902] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13608.448004] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13608.458417] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13608.460927] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13616.057937] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13616.067916] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13616.108820] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13616.120106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13616.132025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13616.137248] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13616.164582] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13616.171583] RAX: 0000000000006483 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13616.176487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13616.185625] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13616.195957] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13616.205517] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13616.212479] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13616.217295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13616.222847] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13616.226923] Call Trace: [13616.227790] [13616.230397] ? show_regs.cold.9+0x22/0x2f [13616.232593] ? watchdog_timer_fn.cold.6+0x62/0x10a [13616.240944] ? __hrtimer_run_queues+0x1c2/0x4d0 [13616.245636] ? watchdog+0x40/0x40 [13616.247316] ? hrtimer_interrupt+0x128/0x2c0 [13616.249670] ? smp_apic_timer_interrupt+0xa6/0x270 [13616.253070] ? apic_timer_interrupt+0xf/0x20 [13616.256817] [13616.258455] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13616.272407] do_raw_spin_lock+0x177/0x190 [13616.275630] _raw_spin_lock+0x21/0x30 [13616.279265] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13616.291361] ? do_raw_spin_unlock+0x75/0x190 [13616.293973] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13616.301452] ? do_wait_intr+0xf0/0xf0 [13616.307417] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13616.311441] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13616.320061] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13616.327963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13616.333639] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13616.339008] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13616.344528] mdc_intent_lock+0x353/0x7e0 [mdc] [13616.346363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13616.348602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13616.351237] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13616.353672] lmv_intent_open+0x338/0x1150 [lmv] [13616.357625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13616.360642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13616.363521] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13616.366696] ll_intent_lock+0xc5/0xaa0 [lustre] [13616.369535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13616.374346] ll_lookup_it+0xe0e/0x2270 [lustre] [13616.376627] ? do_raw_spin_unlock+0x75/0x190 [13616.378665] ? _raw_spin_unlock+0x12/0x30 [13616.380447] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13616.386295] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13616.391067] ll_atomic_open+0x36a/0xd70 [lustre] [13616.394401] lookup_open+0x356/0xb40 [13616.396118] do_last+0x313/0xfc0 [13616.397600] ? nd_jump_root+0xe5/0x160 [13616.399565] ? path_init+0x437/0x520 [13616.400814] path_openat+0xf7/0x500 [13616.403456] ? do_raw_spin_unlock+0x75/0x190 [13616.405624] do_filp_open+0x99/0x140 [13616.408854] ? getname_flags+0x6e/0x330 [13616.411130] ? __check_object_size+0xff/0x256 [13616.413336] ? do_raw_spin_unlock+0x75/0x190 [13616.416103] ? _raw_spin_unlock+0x12/0x30 [13616.418333] do_sys_openat2+0x2b4/0x410 [13616.420981] do_sys_open+0x73/0xa0 [13616.422471] __x64_sys_openat+0x24/0x30 [13616.424090] do_syscall_64+0xc1/0x440 [13616.425985] entry_SYSCALL_64_after_hwframe+0x49/0xae [13616.428769] RIP: 0033:0x7fa749552332 [13616.430936] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13616.440685] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13616.444951] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13616.448976] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13616.452964] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13616.456720] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13616.460356] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13621.221272] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11704s! [13621.225282] Showing busy workqueues and worker pools: [13621.232720] workqueue events: flags=0x0 [13621.238245] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13621.241741] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13621.247054] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13621.250769] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13621.258442] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.266574] pending: kfree_rcu_monitor [13621.279239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13621.285957] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13621.292172] workqueue events_long: flags=0x0 [13621.293829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.298537] pending: lu_objects_flush [obdclass] [13621.304828] workqueue events_unbound: flags=0x2 [13621.308419] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13621.313098] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13621.318560] workqueue events_power_efficient: flags=0x80 [13621.331738] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.334142] pending: check_lifetime [13621.340213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.346009] pending: neigh_periodic_work [13621.350806] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.354448] pending: neigh_periodic_work [13621.355983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.368488] pending: do_cache_clean [13621.371030] workqueue mm_percpu_wq: flags=0x8 [13621.372964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.376121] pending: vmstat_update [13621.377730] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.381956] pending: vmstat_update [13621.384479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13621.393355] pending: vmstat_update [13621.394964] workqueue writeback: flags=0x4a [13621.397012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13621.400764] pending: wb_workfn, wb_workfn [13621.402299] workqueue xprtiod: flags=0xa [13621.406393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13621.408811] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13621.413047] workqueue ipv6_addrconf: flags=0x40008 [13621.417557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13621.422932] pending: addrconf_verify_work [13621.427151] workqueue knbd0-recv: flags=0x1a [13621.429140] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13621.431596] in-flight: 651:recv_work [13621.433316] workqueue ptlrpc_pinger: flags=0x2 [13621.435329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13621.437740] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13621.440738] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11704s workers=4 idle: 37 31846 28897 [13621.448386] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13611s workers=2 idle: 653 [13624.057386] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13624.063993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13624.100944] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13624.115805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13624.123914] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13624.141281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13624.142850] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13624.171586] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13624.191596] RAX: 0000000000007f85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13624.201561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13624.220069] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13624.227754] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13624.238791] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13624.259422] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13624.271937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13624.280916] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13624.293921] Call Trace: [13624.297424] [13624.302438] ? show_regs.cold.9+0x22/0x2f [13624.305874] ? watchdog_timer_fn.cold.6+0x62/0x10a [13624.318708] ? __hrtimer_run_queues+0x1c2/0x4d0 [13624.326611] ? watchdog+0x40/0x40 [13624.333971] ? hrtimer_interrupt+0x128/0x2c0 [13624.345690] ? smp_apic_timer_interrupt+0xa6/0x270 [13624.353947] ? apic_timer_interrupt+0xf/0x20 [13624.358786] [13624.365604] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13624.377550] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13624.382886] do_raw_spin_lock+0x177/0x190 [13624.389986] _raw_spin_lock+0x21/0x30 [13624.393335] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13624.400796] process_one_work+0x2c8/0x700 [13624.406534] worker_thread+0x296/0x6e0 [13624.412304] ? rescuer_thread+0x570/0x570 [13624.417106] kthread+0x1d1/0x200 [13624.424512] ? set_kthread_struct+0x70/0x70 [13624.428946] ret_from_fork+0x1f/0x30 [13632.059479] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13632.069408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13632.112738] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13632.122189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13632.130397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13632.136022] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13632.153759] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13632.157321] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13632.165563] RDX: 0000000000002429 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13632.170072] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13632.174853] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13632.180305] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13632.186672] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13632.195825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13632.201785] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13632.203812] Call Trace: [13632.204532] [13632.205096] ? show_regs.cold.9+0x22/0x2f [13632.209886] ? watchdog_timer_fn.cold.6+0x62/0x10a [13632.213606] ? __hrtimer_run_queues+0x1c2/0x4d0 [13632.217744] ? watchdog+0x40/0x40 [13632.222025] ? hrtimer_interrupt+0x128/0x2c0 [13632.224776] ? smp_apic_timer_interrupt+0xa6/0x270 [13632.229249] ? apic_timer_interrupt+0xf/0x20 [13632.230723] [13632.232339] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13632.236627] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13632.241853] do_raw_spin_lock+0x177/0x190 [13632.244278] _raw_spin_lock+0x21/0x30 [13632.246366] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13632.250678] obd_mod_free+0x46/0x140 [ptlrpc] [13632.257267] mdc_commit_open+0x91/0xb0 [mdc] [13632.260962] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13632.266610] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13632.271367] after_reply+0xe2c/0x1780 [ptlrpc] [13632.277078] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13632.283147] ? __next_timer_interrupt+0x160/0x160 [13632.289366] ptlrpcd+0xa82/0xe40 [ptlrpc] [13632.293375] ? do_wait_intr+0xf0/0xf0 [13632.295918] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13632.299359] kthread+0x1d1/0x200 [13632.302275] ? set_kthread_struct+0x70/0x70 [13632.306382] ret_from_fork+0x1f/0x30 [13636.060318] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13636.068381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13636.124712] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13636.141614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13636.153247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13636.159155] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13636.189302] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13636.198656] RAX: 0000000000007c85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13636.207514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13636.215408] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13636.221916] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13636.228971] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13636.239800] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13636.248548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13636.257420] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13636.267184] Call Trace: [13636.271022] [13636.271777] ? show_regs.cold.9+0x22/0x2f [13636.282524] ? watchdog_timer_fn.cold.6+0x62/0x10a [13636.288912] ? __hrtimer_run_queues+0x1c2/0x4d0 [13636.295590] ? watchdog+0x40/0x40 [13636.301309] ? hrtimer_interrupt+0x128/0x2c0 [13636.305951] ? smp_apic_timer_interrupt+0xa6/0x270 [13636.315772] ? apic_timer_interrupt+0xf/0x20 [13636.325600] [13636.326418] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13636.340620] do_raw_spin_lock+0x177/0x190 [13636.344962] _raw_spin_lock+0x21/0x30 [13636.348610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13636.356630] ? do_raw_spin_unlock+0x75/0x190 [13636.364282] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13636.370358] ? do_wait_intr+0xf0/0xf0 [13636.371604] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13636.384726] mdc_close+0x53b/0xd70 [mdc] [13636.390098] lmv_close+0x18d/0x490 [lmv] [13636.395554] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13636.404668] ll_md_real_close+0xa5/0x2a0 [lustre] [13636.412096] ll_md_close+0x274/0x800 [lustre] [13636.419634] ll_file_release+0x128/0xc50 [lustre] [13636.421464] __fput+0x10d/0x440 [13636.426191] ____fput+0x12/0x20 [13636.427557] task_work_run+0xc6/0x110 [13636.431273] exit_to_usermode_loop+0x1dd/0x1f0 [13636.432756] do_syscall_64+0x3d6/0x440 [13636.438960] entry_SYSCALL_64_after_hwframe+0x49/0xae [13636.443012] RIP: 0033:0x7fa749552cd8 [13636.444484] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13636.459718] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13636.463366] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13636.466324] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13636.469284] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13636.471994] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13636.474695] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13648.054243] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13648.060070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13648.109764] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13648.125401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13648.147754] RIP: 0010:native_safe_halt+0x2e/0x50 [13648.155863] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13648.171145] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13648.178159] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13648.181531] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [13648.197384] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13648.203199] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13648.210807] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13648.212868] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13648.222646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13648.232223] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13648.242382] Call Trace: [13648.244015] [13648.245027] ? show_regs.cold.9+0x22/0x2f [13648.246941] ? watchdog_timer_fn.cold.6+0x62/0x10a [13648.251741] ? __hrtimer_run_queues+0x1c2/0x4d0 [13648.258335] ? watchdog+0x40/0x40 [13648.268938] ? hrtimer_interrupt+0x128/0x2c0 [13648.273718] ? smp_apic_timer_interrupt+0xa6/0x270 [13648.282216] ? apic_timer_interrupt+0xf/0x20 [13648.289034] [13648.290328] ? native_safe_halt+0x2e/0x50 [13648.297351] kvm_wait+0xcd/0xe0 [13648.298533] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13648.304592] do_raw_spin_lock+0x177/0x190 [13648.306993] _raw_spin_lock+0x21/0x30 [13648.310723] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13648.318772] ? do_raw_spin_unlock+0x75/0x190 [13648.320668] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13648.322599] ? do_wait_intr+0xf0/0xf0 [13648.327626] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13648.333244] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13648.344523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13648.351403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.362505] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13648.370475] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13648.376314] mdc_intent_lock+0x353/0x7e0 [mdc] [13648.380743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.389631] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13648.394399] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13648.407316] lmv_intent_open+0x338/0x1150 [lmv] [13648.415192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.419623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.430244] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13648.440331] ll_intent_lock+0xc5/0xaa0 [lustre] [13648.445523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.453355] ll_lookup_it+0xe0e/0x2270 [lustre] [13648.463192] ? do_raw_spin_unlock+0x75/0x190 [13648.469251] ? _raw_spin_unlock+0x12/0x30 [13648.472294] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13648.474126] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13648.479389] ll_atomic_open+0x36a/0xd70 [lustre] [13648.482051] lookup_open+0x356/0xb40 [13648.485358] do_last+0x313/0xfc0 [13648.492519] ? nd_jump_root+0xe5/0x160 [13648.496272] ? path_init+0x437/0x520 [13648.497826] path_openat+0xf7/0x500 [13648.499408] ? do_raw_spin_unlock+0x75/0x190 [13648.502347] do_filp_open+0x99/0x140 [13648.504017] ? getname_flags+0x6e/0x330 [13648.506094] ? __check_object_size+0xff/0x256 [13648.510266] ? do_raw_spin_unlock+0x75/0x190 [13648.512463] ? _raw_spin_unlock+0x12/0x30 [13648.514809] do_sys_openat2+0x2b4/0x410 [13648.516530] do_sys_open+0x73/0xa0 [13648.518407] __x64_sys_openat+0x24/0x30 [13648.523517] do_syscall_64+0xc1/0x440 [13648.532310] entry_SYSCALL_64_after_hwframe+0x49/0xae [13648.540253] RIP: 0033:0x7fa749552332 [13648.545278] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13648.572420] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13648.581570] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13648.590727] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13648.595286] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13648.609183] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13648.624344] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13651.937013] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11734s! [13651.941714] Showing busy workqueues and worker pools: [13651.949196] workqueue events: flags=0x0 [13651.950243] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13651.961481] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13651.972722] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13651.977148] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13651.986145] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13651.989487] pending: kfree_rcu_monitor [13651.993412] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13651.999155] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13652.002773] workqueue events_long: flags=0x0 [13652.004306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13652.010036] pending: lu_objects_flush [obdclass] [13652.015043] workqueue events_unbound: flags=0x2 [13652.016634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13652.022116] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13652.034227] workqueue events_power_efficient: flags=0x80 [13652.045299] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13652.052065] pending: check_lifetime [13652.057067] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13652.058222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13652.058240] pending: neigh_periodic_work [13652.062136] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13652.062214] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13652.062231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13652.062242] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13652.062778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13652.062802] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13652.062815] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13652.062823] RAX: 0000000000007487 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13652.062825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13652.062830] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13652.062832] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13652.062837] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13652.063172] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13652.063175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13652.063178] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13652.063185] Call Trace: [13652.063196] [13652.063201] ? show_regs.cold.9+0x22/0x2f [13652.063211] ? watchdog_timer_fn.cold.6+0x62/0x10a [13652.063223] ? __hrtimer_run_queues+0x1c2/0x4d0 [13652.063229] ? watchdog+0x40/0x40 [13652.063234] ? hrtimer_interrupt+0x128/0x2c0 [13652.063237] ? smp_apic_timer_interrupt+0xa6/0x270 [13652.063247] ? apic_timer_interrupt+0xf/0x20 [13652.063257] [13652.063260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13652.063265] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13652.063271] do_raw_spin_lock+0x177/0x190 [13652.063277] _raw_spin_lock+0x21/0x30 [13652.063281] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13652.063686] process_one_work+0x2c8/0x700 [13652.063695] worker_thread+0x296/0x6e0 [13652.063698] ? rescuer_thread+0x570/0x570 [13652.063702] kthread+0x1d1/0x200 [13652.063706] ? set_kthread_struct+0x70/0x70 [13652.063709] ret_from_fork+0x1f/0x30 [13652.318277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13652.327943] pending: neigh_periodic_work [13652.329289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13652.343777] pending: do_cache_clean [13652.346473] workqueue mm_percpu_wq: flags=0x8 [13652.349014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13652.352442] pending: vmstat_update [13652.359651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13652.361784] pending: vmstat_update [13652.367201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13652.374381] pending: vmstat_update [13652.380643] workqueue writeback: flags=0x4a [13652.384158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13652.391914] pending: wb_workfn, wb_workfn [13652.394287] workqueue xprtiod: flags=0xa [13652.396959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13652.410867] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13652.417375] workqueue ipv6_addrconf: flags=0x40008 [13652.421489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13652.426845] pending: addrconf_verify_work [13652.430934] workqueue knbd0-recv: flags=0x1a [13652.435968] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13652.442018] in-flight: 651:recv_work [13652.455911] workqueue ptlrpc_pinger: flags=0x2 [13652.467152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13652.475311] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13652.478916] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11735s workers=4 idle: 37 31846 28897 [13652.485021] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13642s workers=2 idle: 653 [13660.058688] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13660.069742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13660.141188] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13660.153904] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13660.162644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13660.169953] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13660.194599] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13660.203719] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13660.215167] RDX: 0000000000003d36 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13660.222719] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13660.231007] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13660.240185] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13660.246301] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13660.251254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13660.258163] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13660.263222] Call Trace: [13660.264103] [13660.265604] ? show_regs.cold.9+0x22/0x2f [13660.271046] ? watchdog_timer_fn.cold.6+0x62/0x10a [13660.281901] ? __hrtimer_run_queues+0x1c2/0x4d0 [13660.291638] ? watchdog+0x40/0x40 [13660.295126] ? hrtimer_interrupt+0x128/0x2c0 [13660.300018] ? smp_apic_timer_interrupt+0xa6/0x270 [13660.315764] ? apic_timer_interrupt+0xf/0x20 [13660.322943] [13660.323748] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13660.333787] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13660.352096] do_raw_spin_lock+0x177/0x190 [13660.353484] _raw_spin_lock+0x21/0x30 [13660.354766] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13660.368961] obd_mod_free+0x46/0x140 [ptlrpc] [13660.373456] mdc_commit_open+0x91/0xb0 [mdc] [13660.379704] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13660.388473] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13660.390422] after_reply+0xe2c/0x1780 [ptlrpc] [13660.404055] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13660.410268] ? __next_timer_interrupt+0x160/0x160 [13660.420796] ptlrpcd+0xa82/0xe40 [ptlrpc] [13660.425982] ? do_wait_intr+0xf0/0xf0 [13660.434596] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13660.436555] kthread+0x1d1/0x200 [13660.437623] ? set_kthread_struct+0x70/0x70 [13660.456951] ret_from_fork+0x1f/0x30 [13664.062777] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13664.077396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13664.153782] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13664.156482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13664.176889] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13664.179024] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13664.221995] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13664.234583] RAX: 0000000000007e16 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13664.239526] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13664.249415] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13664.258358] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13664.266863] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13664.272508] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13664.287881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13664.294810] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13664.309802] Call Trace: [13664.313865] [13664.321938] ? show_regs.cold.9+0x22/0x2f [13664.324094] ? watchdog_timer_fn.cold.6+0x62/0x10a [13664.329821] ? __hrtimer_run_queues+0x1c2/0x4d0 [13664.334955] ? watchdog+0x40/0x40 [13664.341482] ? hrtimer_interrupt+0x128/0x2c0 [13664.352916] ? smp_apic_timer_interrupt+0xa6/0x270 [13664.354562] ? apic_timer_interrupt+0xf/0x20 [13664.368868] [13664.369576] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13664.381231] do_raw_spin_lock+0x177/0x190 [13664.389152] _raw_spin_lock+0x21/0x30 [13664.391483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13664.402377] ? do_raw_spin_unlock+0x75/0x190 [13664.408208] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13664.418788] ? do_wait_intr+0xf0/0xf0 [13664.420127] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13664.429849] mdc_close+0x53b/0xd70 [mdc] [13664.438663] lmv_close+0x18d/0x490 [lmv] [13664.447390] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13664.462635] ll_md_real_close+0xa5/0x2a0 [lustre] [13664.469820] ll_md_close+0x274/0x800 [lustre] [13664.481470] ll_file_release+0x128/0xc50 [lustre] [13664.485288] __fput+0x10d/0x440 [13664.490222] ____fput+0x12/0x20 [13664.501387] task_work_run+0xc6/0x110 [13664.506581] exit_to_usermode_loop+0x1dd/0x1f0 [13664.508133] do_syscall_64+0x3d6/0x440 [13664.516089] entry_SYSCALL_64_after_hwframe+0x49/0xae [13664.527476] RIP: 0033:0x7fa749552cd8 [13664.532636] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13664.573140] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13664.589080] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13664.597854] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13664.634935] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13664.654396] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13664.656977] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13673.948876] rcu: INFO: rcu_sched self-detected stall on CPU [13673.953023] rcu: 2-...!: (6693845 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [13673.964554] (t=11760259 jiffies g=258701 q=4791) [13673.967553] rcu: rcu_sched kthread starved for 11757054 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [13673.976350] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13673.990003] rcu: RCU grace-period kthread stack dump: [13673.997544] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13674.003281] Call Trace: [13674.005893] __schedule+0x351/0xcb0 [13674.008721] schedule+0xc0/0x180 [13674.013437] schedule_timeout+0xb4/0x190 [13674.018899] ? do_raw_spin_unlock+0x75/0x190 [13674.021017] ? __next_timer_interrupt+0x160/0x160 [13674.025000] ? prepare_to_swait_event+0xf3/0x140 [13674.029025] rcu_gp_kthread+0x8dd/0x15b0 [13674.031636] ? force_qs_rnp+0x2c0/0x2c0 [13674.034967] kthread+0x1d1/0x200 [13674.040170] ? set_kthread_struct+0x70/0x70 [13674.041967] ret_from_fork+0x1f/0x30 [13674.043900] rcu: Stack dump where RCU GP kthread last ran: [13674.050257] NMI backtrace for cpu 2 [13674.055040] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13674.061974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13674.072344] Call Trace: [13674.073243] [13674.079307] ? dump_stack+0xbb/0x10e [13674.081115] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13674.096245] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13674.102063] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13674.112875] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13674.129266] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13674.131138] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [13674.145310] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [13674.156539] ? update_process_times+0x6d/0xc0 [13674.162977] ? tick_sched_handle.isra.18+0x27/0x90 [13674.169280] ? tick_sched_timer+0x53/0xb0 [13674.172327] ? __hrtimer_run_queues+0x1c2/0x4d0 [13674.175945] ? tick_sched_handle.isra.18+0x90/0x90 [13674.181338] ? hrtimer_interrupt+0x128/0x2c0 [13674.184834] ? smp_apic_timer_interrupt+0xa6/0x270 [13674.191193] ? apic_timer_interrupt+0xf/0x20 [13674.193952] [13674.195339] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13674.198876] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13674.200832] ? do_raw_spin_lock+0x177/0x190 [13674.201920] ? _raw_spin_lock+0x21/0x30 [13674.207072] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13674.215121] ? obd_mod_free+0x46/0x140 [ptlrpc] [13674.219371] ? mdc_commit_open+0x91/0xb0 [mdc] [13674.222068] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13674.227537] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13674.232716] ? after_reply+0xe2c/0x1780 [ptlrpc] [13674.238258] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13674.242006] ? __next_timer_interrupt+0x160/0x160 [13674.246874] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13674.251585] ? do_wait_intr+0xf0/0xf0 [13674.253931] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13674.258320] ? kthread+0x1d1/0x200 [13674.262056] ? set_kthread_struct+0x70/0x70 [13674.264664] ? ret_from_fork+0x1f/0x30 [13674.266318] NMI backtrace for cpu 2 [13674.268463] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13674.275112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13674.284996] Call Trace: [13674.289452] [13674.290167] ? dump_stack+0xbb/0x10e [13674.292919] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13674.299015] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13674.304438] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13674.307004] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13674.309929] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13674.314509] ? rcu_dump_cpu_stacks+0x11b/0x171 [13674.316090] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13674.321557] ? update_process_times+0x6d/0xc0 [13674.326218] ? tick_sched_handle.isra.18+0x27/0x90 [13674.330289] ? tick_sched_timer+0x53/0xb0 [13674.332459] ? __hrtimer_run_queues+0x1c2/0x4d0 [13674.334587] ? tick_sched_handle.isra.18+0x90/0x90 [13674.345546] ? hrtimer_interrupt+0x128/0x2c0 [13674.355872] ? smp_apic_timer_interrupt+0xa6/0x270 [13674.364941] ? apic_timer_interrupt+0xf/0x20 [13674.372189] [13674.372903] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13674.384608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13674.395028] ? do_raw_spin_lock+0x177/0x190 [13674.403451] ? _raw_spin_lock+0x21/0x30 [13674.415017] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13674.432194] ? obd_mod_free+0x46/0x140 [ptlrpc] [13674.434132] ? mdc_commit_open+0x91/0xb0 [mdc] [13674.455294] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13674.457388] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13674.469282] ? after_reply+0xe2c/0x1780 [ptlrpc] [13674.476481] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13674.487009] ? __next_timer_interrupt+0x160/0x160 [13674.492694] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13674.497026] ? do_wait_intr+0xf0/0xf0 [13674.500490] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13674.503912] ? kthread+0x1d1/0x200 [13674.508913] ? set_kthread_struct+0x70/0x70 [13674.511530] ? ret_from_fork+0x1f/0x30 [13680.054202] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13680.057115] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13680.060062] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13680.060144] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13680.060151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13680.060154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13680.060167] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13680.060171] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13680.060176] RAX: 0000000000006d11 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13680.060178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13680.060181] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13680.060183] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13680.060186] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13680.060189] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13680.060192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13680.060194] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13680.060261] Call Trace: [13680.060266] [13680.060271] ? show_regs.cold.9+0x22/0x2f [13680.060280] ? watchdog_timer_fn.cold.6+0x62/0x10a [13680.060286] ? __hrtimer_run_queues+0x1c2/0x4d0 [13680.060290] ? watchdog+0x40/0x40 [13680.060294] ? hrtimer_interrupt+0x128/0x2c0 [13680.060299] ? smp_apic_timer_interrupt+0xa6/0x270 [13680.060309] ? apic_timer_interrupt+0xf/0x20 [13680.060377] [13680.060380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13680.060388] do_raw_spin_lock+0x177/0x190 [13680.060394] _raw_spin_lock+0x21/0x30 [13680.060399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13680.060773] ? do_raw_spin_unlock+0x75/0x190 [13680.060879] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13680.061106] ? do_wait_intr+0xf0/0xf0 [13680.061114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13680.061567] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13680.062100] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13680.062402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13680.062587] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13680.062624] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13680.062661] mdc_intent_lock+0x353/0x7e0 [mdc] [13680.062694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13680.062778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13680.074818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13680.074953] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13680.074961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13680.074967] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13680.075282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13680.075295] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13680.075299] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13680.075303] RAX: 0000000000007384 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13680.075306] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13680.075308] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13680.075309] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13680.075312] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13680.075404] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13680.075408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13680.075411] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13680.075489] Call Trace: [13680.075496] [13680.075501] ? show_regs.cold.9+0x22/0x2f [13680.075510] ? watchdog_timer_fn.cold.6+0x62/0x10a [13680.075518] ? __hrtimer_run_queues+0x1c2/0x4d0 [13680.075521] ? watchdog+0x40/0x40 [13680.075525] ? hrtimer_interrupt+0x128/0x2c0 [13680.075529] ? smp_apic_timer_interrupt+0xa6/0x270 [13680.075538] ? apic_timer_interrupt+0xf/0x20 [13680.075542] [13680.075545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13680.075550] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13680.075556] do_raw_spin_lock+0x177/0x190 [13680.075561] _raw_spin_lock+0x21/0x30 [13680.075565] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13680.075772] process_one_work+0x2c8/0x700 [13680.075779] worker_thread+0x296/0x6e0 [13680.075783] ? rescuer_thread+0x570/0x570 [13680.075787] kthread+0x1d1/0x200 [13680.075792] ? set_kthread_struct+0x70/0x70 [13680.075795] ret_from_fork+0x1f/0x30 [13680.374937] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13680.379764] lmv_intent_open+0x338/0x1150 [lmv] [13680.383281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13680.386916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13680.389895] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13680.391158] ll_intent_lock+0xc5/0xaa0 [lustre] [13680.393898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13680.396651] ll_lookup_it+0xe0e/0x2270 [lustre] [13680.399820] ? do_raw_spin_unlock+0x75/0x190 [13680.404942] ? _raw_spin_unlock+0x12/0x30 [13680.409942] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13680.414606] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13680.418356] ll_atomic_open+0x36a/0xd70 [lustre] [13680.421186] lookup_open+0x356/0xb40 [13680.423759] do_last+0x313/0xfc0 [13680.425075] ? nd_jump_root+0xe5/0x160 [13680.426599] ? path_init+0x437/0x520 [13680.429070] path_openat+0xf7/0x500 [13680.431327] ? do_raw_spin_unlock+0x75/0x190 [13680.433602] do_filp_open+0x99/0x140 [13680.435015] ? getname_flags+0x6e/0x330 [13680.438144] ? __check_object_size+0xff/0x256 [13680.439817] ? do_raw_spin_unlock+0x75/0x190 [13680.441662] ? _raw_spin_unlock+0x12/0x30 [13680.443103] do_sys_openat2+0x2b4/0x410 [13680.445205] do_sys_open+0x73/0xa0 [13680.447151] __x64_sys_openat+0x24/0x30 [13680.448574] do_syscall_64+0xc1/0x440 [13680.449984] entry_SYSCALL_64_after_hwframe+0x49/0xae [13680.453140] RIP: 0033:0x7fa749552332 [13680.455937] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13680.473226] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13680.481739] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13680.490698] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13680.502748] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13680.508647] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13680.514578] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13682.658826] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11765s! [13682.667089] Showing busy workqueues and worker pools: [13682.669683] workqueue events: flags=0x0 [13682.671689] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13682.685472] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13682.695168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13682.700185] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13682.704106] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.708913] pending: kfree_rcu_monitor [13682.711782] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13682.715001] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13682.719104] workqueue events_long: flags=0x0 [13682.720317] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.722351] pending: lu_objects_flush [obdclass] [13682.724051] workqueue events_unbound: flags=0x2 [13682.725423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13682.727312] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13682.729959] workqueue events_power_efficient: flags=0x80 [13682.731812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.734083] pending: check_lifetime [13682.735445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.737977] pending: neigh_periodic_work [13682.739740] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.742196] pending: neigh_periodic_work [13682.743839] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.746605] pending: do_cache_clean [13682.748212] workqueue mm_percpu_wq: flags=0x8 [13682.749915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.752401] pending: vmstat_update [13682.753872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.756429] pending: vmstat_update [13682.757980] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13682.760952] pending: vmstat_update [13682.762464] workqueue writeback: flags=0x4a [13682.764123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13682.766757] pending: wb_workfn, wb_workfn [13682.768515] workqueue xprtiod: flags=0xa [13682.770102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13682.772653] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13682.776491] workqueue ipv6_addrconf: flags=0x40008 [13682.779891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13682.783356] pending: addrconf_verify_work [13682.787171] workqueue knbd0-recv: flags=0x1a [13682.790074] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13682.793369] in-flight: 651:recv_work [13682.794908] workqueue ptlrpc_pinger: flags=0x2 [13682.796407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13682.799167] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13682.802904] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11765s workers=4 idle: 37 31846 28897 [13682.807077] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13672s workers=2 idle: 653 [13692.061454] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13692.068512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13692.111393] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13692.123182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13692.128898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13692.131025] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13692.143027] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13692.146579] RAX: 0000000000007389 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13692.153075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13692.160622] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13692.166756] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13692.172729] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13692.178589] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13692.182437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13692.184841] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13692.187379] Call Trace: [13692.188062] [13692.188552] ? show_regs.cold.9+0x22/0x2f [13692.190895] ? watchdog_timer_fn.cold.6+0x62/0x10a [13692.192209] ? __hrtimer_run_queues+0x1c2/0x4d0 [13692.197639] ? watchdog+0x40/0x40 [13692.198812] ? hrtimer_interrupt+0x128/0x2c0 [13692.200009] ? smp_apic_timer_interrupt+0xa6/0x270 [13692.201610] ? apic_timer_interrupt+0xf/0x20 [13692.203234] [13692.204017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13692.206265] do_raw_spin_lock+0x177/0x190 [13692.207904] _raw_spin_lock+0x21/0x30 [13692.209502] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13692.212099] ? do_raw_spin_unlock+0x75/0x190 [13692.214467] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13692.216733] ? do_wait_intr+0xf0/0xf0 [13692.218399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13692.221450] mdc_close+0x53b/0xd70 [mdc] [13692.224714] lmv_close+0x18d/0x490 [lmv] [13692.226693] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13692.229426] ll_md_real_close+0xa5/0x2a0 [lustre] [13692.232465] ll_md_close+0x274/0x800 [lustre] [13692.235159] ll_file_release+0x128/0xc50 [lustre] [13692.237633] __fput+0x10d/0x440 [13692.239055] ____fput+0x12/0x20 [13692.240587] task_work_run+0xc6/0x110 [13692.242202] exit_to_usermode_loop+0x1dd/0x1f0 [13692.244626] do_syscall_64+0x3d6/0x440 [13692.246488] entry_SYSCALL_64_after_hwframe+0x49/0xae [13692.249371] RIP: 0033:0x7fa749552cd8 [13692.251067] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13692.259610] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13692.263251] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13692.269506] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13692.274163] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13692.278112] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13692.281343] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13700.059348] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13700.068783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13700.119905] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13700.128543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13700.134331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13700.141332] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13700.153054] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13700.159051] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13700.163693] RDX: 000000000000657c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13700.166248] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13700.173312] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13700.177985] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13700.180685] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13700.183984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13700.190759] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13700.192649] Call Trace: [13700.193555] [13700.194225] ? show_regs.cold.9+0x22/0x2f [13700.195760] ? watchdog_timer_fn.cold.6+0x62/0x10a [13700.197455] ? __hrtimer_run_queues+0x1c2/0x4d0 [13700.198748] ? watchdog+0x40/0x40 [13700.199751] ? hrtimer_interrupt+0x128/0x2c0 [13700.201215] ? smp_apic_timer_interrupt+0xa6/0x270 [13700.202906] ? apic_timer_interrupt+0xf/0x20 [13700.204458] [13700.205902] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13700.207832] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13700.209745] do_raw_spin_lock+0x177/0x190 [13700.211147] _raw_spin_lock+0x21/0x30 [13700.212483] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13700.214477] obd_mod_free+0x46/0x140 [ptlrpc] [13700.216363] mdc_commit_open+0x91/0xb0 [mdc] [13700.218416] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13700.220171] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13700.225315] after_reply+0xe2c/0x1780 [ptlrpc] [13700.229667] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13700.233305] ? __next_timer_interrupt+0x160/0x160 [13700.236012] ptlrpcd+0xa82/0xe40 [ptlrpc] [13700.238367] ? do_wait_intr+0xf0/0xf0 [13700.239971] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13700.244939] kthread+0x1d1/0x200 [13700.246165] ? set_kthread_struct+0x70/0x70 [13700.247941] ret_from_fork+0x1f/0x30 [13708.056655] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13708.057940] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13708.074164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13708.076032] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13708.076042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13708.076045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13708.076060] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13708.076064] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13708.076069] RAX: 000000000000248b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13708.076071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13708.076073] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13708.076075] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13708.076078] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13708.076081] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13708.076084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13708.076086] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13708.076096] Call Trace: [13708.076101] [13708.076107] ? show_regs.cold.9+0x22/0x2f [13708.076608] ? watchdog_timer_fn.cold.6+0x62/0x10a [13708.076618] ? __hrtimer_run_queues+0x1c2/0x4d0 [13708.076622] ? watchdog+0x40/0x40 [13708.076626] ? hrtimer_interrupt+0x128/0x2c0 [13708.076631] ? smp_apic_timer_interrupt+0xa6/0x270 [13708.076639] ? apic_timer_interrupt+0xf/0x20 [13708.076644] [13708.076647] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13708.076655] do_raw_spin_lock+0x177/0x190 [13708.076660] _raw_spin_lock+0x21/0x30 [13708.076800] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13708.081019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13708.081099] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13708.081107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13708.081113] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13708.081368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13708.081380] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13708.081384] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13708.081388] RAX: 00000000000041f5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13708.081391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13708.081393] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13708.081395] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13708.081398] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13708.081402] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13708.081406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13708.081408] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13708.081418] Call Trace: [13708.081424] [13708.081429] ? show_regs.cold.9+0x22/0x2f [13708.081439] ? watchdog_timer_fn.cold.6+0x62/0x10a [13708.081446] ? __hrtimer_run_queues+0x1c2/0x4d0 [13708.081452] ? watchdog+0x40/0x40 [13708.081457] ? hrtimer_interrupt+0x128/0x2c0 [13708.081461] ? smp_apic_timer_interrupt+0xa6/0x270 [13708.081469] ? apic_timer_interrupt+0xf/0x20 [13708.081474] [13708.081478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13708.081483] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13708.081488] do_raw_spin_lock+0x177/0x190 [13708.081493] _raw_spin_lock+0x21/0x30 [13708.081498] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13708.081703] process_one_work+0x2c8/0x700 [13708.081710] worker_thread+0x296/0x6e0 [13708.081714] ? rescuer_thread+0x570/0x570 [13708.081718] kthread+0x1d1/0x200 [13708.081722] ? set_kthread_struct+0x70/0x70 [13708.081726] ret_from_fork+0x1f/0x30 [13708.446096] ? do_raw_spin_unlock+0x75/0x190 [13708.448685] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13708.460524] ? do_wait_intr+0xf0/0xf0 [13708.464292] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13708.474645] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13708.488413] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13708.498063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13708.501226] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13708.504833] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13708.511638] mdc_intent_lock+0x353/0x7e0 [mdc] [13708.517313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13708.520202] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13708.525301] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13708.528720] lmv_intent_open+0x338/0x1150 [lmv] [13708.530877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13708.538232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13708.541068] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13708.545079] ll_intent_lock+0xc5/0xaa0 [lustre] [13708.547614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13708.554128] ll_lookup_it+0xe0e/0x2270 [lustre] [13708.560113] ? do_raw_spin_unlock+0x75/0x190 [13708.565259] ? _raw_spin_unlock+0x12/0x30 [13708.572204] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13708.576656] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13708.582104] ll_atomic_open+0x36a/0xd70 [lustre] [13708.585076] lookup_open+0x356/0xb40 [13708.597337] do_last+0x313/0xfc0 [13708.599231] ? nd_jump_root+0xe5/0x160 [13708.604715] ? path_init+0x437/0x520 [13708.606482] path_openat+0xf7/0x500 [13708.610433] ? do_raw_spin_unlock+0x75/0x190 [13708.612481] do_filp_open+0x99/0x140 [13708.614340] ? getname_flags+0x6e/0x330 [13708.618004] ? __check_object_size+0xff/0x256 [13708.619944] ? do_raw_spin_unlock+0x75/0x190 [13708.622065] ? _raw_spin_unlock+0x12/0x30 [13708.624059] do_sys_openat2+0x2b4/0x410 [13708.629951] do_sys_open+0x73/0xa0 [13708.631274] __x64_sys_openat+0x24/0x30 [13708.632317] do_syscall_64+0xc1/0x440 [13708.636358] entry_SYSCALL_64_after_hwframe+0x49/0xae [13708.639733] RIP: 0033:0x7fa749552332 [13708.641410] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13708.657170] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13708.666637] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13708.671487] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13708.679063] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13708.684607] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13708.688509] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13712.866045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11795s! [13712.868777] Showing busy workqueues and worker pools: [13712.875910] workqueue events: flags=0x0 [13712.878878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13712.890110] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13712.901448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13712.907177] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13712.910668] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.914049] pending: kfree_rcu_monitor [13712.915703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13712.920872] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13712.925127] workqueue events_long: flags=0x0 [13712.929840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.933287] pending: lu_objects_flush [obdclass] [13712.935804] workqueue events_unbound: flags=0x2 [13712.937838] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13712.940652] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13712.944382] workqueue events_power_efficient: flags=0x80 [13712.949307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.953344] pending: check_lifetime [13712.955011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.957762] pending: neigh_periodic_work [13712.959691] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.966300] pending: neigh_periodic_work [13712.968509] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.974304] pending: do_cache_clean [13712.976202] workqueue mm_percpu_wq: flags=0x8 [13712.981193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.984849] pending: vmstat_update [13712.986346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.989036] pending: vmstat_update [13712.990287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13712.994136] pending: vmstat_update [13712.996801] workqueue writeback: flags=0x4a [13712.998626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13713.000989] pending: wb_workfn, wb_workfn [13713.005688] workqueue xprtiod: flags=0xa [13713.007072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13713.012055] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13713.014960] workqueue ipv6_addrconf: flags=0x40008 [13713.016710] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13713.019085] pending: addrconf_verify_work [13713.020831] workqueue knbd0-recv: flags=0x1a [13713.022563] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13713.024861] in-flight: 651:recv_work [13713.026445] workqueue ptlrpc_pinger: flags=0x2 [13713.028225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13713.030886] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13713.033238] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11795s workers=4 idle: 37 31846 28897 [13713.036324] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13702s workers=2 idle: 653 [13720.060368] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13720.068704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13720.115608] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13720.124238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13720.131128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13720.136060] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13720.151093] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13720.157253] RAX: 0000000000006982 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13720.160151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13720.169051] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13720.172816] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13720.184319] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13720.189247] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13720.194672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13720.201278] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13720.205663] Call Trace: [13720.206893] [13720.207944] ? show_regs.cold.9+0x22/0x2f [13720.210115] ? watchdog_timer_fn.cold.6+0x62/0x10a [13720.220316] ? __hrtimer_run_queues+0x1c2/0x4d0 [13720.223991] ? watchdog+0x40/0x40 [13720.228072] ? hrtimer_interrupt+0x128/0x2c0 [13720.231820] ? smp_apic_timer_interrupt+0xa6/0x270 [13720.234125] ? apic_timer_interrupt+0xf/0x20 [13720.239668] [13720.240693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13720.244655] do_raw_spin_lock+0x177/0x190 [13720.247967] _raw_spin_lock+0x21/0x30 [13720.249904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13720.253160] ? do_raw_spin_unlock+0x75/0x190 [13720.254956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13720.258733] ? do_wait_intr+0xf0/0xf0 [13720.260624] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13720.268419] mdc_close+0x53b/0xd70 [mdc] [13720.270133] lmv_close+0x18d/0x490 [lmv] [13720.272061] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13720.276532] ll_md_real_close+0xa5/0x2a0 [lustre] [13720.279683] ll_md_close+0x274/0x800 [lustre] [13720.281872] ll_file_release+0x128/0xc50 [lustre] [13720.284127] __fput+0x10d/0x440 [13720.284922] ____fput+0x12/0x20 [13720.288691] task_work_run+0xc6/0x110 [13720.290346] exit_to_usermode_loop+0x1dd/0x1f0 [13720.292969] do_syscall_64+0x3d6/0x440 [13720.294489] entry_SYSCALL_64_after_hwframe+0x49/0xae [13720.296504] RIP: 0033:0x7fa749552cd8 [13720.297842] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13720.306760] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13720.310411] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13720.314063] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13720.317221] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13720.321287] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13720.324298] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13728.058088] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13728.062742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13728.088575] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13728.104289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13728.111917] RIP: 0010:native_safe_halt+0x2e/0x50 [13728.113633] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13728.141869] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13728.156609] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13728.163151] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13728.172907] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13728.182072] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13728.193074] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13728.203569] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13728.209137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13728.212715] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13728.217849] Call Trace: [13728.219124] [13728.220675] ? show_regs.cold.9+0x22/0x2f [13728.223138] ? watchdog_timer_fn.cold.6+0x62/0x10a [13728.225696] ? __hrtimer_run_queues+0x1c2/0x4d0 [13728.227827] ? watchdog+0x40/0x40 [13728.229917] ? hrtimer_interrupt+0x128/0x2c0 [13728.233662] ? smp_apic_timer_interrupt+0xa6/0x270 [13728.236125] ? apic_timer_interrupt+0xf/0x20 [13728.238997] [13728.240549] ? native_safe_halt+0x2e/0x50 [13728.245699] kvm_wait+0xcd/0xe0 [13728.248958] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13728.258815] do_raw_spin_lock+0x177/0x190 [13728.263742] _raw_spin_lock+0x21/0x30 [13728.268656] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13728.272750] obd_mod_free+0x46/0x140 [ptlrpc] [13728.275413] mdc_commit_open+0x91/0xb0 [mdc] [13728.276818] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13728.278836] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13728.281353] after_reply+0xe2c/0x1780 [ptlrpc] [13728.283920] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13728.286544] ? __next_timer_interrupt+0x160/0x160 [13728.288703] ptlrpcd+0xa82/0xe40 [ptlrpc] [13728.290925] ? do_wait_intr+0xf0/0xf0 [13728.292638] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13728.296098] kthread+0x1d1/0x200 [13728.297165] ? set_kthread_struct+0x70/0x70 [13728.298945] ret_from_fork+0x1f/0x30 [13736.057029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13736.061094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13736.080232] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13736.083456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13736.086905] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13736.089425] RIP: 0010:native_safe_halt+0x2e/0x50 [13736.090866] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13736.098461] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13736.101559] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13736.104055] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13736.106545] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13736.109020] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13736.111916] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13736.114955] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13736.118455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13736.119748] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13736.121312] Call Trace: [13736.121788] [13736.122261] ? show_regs.cold.9+0x22/0x2f [13736.123374] ? watchdog_timer_fn.cold.6+0x62/0x10a [13736.124541] ? __hrtimer_run_queues+0x1c2/0x4d0 [13736.125507] ? watchdog+0x40/0x40 [13736.126369] ? hrtimer_interrupt+0x128/0x2c0 [13736.127499] ? smp_apic_timer_interrupt+0xa6/0x270 [13736.128640] ? apic_timer_interrupt+0xf/0x20 [13736.129642] [13736.130115] ? native_safe_halt+0x2e/0x50 [13736.130971] kvm_wait+0xcd/0xe0 [13736.131990] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13736.133236] do_raw_spin_lock+0x177/0x190 [13736.134443] _raw_spin_lock+0x21/0x30 [13736.135236] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13736.137714] process_one_work+0x2c8/0x700 [13736.138950] worker_thread+0x296/0x6e0 [13736.140596] ? rescuer_thread+0x570/0x570 [13736.142026] kthread+0x1d1/0x200 [13736.143576] ? set_kthread_struct+0x70/0x70 [13736.145203] ret_from_fork+0x1f/0x30 [13740.054071] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13740.056975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13740.089348] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13740.093630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13740.098249] RIP: 0010:native_safe_halt+0x2e/0x50 [13740.101640] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13740.111211] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13740.116174] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13740.120769] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [13740.124466] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13740.129404] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13740.134696] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13740.138389] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13740.142300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13740.145035] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13740.148482] Call Trace: [13740.149675] [13740.150576] ? show_regs.cold.9+0x22/0x2f [13740.152536] ? watchdog_timer_fn.cold.6+0x62/0x10a [13740.154938] ? __hrtimer_run_queues+0x1c2/0x4d0 [13740.156948] ? watchdog+0x40/0x40 [13740.158598] ? hrtimer_interrupt+0x128/0x2c0 [13740.160798] ? smp_apic_timer_interrupt+0xa6/0x270 [13740.163272] ? apic_timer_interrupt+0xf/0x20 [13740.165063] [13740.165991] ? native_safe_halt+0x2e/0x50 [13740.167755] kvm_wait+0xcd/0xe0 [13740.168972] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13740.170965] do_raw_spin_lock+0x177/0x190 [13740.172676] _raw_spin_lock+0x21/0x30 [13740.174553] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13740.177095] ? do_raw_spin_unlock+0x75/0x190 [13740.178951] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13740.181706] ? do_wait_intr+0xf0/0xf0 [13740.183085] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13740.185928] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13740.188567] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13740.192255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13740.197369] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13740.203408] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13740.204817] mdc_intent_lock+0x353/0x7e0 [mdc] [13740.209045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13740.210611] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13740.212920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13740.214442] lmv_intent_open+0x338/0x1150 [lmv] [13740.216086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13740.217753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13740.220040] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13740.221968] ll_intent_lock+0xc5/0xaa0 [lustre] [13740.224034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13740.226496] ll_lookup_it+0xe0e/0x2270 [lustre] [13740.228635] ? do_raw_spin_unlock+0x75/0x190 [13740.230314] ? _raw_spin_unlock+0x12/0x30 [13740.232016] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13740.234215] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13740.236461] ll_atomic_open+0x36a/0xd70 [lustre] [13740.238698] lookup_open+0x356/0xb40 [13740.240106] do_last+0x313/0xfc0 [13740.241462] ? nd_jump_root+0xe5/0x160 [13740.243091] ? path_init+0x437/0x520 [13740.244705] path_openat+0xf7/0x500 [13740.246177] ? do_raw_spin_unlock+0x75/0x190 [13740.247931] do_filp_open+0x99/0x140 [13740.249624] ? getname_flags+0x6e/0x330 [13740.251253] ? __check_object_size+0xff/0x256 [13740.253041] ? do_raw_spin_unlock+0x75/0x190 [13740.254786] ? _raw_spin_unlock+0x12/0x30 [13740.256656] do_sys_openat2+0x2b4/0x410 [13740.258222] do_sys_open+0x73/0xa0 [13740.259721] __x64_sys_openat+0x24/0x30 [13740.261406] do_syscall_64+0xc1/0x440 [13740.263073] entry_SYSCALL_64_after_hwframe+0x49/0xae [13740.265298] RIP: 0033:0x7fa749552332 [13740.266954] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13740.275334] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13740.278569] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13740.281840] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13740.284890] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13740.287732] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13740.290721] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13743.074252] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11825s! [13743.079318] Showing busy workqueues and worker pools: [13743.081680] workqueue events: flags=0x0 [13743.084024] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13743.089620] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13743.099405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13743.102941] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13743.113338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.116136] pending: kfree_rcu_monitor [13743.118795] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13743.121828] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13743.127647] workqueue events_long: flags=0x0 [13743.129708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.132770] pending: lu_objects_flush [obdclass] [13743.135207] workqueue events_unbound: flags=0x2 [13743.138154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13743.140697] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13743.146423] workqueue events_power_efficient: flags=0x80 [13743.148618] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.151416] pending: check_lifetime [13743.152915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.155784] pending: neigh_periodic_work [13743.157424] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.160385] pending: neigh_periodic_work [13743.162067] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.164480] pending: do_cache_clean [13743.166322] workqueue mm_percpu_wq: flags=0x8 [13743.168435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.171527] pending: vmstat_update [13743.173169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.176171] pending: vmstat_update [13743.177987] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13743.181205] pending: vmstat_update [13743.182783] workqueue writeback: flags=0x4a [13743.184824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13743.187843] pending: wb_workfn, wb_workfn [13743.189931] workqueue xprtiod: flags=0xa [13743.191837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13743.196484] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13743.200860] workqueue ipv6_addrconf: flags=0x40008 [13743.203209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13743.206743] pending: addrconf_verify_work [13743.208707] workqueue knbd0-recv: flags=0x1a [13743.210985] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13743.214174] in-flight: 651:recv_work [13743.216012] workqueue ptlrpc_pinger: flags=0x2 [13743.217821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13743.220305] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13743.223316] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11826s workers=4 idle: 37 31846 28897 [13743.227759] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13733s workers=2 idle: 653 [13748.061952] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13748.069179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13748.108462] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13748.114786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13748.123254] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13748.125978] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13748.154800] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13748.164276] RAX: 0000000000005bb1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13748.176690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13748.182077] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13748.193452] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13748.195568] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13748.205075] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13748.213526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13748.216755] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13748.222667] Call Trace: [13748.225268] [13748.227227] ? show_regs.cold.9+0x22/0x2f [13748.231267] ? watchdog_timer_fn.cold.6+0x62/0x10a [13748.237959] ? __hrtimer_run_queues+0x1c2/0x4d0 [13748.245426] ? watchdog+0x40/0x40 [13748.246547] ? hrtimer_interrupt+0x128/0x2c0 [13748.247874] ? smp_apic_timer_interrupt+0xa6/0x270 [13748.251805] ? apic_timer_interrupt+0xf/0x20 [13748.254896] [13748.255700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13748.264654] do_raw_spin_lock+0x177/0x190 [13748.266470] _raw_spin_lock+0x21/0x30 [13748.273386] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13748.279380] ? do_raw_spin_unlock+0x75/0x190 [13748.286443] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13748.289073] ? do_wait_intr+0xf0/0xf0 [13748.290179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13748.293805] mdc_close+0x53b/0xd70 [mdc] [13748.299262] lmv_close+0x18d/0x490 [lmv] [13748.301763] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13748.309367] ll_md_real_close+0xa5/0x2a0 [lustre] [13748.316837] ll_md_close+0x274/0x800 [lustre] [13748.326536] ll_file_release+0x128/0xc50 [lustre] [13748.328814] __fput+0x10d/0x440 [13748.330515] ____fput+0x12/0x20 [13748.335728] task_work_run+0xc6/0x110 [13748.337612] exit_to_usermode_loop+0x1dd/0x1f0 [13748.340517] do_syscall_64+0x3d6/0x440 [13748.342718] entry_SYSCALL_64_after_hwframe+0x49/0xae [13748.344960] RIP: 0033:0x7fa749552cd8 [13748.352089] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13748.363942] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13748.369985] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13748.373816] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13748.379058] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13748.391550] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13748.393734] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13756.058009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13756.064139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13756.116159] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13756.128490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13756.145267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13756.153963] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13756.179468] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13756.185068] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13756.192260] RDX: 000000000000128b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13756.202679] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13756.207243] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13756.217314] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13756.219767] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13756.224647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13756.235707] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13756.239657] Call Trace: [13756.241899] [13756.242649] ? show_regs.cold.9+0x22/0x2f [13756.246359] ? watchdog_timer_fn.cold.6+0x62/0x10a [13756.251623] ? __hrtimer_run_queues+0x1c2/0x4d0 [13756.256119] ? watchdog+0x40/0x40 [13756.259223] ? hrtimer_interrupt+0x128/0x2c0 [13756.262241] ? smp_apic_timer_interrupt+0xa6/0x270 [13756.267046] ? apic_timer_interrupt+0xf/0x20 [13756.268468] [13756.269211] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13756.275277] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13756.286703] do_raw_spin_lock+0x177/0x190 [13756.290166] _raw_spin_lock+0x21/0x30 [13756.293275] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13756.296431] obd_mod_free+0x46/0x140 [ptlrpc] [13756.302344] mdc_commit_open+0x91/0xb0 [mdc] [13756.308178] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13756.310539] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13756.324937] after_reply+0xe2c/0x1780 [ptlrpc] [13756.326712] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13756.341020] ? __next_timer_interrupt+0x160/0x160 [13756.344520] ptlrpcd+0xa82/0xe40 [ptlrpc] [13756.347966] ? do_wait_intr+0xf0/0xf0 [13756.349557] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13756.353319] kthread+0x1d1/0x200 [13756.354405] ? set_kthread_struct+0x70/0x70 [13756.360139] ret_from_fork+0x1f/0x30 [13764.057287] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13764.060047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13764.083154] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13764.089634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13764.096370] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13764.102318] RIP: 0010:native_safe_halt+0x2e/0x50 [13764.105082] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13764.122899] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13764.131077] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13764.135095] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13764.140552] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13764.145808] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13764.151801] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13764.155213] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13764.159879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13764.162692] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13764.165868] Call Trace: [13764.167432] [13764.168461] ? show_regs.cold.9+0x22/0x2f [13764.170233] ? watchdog_timer_fn.cold.6+0x62/0x10a [13764.173984] ? __hrtimer_run_queues+0x1c2/0x4d0 [13764.176039] ? watchdog+0x40/0x40 [13764.177933] ? hrtimer_interrupt+0x128/0x2c0 [13764.180019] ? smp_apic_timer_interrupt+0xa6/0x270 [13764.184840] ? apic_timer_interrupt+0xf/0x20 [13764.191238] [13764.192191] ? native_safe_halt+0x2e/0x50 [13764.196078] kvm_wait+0xcd/0xe0 [13764.197610] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13764.200062] do_raw_spin_lock+0x177/0x190 [13764.202448] _raw_spin_lock+0x21/0x30 [13764.204029] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13764.209407] process_one_work+0x2c8/0x700 [13764.213824] worker_thread+0x296/0x6e0 [13764.216890] ? rescuer_thread+0x570/0x570 [13764.218854] kthread+0x1d1/0x200 [13764.220329] ? set_kthread_struct+0x70/0x70 [13764.221779] ret_from_fork+0x1f/0x30 [13768.054041] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13768.058271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13768.113838] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13768.127712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13768.135088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13768.138020] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13768.153352] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13768.157267] RAX: 0000000000006d1a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13768.164154] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13768.175428] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13768.187421] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13768.191754] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13768.198905] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13768.206919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13768.212125] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13768.219932] Call Trace: [13768.223997] [13768.228094] ? show_regs.cold.9+0x22/0x2f [13768.229606] ? watchdog_timer_fn.cold.6+0x62/0x10a [13768.231189] ? __hrtimer_run_queues+0x1c2/0x4d0 [13768.236248] ? watchdog+0x40/0x40 [13768.238989] ? hrtimer_interrupt+0x128/0x2c0 [13768.247913] ? smp_apic_timer_interrupt+0xa6/0x270 [13768.252288] ? apic_timer_interrupt+0xf/0x20 [13768.255843] [13768.260528] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13768.265129] do_raw_spin_lock+0x177/0x190 [13768.271398] _raw_spin_lock+0x21/0x30 [13768.272882] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13768.281739] ? do_raw_spin_unlock+0x75/0x190 [13768.288451] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13768.293103] ? do_wait_intr+0xf0/0xf0 [13768.295475] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13768.301427] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13768.307077] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13768.310073] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13768.312982] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13768.318280] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13768.321371] mdc_intent_lock+0x353/0x7e0 [mdc] [13768.325805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13768.332356] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13768.336058] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13768.341477] lmv_intent_open+0x338/0x1150 [lmv] [13768.343404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13768.345780] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13768.351912] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13768.355239] ll_intent_lock+0xc5/0xaa0 [lustre] [13768.363441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13768.371730] ll_lookup_it+0xe0e/0x2270 [lustre] [13768.376495] ? do_raw_spin_unlock+0x75/0x190 [13768.378093] ? _raw_spin_unlock+0x12/0x30 [13768.387093] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13768.390404] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13768.395086] ll_atomic_open+0x36a/0xd70 [lustre] [13768.398388] lookup_open+0x356/0xb40 [13768.401463] do_last+0x313/0xfc0 [13768.403160] ? nd_jump_root+0xe5/0x160 [13768.408726] ? path_init+0x437/0x520 [13768.412982] path_openat+0xf7/0x500 [13768.419025] ? do_raw_spin_unlock+0x75/0x190 [13768.422481] do_filp_open+0x99/0x140 [13768.427789] ? getname_flags+0x6e/0x330 [13768.431297] ? __check_object_size+0xff/0x256 [13768.434964] ? do_raw_spin_unlock+0x75/0x190 [13768.437688] ? _raw_spin_unlock+0x12/0x30 [13768.440396] do_sys_openat2+0x2b4/0x410 [13768.442678] do_sys_open+0x73/0xa0 [13768.447440] __x64_sys_openat+0x24/0x30 [13768.453585] do_syscall_64+0xc1/0x440 [13768.457396] entry_SYSCALL_64_after_hwframe+0x49/0xae [13768.461662] RIP: 0033:0x7fa749552332 [13768.462867] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13768.486630] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13768.493709] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13768.499221] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13768.506036] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13768.516269] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13768.527193] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13773.281249] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11856s! [13773.289712] Showing busy workqueues and worker pools: [13773.293286] workqueue events: flags=0x0 [13773.297936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13773.305082] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13773.315099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13773.322316] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13773.330744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.337388] pending: kfree_rcu_monitor [13773.342833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13773.349657] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13773.358516] workqueue events_long: flags=0x0 [13773.361568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.371453] pending: lu_objects_flush [obdclass] [13773.377750] workqueue events_unbound: flags=0x2 [13773.385257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13773.391128] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13773.399833] workqueue events_power_efficient: flags=0x80 [13773.404543] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.410808] pending: check_lifetime [13773.413720] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.417558] pending: neigh_periodic_work [13773.421612] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.429890] pending: neigh_periodic_work [13773.432260] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.438319] pending: do_cache_clean [13773.441921] workqueue mm_percpu_wq: flags=0x8 [13773.446129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.455786] pending: vmstat_update [13773.459641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.465137] pending: vmstat_update [13773.466437] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13773.472092] pending: vmstat_update [13773.474832] workqueue writeback: flags=0x4a [13773.479543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13773.481858] pending: wb_workfn, wb_workfn [13773.487119] workqueue xprtiod: flags=0xa [13773.488403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13773.494540] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13773.501108] workqueue ipv6_addrconf: flags=0x40008 [13773.506153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13773.511982] pending: addrconf_verify_work [13773.514361] workqueue knbd0-recv: flags=0x1a [13773.518956] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13773.521351] in-flight: 651:recv_work [13773.526444] workqueue ptlrpc_pinger: flags=0x2 [13773.528433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13773.534262] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13773.539914] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11856s workers=4 idle: 37 31846 28897 [13773.546543] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13763s workers=2 idle: 653 [13776.059126] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13776.062860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13776.083385] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13776.088797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13776.093206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13776.095821] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13776.103571] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13776.106706] RAX: 0000000000005f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13776.109550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13776.112573] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13776.119541] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13776.127628] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13776.136598] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13776.140921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13776.142600] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13776.144913] Call Trace: [13776.145837] [13776.146494] ? show_regs.cold.9+0x22/0x2f [13776.147656] ? watchdog_timer_fn.cold.6+0x62/0x10a [13776.148972] ? __hrtimer_run_queues+0x1c2/0x4d0 [13776.150909] ? watchdog+0x40/0x40 [13776.151979] ? hrtimer_interrupt+0x128/0x2c0 [13776.153545] ? smp_apic_timer_interrupt+0xa6/0x270 [13776.155267] ? apic_timer_interrupt+0xf/0x20 [13776.156699] [13776.157415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13776.159448] do_raw_spin_lock+0x177/0x190 [13776.160999] _raw_spin_lock+0x21/0x30 [13776.162481] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13776.164574] ? do_raw_spin_unlock+0x75/0x190 [13776.166463] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13776.168882] ? do_wait_intr+0xf0/0xf0 [13776.170433] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13776.172514] mdc_close+0x53b/0xd70 [mdc] [13776.174114] lmv_close+0x18d/0x490 [lmv] [13776.175882] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13776.177992] ll_md_real_close+0xa5/0x2a0 [lustre] [13776.179932] ll_md_close+0x274/0x800 [lustre] [13776.181854] ll_file_release+0x128/0xc50 [lustre] [13776.183513] __fput+0x10d/0x440 [13776.184963] ____fput+0x12/0x20 [13776.186159] task_work_run+0xc6/0x110 [13776.187427] exit_to_usermode_loop+0x1dd/0x1f0 [13776.188917] do_syscall_64+0x3d6/0x440 [13776.190394] entry_SYSCALL_64_after_hwframe+0x49/0xae [13776.191980] RIP: 0033:0x7fa749552cd8 [13776.193452] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13776.199814] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13776.202514] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13776.205164] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13776.207498] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13776.209979] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13776.214567] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13784.058470] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13784.070640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13784.109038] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13784.122143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13784.126192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13784.130023] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13784.150926] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13784.166697] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13784.171120] RDX: 000000000000277b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13784.175840] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13784.182586] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13784.192624] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13784.199968] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13784.208509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13784.211709] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13784.222045] Call Trace: [13784.223669] [13784.227883] ? show_regs.cold.9+0x22/0x2f [13784.231584] ? watchdog_timer_fn.cold.6+0x62/0x10a [13784.235053] ? __hrtimer_run_queues+0x1c2/0x4d0 [13784.237803] ? watchdog+0x40/0x40 [13784.242618] ? hrtimer_interrupt+0x128/0x2c0 [13784.246845] ? smp_apic_timer_interrupt+0xa6/0x270 [13784.251919] ? apic_timer_interrupt+0xf/0x20 [13784.255680] [13784.259164] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13784.261911] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13784.265710] do_raw_spin_lock+0x177/0x190 [13784.269921] _raw_spin_lock+0x21/0x30 [13784.274297] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13784.280039] obd_mod_free+0x46/0x140 [ptlrpc] [13784.285003] mdc_commit_open+0x91/0xb0 [mdc] [13784.288037] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13784.291816] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13784.295034] after_reply+0xe2c/0x1780 [ptlrpc] [13784.298052] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13784.304687] ? __next_timer_interrupt+0x160/0x160 [13784.310783] ptlrpcd+0xa82/0xe40 [ptlrpc] [13784.313736] ? do_wait_intr+0xf0/0xf0 [13784.322451] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13784.327676] kthread+0x1d1/0x200 [13784.331643] ? set_kthread_struct+0x70/0x70 [13784.334556] ret_from_fork+0x1f/0x30 [13792.057392] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13792.075068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13792.128379] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13792.140596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13792.158344] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13792.169937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13792.178825] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13792.197033] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13792.200799] RAX: 0000000000002f18 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13792.204775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13792.214856] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13792.225385] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13792.247885] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13792.257367] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13792.273214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13792.289154] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13792.292476] Call Trace: [13792.293194] [13792.315565] ? show_regs.cold.9+0x22/0x2f [13792.327763] ? watchdog_timer_fn.cold.6+0x62/0x10a [13792.341992] ? __hrtimer_run_queues+0x1c2/0x4d0 [13792.352454] ? watchdog+0x40/0x40 [13792.356096] ? hrtimer_interrupt+0x128/0x2c0 [13792.358795] ? smp_apic_timer_interrupt+0xa6/0x270 [13792.369727] ? apic_timer_interrupt+0xf/0x20 [13792.371319] [13792.382882] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13792.391717] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13792.402497] do_raw_spin_lock+0x177/0x190 [13792.416896] _raw_spin_lock+0x21/0x30 [13792.418175] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13792.443689] process_one_work+0x2c8/0x700 [13792.448529] worker_thread+0x296/0x6e0 [13792.455995] ? rescuer_thread+0x570/0x570 [13792.465995] kthread+0x1d1/0x200 [13792.467085] ? set_kthread_struct+0x70/0x70 [13792.468180] ret_from_fork+0x1f/0x30 [13800.054011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13800.057385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13800.095065] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13800.101493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13800.104794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13800.107475] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13800.116204] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13800.119556] RAX: 000000000000010d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13800.123078] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13800.134975] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13800.138415] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13800.142078] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13800.145087] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13800.148264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13800.150858] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13800.154104] Call Trace: [13800.155169] [13800.156157] ? show_regs.cold.9+0x22/0x2f [13800.158040] ? watchdog_timer_fn.cold.6+0x62/0x10a [13800.160086] ? __hrtimer_run_queues+0x1c2/0x4d0 [13800.162181] ? watchdog+0x40/0x40 [13800.163697] ? hrtimer_interrupt+0x128/0x2c0 [13800.165608] ? smp_apic_timer_interrupt+0xa6/0x270 [13800.167827] ? apic_timer_interrupt+0xf/0x20 [13800.169806] [13800.170783] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13800.173118] do_raw_spin_lock+0x177/0x190 [13800.175028] _raw_spin_lock+0x21/0x30 [13800.176741] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13800.179467] ? do_raw_spin_unlock+0x75/0x190 [13800.181475] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13800.185227] ? do_wait_intr+0xf0/0xf0 [13800.187245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13800.189456] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13800.191530] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13800.193920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13800.196125] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13800.201784] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13800.204558] mdc_intent_lock+0x353/0x7e0 [mdc] [13800.206710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13800.209337] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13800.212245] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13800.215874] lmv_intent_open+0x338/0x1150 [lmv] [13800.218546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13800.221236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13800.225177] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13800.227750] ll_intent_lock+0xc5/0xaa0 [lustre] [13800.230395] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13800.241798] ll_lookup_it+0xe0e/0x2270 [lustre] [13800.248999] ? do_raw_spin_unlock+0x75/0x190 [13800.255815] ? _raw_spin_unlock+0x12/0x30 [13800.261293] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13800.265440] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13800.268371] ll_atomic_open+0x36a/0xd70 [lustre] [13800.272445] lookup_open+0x356/0xb40 [13800.276028] do_last+0x313/0xfc0 [13800.279186] ? nd_jump_root+0xe5/0x160 [13800.283233] ? path_init+0x437/0x520 [13800.291336] path_openat+0xf7/0x500 [13800.295283] ? do_raw_spin_unlock+0x75/0x190 [13800.302510] do_filp_open+0x99/0x140 [13800.305329] ? getname_flags+0x6e/0x330 [13800.309375] ? __check_object_size+0xff/0x256 [13800.312404] ? do_raw_spin_unlock+0x75/0x190 [13800.314859] ? _raw_spin_unlock+0x12/0x30 [13800.317362] do_sys_openat2+0x2b4/0x410 [13800.320927] do_sys_open+0x73/0xa0 [13800.325282] __x64_sys_openat+0x24/0x30 [13800.328773] do_syscall_64+0xc1/0x440 [13800.331455] entry_SYSCALL_64_after_hwframe+0x49/0xae [13800.342271] RIP: 0033:0x7fa749552332 [13800.346683] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13800.377182] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13800.395985] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13800.412349] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13800.422618] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13800.433667] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13800.438627] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13804.001312] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11886s! [13804.007498] Showing busy workqueues and worker pools: [13804.009930] workqueue events: flags=0x0 [13804.011540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13804.014634] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13804.020117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13804.023432] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13804.027704] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.030743] pending: kfree_rcu_monitor [13804.032682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13804.036194] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13804.041464] workqueue events_long: flags=0x0 [13804.043342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.056079] pending: lu_objects_flush [obdclass] [13804.059058] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13804.062534] workqueue events_unbound: flags=0x2 [13804.062545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13804.062785] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13804.062811] workqueue events_power_efficient: flags=0x80 [13804.062814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.062823] pending: check_lifetime [13804.062830] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.062839] pending: neigh_periodic_work [13804.062846] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.062853] pending: neigh_periodic_work [13804.063057] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.063067] pending: do_cache_clean [13804.063078] workqueue mm_percpu_wq: flags=0x8 [13804.063080] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.063086] pending: vmstat_update [13804.063093] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.063099] pending: vmstat_update [13804.063103] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13804.063110] pending: vmstat_update [13804.063117] workqueue writeback: flags=0x4a [13804.063119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13804.063126] pending: wb_workfn, wb_workfn [13804.063139] workqueue xprtiod: flags=0xa [13804.063141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13804.063455] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13804.063474] workqueue ipv6_addrconf: flags=0x40008 [13804.063477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13804.063484] pending: addrconf_verify_work [13804.063494] workqueue knbd0-recv: flags=0x1a [13804.063497] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13804.063504] in-flight: 651:recv_work [13804.063513] workqueue ptlrpc_pinger: flags=0x2 [13804.063516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13804.063521] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13804.064729] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11886s workers=4 idle: 37 31846 28897 [13804.064746] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13793s workers=2 idle: 653 [13804.222351] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13804.244366] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13804.251938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13804.255457] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [13804.257195] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [13804.263341] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13804.265474] RAX: 0000000000002cf8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13804.267384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13804.269305] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13804.270881] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13804.272801] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13804.274610] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13804.276830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13804.281166] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13804.283102] Call Trace: [13804.284475] [13804.285438] ? show_regs.cold.9+0x22/0x2f [13804.288415] ? watchdog_timer_fn.cold.6+0x62/0x10a [13804.292379] ? __hrtimer_run_queues+0x1c2/0x4d0 [13804.295072] ? watchdog+0x40/0x40 [13804.297715] ? hrtimer_interrupt+0x128/0x2c0 [13804.300408] ? smp_apic_timer_interrupt+0xa6/0x270 [13804.305079] ? apic_timer_interrupt+0xf/0x20 [13804.313745] [13804.316968] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [13804.319548] do_raw_spin_lock+0x177/0x190 [13804.321624] _raw_spin_lock+0x21/0x30 [13804.323465] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13804.326404] ? do_raw_spin_unlock+0x75/0x190 [13804.328714] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13804.331396] ? do_wait_intr+0xf0/0xf0 [13804.333094] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13804.336059] mdc_close+0x53b/0xd70 [mdc] [13804.337996] lmv_close+0x18d/0x490 [lmv] [13804.340379] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13804.343211] ll_md_real_close+0xa5/0x2a0 [lustre] [13804.345694] ll_md_close+0x274/0x800 [lustre] [13804.347940] ll_file_release+0x128/0xc50 [lustre] [13804.352035] __fput+0x10d/0x440 [13804.354100] ____fput+0x12/0x20 [13804.357345] task_work_run+0xc6/0x110 [13804.359210] exit_to_usermode_loop+0x1dd/0x1f0 [13804.361782] do_syscall_64+0x3d6/0x440 [13804.363799] entry_SYSCALL_64_after_hwframe+0x49/0xae [13804.366684] RIP: 0033:0x7fa749552cd8 [13804.368638] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13804.377979] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13804.381876] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13804.384452] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13804.386840] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13804.391992] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13804.396798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13812.058059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13812.068504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13812.158932] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13812.189335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13812.200465] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13812.213835] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13812.237247] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13812.247861] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13812.250912] RDX: 000000000000721f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13812.253920] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13812.262945] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13812.269009] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13812.280707] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13812.296245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13812.302246] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13812.305915] Call Trace: [13812.316860] [13812.318219] ? show_regs.cold.9+0x22/0x2f [13812.329846] ? watchdog_timer_fn.cold.6+0x62/0x10a [13812.331332] ? __hrtimer_run_queues+0x1c2/0x4d0 [13812.339668] ? watchdog+0x40/0x40 [13812.341988] ? hrtimer_interrupt+0x128/0x2c0 [13812.347117] ? smp_apic_timer_interrupt+0xa6/0x270 [13812.356284] ? apic_timer_interrupt+0xf/0x20 [13812.357840] [13812.358461] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13812.364141] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13812.369900] do_raw_spin_lock+0x177/0x190 [13812.373074] _raw_spin_lock+0x21/0x30 [13812.375556] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13812.377508] obd_mod_free+0x46/0x140 [ptlrpc] [13812.381220] mdc_commit_open+0x91/0xb0 [mdc] [13812.386636] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13812.390362] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13812.408047] after_reply+0xe2c/0x1780 [ptlrpc] [13812.418050] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13812.435018] ? __next_timer_interrupt+0x160/0x160 [13812.436618] ptlrpcd+0xa82/0xe40 [ptlrpc] [13812.442880] ? do_wait_intr+0xf0/0xf0 [13812.444446] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13812.458622] kthread+0x1d1/0x200 [13812.461316] ? set_kthread_struct+0x70/0x70 [13812.469538] ret_from_fork+0x1f/0x30 [13820.057093] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13820.071088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13820.133154] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13820.139550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13820.146107] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13820.152096] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13820.155783] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13820.181837] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13820.187044] RAX: 000000000000791c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13820.199296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13820.206068] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13820.209221] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13820.213462] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13820.217368] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13820.223965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13820.230890] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13820.237424] Call Trace: [13820.238446] [13820.239143] ? show_regs.cold.9+0x22/0x2f [13820.240609] ? watchdog_timer_fn.cold.6+0x62/0x10a [13820.245763] ? __hrtimer_run_queues+0x1c2/0x4d0 [13820.247889] ? watchdog+0x40/0x40 [13820.253334] ? hrtimer_interrupt+0x128/0x2c0 [13820.255269] ? smp_apic_timer_interrupt+0xa6/0x270 [13820.259111] ? apic_timer_interrupt+0xf/0x20 [13820.261327] [13820.262813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13820.266112] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13820.269106] do_raw_spin_lock+0x177/0x190 [13820.274055] _raw_spin_lock+0x21/0x30 [13820.276294] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13820.281093] process_one_work+0x2c8/0x700 [13820.283944] worker_thread+0x296/0x6e0 [13820.287016] ? rescuer_thread+0x570/0x570 [13820.288260] kthread+0x1d1/0x200 [13820.289387] ? set_kthread_struct+0x70/0x70 [13820.294469] ret_from_fork+0x1f/0x30 [13832.054100] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13832.059041] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13832.059988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13832.060060] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13832.060067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13832.060070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13832.060084] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13832.060089] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13832.060093] RAX: 00000000000003b1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13832.060096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13832.060098] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13832.060101] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13832.060169] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13832.060173] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13832.060176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13832.060178] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13832.060186] Call Trace: [13832.060191] [13832.060198] ? show_regs.cold.9+0x22/0x2f [13832.060208] ? watchdog_timer_fn.cold.6+0x62/0x10a [13832.060214] ? __hrtimer_run_queues+0x1c2/0x4d0 [13832.060219] ? watchdog+0x40/0x40 [13832.060223] ? hrtimer_interrupt+0x128/0x2c0 [13832.060227] ? smp_apic_timer_interrupt+0xa6/0x270 [13832.060236] ? apic_timer_interrupt+0xf/0x20 [13832.060241] [13832.060244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13832.060263] do_raw_spin_lock+0x177/0x190 [13832.060269] _raw_spin_lock+0x21/0x30 [13832.060315] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13832.060843] ? do_raw_spin_unlock+0x75/0x190 [13832.060849] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13832.061046] ? do_wait_intr+0xf0/0xf0 [13832.061054] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13832.061247] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13832.063321] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13832.070361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13832.070436] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13832.070444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13832.070446] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13832.070460] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13832.070465] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13832.070471] RAX: 0000000000002e51 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13832.070474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13832.070476] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13832.070479] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13832.070481] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13832.070484] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13832.070487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13832.070489] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13832.070496] Call Trace: [13832.070501] [13832.070507] ? show_regs.cold.9+0x22/0x2f [13832.070516] ? watchdog_timer_fn.cold.6+0x62/0x10a [13832.070523] ? __hrtimer_run_queues+0x1c2/0x4d0 [13832.070527] ? watchdog+0x40/0x40 [13832.070531] ? hrtimer_interrupt+0x128/0x2c0 [13832.070535] ? smp_apic_timer_interrupt+0xa6/0x270 [13832.070543] ? apic_timer_interrupt+0xf/0x20 [13832.070548] [13832.070551] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13832.070558] do_raw_spin_lock+0x177/0x190 [13832.070563] _raw_spin_lock+0x21/0x30 [13832.070568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13832.106972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13832.107096] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13832.107141] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13832.107173] mdc_intent_lock+0x353/0x7e0 [mdc] [13832.107199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13832.107495] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13832.107809] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13832.107846] lmv_intent_open+0x338/0x1150 [lmv] [13832.107880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13832.107969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13832.108051] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13832.108072] ll_intent_lock+0xc5/0xaa0 [lustre] [13832.108155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13832.108235] ll_lookup_it+0xe0e/0x2270 [lustre] [13832.113580] ? do_raw_spin_unlock+0x75/0x190 [13832.120488] ? do_raw_spin_unlock+0x75/0x190 [13832.122600] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13832.123399] ? do_wait_intr+0xf0/0xf0 [13832.123409] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13832.123613] mdc_close+0x53b/0xd70 [mdc] [13832.123732] lmv_close+0x18d/0x490 [lmv] [13832.123768] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13832.123904] ll_md_real_close+0xa5/0x2a0 [lustre] [13832.124040] ll_md_close+0x274/0x800 [lustre] [13832.124131] ll_file_release+0x128/0xc50 [lustre] [13832.124431] __fput+0x10d/0x440 [13832.124440] ____fput+0x12/0x20 [13832.124445] task_work_run+0xc6/0x110 [13832.124454] exit_to_usermode_loop+0x1dd/0x1f0 [13832.124465] do_syscall_64+0x3d6/0x440 [13832.124469] entry_SYSCALL_64_after_hwframe+0x49/0xae [13832.124478] RIP: 0033:0x7fa749552cd8 [13832.124485] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13832.124490] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13832.124495] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13832.124497] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13832.124499] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13832.124501] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13832.124504] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13832.507769] ? _raw_spin_unlock+0x12/0x30 [13832.509152] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13832.510932] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13832.512595] ll_atomic_open+0x36a/0xd70 [lustre] [13832.514161] lookup_open+0x356/0xb40 [13832.515857] do_last+0x313/0xfc0 [13832.516723] ? nd_jump_root+0xe5/0x160 [13832.517886] ? path_init+0x437/0x520 [13832.518983] path_openat+0xf7/0x500 [13832.520207] ? do_raw_spin_unlock+0x75/0x190 [13832.521738] do_filp_open+0x99/0x140 [13832.522996] ? getname_flags+0x6e/0x330 [13832.524271] ? __check_object_size+0xff/0x256 [13832.525829] ? do_raw_spin_unlock+0x75/0x190 [13832.527630] ? _raw_spin_unlock+0x12/0x30 [13832.529186] do_sys_openat2+0x2b4/0x410 [13832.530836] do_sys_open+0x73/0xa0 [13832.532188] __x64_sys_openat+0x24/0x30 [13832.533987] do_syscall_64+0xc1/0x440 [13832.535712] entry_SYSCALL_64_after_hwframe+0x49/0xae [13832.537633] RIP: 0033:0x7fa749552332 [13832.539209] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13832.549177] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13832.553514] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13832.557096] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13832.559950] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13832.564299] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13832.568774] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13834.213468] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11917s! [13834.225264] Showing busy workqueues and worker pools: [13834.228145] workqueue events: flags=0x0 [13834.233866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13834.239379] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13834.261315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13834.268866] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13834.285090] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.297791] pending: kfree_rcu_monitor [13834.306795] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13834.308701] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13834.328900] workqueue events_long: flags=0x0 [13834.335232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.352291] pending: lu_objects_flush [obdclass] [13834.359874] workqueue events_unbound: flags=0x2 [13834.369715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13834.374708] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13834.384288] workqueue events_power_efficient: flags=0x80 [13834.388748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.393520] pending: check_lifetime [13834.396726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.402138] pending: neigh_periodic_work [13834.406253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.407871] pending: neigh_periodic_work [13834.413721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.419171] pending: do_cache_clean [13834.421839] workqueue mm_percpu_wq: flags=0x8 [13834.424831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.428486] pending: vmstat_update [13834.430534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.433316] pending: vmstat_update [13834.436169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13834.438845] pending: vmstat_update [13834.440223] workqueue writeback: flags=0x4a [13834.442162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13834.444886] pending: wb_workfn, wb_workfn [13834.446865] workqueue xprtiod: flags=0xa [13834.448293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13834.450723] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13834.453488] workqueue ipv6_addrconf: flags=0x40008 [13834.455247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13834.457926] pending: addrconf_verify_work [13834.459607] workqueue knbd0-recv: flags=0x1a [13834.461171] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13834.463691] in-flight: 651:recv_work [13834.464984] workqueue ptlrpc_pinger: flags=0x2 [13834.466794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13834.468807] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13834.471027] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11917s workers=4 idle: 37 31846 28897 [13834.473970] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13824s workers=2 idle: 653 [13840.058024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13840.063766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13840.121142] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13840.133850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13840.145062] RIP: 0010:native_safe_halt+0x2e/0x50 [13840.149134] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13840.159651] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13840.162865] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13840.169203] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13840.175372] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13840.184958] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13840.190602] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13840.195820] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13840.206601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13840.211229] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13840.219833] Call Trace: [13840.221832] [13840.223002] ? show_regs.cold.9+0x22/0x2f [13840.224713] ? watchdog_timer_fn.cold.6+0x62/0x10a [13840.227140] ? __hrtimer_run_queues+0x1c2/0x4d0 [13840.229603] ? watchdog+0x40/0x40 [13840.230999] ? hrtimer_interrupt+0x128/0x2c0 [13840.233716] ? smp_apic_timer_interrupt+0xa6/0x270 [13840.235635] ? apic_timer_interrupt+0xf/0x20 [13840.237527] [13840.238191] ? native_safe_halt+0x2e/0x50 [13840.240959] kvm_wait+0xcd/0xe0 [13840.242674] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13840.245756] do_raw_spin_lock+0x177/0x190 [13840.247088] _raw_spin_lock+0x21/0x30 [13840.248297] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13840.256899] obd_mod_free+0x46/0x140 [ptlrpc] [13840.259685] mdc_commit_open+0x91/0xb0 [mdc] [13840.261672] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13840.267302] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13840.273582] after_reply+0xe2c/0x1780 [ptlrpc] [13840.276024] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13840.287643] ? __next_timer_interrupt+0x160/0x160 [13840.290937] ptlrpcd+0xa82/0xe40 [ptlrpc] [13840.294044] ? do_wait_intr+0xf0/0xf0 [13840.295947] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13840.298671] kthread+0x1d1/0x200 [13840.300055] ? set_kthread_struct+0x70/0x70 [13840.302009] ret_from_fork+0x1f/0x30 [13848.060511] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13848.069586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13848.104902] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13848.113347] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13848.121843] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13848.127386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13848.135552] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13848.149764] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13848.153463] RAX: 0000000000007c96 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13848.157629] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13848.161957] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13848.171890] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13848.179294] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13848.186273] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13848.191097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13848.194264] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13848.198339] Call Trace: [13848.199816] [13848.201233] ? show_regs.cold.9+0x22/0x2f [13848.203785] ? watchdog_timer_fn.cold.6+0x62/0x10a [13848.207082] ? __hrtimer_run_queues+0x1c2/0x4d0 [13848.209799] ? watchdog+0x40/0x40 [13848.211380] ? hrtimer_interrupt+0x128/0x2c0 [13848.213783] ? smp_apic_timer_interrupt+0xa6/0x270 [13848.217209] ? apic_timer_interrupt+0xf/0x20 [13848.219944] [13848.220955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13848.223775] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13848.226380] do_raw_spin_lock+0x177/0x190 [13848.228229] _raw_spin_lock+0x21/0x30 [13848.230032] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13848.232823] process_one_work+0x2c8/0x700 [13848.244029] worker_thread+0x296/0x6e0 [13848.248435] ? rescuer_thread+0x570/0x570 [13848.251456] kthread+0x1d1/0x200 [13848.253177] ? set_kthread_struct+0x70/0x70 [13848.255366] ret_from_fork+0x1f/0x30 [13853.951051] rcu: INFO: rcu_sched self-detected stall on CPU [13853.953079] rcu: 2-...!: (6791903 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [13853.956297] (t=11940262 jiffies g=258701 q=4845) [13853.958359] rcu: rcu_sched kthread starved for 11937057 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [13853.962134] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13853.965143] rcu: RCU grace-period kthread stack dump: [13853.966733] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13853.970256] Call Trace: [13853.971169] __schedule+0x351/0xcb0 [13853.972512] schedule+0xc0/0x180 [13853.973708] schedule_timeout+0xb4/0x190 [13853.975068] ? do_raw_spin_unlock+0x75/0x190 [13853.976522] ? __next_timer_interrupt+0x160/0x160 [13853.978151] ? prepare_to_swait_event+0xf3/0x140 [13853.979726] rcu_gp_kthread+0x8dd/0x15b0 [13853.981104] ? force_qs_rnp+0x2c0/0x2c0 [13853.982480] kthread+0x1d1/0x200 [13853.983763] ? set_kthread_struct+0x70/0x70 [13853.985247] ret_from_fork+0x1f/0x30 [13853.986628] rcu: Stack dump where RCU GP kthread last ran: [13853.989070] Sending NMI from CPU 2 to CPUs 0: [13853.993506] NMI backtrace for cpu 0 [13853.993513] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13853.993515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13853.993517] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13853.993519] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13853.993521] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 [13853.993523] RAX: 0000000000007f90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13853.993525] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13853.993526] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13853.993527] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13853.993528] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13853.993529] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13853.993531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13853.993532] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13853.993533] Call Trace: [13853.993534] [13853.993534] ? show_regs.cold.9+0x22/0x2f [13853.993535] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13853.993537] ? nmi_cpu_backtrace_handler+0x15/0x40 [13853.993538] ? nmi_handle+0xbd/0x210 [13853.993538] ? default_do_nmi+0x4b/0x1b0 [13853.993539] ? do_nmi+0x280/0x380 [13853.993540] ? end_repeat_nmi+0x16/0x1f [13853.993541] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13853.993542] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13853.993543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13853.993544] [13853.993545] do_raw_spin_lock+0x177/0x190 [13853.993546] _raw_spin_lock+0x21/0x30 [13853.993547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13853.993548] ? do_raw_spin_unlock+0x75/0x190 [13853.993549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13853.993550] ? do_wait_intr+0xf0/0xf0 [13853.993551] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13853.993552] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13853.993553] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13853.993555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13853.993556] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13853.993557] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13853.993558] mdc_intent_lock+0x353/0x7e0 [mdc] [13853.993559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13853.993560] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13853.993562] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13853.993563] lmv_intent_open+0x338/0x1150 [lmv] [13853.993564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13853.993565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13853.993566] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13853.993568] ll_intent_lock+0xc5/0xaa0 [lustre] [13853.993569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13853.993570] ll_lookup_it+0xe0e/0x2270 [lustre] [13853.993571] ? do_raw_spin_unlock+0x75/0x190 [13853.993572] ? _raw_spin_unlock+0x12/0x30 [13853.993573] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13853.993574] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13853.993575] ll_atomic_open+0x36a/0xd70 [lustre] [13853.993576] lookup_open+0x356/0xb40 [13853.993577] do_last+0x313/0xfc0 [13853.993578] ? nd_jump_root+0xe5/0x160 [13853.993580] ? path_init+0x437/0x520 [13853.993581] path_openat+0xf7/0x500 [13853.993582] ? do_raw_spin_unlock+0x75/0x190 [13853.993583] do_filp_open+0x99/0x140 [13853.993584] ? getname_flags+0x6e/0x330 [13853.993585] ? __check_object_size+0xff/0x256 [13853.993587] ? do_raw_spin_unlock+0x75/0x190 [13853.993588] ? _raw_spin_unlock+0x12/0x30 [13853.993589] do_sys_openat2+0x2b4/0x410 [13853.993590] do_sys_open+0x73/0xa0 [13853.993591] __x64_sys_openat+0x24/0x30 [13853.993592] do_syscall_64+0xc1/0x440 [13853.993594] entry_SYSCALL_64_after_hwframe+0x49/0xae [13853.993595] RIP: 0033:0x7fa749552332 [13853.993597] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13853.993599] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13853.993602] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13853.993603] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13853.993605] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13853.993607] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13853.993627] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13853.994922] NMI backtrace for cpu 2 [13854.140529] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13854.144785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13854.148370] Call Trace: [13854.149308] [13854.150313] ? dump_stack+0xbb/0x10e [13854.151716] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13854.153687] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13854.155771] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13854.157372] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13854.159400] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13854.161231] ? rcu_dump_cpu_stacks+0x11b/0x171 [13854.163350] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13854.165452] ? update_process_times+0x6d/0xc0 [13854.166907] ? tick_sched_handle.isra.18+0x27/0x90 [13854.168690] ? tick_sched_timer+0x53/0xb0 [13854.170473] ? __hrtimer_run_queues+0x1c2/0x4d0 [13854.172840] ? tick_sched_handle.isra.18+0x90/0x90 [13854.174669] ? hrtimer_interrupt+0x128/0x2c0 [13854.177631] ? smp_apic_timer_interrupt+0xa6/0x270 [13854.179443] ? apic_timer_interrupt+0xf/0x20 [13854.182736] [13854.183557] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13854.187653] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13854.189726] ? do_raw_spin_lock+0x177/0x190 [13854.192262] ? _raw_spin_lock+0x21/0x30 [13854.193527] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13854.196495] ? obd_mod_free+0x46/0x140 [ptlrpc] [13854.198872] ? mdc_commit_open+0x91/0xb0 [mdc] [13854.200499] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13854.203176] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13854.205806] ? after_reply+0xe2c/0x1780 [ptlrpc] [13854.208142] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13854.210506] ? __next_timer_interrupt+0x160/0x160 [13854.212316] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13854.214331] ? do_wait_intr+0xf0/0xf0 [13854.215897] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13854.217969] ? kthread+0x1d1/0x200 [13854.219405] ? set_kthread_struct+0x70/0x70 [13854.220962] ? ret_from_fork+0x1f/0x30 [13860.059091] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13860.067768] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13860.114323] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13860.131432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13860.139322] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13860.150080] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13860.167812] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13860.173455] RAX: 000000000000602c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13860.181261] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13860.188365] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13860.192463] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13860.196046] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13860.199049] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13860.203281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13860.206141] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13860.208511] Call Trace: [13860.209159] [13860.210287] ? show_regs.cold.9+0x22/0x2f [13860.211802] ? watchdog_timer_fn.cold.6+0x62/0x10a [13860.214547] ? __hrtimer_run_queues+0x1c2/0x4d0 [13860.221513] ? watchdog+0x40/0x40 [13860.223679] ? hrtimer_interrupt+0x128/0x2c0 [13860.225220] ? smp_apic_timer_interrupt+0xa6/0x270 [13860.226453] ? apic_timer_interrupt+0xf/0x20 [13860.228828] [13860.230595] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13860.236311] do_raw_spin_lock+0x177/0x190 [13860.240011] _raw_spin_lock+0x21/0x30 [13860.243091] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13860.245944] ? do_raw_spin_unlock+0x75/0x190 [13860.247756] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13860.250025] ? do_wait_intr+0xf0/0xf0 [13860.251551] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13860.253991] mdc_close+0x53b/0xd70 [mdc] [13860.255225] lmv_close+0x18d/0x490 [lmv] [13860.256809] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13860.259095] ll_md_real_close+0xa5/0x2a0 [lustre] [13860.261018] ll_md_close+0x274/0x800 [lustre] [13860.263726] ll_file_release+0x128/0xc50 [lustre] [13860.266353] __fput+0x10d/0x440 [13860.267671] ____fput+0x12/0x20 [13860.269015] task_work_run+0xc6/0x110 [13860.270699] exit_to_usermode_loop+0x1dd/0x1f0 [13860.272335] do_syscall_64+0x3d6/0x440 [13860.273767] entry_SYSCALL_64_after_hwframe+0x49/0xae [13860.275629] RIP: 0033:0x7fa749552cd8 [13860.277496] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13860.285668] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13860.295767] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13860.306025] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13860.311625] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13860.315852] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13860.318131] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13864.932024] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11947s! [13864.935784] Showing busy workqueues and worker pools: [13864.938327] workqueue events: flags=0x0 [13864.940535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13864.944100] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13864.949838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13864.953787] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13864.959241] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13864.962836] pending: kfree_rcu_monitor [13864.964859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13864.968775] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13864.973564] workqueue events_long: flags=0x0 [13864.975919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13864.979510] pending: lu_objects_flush [obdclass] [13864.982252] workqueue events_unbound: flags=0x2 [13864.985231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13864.989190] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13864.994691] workqueue events_power_efficient: flags=0x80 [13864.997345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13865.001015] pending: check_lifetime [13865.002591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13865.006070] pending: neigh_periodic_work [13865.008476] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13865.012602] pending: neigh_periodic_work [13865.015811] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13865.019277] pending: do_cache_clean [13865.021278] workqueue mm_percpu_wq: flags=0x8 [13865.023735] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13865.028514] pending: vmstat_update [13865.031461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13865.035173] pending: vmstat_update [13865.037469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13865.041535] pending: vmstat_update [13865.044455] workqueue writeback: flags=0x4a [13865.046607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13865.050676] pending: wb_workfn, wb_workfn [13865.052989] workqueue xprtiod: flags=0xa [13865.054857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13865.059272] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13865.063844] workqueue ipv6_addrconf: flags=0x40008 [13865.066472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13865.070430] pending: addrconf_verify_work [13865.072655] workqueue knbd0-recv: flags=0x1a [13865.074849] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13865.078505] in-flight: 651:recv_work [13865.080863] workqueue ptlrpc_pinger: flags=0x2 [13865.083715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13865.087306] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13865.090675] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11947s workers=4 idle: 37 31846 28897 [13865.095284] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13854s workers=2 idle: 653 [13876.057130] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13876.062221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13876.100753] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13876.106278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13876.110189] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13876.113474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13876.116157] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13876.124486] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13876.127964] RAX: 0000000000005208 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13876.131702] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13876.134828] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13876.137938] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13876.141439] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13876.145187] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13876.148871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13876.151415] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13876.154933] Call Trace: [13876.156028] [13876.157429] ? show_regs.cold.9+0x22/0x2f [13876.159363] ? watchdog_timer_fn.cold.6+0x62/0x10a [13876.161733] ? __hrtimer_run_queues+0x1c2/0x4d0 [13876.163932] ? watchdog+0x40/0x40 [13876.165262] ? hrtimer_interrupt+0x128/0x2c0 [13876.167154] ? smp_apic_timer_interrupt+0xa6/0x270 [13876.169256] ? apic_timer_interrupt+0xf/0x20 [13876.171376] [13876.172505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13876.174899] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13876.178191] do_raw_spin_lock+0x177/0x190 [13876.180917] _raw_spin_lock+0x21/0x30 [13876.183840] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13876.186180] process_one_work+0x2c8/0x700 [13876.187210] worker_thread+0x296/0x6e0 [13876.188361] ? rescuer_thread+0x570/0x570 [13876.190143] kthread+0x1d1/0x200 [13876.191137] ? set_kthread_struct+0x70/0x70 [13876.192240] ret_from_fork+0x1f/0x30 [13880.058576] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13880.066240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13880.108873] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13880.116170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13880.121774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13880.125107] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13880.141207] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13880.145552] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13880.157390] RDX: 0000000000004370 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13880.166869] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13880.175769] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13880.184787] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13880.189329] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13880.194780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13880.196746] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13880.202693] Call Trace: [13880.203628] [13880.204376] ? show_regs.cold.9+0x22/0x2f [13880.208748] ? watchdog_timer_fn.cold.6+0x62/0x10a [13880.210726] ? __hrtimer_run_queues+0x1c2/0x4d0 [13880.212776] ? watchdog+0x40/0x40 [13880.214276] ? hrtimer_interrupt+0x128/0x2c0 [13880.215978] ? smp_apic_timer_interrupt+0xa6/0x270 [13880.218063] ? apic_timer_interrupt+0xf/0x20 [13880.221821] [13880.224496] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13880.229562] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13880.231509] do_raw_spin_lock+0x177/0x190 [13880.232830] _raw_spin_lock+0x21/0x30 [13880.234847] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13880.238513] obd_mod_free+0x46/0x140 [ptlrpc] [13880.243861] mdc_commit_open+0x91/0xb0 [mdc] [13880.251661] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13880.253523] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13880.259603] after_reply+0xe2c/0x1780 [ptlrpc] [13880.263625] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13880.267337] ? __next_timer_interrupt+0x160/0x160 [13880.273596] ptlrpcd+0xa82/0xe40 [ptlrpc] [13880.279153] ? do_wait_intr+0xf0/0xf0 [13880.282957] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13880.294872] kthread+0x1d1/0x200 [13880.297379] ? set_kthread_struct+0x70/0x70 [13880.302250] ret_from_fork+0x1f/0x30 [13888.060827] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13888.066456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13888.128049] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13888.141997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13888.151310] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13888.155428] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13888.165684] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13888.172040] RAX: 0000000000005b8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13888.176885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13888.186686] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13888.190953] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13888.194785] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13888.202918] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13888.209076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13888.215539] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13888.221954] Call Trace: [13888.224333] [13888.226505] ? show_regs.cold.9+0x22/0x2f [13888.230891] ? watchdog_timer_fn.cold.6+0x62/0x10a [13888.235998] ? __hrtimer_run_queues+0x1c2/0x4d0 [13888.239931] ? watchdog+0x40/0x40 [13888.242151] ? hrtimer_interrupt+0x128/0x2c0 [13888.244823] ? smp_apic_timer_interrupt+0xa6/0x270 [13888.251998] ? apic_timer_interrupt+0xf/0x20 [13888.257480] [13888.262096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13888.271004] do_raw_spin_lock+0x177/0x190 [13888.273706] _raw_spin_lock+0x21/0x30 [13888.278546] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13888.284612] ? do_raw_spin_unlock+0x75/0x190 [13888.291293] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13888.297695] ? do_wait_intr+0xf0/0xf0 [13888.302946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13888.309316] mdc_close+0x53b/0xd70 [mdc] [13888.316072] lmv_close+0x18d/0x490 [lmv] [13888.318603] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13888.323908] ll_md_real_close+0xa5/0x2a0 [lustre] [13888.327133] ll_md_close+0x274/0x800 [lustre] [13888.331379] ll_file_release+0x128/0xc50 [lustre] [13888.335049] __fput+0x10d/0x440 [13888.337566] ____fput+0x12/0x20 [13888.344131] task_work_run+0xc6/0x110 [13888.348572] exit_to_usermode_loop+0x1dd/0x1f0 [13888.354868] do_syscall_64+0x3d6/0x440 [13888.357668] entry_SYSCALL_64_after_hwframe+0x49/0xae [13888.363232] RIP: 0033:0x7fa749552cd8 [13888.365852] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13888.391027] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13888.393976] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13888.404145] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13888.409677] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13888.415687] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13888.422526] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13892.054674] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13892.064677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13892.106490] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13892.111668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13892.116182] RIP: 0010:native_safe_halt+0x2e/0x50 [13892.118686] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13892.127389] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13892.130613] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13892.134137] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13892.137164] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13892.141870] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13892.145896] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13892.149829] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13892.155014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13892.158009] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13892.161623] Call Trace: [13892.162538] [13892.163698] ? show_regs.cold.9+0x22/0x2f [13892.164984] ? watchdog_timer_fn.cold.6+0x62/0x10a [13892.166644] ? __hrtimer_run_queues+0x1c2/0x4d0 [13892.168762] ? watchdog+0x40/0x40 [13892.170949] ? hrtimer_interrupt+0x128/0x2c0 [13892.173395] ? smp_apic_timer_interrupt+0xa6/0x270 [13892.175789] ? apic_timer_interrupt+0xf/0x20 [13892.178465] [13892.179464] ? native_safe_halt+0x2e/0x50 [13892.180900] kvm_wait+0xcd/0xe0 [13892.182254] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13892.184638] do_raw_spin_lock+0x177/0x190 [13892.186557] _raw_spin_lock+0x21/0x30 [13892.188067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13892.191433] ? do_raw_spin_unlock+0x75/0x190 [13892.193583] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13892.196402] ? do_wait_intr+0xf0/0xf0 [13892.198205] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13892.201374] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13892.204698] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13892.206990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13892.209045] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13892.211691] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13892.213512] mdc_intent_lock+0x353/0x7e0 [mdc] [13892.215300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13892.217936] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13892.221515] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13892.223582] lmv_intent_open+0x338/0x1150 [lmv] [13892.225964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13892.228671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13892.231010] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13892.232773] ll_intent_lock+0xc5/0xaa0 [lustre] [13892.234572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13892.236664] ll_lookup_it+0xe0e/0x2270 [lustre] [13892.238280] ? do_raw_spin_unlock+0x75/0x190 [13892.239398] ? _raw_spin_unlock+0x12/0x30 [13892.240678] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13892.242707] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13892.244512] ll_atomic_open+0x36a/0xd70 [lustre] [13892.246247] lookup_open+0x356/0xb40 [13892.247651] do_last+0x313/0xfc0 [13892.248809] ? nd_jump_root+0xe5/0x160 [13892.249961] ? path_init+0x437/0x520 [13892.251302] path_openat+0xf7/0x500 [13892.252509] ? do_raw_spin_unlock+0x75/0x190 [13892.254081] do_filp_open+0x99/0x140 [13892.255480] ? getname_flags+0x6e/0x330 [13892.257205] ? __check_object_size+0xff/0x256 [13892.259297] ? do_raw_spin_unlock+0x75/0x190 [13892.260778] ? _raw_spin_unlock+0x12/0x30 [13892.262763] do_sys_openat2+0x2b4/0x410 [13892.264084] do_sys_open+0x73/0xa0 [13892.265493] __x64_sys_openat+0x24/0x30 [13892.267419] do_syscall_64+0xc1/0x440 [13892.269459] entry_SYSCALL_64_after_hwframe+0x49/0xae [13892.271356] RIP: 0033:0x7fa749552332 [13892.272786] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13892.282285] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13892.286002] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13892.289457] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13892.293016] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13892.295614] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13892.300141] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13895.139009] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11977s! [13895.142531] Showing busy workqueues and worker pools: [13895.151106] workqueue events: flags=0x0 [13895.157524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13895.160566] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13895.164676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13895.166958] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13895.175814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.186288] pending: kfree_rcu_monitor [13895.189729] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13895.195536] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13895.200517] workqueue events_long: flags=0x0 [13895.202113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.207467] pending: lu_objects_flush [obdclass] [13895.214084] workqueue events_unbound: flags=0x2 [13895.217903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13895.224717] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13895.228852] workqueue events_power_efficient: flags=0x80 [13895.235691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.242120] pending: check_lifetime [13895.253344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.255638] pending: neigh_periodic_work [13895.263742] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.266957] pending: neigh_periodic_work [13895.273620] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.282937] pending: do_cache_clean [13895.288104] workqueue mm_percpu_wq: flags=0x8 [13895.290159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.296447] pending: vmstat_update [13895.301624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.310467] pending: vmstat_update [13895.315195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13895.325829] pending: vmstat_update [13895.333879] workqueue writeback: flags=0x4a [13895.335261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13895.346276] pending: wb_workfn, wb_workfn [13895.347708] workqueue xprtiod: flags=0xa [13895.355822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13895.369044] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13895.381204] workqueue ipv6_addrconf: flags=0x40008 [13895.391376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13895.396724] pending: addrconf_verify_work [13895.402978] workqueue knbd0-recv: flags=0x1a [13895.405381] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13895.411653] in-flight: 651:recv_work [13895.413245] workqueue ptlrpc_pinger: flags=0x2 [13895.415015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13895.420985] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13895.428310] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11978s workers=4 idle: 37 31846 28897 [13895.432304] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13885s workers=2 idle: 653 [13904.057088] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13904.061496] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13904.113927] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13904.136287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13904.144340] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13904.158239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13904.166381] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13904.208192] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13904.225031] RAX: 0000000000006fb5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13904.232631] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13904.251464] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13904.261194] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13904.279286] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13904.293059] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13904.308848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13904.329922] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13904.332201] Call Trace: [13904.341174] [13904.341936] ? show_regs.cold.9+0x22/0x2f [13904.343385] ? watchdog_timer_fn.cold.6+0x62/0x10a [13904.364202] ? __hrtimer_run_queues+0x1c2/0x4d0 [13904.367626] ? watchdog+0x40/0x40 [13904.378220] ? hrtimer_interrupt+0x128/0x2c0 [13904.387230] ? smp_apic_timer_interrupt+0xa6/0x270 [13904.396443] ? apic_timer_interrupt+0xf/0x20 [13904.401281] [13904.402747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13904.419535] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13904.429792] do_raw_spin_lock+0x177/0x190 [13904.432040] _raw_spin_lock+0x21/0x30 [13904.435409] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13904.441317] process_one_work+0x2c8/0x700 [13904.446442] worker_thread+0x296/0x6e0 [13904.448762] ? rescuer_thread+0x570/0x570 [13904.451013] kthread+0x1d1/0x200 [13904.453004] ? set_kthread_struct+0x70/0x70 [13904.457254] ret_from_fork+0x1f/0x30 [13908.058006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13908.065383] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13908.110268] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13908.119309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13908.126500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13908.131966] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13908.158017] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13908.167487] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13908.175981] RDX: 00000000000076c1 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13908.183747] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13908.197970] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13908.203777] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13908.206257] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13908.214467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13908.222381] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13908.233218] Call Trace: [13908.234231] [13908.235588] ? show_regs.cold.9+0x22/0x2f [13908.242666] ? watchdog_timer_fn.cold.6+0x62/0x10a [13908.246951] ? __hrtimer_run_queues+0x1c2/0x4d0 [13908.250538] ? watchdog+0x40/0x40 [13908.251757] ? hrtimer_interrupt+0x128/0x2c0 [13908.258879] ? smp_apic_timer_interrupt+0xa6/0x270 [13908.264916] ? apic_timer_interrupt+0xf/0x20 [13908.268813] [13908.272265] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13908.277923] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13908.284671] do_raw_spin_lock+0x177/0x190 [13908.287963] _raw_spin_lock+0x21/0x30 [13908.289734] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13908.293084] obd_mod_free+0x46/0x140 [ptlrpc] [13908.295664] mdc_commit_open+0x91/0xb0 [mdc] [13908.302026] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13908.309105] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13908.313211] after_reply+0xe2c/0x1780 [ptlrpc] [13908.316166] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13908.319728] ? __next_timer_interrupt+0x160/0x160 [13908.322002] ptlrpcd+0xa82/0xe40 [ptlrpc] [13908.331235] ? do_wait_intr+0xf0/0xf0 [13908.332909] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13908.337196] kthread+0x1d1/0x200 [13908.339441] ? set_kthread_struct+0x70/0x70 [13908.340864] ret_from_fork+0x1f/0x30 [13916.059011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [13916.064513] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13916.098143] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13916.108668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13916.113530] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13916.119442] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13916.127151] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13916.131016] RAX: 0000000000001818 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13916.136730] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13916.144071] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13916.151013] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13916.154415] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13916.161371] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13916.167090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13916.174123] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13916.181006] Call Trace: [13916.183699] [13916.185660] ? show_regs.cold.9+0x22/0x2f [13916.189821] ? watchdog_timer_fn.cold.6+0x62/0x10a [13916.196393] ? __hrtimer_run_queues+0x1c2/0x4d0 [13916.198391] ? watchdog+0x40/0x40 [13916.201412] ? hrtimer_interrupt+0x128/0x2c0 [13916.203719] ? smp_apic_timer_interrupt+0xa6/0x270 [13916.206012] ? apic_timer_interrupt+0xf/0x20 [13916.208158] [13916.211493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13916.216263] do_raw_spin_lock+0x177/0x190 [13916.220448] _raw_spin_lock+0x21/0x30 [13916.224783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13916.228545] ? do_raw_spin_unlock+0x75/0x190 [13916.229585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13916.232337] ? do_wait_intr+0xf0/0xf0 [13916.238272] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13916.244880] mdc_close+0x53b/0xd70 [mdc] [13916.247411] lmv_close+0x18d/0x490 [lmv] [13916.250180] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13916.254811] ll_md_real_close+0xa5/0x2a0 [lustre] [13916.259760] ll_md_close+0x274/0x800 [lustre] [13916.262083] ll_file_release+0x128/0xc50 [lustre] [13916.264777] __fput+0x10d/0x440 [13916.266456] ____fput+0x12/0x20 [13916.267649] task_work_run+0xc6/0x110 [13916.269288] exit_to_usermode_loop+0x1dd/0x1f0 [13916.271120] do_syscall_64+0x3d6/0x440 [13916.272729] entry_SYSCALL_64_after_hwframe+0x49/0xae [13916.274734] RIP: 0033:0x7fa749552cd8 [13916.276243] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13916.287463] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13916.290286] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13916.293157] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13916.296472] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13916.303484] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13916.310706] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13920.054314] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [13920.064935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13920.107823] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13920.124744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13920.132172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13920.135647] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13920.148131] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13920.152089] RAX: 0000000000004209 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13920.155692] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13920.159063] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13920.170437] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13920.175676] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13920.182686] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13920.190701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13920.194151] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13920.198944] Call Trace: [13920.200016] [13920.203468] ? show_regs.cold.9+0x22/0x2f [13920.206784] ? watchdog_timer_fn.cold.6+0x62/0x10a [13920.210411] ? __hrtimer_run_queues+0x1c2/0x4d0 [13920.215539] ? watchdog+0x40/0x40 [13920.218576] ? hrtimer_interrupt+0x128/0x2c0 [13920.226557] ? smp_apic_timer_interrupt+0xa6/0x270 [13920.229606] ? apic_timer_interrupt+0xf/0x20 [13920.234263] [13920.235028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13920.240300] do_raw_spin_lock+0x177/0x190 [13920.242690] _raw_spin_lock+0x21/0x30 [13920.244845] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13920.249004] ? do_raw_spin_unlock+0x75/0x190 [13920.251396] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13920.256276] ? do_wait_intr+0xf0/0xf0 [13920.260853] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13920.264025] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13920.267947] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13920.271161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13920.274020] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13920.276932] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13920.279062] mdc_intent_lock+0x353/0x7e0 [mdc] [13920.283536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13920.288814] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13920.293066] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13920.297697] lmv_intent_open+0x338/0x1150 [lmv] [13920.301493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13920.304645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13920.307496] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13920.309623] ll_intent_lock+0xc5/0xaa0 [lustre] [13920.318014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13920.322555] ll_lookup_it+0xe0e/0x2270 [lustre] [13920.326499] ? do_raw_spin_unlock+0x75/0x190 [13920.329437] ? _raw_spin_unlock+0x12/0x30 [13920.332622] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13920.334608] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13920.338488] ll_atomic_open+0x36a/0xd70 [lustre] [13920.341500] lookup_open+0x356/0xb40 [13920.343566] do_last+0x313/0xfc0 [13920.347222] ? nd_jump_root+0xe5/0x160 [13920.352458] ? path_init+0x437/0x520 [13920.353801] path_openat+0xf7/0x500 [13920.358037] ? do_raw_spin_unlock+0x75/0x190 [13920.360185] do_filp_open+0x99/0x140 [13920.362397] ? getname_flags+0x6e/0x330 [13920.366344] ? __check_object_size+0xff/0x256 [13920.370484] ? do_raw_spin_unlock+0x75/0x190 [13920.374063] ? _raw_spin_unlock+0x12/0x30 [13920.376181] do_sys_openat2+0x2b4/0x410 [13920.377959] do_sys_open+0x73/0xa0 [13920.379786] __x64_sys_openat+0x24/0x30 [13920.382059] do_syscall_64+0xc1/0x440 [13920.383432] entry_SYSCALL_64_after_hwframe+0x49/0xae [13920.388004] RIP: 0033:0x7fa749552332 [13920.389512] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13920.400204] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13920.404657] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13920.408080] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13920.413372] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13920.419142] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13920.423300] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13925.856594] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12008s! [13925.861332] Showing busy workqueues and worker pools: [13925.863800] workqueue events: flags=0x0 [13925.865859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13925.869171] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13925.874575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13925.877470] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13925.882289] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.885563] pending: kfree_rcu_monitor [13925.887377] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13925.890423] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13925.894617] workqueue events_long: flags=0x0 [13925.896606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.899684] pending: lu_objects_flush [obdclass] [13925.903074] workqueue events_unbound: flags=0x2 [13925.905115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13925.908155] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13925.912644] workqueue events_power_efficient: flags=0x80 [13925.914785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.917970] pending: check_lifetime [13925.919580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.922539] pending: neigh_periodic_work [13925.924601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.928432] pending: neigh_periodic_work [13925.933971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.939133] pending: do_cache_clean [13925.941160] workqueue mm_percpu_wq: flags=0x8 [13925.946494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.950346] pending: vmstat_update [13925.951642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.955336] pending: vmstat_update [13925.956505] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13925.959860] pending: vmstat_update [13925.961566] workqueue writeback: flags=0x4a [13925.965630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13925.968171] pending: wb_workfn, wb_workfn [13925.970275] workqueue xprtiod: flags=0xa [13925.972314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13925.976736] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13925.979696] workqueue ipv6_addrconf: flags=0x40008 [13925.981171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13925.983328] pending: addrconf_verify_work [13925.985150] workqueue knbd0-recv: flags=0x1a [13925.986575] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13925.988458] in-flight: 651:recv_work [13925.992782] workqueue ptlrpc_pinger: flags=0x2 [13925.996666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13925.998785] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13926.004498] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12008s workers=4 idle: 37 31846 28897 [13926.008616] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13915s workers=2 idle: 653 [13932.057022] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13932.059388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13932.073551] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13932.076971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13932.079749] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13932.082027] RIP: 0010:native_safe_halt+0x2e/0x50 [13932.083363] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13932.088876] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13932.091254] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13932.093362] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13932.095633] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13932.097933] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13932.100747] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13932.102740] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13932.105061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13932.106764] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13932.108976] Call Trace: [13932.109634] [13932.110165] ? show_regs.cold.9+0x22/0x2f [13932.111520] ? watchdog_timer_fn.cold.6+0x62/0x10a [13932.112885] ? __hrtimer_run_queues+0x1c2/0x4d0 [13932.114241] ? watchdog+0x40/0x40 [13932.115275] ? hrtimer_interrupt+0x128/0x2c0 [13932.116620] ? smp_apic_timer_interrupt+0xa6/0x270 [13932.118146] ? apic_timer_interrupt+0xf/0x20 [13932.119465] [13932.120143] ? native_safe_halt+0x2e/0x50 [13932.121258] kvm_wait+0xcd/0xe0 [13932.122198] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13932.123807] do_raw_spin_lock+0x177/0x190 [13932.124919] _raw_spin_lock+0x21/0x30 [13932.125944] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13932.127865] process_one_work+0x2c8/0x700 [13932.129049] worker_thread+0x296/0x6e0 [13932.130248] ? rescuer_thread+0x570/0x570 [13932.131460] kthread+0x1d1/0x200 [13932.132390] ? set_kthread_struct+0x70/0x70 [13932.133576] ret_from_fork+0x1f/0x30 [13936.058023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [13936.062346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13936.081958] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13936.087521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13936.094330] RIP: 0010:native_safe_halt+0x2e/0x50 [13936.096227] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13936.103290] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13936.106728] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13936.109781] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13936.112852] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13936.116890] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13936.119828] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13936.122736] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13936.126143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.128319] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13936.131139] Call Trace: [13936.132032] [13936.132899] ? show_regs.cold.9+0x22/0x2f [13936.134369] ? watchdog_timer_fn.cold.6+0x62/0x10a [13936.136387] ? __hrtimer_run_queues+0x1c2/0x4d0 [13936.138168] ? watchdog+0x40/0x40 [13936.139994] ? hrtimer_interrupt+0x128/0x2c0 [13936.141489] ? smp_apic_timer_interrupt+0xa6/0x270 [13936.143278] ? apic_timer_interrupt+0xf/0x20 [13936.144810] [13936.145469] ? native_safe_halt+0x2e/0x50 [13936.146947] kvm_wait+0xcd/0xe0 [13936.148057] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13936.150168] do_raw_spin_lock+0x177/0x190 [13936.152013] _raw_spin_lock+0x21/0x30 [13936.153418] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13936.155643] obd_mod_free+0x46/0x140 [ptlrpc] [13936.157525] mdc_commit_open+0x91/0xb0 [mdc] [13936.159340] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13936.161568] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13936.163902] after_reply+0xe2c/0x1780 [ptlrpc] [13936.165901] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13936.168081] ? __next_timer_interrupt+0x160/0x160 [13936.169790] ptlrpcd+0xa82/0xe40 [ptlrpc] [13936.171450] ? do_wait_intr+0xf0/0xf0 [13936.173065] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13936.175031] kthread+0x1d1/0x200 [13936.176136] ? set_kthread_struct+0x70/0x70 [13936.177468] ret_from_fork+0x1f/0x30 [13944.061913] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13944.072171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13944.099050] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13944.104246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13944.108399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13944.111528] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13944.131311] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13944.138251] RAX: 0000000000003f97 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13944.145520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13944.152254] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13944.156314] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13944.160238] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13944.163912] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13944.167742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13944.170760] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13944.174460] Call Trace: [13944.175800] [13944.176704] ? show_regs.cold.9+0x22/0x2f [13944.178661] ? watchdog_timer_fn.cold.6+0x62/0x10a [13944.181212] ? __hrtimer_run_queues+0x1c2/0x4d0 [13944.183345] ? watchdog+0x40/0x40 [13944.185213] ? hrtimer_interrupt+0x128/0x2c0 [13944.187307] ? smp_apic_timer_interrupt+0xa6/0x270 [13944.189779] ? apic_timer_interrupt+0xf/0x20 [13944.191836] [13944.192835] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13944.194815] do_raw_spin_lock+0x177/0x190 [13944.196677] _raw_spin_lock+0x21/0x30 [13944.199391] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13944.202339] ? do_raw_spin_unlock+0x75/0x190 [13944.204278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13944.206661] ? do_wait_intr+0xf0/0xf0 [13944.207945] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13944.210324] mdc_close+0x53b/0xd70 [mdc] [13944.211801] lmv_close+0x18d/0x490 [lmv] [13944.213340] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13944.216081] ll_md_real_close+0xa5/0x2a0 [lustre] [13944.218526] ll_md_close+0x274/0x800 [lustre] [13944.220559] ll_file_release+0x128/0xc50 [lustre] [13944.222542] __fput+0x10d/0x440 [13944.226858] ____fput+0x12/0x20 [13944.228113] task_work_run+0xc6/0x110 [13944.233979] exit_to_usermode_loop+0x1dd/0x1f0 [13944.237507] do_syscall_64+0x3d6/0x440 [13944.239940] entry_SYSCALL_64_after_hwframe+0x49/0xae [13944.245063] RIP: 0033:0x7fa749552cd8 [13944.246669] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13944.260830] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13944.264496] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13944.270722] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13944.275364] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13944.281221] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13944.290845] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13952.054029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13952.057259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13952.086562] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13952.099705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13952.108611] RIP: 0010:native_safe_halt+0x2e/0x50 [13952.115034] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13952.134187] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13952.143855] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13952.156544] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [13952.164431] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13952.174850] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13952.182002] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13952.189362] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13952.193526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13952.199953] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13952.202851] Call Trace: [13952.203754] [13952.207851] ? show_regs.cold.9+0x22/0x2f [13952.210222] ? watchdog_timer_fn.cold.6+0x62/0x10a [13952.213759] ? __hrtimer_run_queues+0x1c2/0x4d0 [13952.217604] ? watchdog+0x40/0x40 [13952.220529] ? hrtimer_interrupt+0x128/0x2c0 [13952.224127] ? smp_apic_timer_interrupt+0xa6/0x270 [13952.225773] ? apic_timer_interrupt+0xf/0x20 [13952.231862] [13952.233369] ? native_safe_halt+0x2e/0x50 [13952.237107] kvm_wait+0xcd/0xe0 [13952.239107] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13952.241590] do_raw_spin_lock+0x177/0x190 [13952.244212] _raw_spin_lock+0x21/0x30 [13952.245766] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13952.250404] ? do_raw_spin_unlock+0x75/0x190 [13952.253098] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13952.255923] ? do_wait_intr+0xf0/0xf0 [13952.258350] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13952.260669] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13952.264898] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13952.269242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13952.271898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13952.275827] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13952.278750] mdc_intent_lock+0x353/0x7e0 [mdc] [13952.281751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13952.285840] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13952.290767] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13952.294472] lmv_intent_open+0x338/0x1150 [lmv] [13952.296178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13952.299927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13952.304079] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13952.316201] ll_intent_lock+0xc5/0xaa0 [lustre] [13952.322085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13952.332450] ll_lookup_it+0xe0e/0x2270 [lustre] [13952.336427] ? do_raw_spin_unlock+0x75/0x190 [13952.340722] ? _raw_spin_unlock+0x12/0x30 [13952.352220] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13952.357320] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13952.367659] ll_atomic_open+0x36a/0xd70 [lustre] [13952.376870] lookup_open+0x356/0xb40 [13952.379281] do_last+0x313/0xfc0 [13952.382141] ? nd_jump_root+0xe5/0x160 [13952.384495] ? path_init+0x437/0x520 [13952.386859] path_openat+0xf7/0x500 [13952.388983] ? do_raw_spin_unlock+0x75/0x190 [13952.391447] do_filp_open+0x99/0x140 [13952.393727] ? getname_flags+0x6e/0x330 [13952.397735] ? __check_object_size+0xff/0x256 [13952.400887] ? do_raw_spin_unlock+0x75/0x190 [13952.404904] ? _raw_spin_unlock+0x12/0x30 [13952.407806] do_sys_openat2+0x2b4/0x410 [13952.409256] do_sys_open+0x73/0xa0 [13952.411463] __x64_sys_openat+0x24/0x30 [13952.414125] do_syscall_64+0xc1/0x440 [13952.418081] entry_SYSCALL_64_after_hwframe+0x49/0xae [13952.422857] RIP: 0033:0x7fa749552332 [13952.429133] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13952.450699] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13952.462753] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13952.475984] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13952.485461] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13952.492354] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13952.496316] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13956.065060] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12038s! [13956.068624] Showing busy workqueues and worker pools: [13956.070717] workqueue events: flags=0x0 [13956.072085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13956.075054] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13956.080268] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13956.082931] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13956.087400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.089855] pending: kfree_rcu_monitor [13956.091354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13956.094077] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13956.097926] workqueue events_long: flags=0x0 [13956.099294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.102020] pending: lu_objects_flush [obdclass] [13956.103834] workqueue events_unbound: flags=0x2 [13956.105602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13956.107877] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13956.111279] workqueue events_power_efficient: flags=0x80 [13956.115122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.118889] pending: check_lifetime [13956.120953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.124576] pending: neigh_periodic_work [13956.126978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.136520] pending: neigh_periodic_work [13956.139780] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.147728] pending: do_cache_clean [13956.151792] workqueue mm_percpu_wq: flags=0x8 [13956.155060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.158991] pending: vmstat_update [13956.161088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.165114] pending: vmstat_update [13956.166969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13956.170722] pending: vmstat_update [13956.176460] workqueue writeback: flags=0x4a [13956.179755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13956.186667] pending: wb_workfn, wb_workfn [13956.189072] workqueue xprtiod: flags=0xa [13956.190387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13956.192398] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13956.195519] workqueue ipv6_addrconf: flags=0x40008 [13956.198043] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13956.200887] pending: addrconf_verify_work [13956.202931] workqueue knbd0-recv: flags=0x1a [13956.204917] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13956.207171] in-flight: 651:recv_work [13956.208705] workqueue ptlrpc_pinger: flags=0x2 [13956.210079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13956.212137] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13956.214738] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12039s workers=4 idle: 37 31846 28897 [13956.218460] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13946s workers=2 idle: 653 [13960.057060] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [13960.061394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13960.096648] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13960.111866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13960.119896] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13960.135623] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13960.149231] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13960.171469] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13960.177525] RAX: 000000000000530f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13960.182859] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13960.192097] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13960.198785] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13960.212621] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13960.219405] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13960.233271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13960.242573] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13960.249645] Call Trace: [13960.256392] [13960.258562] ? show_regs.cold.9+0x22/0x2f [13960.260952] ? watchdog_timer_fn.cold.6+0x62/0x10a [13960.267478] ? __hrtimer_run_queues+0x1c2/0x4d0 [13960.269142] ? watchdog+0x40/0x40 [13960.276880] ? hrtimer_interrupt+0x128/0x2c0 [13960.278476] ? smp_apic_timer_interrupt+0xa6/0x270 [13960.283606] ? apic_timer_interrupt+0xf/0x20 [13960.288941] [13960.293552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13960.298872] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13960.303909] do_raw_spin_lock+0x177/0x190 [13960.307550] _raw_spin_lock+0x21/0x30 [13960.310935] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13960.316772] process_one_work+0x2c8/0x700 [13960.325367] worker_thread+0x296/0x6e0 [13960.329668] ? rescuer_thread+0x570/0x570 [13960.334918] kthread+0x1d1/0x200 [13960.339429] ? set_kthread_struct+0x70/0x70 [13960.340988] ret_from_fork+0x1f/0x30 [13964.058047] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13964.060984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13964.079031] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13964.083396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13964.087035] RIP: 0010:native_safe_halt+0x2e/0x50 [13964.089033] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13964.097613] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13964.102632] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13964.106868] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13964.112031] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13964.118937] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13964.122335] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13964.127554] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13964.131967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13964.134820] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13964.138490] Call Trace: [13964.139579] [13964.141888] ? show_regs.cold.9+0x22/0x2f [13964.145841] ? watchdog_timer_fn.cold.6+0x62/0x10a [13964.152965] ? __hrtimer_run_queues+0x1c2/0x4d0 [13964.155192] ? watchdog+0x40/0x40 [13964.160984] ? hrtimer_interrupt+0x128/0x2c0 [13964.164563] ? smp_apic_timer_interrupt+0xa6/0x270 [13964.167803] ? apic_timer_interrupt+0xf/0x20 [13964.170654] [13964.172007] ? native_safe_halt+0x2e/0x50 [13964.174493] kvm_wait+0xcd/0xe0 [13964.176357] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13964.179142] do_raw_spin_lock+0x177/0x190 [13964.181231] _raw_spin_lock+0x21/0x30 [13964.183874] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13964.187277] obd_mod_free+0x46/0x140 [ptlrpc] [13964.190330] mdc_commit_open+0x91/0xb0 [mdc] [13964.193253] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13964.196557] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13964.207677] after_reply+0xe2c/0x1780 [ptlrpc] [13964.215155] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13964.217436] ? __next_timer_interrupt+0x160/0x160 [13964.223898] ptlrpcd+0xa82/0xe40 [ptlrpc] [13964.229184] ? do_wait_intr+0xf0/0xf0 [13964.235368] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13964.241630] kthread+0x1d1/0x200 [13964.248621] ? set_kthread_struct+0x70/0x70 [13964.251500] ret_from_fork+0x1f/0x30 [13972.059081] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [13972.067825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13972.111643] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13972.127635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13972.143008] RIP: 0010:native_safe_halt+0x2e/0x50 [13972.150825] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13972.168678] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13972.171940] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13972.174194] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [13972.186454] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [13972.191687] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [13972.202079] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [13972.213739] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [13972.222285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13972.226426] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [13972.241949] Call Trace: [13972.242812] [13972.246736] ? show_regs.cold.9+0x22/0x2f [13972.249899] ? watchdog_timer_fn.cold.6+0x62/0x10a [13972.254125] ? __hrtimer_run_queues+0x1c2/0x4d0 [13972.255983] ? watchdog+0x40/0x40 [13972.263615] ? hrtimer_interrupt+0x128/0x2c0 [13972.268658] ? smp_apic_timer_interrupt+0xa6/0x270 [13972.273498] ? apic_timer_interrupt+0xf/0x20 [13972.278609] [13972.279642] ? native_safe_halt+0x2e/0x50 [13972.282591] kvm_wait+0xcd/0xe0 [13972.285282] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13972.288715] do_raw_spin_lock+0x177/0x190 [13972.290748] _raw_spin_lock+0x21/0x30 [13972.292626] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13972.296466] ? do_raw_spin_unlock+0x75/0x190 [13972.298589] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13972.303298] ? do_wait_intr+0xf0/0xf0 [13972.309295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13972.311858] mdc_close+0x53b/0xd70 [mdc] [13972.315717] lmv_close+0x18d/0x490 [lmv] [13972.318343] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13972.322279] ll_md_real_close+0xa5/0x2a0 [lustre] [13972.324763] ll_md_close+0x274/0x800 [lustre] [13972.327317] ll_file_release+0x128/0xc50 [lustre] [13972.330537] __fput+0x10d/0x440 [13972.332477] ____fput+0x12/0x20 [13972.333749] task_work_run+0xc6/0x110 [13972.335890] exit_to_usermode_loop+0x1dd/0x1f0 [13972.337615] do_syscall_64+0x3d6/0x440 [13972.338686] entry_SYSCALL_64_after_hwframe+0x49/0xae [13972.340614] RIP: 0033:0x7fa749552cd8 [13972.342410] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13972.356825] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13972.368437] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [13972.373863] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [13972.377600] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [13972.383860] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [13972.386111] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13984.057069] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [13984.066692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13984.104194] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13984.114813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13984.120836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13984.122948] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13984.143470] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13984.153315] RAX: 0000000000007e8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13984.157874] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [13984.165607] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [13984.168553] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [13984.175081] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [13984.182204] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [13984.189425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13984.195517] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [13984.204363] Call Trace: [13984.205241] [13984.205808] ? show_regs.cold.9+0x22/0x2f [13984.210261] ? watchdog_timer_fn.cold.6+0x62/0x10a [13984.225213] ? __hrtimer_run_queues+0x1c2/0x4d0 [13984.226787] ? watchdog+0x40/0x40 [13984.227890] ? hrtimer_interrupt+0x128/0x2c0 [13984.231459] ? smp_apic_timer_interrupt+0xa6/0x270 [13984.234021] ? apic_timer_interrupt+0xf/0x20 [13984.236238] [13984.236918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13984.241226] do_raw_spin_lock+0x177/0x190 [13984.244437] _raw_spin_lock+0x21/0x30 [13984.247838] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13984.254386] ? do_raw_spin_unlock+0x75/0x190 [13984.257223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13984.260629] ? do_wait_intr+0xf0/0xf0 [13984.264540] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13984.269396] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13984.272429] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13984.275688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.277977] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13984.281603] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13984.288166] mdc_intent_lock+0x353/0x7e0 [mdc] [13984.292108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.299863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13984.306878] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13984.308758] lmv_intent_open+0x338/0x1150 [lmv] [13984.314645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.318116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.321098] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13984.331595] ll_intent_lock+0xc5/0xaa0 [lustre] [13984.335651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.339562] ll_lookup_it+0xe0e/0x2270 [lustre] [13984.342621] ? do_raw_spin_unlock+0x75/0x190 [13984.344855] ? _raw_spin_unlock+0x12/0x30 [13984.349875] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13984.352549] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13984.357356] ll_atomic_open+0x36a/0xd70 [lustre] [13984.363343] lookup_open+0x356/0xb40 [13984.364586] do_last+0x313/0xfc0 [13984.368694] ? nd_jump_root+0xe5/0x160 [13984.374298] ? path_init+0x437/0x520 [13984.376980] path_openat+0xf7/0x500 [13984.379189] ? do_raw_spin_unlock+0x75/0x190 [13984.383345] do_filp_open+0x99/0x140 [13984.385411] ? getname_flags+0x6e/0x330 [13984.393884] ? __check_object_size+0xff/0x256 [13984.401189] ? do_raw_spin_unlock+0x75/0x190 [13984.405112] ? _raw_spin_unlock+0x12/0x30 [13984.409227] do_sys_openat2+0x2b4/0x410 [13984.410480] do_sys_open+0x73/0xa0 [13984.411253] __x64_sys_openat+0x24/0x30 [13984.412846] do_syscall_64+0xc1/0x440 [13984.413955] entry_SYSCALL_64_after_hwframe+0x49/0xae [13984.415965] RIP: 0033:0x7fa749552332 [13984.421536] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13984.434656] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13984.440109] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [13984.443149] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [13984.447525] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [13984.454198] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [13984.460042] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [13986.274039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12069s! [13986.277889] Showing busy workqueues and worker pools: [13986.285455] workqueue events: flags=0x0 [13986.287882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13986.298035] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [13986.308272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13986.315858] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13986.320780] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.323560] pending: kfree_rcu_monitor [13986.325497] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13986.329941] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [13986.333530] workqueue events_long: flags=0x0 [13986.337400] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.340586] pending: lu_objects_flush [obdclass] [13986.343576] workqueue events_unbound: flags=0x2 [13986.345522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [13986.347627] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13986.361470] workqueue events_power_efficient: flags=0x80 [13986.365940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.370317] pending: check_lifetime [13986.372387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.377938] pending: neigh_periodic_work [13986.379639] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.387223] pending: neigh_periodic_work [13986.391533] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.397110] pending: do_cache_clean [13986.399334] workqueue mm_percpu_wq: flags=0x8 [13986.404462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.408963] pending: vmstat_update [13986.409864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.412375] pending: vmstat_update [13986.413637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13986.417603] pending: vmstat_update [13986.419534] workqueue writeback: flags=0x4a [13986.422153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13986.427154] pending: wb_workfn, wb_workfn [13986.433419] workqueue xprtiod: flags=0xa [13986.435103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13986.437704] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [13986.442743] workqueue ipv6_addrconf: flags=0x40008 [13986.444876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13986.447772] pending: addrconf_verify_work [13986.449574] workqueue knbd0-recv: flags=0x1a [13986.451478] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13986.454657] in-flight: 651:recv_work [13986.456201] workqueue ptlrpc_pinger: flags=0x2 [13986.459244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13986.462405] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [13986.464990] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12069s workers=4 idle: 37 31846 28897 [13986.470515] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13976s workers=2 idle: 653 [13988.063801] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [13988.073050] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13988.123707] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13988.130428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13988.136833] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13988.141354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13988.148456] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13988.174742] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13988.188112] RAX: 0000000000007b0f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [13988.190994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [13988.197435] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [13988.202626] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [13988.209977] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [13988.217352] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [13988.227803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13988.229481] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [13988.238227] Call Trace: [13988.241442] [13988.242370] ? show_regs.cold.9+0x22/0x2f [13988.246442] ? watchdog_timer_fn.cold.6+0x62/0x10a [13988.250326] ? __hrtimer_run_queues+0x1c2/0x4d0 [13988.256553] ? watchdog+0x40/0x40 [13988.258208] ? hrtimer_interrupt+0x128/0x2c0 [13988.259972] ? smp_apic_timer_interrupt+0xa6/0x270 [13988.262648] ? apic_timer_interrupt+0xf/0x20 [13988.265562] [13988.266710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13988.269909] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13988.277387] do_raw_spin_lock+0x177/0x190 [13988.279582] _raw_spin_lock+0x21/0x30 [13988.285007] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13988.291346] process_one_work+0x2c8/0x700 [13988.294437] worker_thread+0x296/0x6e0 [13988.299778] ? rescuer_thread+0x570/0x570 [13988.303912] kthread+0x1d1/0x200 [13988.308014] ? set_kthread_struct+0x70/0x70 [13988.311650] ret_from_fork+0x1f/0x30 [13992.058550] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [13992.069254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13992.114821] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13992.120194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13992.123777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13992.126683] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13992.140377] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13992.149216] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [13992.158517] RDX: 0000000000000404 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [13992.168270] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [13992.177267] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [13992.184694] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13992.192544] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [13992.200206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13992.204216] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [13992.213640] Call Trace: [13992.217275] [13992.218850] ? show_regs.cold.9+0x22/0x2f [13992.222338] ? watchdog_timer_fn.cold.6+0x62/0x10a [13992.225538] ? __hrtimer_run_queues+0x1c2/0x4d0 [13992.229179] ? watchdog+0x40/0x40 [13992.230684] ? hrtimer_interrupt+0x128/0x2c0 [13992.233000] ? smp_apic_timer_interrupt+0xa6/0x270 [13992.236419] ? apic_timer_interrupt+0xf/0x20 [13992.245449] [13992.249370] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13992.260215] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13992.267139] do_raw_spin_lock+0x177/0x190 [13992.270132] _raw_spin_lock+0x21/0x30 [13992.272118] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13992.278903] obd_mod_free+0x46/0x140 [ptlrpc] [13992.285316] mdc_commit_open+0x91/0xb0 [mdc] [13992.289422] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13992.292694] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13992.299824] after_reply+0xe2c/0x1780 [ptlrpc] [13992.306234] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13992.312204] ? __next_timer_interrupt+0x160/0x160 [13992.317722] ptlrpcd+0xa82/0xe40 [ptlrpc] [13992.325645] ? do_wait_intr+0xf0/0xf0 [13992.329298] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13992.337556] kthread+0x1d1/0x200 [13992.340188] ? set_kthread_struct+0x70/0x70 [13992.351602] ret_from_fork+0x1f/0x30 [14000.059380] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14000.063300] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14000.096887] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14000.102908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14000.107401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14000.110995] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14000.117985] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14000.120938] RAX: 0000000000003c86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14000.123517] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14000.126188] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14000.129106] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14000.132026] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14000.134546] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14000.137894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14000.140538] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14000.143826] Call Trace: [14000.144729] [14000.145780] ? show_regs.cold.9+0x22/0x2f [14000.147428] ? watchdog_timer_fn.cold.6+0x62/0x10a [14000.149419] ? __hrtimer_run_queues+0x1c2/0x4d0 [14000.151282] ? watchdog+0x40/0x40 [14000.152511] ? hrtimer_interrupt+0x128/0x2c0 [14000.154340] ? smp_apic_timer_interrupt+0xa6/0x270 [14000.156305] ? apic_timer_interrupt+0xf/0x20 [14000.158081] [14000.159237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14000.165036] do_raw_spin_lock+0x177/0x190 [14000.168906] _raw_spin_lock+0x21/0x30 [14000.171200] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14000.178939] ? do_raw_spin_unlock+0x75/0x190 [14000.183155] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14000.188211] ? do_wait_intr+0xf0/0xf0 [14000.192700] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14000.197899] mdc_close+0x53b/0xd70 [mdc] [14000.201315] lmv_close+0x18d/0x490 [lmv] [14000.202850] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14000.206283] ll_md_real_close+0xa5/0x2a0 [lustre] [14000.210265] ll_md_close+0x274/0x800 [lustre] [14000.212839] ll_file_release+0x128/0xc50 [lustre] [14000.217184] __fput+0x10d/0x440 [14000.218223] ____fput+0x12/0x20 [14000.220216] task_work_run+0xc6/0x110 [14000.224369] exit_to_usermode_loop+0x1dd/0x1f0 [14000.226770] do_syscall_64+0x3d6/0x440 [14000.229854] entry_SYSCALL_64_after_hwframe+0x49/0xae [14000.232488] RIP: 0033:0x7fa749552cd8 [14000.234292] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14000.246614] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14000.251124] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14000.258721] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14000.264387] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14000.268201] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14000.273528] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14012.054030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14012.059202] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14012.093237] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14012.100371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14012.105674] RIP: 0010:native_safe_halt+0x2e/0x50 [14012.108135] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14012.118125] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14012.122026] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14012.125343] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14012.128758] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14012.132049] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14012.135186] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14012.138360] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14012.142072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14012.144651] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14012.148333] Call Trace: [14012.150124] [14012.151232] ? show_regs.cold.9+0x22/0x2f [14012.153291] ? watchdog_timer_fn.cold.6+0x62/0x10a [14012.155515] ? __hrtimer_run_queues+0x1c2/0x4d0 [14012.157747] ? watchdog+0x40/0x40 [14012.159348] ? hrtimer_interrupt+0x128/0x2c0 [14012.161866] ? smp_apic_timer_interrupt+0xa6/0x270 [14012.164276] ? apic_timer_interrupt+0xf/0x20 [14012.166182] [14012.167373] ? native_safe_halt+0x2e/0x50 [14012.169329] kvm_wait+0xcd/0xe0 [14012.171020] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14012.173201] do_raw_spin_lock+0x177/0x190 [14012.175023] _raw_spin_lock+0x21/0x30 [14012.176636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14012.179293] ? do_raw_spin_unlock+0x75/0x190 [14012.181365] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14012.184104] ? do_wait_intr+0xf0/0xf0 [14012.185667] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14012.188426] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14012.191385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14012.194344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.196924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14012.199275] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14012.201372] mdc_intent_lock+0x353/0x7e0 [mdc] [14012.203096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.205474] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14012.208523] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14012.210847] lmv_intent_open+0x338/0x1150 [lmv] [14012.212490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.214596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.216653] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14012.218752] ll_intent_lock+0xc5/0xaa0 [lustre] [14012.220965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.223177] ll_lookup_it+0xe0e/0x2270 [lustre] [14012.225096] ? do_raw_spin_unlock+0x75/0x190 [14012.226650] ? _raw_spin_unlock+0x12/0x30 [14012.228560] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14012.230405] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14012.232231] ll_atomic_open+0x36a/0xd70 [lustre] [14012.234012] lookup_open+0x356/0xb40 [14012.235297] do_last+0x313/0xfc0 [14012.236484] ? nd_jump_root+0xe5/0x160 [14012.238199] ? path_init+0x437/0x520 [14012.239601] path_openat+0xf7/0x500 [14012.240834] ? do_raw_spin_unlock+0x75/0x190 [14012.242492] do_filp_open+0x99/0x140 [14012.243914] ? getname_flags+0x6e/0x330 [14012.245142] ? __check_object_size+0xff/0x256 [14012.246587] ? do_raw_spin_unlock+0x75/0x190 [14012.249307] ? _raw_spin_unlock+0x12/0x30 [14012.251139] do_sys_openat2+0x2b4/0x410 [14012.252893] do_sys_open+0x73/0xa0 [14012.254018] __x64_sys_openat+0x24/0x30 [14012.255736] do_syscall_64+0xc1/0x440 [14012.257112] entry_SYSCALL_64_after_hwframe+0x49/0xae [14012.258677] RIP: 0033:0x7fa749552332 [14012.260450] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14012.267166] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14012.270528] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14012.273453] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14012.276090] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14012.278714] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14012.281229] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14016.057834] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14016.062888] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14016.122892] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14016.133827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14016.139990] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14016.149225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14016.152488] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14016.163121] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14016.170526] RAX: 0000000000007a1a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14016.172950] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14016.177082] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14016.187019] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14016.194327] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14016.197025] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14016.203848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14016.205661] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14016.213101] Call Trace: [14016.214330] [14016.215057] ? show_regs.cold.9+0x22/0x2f [14016.217354] ? watchdog_timer_fn.cold.6+0x62/0x10a [14016.219592] ? __hrtimer_run_queues+0x1c2/0x4d0 [14016.221607] ? watchdog+0x40/0x40 [14016.223627] ? hrtimer_interrupt+0x128/0x2c0 [14016.231075] ? smp_apic_timer_interrupt+0xa6/0x270 [14016.234182] ? apic_timer_interrupt+0xf/0x20 [14016.237575] [14016.238319] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14016.241871] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14016.246611] do_raw_spin_lock+0x177/0x190 [14016.250846] _raw_spin_lock+0x21/0x30 [14016.252732] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14016.257758] process_one_work+0x2c8/0x700 [14016.260638] worker_thread+0x296/0x6e0 [14016.263495] ? rescuer_thread+0x570/0x570 [14016.265262] kthread+0x1d1/0x200 [14016.267129] ? set_kthread_struct+0x70/0x70 [14016.269181] ret_from_fork+0x1f/0x30 [14016.480050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12099s! [14016.494039] Showing busy workqueues and worker pools: [14016.499723] workqueue events: flags=0x0 [14016.504405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14016.509265] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14016.516552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14016.520624] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14016.528892] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.532608] pending: kfree_rcu_monitor [14016.536764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14016.541552] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14016.547543] workqueue events_long: flags=0x0 [14016.549218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.554525] pending: lu_objects_flush [obdclass] [14016.558573] workqueue events_unbound: flags=0x2 [14016.562732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14016.566699] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14016.575979] workqueue events_power_efficient: flags=0x80 [14016.583598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.590908] pending: check_lifetime [14016.593588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.601060] pending: neigh_periodic_work [14016.605812] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.612933] pending: neigh_periodic_work [14016.615708] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.619970] pending: do_cache_clean [14016.623755] workqueue mm_percpu_wq: flags=0x8 [14016.626938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.631223] pending: vmstat_update [14016.632726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.637577] pending: vmstat_update [14016.638796] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.647314] pending: vmstat_update [14016.651409] workqueue writeback: flags=0x4a [14016.653709] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14016.665878] pending: wb_workfn, wb_workfn [14016.667281] workqueue xprtiod: flags=0xa [14016.668366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14016.673592] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14016.677708] workqueue ipv6_addrconf: flags=0x40008 [14016.687109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14016.698686] pending: addrconf_verify_work [14016.700223] workqueue knbd0-recv: flags=0x1a [14016.703845] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14016.706923] in-flight: 651:recv_work [14016.708847] workqueue ptlrpc_pinger: flags=0x2 [14016.711593] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14016.715676] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14016.725465] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12099s workers=4 idle: 37 31846 28897 [14016.734756] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14006s workers=2 idle: 653 [14020.058022] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14020.067552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14020.099808] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14020.104728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14020.108977] RIP: 0010:native_safe_halt+0x2e/0x50 [14020.110958] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14020.121191] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14020.127972] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14020.134272] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14020.138265] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14020.141321] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14020.144731] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14020.147441] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14020.150778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14020.153322] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14020.157819] Call Trace: [14020.159052] [14020.160811] ? show_regs.cold.9+0x22/0x2f [14020.162719] ? watchdog_timer_fn.cold.6+0x62/0x10a [14020.164994] ? __hrtimer_run_queues+0x1c2/0x4d0 [14020.167281] ? watchdog+0x40/0x40 [14020.168748] ? hrtimer_interrupt+0x128/0x2c0 [14020.170899] ? smp_apic_timer_interrupt+0xa6/0x270 [14020.172983] ? apic_timer_interrupt+0xf/0x20 [14020.175488] [14020.176497] ? native_safe_halt+0x2e/0x50 [14020.178733] kvm_wait+0xcd/0xe0 [14020.180442] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14020.183288] do_raw_spin_lock+0x177/0x190 [14020.185708] _raw_spin_lock+0x21/0x30 [14020.187795] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14020.190602] obd_mod_free+0x46/0x140 [ptlrpc] [14020.193888] mdc_commit_open+0x91/0xb0 [mdc] [14020.197705] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14020.200802] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14020.205015] after_reply+0xe2c/0x1780 [ptlrpc] [14020.208412] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14020.212995] ? __next_timer_interrupt+0x160/0x160 [14020.221250] ptlrpcd+0xa82/0xe40 [ptlrpc] [14020.226375] ? do_wait_intr+0xf0/0xf0 [14020.229475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14020.233223] kthread+0x1d1/0x200 [14020.235154] ? set_kthread_struct+0x70/0x70 [14020.238417] ret_from_fork+0x1f/0x30 [14028.059027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14028.062214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14028.100795] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14028.108668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14028.112291] RIP: 0010:native_safe_halt+0x2e/0x50 [14028.114007] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14028.120802] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14028.123156] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14028.132519] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14028.135178] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14028.138291] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14028.141246] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14028.149499] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14028.152284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14028.154435] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14028.157429] Call Trace: [14028.158413] [14028.159247] ? show_regs.cold.9+0x22/0x2f [14028.160722] ? watchdog_timer_fn.cold.6+0x62/0x10a [14028.162663] ? __hrtimer_run_queues+0x1c2/0x4d0 [14028.164487] ? watchdog+0x40/0x40 [14028.165704] ? hrtimer_interrupt+0x128/0x2c0 [14028.166769] ? smp_apic_timer_interrupt+0xa6/0x270 [14028.168004] ? apic_timer_interrupt+0xf/0x20 [14028.168970] [14028.169849] ? native_safe_halt+0x2e/0x50 [14028.170806] kvm_wait+0xcd/0xe0 [14028.171585] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14028.172753] do_raw_spin_lock+0x177/0x190 [14028.174437] _raw_spin_lock+0x21/0x30 [14028.175481] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14028.177681] ? do_raw_spin_unlock+0x75/0x190 [14028.180573] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14028.184058] ? do_wait_intr+0xf0/0xf0 [14028.186350] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14028.189330] mdc_close+0x53b/0xd70 [mdc] [14028.190826] lmv_close+0x18d/0x490 [lmv] [14028.196095] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14028.198855] ll_md_real_close+0xa5/0x2a0 [lustre] [14028.200795] ll_md_close+0x274/0x800 [lustre] [14028.202641] ll_file_release+0x128/0xc50 [lustre] [14028.205350] __fput+0x10d/0x440 [14028.206816] ____fput+0x12/0x20 [14028.208119] task_work_run+0xc6/0x110 [14028.208989] exit_to_usermode_loop+0x1dd/0x1f0 [14028.210591] do_syscall_64+0x3d6/0x440 [14028.212200] entry_SYSCALL_64_after_hwframe+0x49/0xae [14028.214251] RIP: 0033:0x7fa749552cd8 [14028.216203] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14028.228134] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14028.231670] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14028.237144] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14028.241963] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14028.247264] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14028.249480] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14033.954060] rcu: INFO: rcu_sched self-detected stall on CPU [14033.957026] rcu: 2-...!: (6913491 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [14033.962972] (t=12120265 jiffies g=258701 q=4845) [14033.965613] rcu: rcu_sched kthread starved for 12117060 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [14033.970633] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14033.974897] rcu: RCU grace-period kthread stack dump: [14033.977181] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14033.986815] Call Trace: [14033.990926] __schedule+0x351/0xcb0 [14033.992958] schedule+0xc0/0x180 [14033.994722] schedule_timeout+0xb4/0x190 [14033.997306] ? do_raw_spin_unlock+0x75/0x190 [14034.005294] ? __next_timer_interrupt+0x160/0x160 [14034.007629] ? prepare_to_swait_event+0xf3/0x140 [14034.010500] rcu_gp_kthread+0x8dd/0x15b0 [14034.013042] ? force_qs_rnp+0x2c0/0x2c0 [14034.015011] kthread+0x1d1/0x200 [14034.016988] ? set_kthread_struct+0x70/0x70 [14034.019776] ret_from_fork+0x1f/0x30 [14034.025104] rcu: Stack dump where RCU GP kthread last ran: [14034.032583] NMI backtrace for cpu 2 [14034.033708] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14034.042145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14034.048945] Call Trace: [14034.052090] [14034.055221] ? dump_stack+0xbb/0x10e [14034.066353] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14034.072014] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14034.074742] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14034.077928] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14034.087907] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14034.094745] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [14034.099962] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [14034.106698] ? update_process_times+0x6d/0xc0 [14034.113633] ? tick_sched_handle.isra.18+0x27/0x90 [14034.118495] ? tick_sched_timer+0x53/0xb0 [14034.120671] ? __hrtimer_run_queues+0x1c2/0x4d0 [14034.123211] ? tick_sched_handle.isra.18+0x90/0x90 [14034.128297] ? hrtimer_interrupt+0x128/0x2c0 [14034.138527] ? smp_apic_timer_interrupt+0xa6/0x270 [14034.145419] ? apic_timer_interrupt+0xf/0x20 [14034.147934] [14034.148604] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14034.152413] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14034.155682] ? do_raw_spin_lock+0x177/0x190 [14034.157243] ? _raw_spin_lock+0x21/0x30 [14034.158925] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14034.163725] ? obd_mod_free+0x46/0x140 [ptlrpc] [14034.166281] ? mdc_commit_open+0x91/0xb0 [mdc] [14034.170794] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14034.177284] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14034.182357] ? after_reply+0xe2c/0x1780 [ptlrpc] [14034.187378] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14034.189297] ? __next_timer_interrupt+0x160/0x160 [14034.190497] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14034.191984] ? do_wait_intr+0xf0/0xf0 [14034.193383] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14034.195739] ? kthread+0x1d1/0x200 [14034.197167] ? set_kthread_struct+0x70/0x70 [14034.198788] ? ret_from_fork+0x1f/0x30 [14034.200373] NMI backtrace for cpu 2 [14034.202400] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14034.207824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14034.211324] Call Trace: [14034.212677] [14034.214042] ? dump_stack+0xbb/0x10e [14034.215693] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14034.217649] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14034.219450] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14034.221613] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14034.223395] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14034.225068] ? rcu_dump_cpu_stacks+0x11b/0x171 [14034.226790] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14034.232984] ? update_process_times+0x6d/0xc0 [14034.237765] ? tick_sched_handle.isra.18+0x27/0x90 [14034.244020] ? tick_sched_timer+0x53/0xb0 [14034.247210] ? __hrtimer_run_queues+0x1c2/0x4d0 [14034.251014] ? tick_sched_handle.isra.18+0x90/0x90 [14034.254780] ? hrtimer_interrupt+0x128/0x2c0 [14034.257336] ? smp_apic_timer_interrupt+0xa6/0x270 [14034.260588] ? apic_timer_interrupt+0xf/0x20 [14034.266874] [14034.268106] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14034.278028] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14034.279742] ? do_raw_spin_lock+0x177/0x190 [14034.283690] ? _raw_spin_lock+0x21/0x30 [14034.285516] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14034.289752] ? obd_mod_free+0x46/0x140 [ptlrpc] [14034.292827] ? mdc_commit_open+0x91/0xb0 [mdc] [14034.298585] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14034.302077] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14034.306681] ? after_reply+0xe2c/0x1780 [ptlrpc] [14034.311194] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14034.313811] ? __next_timer_interrupt+0x160/0x160 [14034.320381] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14034.322747] ? do_wait_intr+0xf0/0xf0 [14034.326544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14034.328540] ? kthread+0x1d1/0x200 [14034.332394] ? set_kthread_struct+0x70/0x70 [14034.337751] ? ret_from_fork+0x1f/0x30 [14044.058391] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14044.058544] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14044.072752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14044.072841] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14044.072847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14044.072853] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14044.073151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14044.073162] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14044.073166] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14044.073171] RAX: 000000000000608f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14044.073173] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14044.073176] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14044.073178] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14044.073181] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14044.073184] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14044.073187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14044.073189] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14044.073199] Call Trace: [14044.073205] [14044.073211] ? show_regs.cold.9+0x22/0x2f [14044.073220] ? watchdog_timer_fn.cold.6+0x62/0x10a [14044.073226] ? __hrtimer_run_queues+0x1c2/0x4d0 [14044.073231] ? watchdog+0x40/0x40 [14044.073236] ? hrtimer_interrupt+0x128/0x2c0 [14044.073240] ? smp_apic_timer_interrupt+0xa6/0x270 [14044.073247] ? apic_timer_interrupt+0xf/0x20 [14044.073251] [14044.073254] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14044.073258] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14044.073263] do_raw_spin_lock+0x177/0x190 [14044.073268] _raw_spin_lock+0x21/0x30 [14044.073272] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14044.074057] process_one_work+0x2c8/0x700 [14044.077353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14044.077434] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14044.077442] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14044.077445] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14044.077460] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14044.077464] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14044.077469] RAX: 0000000000003708 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14044.077472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14044.077474] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14044.077476] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14044.077479] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14044.077482] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14044.077485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14044.077488] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14044.077496] Call Trace: [14044.077501] [14044.077507] ? show_regs.cold.9+0x22/0x2f [14044.077516] ? watchdog_timer_fn.cold.6+0x62/0x10a [14044.077523] ? __hrtimer_run_queues+0x1c2/0x4d0 [14044.077527] ? watchdog+0x40/0x40 [14044.077532] ? hrtimer_interrupt+0x128/0x2c0 [14044.077536] ? smp_apic_timer_interrupt+0xa6/0x270 [14044.077544] ? apic_timer_interrupt+0xf/0x20 [14044.077549] [14044.077552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14044.077558] do_raw_spin_lock+0x177/0x190 [14044.077563] _raw_spin_lock+0x21/0x30 [14044.077568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14044.078116] ? do_raw_spin_unlock+0x75/0x190 [14044.078122] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14044.078358] ? do_wait_intr+0xf0/0xf0 [14044.078366] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14044.078751] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14044.079161] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14044.079467] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14044.079845] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14044.079889] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14044.079920] mdc_intent_lock+0x353/0x7e0 [mdc] [14044.079960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14044.080100] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14044.103456] worker_thread+0x296/0x6e0 [14044.103483] ? rescuer_thread+0x570/0x570 [14044.103488] kthread+0x1d1/0x200 [14044.103493] ? set_kthread_struct+0x70/0x70 [14044.103501] ret_from_fork+0x1f/0x30 [14044.306288] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14044.308720] lmv_intent_open+0x338/0x1150 [lmv] [14044.311313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14044.316051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14044.318378] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14044.320630] ll_intent_lock+0xc5/0xaa0 [lustre] [14044.323332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14044.326285] ll_lookup_it+0xe0e/0x2270 [lustre] [14044.328895] ? do_raw_spin_unlock+0x75/0x190 [14044.330725] ? _raw_spin_unlock+0x12/0x30 [14044.332699] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14044.335889] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14044.338877] ll_atomic_open+0x36a/0xd70 [lustre] [14044.340613] lookup_open+0x356/0xb40 [14044.346019] do_last+0x313/0xfc0 [14044.347818] ? nd_jump_root+0xe5/0x160 [14044.349474] ? path_init+0x437/0x520 [14044.352830] path_openat+0xf7/0x500 [14044.354040] ? do_raw_spin_unlock+0x75/0x190 [14044.356603] do_filp_open+0x99/0x140 [14044.362909] ? getname_flags+0x6e/0x330 [14044.365875] ? __check_object_size+0xff/0x256 [14044.370917] ? do_raw_spin_unlock+0x75/0x190 [14044.372438] ? _raw_spin_unlock+0x12/0x30 [14044.389273] do_sys_openat2+0x2b4/0x410 [14044.395590] do_sys_open+0x73/0xa0 [14044.396841] __x64_sys_openat+0x24/0x30 [14044.398175] do_syscall_64+0xc1/0x440 [14044.408189] entry_SYSCALL_64_after_hwframe+0x49/0xae [14044.411252] RIP: 0033:0x7fa749552332 [14044.417361] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14044.449047] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14044.462017] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14044.470956] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14044.479391] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14044.484267] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14044.490054] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14047.201315] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12130s! [14047.205868] Showing busy workqueues and worker pools: [14047.208290] workqueue events: flags=0x0 [14047.210179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14047.214099] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14047.220246] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14047.224917] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14047.236961] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.240866] pending: kfree_rcu_monitor [14047.242802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14047.245459] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14047.265442] workqueue events_long: flags=0x0 [14047.266595] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.273350] pending: lu_objects_flush [obdclass] [14047.279011] workqueue events_unbound: flags=0x2 [14047.283636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14047.292743] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14047.300176] workqueue events_power_efficient: flags=0x80 [14047.305232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.310694] pending: check_lifetime [14047.311726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.322513] pending: neigh_periodic_work [14047.335200] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.337716] pending: neigh_periodic_work [14047.345200] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.351722] pending: do_cache_clean [14047.353932] workqueue mm_percpu_wq: flags=0x8 [14047.359670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.366804] pending: vmstat_update [14047.369988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.376456] pending: vmstat_update [14047.378917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14047.386945] pending: vmstat_update [14047.388447] workqueue writeback: flags=0x4a [14047.394184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14047.401249] pending: wb_workfn, wb_workfn [14047.407058] workqueue xprtiod: flags=0xa [14047.409231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14047.421037] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14047.424035] workqueue ipv6_addrconf: flags=0x40008 [14047.437947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14047.451465] pending: addrconf_verify_work [14047.453127] workqueue knbd0-recv: flags=0x1a [14047.467404] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14047.469744] in-flight: 651:recv_work [14047.484566] workqueue ptlrpc_pinger: flags=0x2 [14047.503774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14047.506621] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14047.522925] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12130s workers=4 idle: 37 31846 28897 [14047.537405] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14037s workers=2 idle: 653 [14056.059021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14056.067680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14056.104194] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14056.113886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14056.121757] RIP: 0010:native_safe_halt+0x2e/0x50 [14056.126778] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14056.144823] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14056.148769] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14056.154481] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14056.159884] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14056.165775] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14056.170086] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14056.174033] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14056.181149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14056.186191] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14056.191077] Call Trace: [14056.192605] [14056.193976] ? show_regs.cold.9+0x22/0x2f [14056.199672] ? watchdog_timer_fn.cold.6+0x62/0x10a [14056.202014] ? __hrtimer_run_queues+0x1c2/0x4d0 [14056.203955] ? watchdog+0x40/0x40 [14056.205189] ? hrtimer_interrupt+0x128/0x2c0 [14056.207599] ? smp_apic_timer_interrupt+0xa6/0x270 [14056.209823] ? apic_timer_interrupt+0xf/0x20 [14056.213594] [14056.215025] ? native_safe_halt+0x2e/0x50 [14056.218913] kvm_wait+0xcd/0xe0 [14056.221004] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14056.223507] do_raw_spin_lock+0x177/0x190 [14056.226061] _raw_spin_lock+0x21/0x30 [14056.228083] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14056.231923] ? do_raw_spin_unlock+0x75/0x190 [14056.233531] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14056.241137] ? do_wait_intr+0xf0/0xf0 [14056.243324] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14056.250286] mdc_close+0x53b/0xd70 [mdc] [14056.251785] lmv_close+0x18d/0x490 [lmv] [14056.255401] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14056.259973] ll_md_real_close+0xa5/0x2a0 [lustre] [14056.262955] ll_md_close+0x274/0x800 [lustre] [14056.265059] ll_file_release+0x128/0xc50 [lustre] [14056.266464] __fput+0x10d/0x440 [14056.267597] ____fput+0x12/0x20 [14056.269191] task_work_run+0xc6/0x110 [14056.270835] exit_to_usermode_loop+0x1dd/0x1f0 [14056.274292] do_syscall_64+0x3d6/0x440 [14056.276028] entry_SYSCALL_64_after_hwframe+0x49/0xae [14056.278990] RIP: 0033:0x7fa749552cd8 [14056.281061] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14056.291141] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14056.300279] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14056.305861] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14056.309552] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14056.317630] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14056.325798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14060.065105] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14060.075942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14060.147010] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14060.163221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14060.182179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14060.190934] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14060.213445] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14060.223135] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14060.238995] RDX: 00000000000076a5 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14060.244253] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14060.253997] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14060.262554] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14060.278402] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14060.281240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14060.294357] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14060.304672] Call Trace: [14060.307319] [14060.313263] ? show_regs.cold.9+0x22/0x2f [14060.323238] ? watchdog_timer_fn.cold.6+0x62/0x10a [14060.330228] ? __hrtimer_run_queues+0x1c2/0x4d0 [14060.344306] ? watchdog+0x40/0x40 [14060.346176] ? hrtimer_interrupt+0x128/0x2c0 [14060.348598] ? smp_apic_timer_interrupt+0xa6/0x270 [14060.351058] ? apic_timer_interrupt+0xf/0x20 [14060.353276] [14060.354608] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14060.357674] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14060.360639] do_raw_spin_lock+0x177/0x190 [14060.363038] _raw_spin_lock+0x21/0x30 [14060.365164] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14060.367809] obd_mod_free+0x46/0x140 [ptlrpc] [14060.370499] mdc_commit_open+0x91/0xb0 [mdc] [14060.372728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14060.375595] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14060.378362] after_reply+0xe2c/0x1780 [ptlrpc] [14060.381196] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14060.384573] ? __next_timer_interrupt+0x160/0x160 [14060.387338] ptlrpcd+0xa82/0xe40 [ptlrpc] [14060.390078] ? do_wait_intr+0xf0/0xf0 [14060.392302] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14060.395338] kthread+0x1d1/0x200 [14060.397429] ? set_kthread_struct+0x70/0x70 [14060.399675] ret_from_fork+0x1f/0x30 [14072.054327] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14072.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14072.057251] Modules linked in: lustre(O) [14072.062007] Modules linked in: lustre(O) osp(O) ofd(O) [14072.063560] osp(O) ofd(O) lod(O) [14072.066695] lod(O) mdt(O) mdd(O) [14072.068535] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14072.068603] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14072.068611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14072.068614] RIP: 0010:native_safe_halt+0x2e/0x50 [14072.068627] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14072.068631] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14072.068635] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14072.068637] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14072.068639] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14072.068642] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14072.068644] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14072.068647] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14072.068650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14072.068652] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14072.068659] Call Trace: [14072.068664] [14072.068669] ? show_regs.cold.9+0x22/0x2f [14072.068677] ? watchdog_timer_fn.cold.6+0x62/0x10a [14072.068682] ? __hrtimer_run_queues+0x1c2/0x4d0 [14072.068686] ? watchdog+0x40/0x40 [14072.068690] ? hrtimer_interrupt+0x128/0x2c0 [14072.068694] ? smp_apic_timer_interrupt+0xa6/0x270 [14072.068700] ? apic_timer_interrupt+0xf/0x20 [14072.068704] [14072.068706] ? native_safe_halt+0x2e/0x50 [14072.068710] kvm_wait+0xcd/0xe0 [14072.068716] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14072.068725] do_raw_spin_lock+0x177/0x190 [14072.068729] _raw_spin_lock+0x21/0x30 [14072.068733] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14072.070813] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14072.070875] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14072.070881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14072.070887] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14072.071472] RIP: 0010:native_safe_halt+0x2e/0x50 [14072.071483] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14072.071487] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14072.071491] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14072.071493] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14072.071496] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14072.071498] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14072.071500] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14072.071504] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14072.071506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14072.071508] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14072.071515] Call Trace: [14072.071519] [14072.071524] ? show_regs.cold.9+0x22/0x2f [14072.071533] ? watchdog_timer_fn.cold.6+0x62/0x10a [14072.071538] ? __hrtimer_run_queues+0x1c2/0x4d0 [14072.071542] ? watchdog+0x40/0x40 [14072.071545] ? hrtimer_interrupt+0x128/0x2c0 [14072.071549] ? smp_apic_timer_interrupt+0xa6/0x270 [14072.071556] ? apic_timer_interrupt+0xf/0x20 [14072.071560] [14072.071564] ? native_safe_halt+0x2e/0x50 [14072.071568] kvm_wait+0xcd/0xe0 [14072.071577] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14072.071585] do_raw_spin_lock+0x177/0x190 [14072.071590] _raw_spin_lock+0x21/0x30 [14072.071594] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14072.072163] process_one_work+0x2c8/0x700 [14072.072170] worker_thread+0x296/0x6e0 [14072.072174] ? rescuer_thread+0x570/0x570 [14072.072178] kthread+0x1d1/0x200 [14072.090645] ? do_raw_spin_unlock+0x75/0x190 [14072.090674] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14072.095030] ? set_kthread_struct+0x70/0x70 [14072.095145] ret_from_fork+0x1f/0x30 [14072.272652] ? do_wait_intr+0xf0/0xf0 [14072.275970] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14072.281016] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14072.289889] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14072.294964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14072.300542] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14072.305656] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14072.309966] mdc_intent_lock+0x353/0x7e0 [mdc] [14072.313849] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14072.316486] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14072.322974] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14072.327208] lmv_intent_open+0x338/0x1150 [lmv] [14072.329096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14072.335770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14072.340285] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14072.343833] ll_intent_lock+0xc5/0xaa0 [lustre] [14072.346160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14072.348143] ll_lookup_it+0xe0e/0x2270 [lustre] [14072.352281] ? do_raw_spin_unlock+0x75/0x190 [14072.354259] ? _raw_spin_unlock+0x12/0x30 [14072.358467] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14072.362392] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14072.365761] ll_atomic_open+0x36a/0xd70 [lustre] [14072.371620] lookup_open+0x356/0xb40 [14072.373730] do_last+0x313/0xfc0 [14072.375708] ? nd_jump_root+0xe5/0x160 [14072.377280] ? path_init+0x437/0x520 [14072.378865] path_openat+0xf7/0x500 [14072.380499] ? do_raw_spin_unlock+0x75/0x190 [14072.382153] do_filp_open+0x99/0x140 [14072.388255] ? getname_flags+0x6e/0x330 [14072.391836] ? __check_object_size+0xff/0x256 [14072.393234] ? do_raw_spin_unlock+0x75/0x190 [14072.395758] ? _raw_spin_unlock+0x12/0x30 [14072.399509] do_sys_openat2+0x2b4/0x410 [14072.401404] do_sys_open+0x73/0xa0 [14072.402358] __x64_sys_openat+0x24/0x30 [14072.403715] do_syscall_64+0xc1/0x440 [14072.405133] entry_SYSCALL_64_after_hwframe+0x49/0xae [14072.407005] RIP: 0033:0x7fa749552332 [14072.408612] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14072.414574] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14072.416723] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14072.419561] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14072.421967] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14072.423952] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14072.426115] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14077.925458] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12160s! [14077.931262] Showing busy workqueues and worker pools: [14077.934382] workqueue events: flags=0x0 [14077.935833] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14077.939056] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14077.945962] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14077.948730] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14077.953049] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14077.956569] pending: kfree_rcu_monitor [14077.958736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14077.961640] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14077.965605] workqueue events_long: flags=0x0 [14077.967609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14077.973700] pending: lu_objects_flush [obdclass] [14077.976696] workqueue events_unbound: flags=0x2 [14077.979633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14077.986567] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14077.991436] workqueue events_power_efficient: flags=0x80 [14077.996030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14078.004586] pending: check_lifetime [14078.007891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14078.012399] pending: neigh_periodic_work [14078.018752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14078.026687] pending: neigh_periodic_work [14078.033703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14078.036236] pending: do_cache_clean [14078.044291] workqueue mm_percpu_wq: flags=0x8 [14078.045988] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14078.052218] pending: vmstat_update [14078.065543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14078.072670] pending: vmstat_update [14078.073978] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14078.077502] pending: vmstat_update [14078.078856] workqueue writeback: flags=0x4a [14078.083294] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14078.085858] pending: wb_workfn, wb_workfn [14078.089136] workqueue xprtiod: flags=0xa [14078.091507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14078.094794] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14078.099814] workqueue ipv6_addrconf: flags=0x40008 [14078.101566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14078.110428] pending: addrconf_verify_work [14078.112655] workqueue knbd0-recv: flags=0x1a [14078.114646] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14078.117879] in-flight: 651:recv_work [14078.119800] workqueue ptlrpc_pinger: flags=0x2 [14078.123132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14078.126783] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14078.129593] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12160s workers=4 idle: 37 31846 28897 [14078.134047] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14068s workers=2 idle: 653 [14084.061547] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14084.069728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14084.105352] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14084.110522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14084.116285] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14084.119439] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14084.130008] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14084.133405] RAX: 00000000000076ed RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14084.137194] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14084.141605] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14084.144884] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14084.148838] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14084.152881] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14084.159446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14084.161861] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14084.171414] Call Trace: [14084.172448] [14084.173789] ? show_regs.cold.9+0x22/0x2f [14084.178170] ? watchdog_timer_fn.cold.6+0x62/0x10a [14084.179921] ? __hrtimer_run_queues+0x1c2/0x4d0 [14084.182292] ? watchdog+0x40/0x40 [14084.183862] ? hrtimer_interrupt+0x128/0x2c0 [14084.190440] ? smp_apic_timer_interrupt+0xa6/0x270 [14084.191828] ? apic_timer_interrupt+0xf/0x20 [14084.192942] [14084.198098] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14084.201462] do_raw_spin_lock+0x177/0x190 [14084.203439] _raw_spin_lock+0x21/0x30 [14084.206716] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14084.209191] ? do_raw_spin_unlock+0x75/0x190 [14084.210924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14084.216861] ? do_wait_intr+0xf0/0xf0 [14084.220322] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14084.222783] mdc_close+0x53b/0xd70 [mdc] [14084.234363] lmv_close+0x18d/0x490 [lmv] [14084.239419] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14084.245002] ll_md_real_close+0xa5/0x2a0 [lustre] [14084.247781] ll_md_close+0x274/0x800 [lustre] [14084.249958] ll_file_release+0x128/0xc50 [lustre] [14084.252713] __fput+0x10d/0x440 [14084.254187] ____fput+0x12/0x20 [14084.255982] task_work_run+0xc6/0x110 [14084.257637] exit_to_usermode_loop+0x1dd/0x1f0 [14084.259911] do_syscall_64+0x3d6/0x440 [14084.261780] entry_SYSCALL_64_after_hwframe+0x49/0xae [14084.264236] RIP: 0033:0x7fa749552cd8 [14084.266113] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14084.274413] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14084.277811] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14084.280976] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14084.284116] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14084.287339] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14084.290551] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14088.058589] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14088.064455] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14088.107227] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14088.123761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14088.130582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14088.134106] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14088.146458] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14088.155316] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14088.160838] RDX: 000000000000385f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14088.172620] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14088.183202] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14088.192561] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14088.200606] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14088.211361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14088.224819] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14088.234908] Call Trace: [14088.235796] [14088.236537] ? show_regs.cold.9+0x22/0x2f [14088.243612] ? watchdog_timer_fn.cold.6+0x62/0x10a [14088.246390] ? __hrtimer_run_queues+0x1c2/0x4d0 [14088.249685] ? watchdog+0x40/0x40 [14088.254496] ? hrtimer_interrupt+0x128/0x2c0 [14088.260081] ? smp_apic_timer_interrupt+0xa6/0x270 [14088.262860] ? apic_timer_interrupt+0xf/0x20 [14088.265261] [14088.266013] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14088.278071] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14088.279916] do_raw_spin_lock+0x177/0x190 [14088.283442] _raw_spin_lock+0x21/0x30 [14088.285648] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14088.291576] obd_mod_free+0x46/0x140 [ptlrpc] [14088.299010] mdc_commit_open+0x91/0xb0 [mdc] [14088.303006] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14088.309594] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14088.314094] after_reply+0xe2c/0x1780 [ptlrpc] [14088.316782] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14088.325009] ? __next_timer_interrupt+0x160/0x160 [14088.329212] ptlrpcd+0xa82/0xe40 [ptlrpc] [14088.332453] ? do_wait_intr+0xf0/0xf0 [14088.334215] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14088.338699] kthread+0x1d1/0x200 [14088.341618] ? set_kthread_struct+0x70/0x70 [14088.345797] ret_from_fork+0x1f/0x30 [14100.057025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14100.062428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14100.092117] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14100.098998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14100.106915] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14100.113165] RIP: 0010:native_safe_halt+0x2e/0x50 [14100.115993] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14100.143580] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14100.151708] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14100.156824] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14100.161400] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14100.164881] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14100.170341] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14100.173084] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14100.175617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14100.177865] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14100.181949] Call Trace: [14100.184420] [14100.185456] ? show_regs.cold.9+0x22/0x2f [14100.188803] ? watchdog_timer_fn.cold.6+0x62/0x10a [14100.192538] ? __hrtimer_run_queues+0x1c2/0x4d0 [14100.196456] ? watchdog+0x40/0x40 [14100.201755] ? hrtimer_interrupt+0x128/0x2c0 [14100.204022] ? smp_apic_timer_interrupt+0xa6/0x270 [14100.209799] ? apic_timer_interrupt+0xf/0x20 [14100.215109] [14100.217313] ? native_safe_halt+0x2e/0x50 [14100.223255] kvm_wait+0xcd/0xe0 [14100.224851] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14100.231368] do_raw_spin_lock+0x177/0x190 [14100.233359] _raw_spin_lock+0x21/0x30 [14100.235833] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14100.239746] process_one_work+0x2c8/0x700 [14100.241890] worker_thread+0x296/0x6e0 [14100.244422] ? rescuer_thread+0x570/0x570 [14100.247395] kthread+0x1d1/0x200 [14100.248548] ? set_kthread_struct+0x70/0x70 [14100.249639] ret_from_fork+0x1f/0x30 [14104.054081] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14104.057391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14104.074859] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14104.079334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14104.088112] RIP: 0010:native_safe_halt+0x2e/0x50 [14104.090920] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14104.098729] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14104.101747] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14104.104305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14104.107077] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14104.109827] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14104.112638] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14104.115650] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14104.118727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14104.120833] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14104.123567] Call Trace: [14104.124454] [14104.125373] ? show_regs.cold.9+0x22/0x2f [14104.127004] ? watchdog_timer_fn.cold.6+0x62/0x10a [14104.128875] ? __hrtimer_run_queues+0x1c2/0x4d0 [14104.130801] ? watchdog+0x40/0x40 [14104.132102] ? hrtimer_interrupt+0x128/0x2c0 [14104.133492] ? smp_apic_timer_interrupt+0xa6/0x270 [14104.135363] ? apic_timer_interrupt+0xf/0x20 [14104.136951] [14104.137938] ? native_safe_halt+0x2e/0x50 [14104.139606] kvm_wait+0xcd/0xe0 [14104.141325] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14104.143237] do_raw_spin_lock+0x177/0x190 [14104.144541] _raw_spin_lock+0x21/0x30 [14104.146102] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14104.148283] ? do_raw_spin_unlock+0x75/0x190 [14104.149836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14104.152110] ? do_wait_intr+0xf0/0xf0 [14104.153815] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14104.156286] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14104.158634] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14104.161136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14104.163625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14104.166119] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14104.167709] mdc_intent_lock+0x353/0x7e0 [mdc] [14104.169369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14104.171288] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14104.173864] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14104.176062] lmv_intent_open+0x338/0x1150 [lmv] [14104.177809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14104.179906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14104.182461] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14104.184671] ll_intent_lock+0xc5/0xaa0 [lustre] [14104.186427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14104.188578] ll_lookup_it+0xe0e/0x2270 [lustre] [14104.191481] ? do_raw_spin_unlock+0x75/0x190 [14104.193230] ? _raw_spin_unlock+0x12/0x30 [14104.194787] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14104.196580] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14104.198519] ll_atomic_open+0x36a/0xd70 [lustre] [14104.200436] lookup_open+0x356/0xb40 [14104.201741] do_last+0x313/0xfc0 [14104.203142] ? nd_jump_root+0xe5/0x160 [14104.204514] ? path_init+0x437/0x520 [14104.205787] path_openat+0xf7/0x500 [14104.207008] ? do_raw_spin_unlock+0x75/0x190 [14104.208446] do_filp_open+0x99/0x140 [14104.209631] ? getname_flags+0x6e/0x330 [14104.211069] ? __check_object_size+0xff/0x256 [14104.212443] ? do_raw_spin_unlock+0x75/0x190 [14104.214068] ? _raw_spin_unlock+0x12/0x30 [14104.215757] do_sys_openat2+0x2b4/0x410 [14104.217228] do_sys_open+0x73/0xa0 [14104.218207] __x64_sys_openat+0x24/0x30 [14104.219619] do_syscall_64+0xc1/0x440 [14104.220964] entry_SYSCALL_64_after_hwframe+0x49/0xae [14104.222850] RIP: 0033:0x7fa749552332 [14104.224216] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14104.230610] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14104.233432] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14104.236101] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14104.238714] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14104.241943] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14104.244591] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14108.651995] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12191s! [14108.659343] Showing busy workqueues and worker pools: [14108.662430] workqueue events: flags=0x0 [14108.664497] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14108.672205] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14108.676892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14108.681801] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14108.688500] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.694217] pending: kfree_rcu_monitor [14108.697500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14108.700254] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14108.708098] workqueue events_long: flags=0x0 [14108.709557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.713123] pending: lu_objects_flush [obdclass] [14108.716713] workqueue events_unbound: flags=0x2 [14108.721704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14108.723420] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14108.729504] workqueue events_power_efficient: flags=0x80 [14108.731461] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.734753] pending: check_lifetime [14108.742840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.750132] pending: neigh_periodic_work [14108.751422] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.755089] pending: neigh_periodic_work [14108.757693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.761291] pending: do_cache_clean [14108.763594] workqueue mm_percpu_wq: flags=0x8 [14108.766235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.774112] pending: vmstat_update [14108.777256] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.780920] pending: vmstat_update [14108.783254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14108.786799] pending: vmstat_update [14108.789104] workqueue writeback: flags=0x4a [14108.794381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14108.796930] pending: wb_workfn, wb_workfn [14108.801658] workqueue xprtiod: flags=0xa [14108.804910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14108.811782] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14108.817754] workqueue ipv6_addrconf: flags=0x40008 [14108.820086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14108.822109] pending: addrconf_verify_work [14108.823382] workqueue knbd0-recv: flags=0x1a [14108.833507] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14108.839710] in-flight: 651:recv_work [14108.842403] workqueue ptlrpc_pinger: flags=0x2 [14108.844406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14108.850465] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14108.866455] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12191s workers=4 idle: 37 31846 28897 [14108.873846] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14098s workers=2 idle: 653 [14112.059511] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14112.070196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14112.095934] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14112.101099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14112.104143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14112.106639] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14112.113883] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14112.117148] RAX: 0000000000006489 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14112.121012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14112.123693] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14112.126133] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14112.134825] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14112.141731] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14112.152744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14112.157583] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14112.169134] Call Trace: [14112.169811] [14112.170505] ? show_regs.cold.9+0x22/0x2f [14112.177968] ? watchdog_timer_fn.cold.6+0x62/0x10a [14112.180265] ? __hrtimer_run_queues+0x1c2/0x4d0 [14112.182232] ? watchdog+0x40/0x40 [14112.188440] ? hrtimer_interrupt+0x128/0x2c0 [14112.192308] ? smp_apic_timer_interrupt+0xa6/0x270 [14112.196055] ? apic_timer_interrupt+0xf/0x20 [14112.198598] [14112.199894] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14112.201719] do_raw_spin_lock+0x177/0x190 [14112.204552] _raw_spin_lock+0x21/0x30 [14112.205740] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14112.209584] ? do_raw_spin_unlock+0x75/0x190 [14112.211210] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14112.213571] ? do_wait_intr+0xf0/0xf0 [14112.214946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14112.216991] mdc_close+0x53b/0xd70 [mdc] [14112.218704] lmv_close+0x18d/0x490 [lmv] [14112.219992] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14112.221778] ll_md_real_close+0xa5/0x2a0 [lustre] [14112.223428] ll_md_close+0x274/0x800 [lustre] [14112.225708] ll_file_release+0x128/0xc50 [lustre] [14112.228115] __fput+0x10d/0x440 [14112.229160] ____fput+0x12/0x20 [14112.230637] task_work_run+0xc6/0x110 [14112.232191] exit_to_usermode_loop+0x1dd/0x1f0 [14112.233922] do_syscall_64+0x3d6/0x440 [14112.235557] entry_SYSCALL_64_after_hwframe+0x49/0xae [14112.237266] RIP: 0033:0x7fa749552cd8 [14112.239116] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14112.247189] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14112.253349] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14112.257062] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14112.264400] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14112.266794] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14112.270006] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14116.058027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14116.063290] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14116.101737] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14116.107598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14116.112258] RIP: 0010:native_safe_halt+0x2e/0x50 [14116.114390] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14116.128767] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14116.136579] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14116.142060] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14116.146748] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14116.149876] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14116.153820] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14116.159181] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14116.168897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14116.175529] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14116.182163] Call Trace: [14116.184259] [14116.185696] ? show_regs.cold.9+0x22/0x2f [14116.187680] ? watchdog_timer_fn.cold.6+0x62/0x10a [14116.190126] ? __hrtimer_run_queues+0x1c2/0x4d0 [14116.193560] ? watchdog+0x40/0x40 [14116.194719] ? hrtimer_interrupt+0x128/0x2c0 [14116.197730] ? smp_apic_timer_interrupt+0xa6/0x270 [14116.206531] ? apic_timer_interrupt+0xf/0x20 [14116.211325] [14116.213330] ? native_safe_halt+0x2e/0x50 [14116.215575] kvm_wait+0xcd/0xe0 [14116.216939] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14116.223657] do_raw_spin_lock+0x177/0x190 [14116.225055] _raw_spin_lock+0x21/0x30 [14116.231562] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14116.236528] obd_mod_free+0x46/0x140 [ptlrpc] [14116.239581] mdc_commit_open+0x91/0xb0 [mdc] [14116.243768] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14116.249964] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14116.255718] after_reply+0xe2c/0x1780 [ptlrpc] [14116.258109] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14116.261344] ? __next_timer_interrupt+0x160/0x160 [14116.264722] ptlrpcd+0xa82/0xe40 [ptlrpc] [14116.267918] ? do_wait_intr+0xf0/0xf0 [14116.269592] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14116.275153] kthread+0x1d1/0x200 [14116.277544] ? set_kthread_struct+0x70/0x70 [14116.279937] ret_from_fork+0x1f/0x30 [14128.057097] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14128.062090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14128.107609] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14128.123132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14128.137284] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14128.142585] RIP: 0010:native_safe_halt+0x2e/0x50 [14128.149054] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14128.166543] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14128.170603] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14128.175073] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14128.178999] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14128.182392] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14128.186156] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14128.188912] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14128.192644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14128.195986] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14128.199828] Call Trace: [14128.201024] [14128.201958] ? show_regs.cold.9+0x22/0x2f [14128.203700] ? watchdog_timer_fn.cold.6+0x62/0x10a [14128.206490] ? __hrtimer_run_queues+0x1c2/0x4d0 [14128.209655] ? watchdog+0x40/0x40 [14128.211637] ? hrtimer_interrupt+0x128/0x2c0 [14128.213791] ? smp_apic_timer_interrupt+0xa6/0x270 [14128.216157] ? apic_timer_interrupt+0xf/0x20 [14128.218587] [14128.219675] ? native_safe_halt+0x2e/0x50 [14128.223985] kvm_wait+0xcd/0xe0 [14128.225669] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14128.232117] do_raw_spin_lock+0x177/0x190 [14128.235515] _raw_spin_lock+0x21/0x30 [14128.238684] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14128.244679] process_one_work+0x2c8/0x700 [14128.248779] worker_thread+0x296/0x6e0 [14128.253980] ? rescuer_thread+0x570/0x570 [14128.255871] kthread+0x1d1/0x200 [14128.257576] ? set_kthread_struct+0x70/0x70 [14128.259022] ret_from_fork+0x1f/0x30 [14136.054136] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14136.059800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14136.084037] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14136.089908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14136.093898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14136.096893] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14136.106347] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14136.110285] RAX: 0000000000002a8d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14136.117931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14136.125751] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14136.132661] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14136.136514] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14136.139232] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14136.149474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14136.155681] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14136.161400] Call Trace: [14136.164662] [14136.165241] ? show_regs.cold.9+0x22/0x2f [14136.168246] ? watchdog_timer_fn.cold.6+0x62/0x10a [14136.174814] ? __hrtimer_run_queues+0x1c2/0x4d0 [14136.176230] ? watchdog+0x40/0x40 [14136.177174] ? hrtimer_interrupt+0x128/0x2c0 [14136.180637] ? smp_apic_timer_interrupt+0xa6/0x270 [14136.184166] ? apic_timer_interrupt+0xf/0x20 [14136.187746] [14136.194808] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14136.201319] do_raw_spin_lock+0x177/0x190 [14136.203696] _raw_spin_lock+0x21/0x30 [14136.208750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14136.212730] ? do_raw_spin_unlock+0x75/0x190 [14136.215913] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14136.220602] ? do_wait_intr+0xf0/0xf0 [14136.221672] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14136.223129] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14136.226267] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14136.227916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14136.230170] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14136.232038] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14136.238473] mdc_intent_lock+0x353/0x7e0 [mdc] [14136.239735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14136.241953] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14136.244386] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14136.248843] lmv_intent_open+0x338/0x1150 [lmv] [14136.256861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14136.258742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14136.274026] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14136.279539] ll_intent_lock+0xc5/0xaa0 [lustre] [14136.285654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14136.293300] ll_lookup_it+0xe0e/0x2270 [lustre] [14136.299804] ? do_raw_spin_unlock+0x75/0x190 [14136.302967] ? _raw_spin_unlock+0x12/0x30 [14136.308753] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14136.310502] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14136.325724] ll_atomic_open+0x36a/0xd70 [lustre] [14136.330290] lookup_open+0x356/0xb40 [14136.332619] do_last+0x313/0xfc0 [14136.334740] ? nd_jump_root+0xe5/0x160 [14136.337015] ? path_init+0x437/0x520 [14136.340007] path_openat+0xf7/0x500 [14136.342162] ? do_raw_spin_unlock+0x75/0x190 [14136.343748] do_filp_open+0x99/0x140 [14136.345947] ? getname_flags+0x6e/0x330 [14136.348039] ? __check_object_size+0xff/0x256 [14136.352569] ? do_raw_spin_unlock+0x75/0x190 [14136.355025] ? _raw_spin_unlock+0x12/0x30 [14136.357338] do_sys_openat2+0x2b4/0x410 [14136.361742] do_sys_open+0x73/0xa0 [14136.366856] __x64_sys_openat+0x24/0x30 [14136.373399] do_syscall_64+0xc1/0x440 [14136.380718] entry_SYSCALL_64_after_hwframe+0x49/0xae [14136.387780] RIP: 0033:0x7fa749552332 [14136.391364] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14136.399539] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14136.402699] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14136.410178] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14136.418357] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14136.425362] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14136.429140] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14139.369145] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12222s! [14139.376842] Showing busy workqueues and worker pools: [14139.381043] workqueue events: flags=0x0 [14139.386964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14139.397398] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14139.419324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14139.429906] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14139.439336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.446469] pending: kfree_rcu_monitor [14139.452062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14139.459436] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14139.469721] workqueue events_long: flags=0x0 [14139.475856] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.481923] pending: lu_objects_flush [obdclass] [14139.487034] workqueue events_unbound: flags=0x2 [14139.493901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14139.500325] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14139.508905] workqueue events_power_efficient: flags=0x80 [14139.513418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.520575] pending: check_lifetime [14139.522637] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.528479] pending: neigh_periodic_work [14139.530914] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.544651] pending: neigh_periodic_work [14139.550027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.558001] pending: do_cache_clean [14139.561420] workqueue mm_percpu_wq: flags=0x8 [14139.564797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.571306] pending: vmstat_update [14139.573433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.577392] pending: vmstat_update [14139.582053] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14139.590426] pending: vmstat_update [14139.593770] workqueue writeback: flags=0x4a [14139.600868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14139.607318] pending: wb_workfn, wb_workfn [14139.609974] workqueue xprtiod: flags=0xa [14139.614401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14139.617265] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14139.622907] workqueue ipv6_addrconf: flags=0x40008 [14139.632070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14139.644311] pending: addrconf_verify_work [14139.651035] workqueue knbd0-recv: flags=0x1a [14139.654133] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14139.658747] in-flight: 651:recv_work [14139.660951] workqueue ptlrpc_pinger: flags=0x2 [14139.663645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14139.671812] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14139.676616] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12222s workers=4 idle: 37 31846 28897 [14139.682212] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14129s workers=2 idle: 653 [14140.059005] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14140.064817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14140.098562] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14140.106496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14140.113698] RIP: 0010:__do_softirq+0xa2/0x4e8 [14140.116283] Code: f6 01 01 c7 44 24 10 0a 00 00 00 48 c7 c0 40 2a 03 00 65 66 c7 00 00 00 48 c7 c0 30 bb b3 b5 48 83 00 01 fb 66 0f 1f 44 00 00 ff ff ff ff 48 83 05 b9 1f f6 01 01 48 c7 c3 00 61 80 b4 41 0f [14140.132606] RSP: 0018:ffffb893c0d8cf78 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14140.138766] RAX: ffffffffb5b3bb30 RBX: ffff8d738219df40 RCX: 0000000000000001 [14140.154432] RDX: 0000000000000115 RSI: 000000000000005a RDI: ffffffffb6a010c0 [14140.162755] RBP: 0000000000000000 R08: 00000cdc3d225a80 R09: 00000cdc3d225a80 [14140.171129] R10: 00000000000f4240 R11: 00000cdc3d225a80 R12: 0000000000000000 [14140.177027] R13: 0000000000000000 R14: 0000000000000080 R15: 0000000000000000 [14140.186632] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14140.197868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14140.201228] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14140.211754] Call Trace: [14140.214377] [14140.221665] ? show_regs.cold.9+0x22/0x2f [14140.225945] ? watchdog_timer_fn.cold.6+0x62/0x10a [14140.230138] ? __hrtimer_run_queues+0x1c2/0x4d0 [14140.235562] ? watchdog+0x40/0x40 [14140.240128] ? hrtimer_interrupt+0x128/0x2c0 [14140.244537] ? smp_apic_timer_interrupt+0xa6/0x270 [14140.246697] ? apic_timer_interrupt+0xf/0x20 [14140.255609] ? apic_timer_interrupt+0xa/0x20 [14140.260338] ? __do_softirq+0xa2/0x4e8 [14140.261554] ? __do_softirq+0x73/0x4e8 [14140.264457] ? kvm_sched_clock_read+0x2c/0x50 [14140.270201] irq_exit_rcu+0x1c6/0x1e0 [14140.273297] irq_exit+0x12/0x30 [14140.274285] smp_apic_timer_interrupt+0xe0/0x270 [14140.276138] apic_timer_interrupt+0xf/0x20 [14140.279136] [14140.281615] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14140.285153] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14140.303865] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14140.311450] RAX: 0000000000007294 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14140.320202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14140.324124] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14140.334132] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14140.342290] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14140.350872] do_raw_spin_lock+0x177/0x190 [14140.354527] _raw_spin_lock+0x21/0x30 [14140.361847] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14140.367519] ? do_raw_spin_unlock+0x75/0x190 [14140.371030] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14140.377846] ? do_wait_intr+0xf0/0xf0 [14140.380310] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14140.388170] mdc_close+0x53b/0xd70 [mdc] [14140.390582] lmv_close+0x18d/0x490 [lmv] [14140.397890] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14140.399919] ll_md_real_close+0xa5/0x2a0 [lustre] [14140.405234] ll_md_close+0x274/0x800 [lustre] [14140.409058] ll_file_release+0x128/0xc50 [lustre] [14140.412104] __fput+0x10d/0x440 [14140.413929] ____fput+0x12/0x20 [14140.415130] task_work_run+0xc6/0x110 [14140.416144] exit_to_usermode_loop+0x1dd/0x1f0 [14140.418372] do_syscall_64+0x3d6/0x440 [14140.419628] entry_SYSCALL_64_after_hwframe+0x49/0xae [14140.422933] RIP: 0033:0x7fa749552cd8 [14140.424191] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14140.431492] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14140.435485] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14140.442422] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14140.445575] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14140.449822] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14140.452276] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14144.058126] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14144.062070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14144.132859] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14144.138993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14144.142307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14144.144469] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14144.153017] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14144.158003] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14144.164140] RDX: 0000000000003629 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14144.171263] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14144.178302] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14144.182499] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14144.186570] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14144.190015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14144.194821] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14144.201248] Call Trace: [14144.201891] [14144.202416] ? show_regs.cold.9+0x22/0x2f [14144.203551] ? watchdog_timer_fn.cold.6+0x62/0x10a [14144.209905] ? __hrtimer_run_queues+0x1c2/0x4d0 [14144.212141] ? watchdog+0x40/0x40 [14144.213488] ? hrtimer_interrupt+0x128/0x2c0 [14144.216956] ? smp_apic_timer_interrupt+0xa6/0x270 [14144.220900] ? apic_timer_interrupt+0xf/0x20 [14144.223394] [14144.226034] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14144.230464] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14144.233399] do_raw_spin_lock+0x177/0x190 [14144.235155] _raw_spin_lock+0x21/0x30 [14144.236536] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14144.239411] obd_mod_free+0x46/0x140 [ptlrpc] [14144.245961] mdc_commit_open+0x91/0xb0 [mdc] [14144.247559] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14144.249857] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14144.252340] after_reply+0xe2c/0x1780 [ptlrpc] [14144.256007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14144.259656] ? __next_timer_interrupt+0x160/0x160 [14144.265457] ptlrpcd+0xa82/0xe40 [ptlrpc] [14144.269497] ? do_wait_intr+0xf0/0xf0 [14144.270745] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14144.275049] kthread+0x1d1/0x200 [14144.281000] ? set_kthread_struct+0x70/0x70 [14144.284567] ret_from_fork+0x1f/0x30 [14156.061477] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14156.073370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14156.128232] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14156.144788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14156.153016] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14156.167127] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14156.171370] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14156.198272] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14156.212886] RAX: 0000000000006a92 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14156.230153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14156.243197] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14156.256389] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14156.266260] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14156.273312] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14156.283063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14156.295989] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14156.299908] Call Trace: [14156.302899] [14156.305696] ? show_regs.cold.9+0x22/0x2f [14156.307275] ? watchdog_timer_fn.cold.6+0x62/0x10a [14156.312759] ? __hrtimer_run_queues+0x1c2/0x4d0 [14156.314827] ? watchdog+0x40/0x40 [14156.316957] ? hrtimer_interrupt+0x128/0x2c0 [14156.320022] ? smp_apic_timer_interrupt+0xa6/0x270 [14156.322178] ? apic_timer_interrupt+0xf/0x20 [14156.326275] [14156.329939] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14156.335782] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14156.338201] do_raw_spin_lock+0x177/0x190 [14156.342242] _raw_spin_lock+0x21/0x30 [14156.346643] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14156.352991] process_one_work+0x2c8/0x700 [14156.360248] worker_thread+0x296/0x6e0 [14156.363755] ? rescuer_thread+0x570/0x570 [14156.369533] kthread+0x1d1/0x200 [14156.371977] ? set_kthread_struct+0x70/0x70 [14156.377669] ret_from_fork+0x1f/0x30 [14164.054640] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14164.058156] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14164.082405] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14164.087413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14164.091755] RIP: 0010:native_safe_halt+0x2e/0x50 [14164.094153] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14164.103284] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14164.106561] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14164.110684] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14164.115002] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14164.119650] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14164.121722] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14164.124689] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14164.128201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14164.130908] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14164.133686] Call Trace: [14164.134687] [14164.135340] ? show_regs.cold.9+0x22/0x2f [14164.137145] ? watchdog_timer_fn.cold.6+0x62/0x10a [14164.139109] ? __hrtimer_run_queues+0x1c2/0x4d0 [14164.141662] ? watchdog+0x40/0x40 [14164.143009] ? hrtimer_interrupt+0x128/0x2c0 [14164.144967] ? smp_apic_timer_interrupt+0xa6/0x270 [14164.147207] ? apic_timer_interrupt+0xf/0x20 [14164.149401] [14164.150406] ? native_safe_halt+0x2e/0x50 [14164.152148] kvm_wait+0xcd/0xe0 [14164.153635] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14164.155821] do_raw_spin_lock+0x177/0x190 [14164.158040] _raw_spin_lock+0x21/0x30 [14164.159880] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14164.162252] ? do_raw_spin_unlock+0x75/0x190 [14164.164519] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14164.166867] ? do_wait_intr+0xf0/0xf0 [14164.168255] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14164.170371] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14164.172896] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14164.175936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14164.178534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14164.184207] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14164.189400] mdc_intent_lock+0x353/0x7e0 [mdc] [14164.192449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14164.198985] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14164.203747] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14164.206476] lmv_intent_open+0x338/0x1150 [lmv] [14164.211152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14164.217272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14164.220950] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14164.225986] ll_intent_lock+0xc5/0xaa0 [lustre] [14164.229332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14164.232147] ll_lookup_it+0xe0e/0x2270 [lustre] [14164.235098] ? do_raw_spin_unlock+0x75/0x190 [14164.237186] ? _raw_spin_unlock+0x12/0x30 [14164.240144] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14164.243270] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14164.245598] ll_atomic_open+0x36a/0xd70 [lustre] [14164.249334] lookup_open+0x356/0xb40 [14164.250885] do_last+0x313/0xfc0 [14164.253032] ? nd_jump_root+0xe5/0x160 [14164.256405] ? path_init+0x437/0x520 [14164.258314] path_openat+0xf7/0x500 [14164.260514] ? do_raw_spin_unlock+0x75/0x190 [14164.262403] do_filp_open+0x99/0x140 [14164.266717] ? getname_flags+0x6e/0x330 [14164.270071] ? __check_object_size+0xff/0x256 [14164.273379] ? do_raw_spin_unlock+0x75/0x190 [14164.276822] ? _raw_spin_unlock+0x12/0x30 [14164.280403] do_sys_openat2+0x2b4/0x410 [14164.282628] do_sys_open+0x73/0xa0 [14164.287511] __x64_sys_openat+0x24/0x30 [14164.289927] do_syscall_64+0xc1/0x440 [14164.292622] entry_SYSCALL_64_after_hwframe+0x49/0xae [14164.296234] RIP: 0033:0x7fa749552332 [14164.298520] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14164.311610] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14164.316538] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14164.326292] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14164.333274] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14164.338142] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14164.341614] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14168.059269] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14168.063362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14168.092005] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14168.099960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14168.106143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14168.109722] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14168.124193] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14168.130467] RAX: 0000000000006aa8 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14168.136943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14168.141516] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14168.146170] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14168.151753] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14168.156058] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14168.160597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14168.164573] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14168.168203] Call Trace: [14168.169449] [14168.170834] ? show_regs.cold.9+0x22/0x2f [14168.172557] ? watchdog_timer_fn.cold.6+0x62/0x10a [14168.174719] ? __hrtimer_run_queues+0x1c2/0x4d0 [14168.178677] ? watchdog+0x40/0x40 [14168.180580] ? hrtimer_interrupt+0x128/0x2c0 [14168.182836] ? smp_apic_timer_interrupt+0xa6/0x270 [14168.185935] ? apic_timer_interrupt+0xf/0x20 [14168.188147] [14168.189401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14168.192751] do_raw_spin_lock+0x177/0x190 [14168.195621] _raw_spin_lock+0x21/0x30 [14168.197463] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14168.201840] ? do_raw_spin_unlock+0x75/0x190 [14168.205004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14168.208443] ? do_wait_intr+0xf0/0xf0 [14168.210589] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14168.213991] mdc_close+0x53b/0xd70 [mdc] [14168.216538] lmv_close+0x18d/0x490 [lmv] [14168.218645] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14168.221808] ll_md_real_close+0xa5/0x2a0 [lustre] [14168.224851] ll_md_close+0x274/0x800 [lustre] [14168.228236] ll_file_release+0x128/0xc50 [lustre] [14168.230896] __fput+0x10d/0x440 [14168.232724] ____fput+0x12/0x20 [14168.236217] task_work_run+0xc6/0x110 [14168.237958] exit_to_usermode_loop+0x1dd/0x1f0 [14168.240416] do_syscall_64+0x3d6/0x440 [14168.242320] entry_SYSCALL_64_after_hwframe+0x49/0xae [14168.245285] RIP: 0033:0x7fa749552cd8 [14168.247112] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14168.257150] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14168.262161] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14168.264581] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14168.268258] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14168.272633] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14168.279649] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14170.080770] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12252s! [14170.087591] Showing busy workqueues and worker pools: [14170.092082] workqueue events: flags=0x0 [14170.094141] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14170.100971] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14170.108933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14170.115762] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14170.124867] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.137700] pending: kfree_rcu_monitor [14170.141419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14170.159834] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14170.170394] workqueue events_long: flags=0x0 [14170.174396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.184115] pending: lu_objects_flush [obdclass] [14170.189269] workqueue events_unbound: flags=0x2 [14170.192257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14170.195336] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14170.203062] workqueue events_power_efficient: flags=0x80 [14170.212965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.218049] pending: check_lifetime [14170.222231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.232572] pending: neigh_periodic_work [14170.236087] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.244011] pending: neigh_periodic_work [14170.252007] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.258997] pending: do_cache_clean [14170.268406] workqueue mm_percpu_wq: flags=0x8 [14170.271415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.280489] pending: vmstat_update [14170.289116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.293299] pending: vmstat_update [14170.299266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14170.304077] pending: vmstat_update [14170.306518] workqueue writeback: flags=0x4a [14170.308899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14170.312943] pending: wb_workfn, wb_workfn [14170.316219] workqueue xprtiod: flags=0xa [14170.321029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14170.326363] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14170.330018] workqueue ipv6_addrconf: flags=0x40008 [14170.334527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14170.341437] pending: addrconf_verify_work [14170.345778] workqueue knbd0-recv: flags=0x1a [14170.348633] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14170.357308] in-flight: 651:recv_work [14170.358757] workqueue ptlrpc_pinger: flags=0x2 [14170.362744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14170.373331] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14170.380214] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12253s workers=4 idle: 37 31846 28897 [14170.385912] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14160s workers=2 idle: 653 [14172.058443] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14172.065106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14172.120430] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14172.127868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14172.130138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14172.144652] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14172.160752] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14172.166113] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14172.172698] RDX: 000000000000447a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14172.178717] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14172.181647] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14172.187090] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14172.190101] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14172.194931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14172.198085] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14172.202211] Call Trace: [14172.206093] [14172.206864] ? show_regs.cold.9+0x22/0x2f [14172.210029] ? watchdog_timer_fn.cold.6+0x62/0x10a [14172.212471] ? __hrtimer_run_queues+0x1c2/0x4d0 [14172.214199] ? watchdog+0x40/0x40 [14172.217358] ? hrtimer_interrupt+0x128/0x2c0 [14172.222092] ? smp_apic_timer_interrupt+0xa6/0x270 [14172.225371] ? apic_timer_interrupt+0xf/0x20 [14172.230044] [14172.231144] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14172.236027] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14172.240136] do_raw_spin_lock+0x177/0x190 [14172.241395] _raw_spin_lock+0x21/0x30 [14172.242444] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14172.244911] obd_mod_free+0x46/0x140 [ptlrpc] [14172.257173] mdc_commit_open+0x91/0xb0 [mdc] [14172.260920] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14172.264032] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14172.266799] after_reply+0xe2c/0x1780 [ptlrpc] [14172.270346] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14172.274162] ? __next_timer_interrupt+0x160/0x160 [14172.276317] ptlrpcd+0xa82/0xe40 [ptlrpc] [14172.284482] ? do_wait_intr+0xf0/0xf0 [14172.286522] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14172.288987] kthread+0x1d1/0x200 [14172.289980] ? set_kthread_struct+0x70/0x70 [14172.292027] ret_from_fork+0x1f/0x30 [14184.062448] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14184.072486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14184.115918] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14184.123988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14184.131881] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14184.135061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14184.136968] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14184.144870] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14184.148364] RAX: 0000000000007c4f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14184.151424] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14184.154648] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14184.157729] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14184.160398] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14184.163183] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14184.166489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14184.168688] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14184.172973] Call Trace: [14184.174203] [14184.175845] ? show_regs.cold.9+0x22/0x2f [14184.180034] ? watchdog_timer_fn.cold.6+0x62/0x10a [14184.182844] ? __hrtimer_run_queues+0x1c2/0x4d0 [14184.185959] ? watchdog+0x40/0x40 [14184.187568] ? hrtimer_interrupt+0x128/0x2c0 [14184.190202] ? smp_apic_timer_interrupt+0xa6/0x270 [14184.192373] ? apic_timer_interrupt+0xf/0x20 [14184.194584] [14184.195992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14184.200376] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14184.204150] do_raw_spin_lock+0x177/0x190 [14184.205886] _raw_spin_lock+0x21/0x30 [14184.208012] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14184.210767] process_one_work+0x2c8/0x700 [14184.212852] worker_thread+0x296/0x6e0 [14184.215753] ? rescuer_thread+0x570/0x570 [14184.217384] kthread+0x1d1/0x200 [14184.218791] ? set_kthread_struct+0x70/0x70 [14184.220398] ret_from_fork+0x1f/0x30 [14196.054506] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14196.059476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [14196.059552] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14196.059559] Modules linked in: lustre(O) [14196.092548] sha512_generic [14196.092564] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14196.092572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14196.092576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14196.092592] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14196.092596] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14196.092600] RAX: 0000000000006dfe RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14196.092603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14196.092606] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14196.092607] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14196.092610] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14196.092613] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14196.092616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14196.092618] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14196.092626] Call Trace: [14196.092632] [14196.092638] ? show_regs.cold.9+0x22/0x2f [14196.092648] ? watchdog_timer_fn.cold.6+0x62/0x10a [14196.092654] ? __hrtimer_run_queues+0x1c2/0x4d0 [14196.092658] ? watchdog+0x40/0x40 [14196.092663] ? hrtimer_interrupt+0x128/0x2c0 [14196.092667] ? smp_apic_timer_interrupt+0xa6/0x270 [14196.092676] ? apic_timer_interrupt+0xf/0x20 [14196.092681] [14196.092684] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14196.092692] do_raw_spin_lock+0x177/0x190 [14196.092697] _raw_spin_lock+0x21/0x30 [14196.092702] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14196.092981] ? do_raw_spin_unlock+0x75/0x190 [14196.092987] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14196.093189] ? do_wait_intr+0xf0/0xf0 [14196.093198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14196.093408] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14196.093614] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14196.096580] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14196.096649] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14196.096655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14196.096658] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14196.096671] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14196.096675] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14196.096679] RAX: 0000000000007598 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14196.096682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14196.096684] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14196.096685] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14196.096688] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14196.096691] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14196.096693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14196.096695] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14196.096705] Call Trace: [14196.096709] [14196.096715] ? show_regs.cold.9+0x22/0x2f [14196.096725] ? watchdog_timer_fn.cold.6+0x62/0x10a [14196.096731] ? __hrtimer_run_queues+0x1c2/0x4d0 [14196.096735] ? watchdog+0x40/0x40 [14196.096740] ? hrtimer_interrupt+0x128/0x2c0 [14196.096744] ? smp_apic_timer_interrupt+0xa6/0x270 [14196.096752] ? apic_timer_interrupt+0xf/0x20 [14196.096757] [14196.096760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14196.096767] do_raw_spin_lock+0x177/0x190 [14196.096772] _raw_spin_lock+0x21/0x30 [14196.096777] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14196.097036] ? do_raw_spin_unlock+0x75/0x190 [14196.097042] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14196.097235] ? do_wait_intr+0xf0/0xf0 [14196.097243] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14196.101365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14196.101492] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14196.101534] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14196.101564] mdc_intent_lock+0x353/0x7e0 [mdc] [14196.101591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14196.101672] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14196.101921] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14196.101947] lmv_intent_open+0x338/0x1150 [lmv] [14196.101976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14196.102060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14196.102142] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14196.102166] ll_intent_lock+0xc5/0xaa0 [lustre] [14196.102433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14196.102520] ll_lookup_it+0xe0e/0x2270 [lustre] [14196.102609] ? do_raw_spin_unlock+0x75/0x190 [14196.102616] ? _raw_spin_unlock+0x12/0x30 [14196.102623] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14196.102648] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14196.102669] ll_atomic_open+0x36a/0xd70 [lustre] [14196.102756] lookup_open+0x356/0xb40 [14196.102765] do_last+0x313/0xfc0 [14196.102770] ? nd_jump_root+0xe5/0x160 [14196.102774] ? path_init+0x437/0x520 [14196.102779] path_openat+0xf7/0x500 [14196.102784] ? do_raw_spin_unlock+0x75/0x190 [14196.102789] do_filp_open+0x99/0x140 [14196.102796] ? getname_flags+0x6e/0x330 [14196.102800] ? __check_object_size+0xff/0x256 [14196.102807] ? do_raw_spin_unlock+0x75/0x190 [14196.102811] ? _raw_spin_unlock+0x12/0x30 [14196.102815] do_sys_openat2+0x2b4/0x410 [14196.102819] do_sys_open+0x73/0xa0 [14196.102824] __x64_sys_openat+0x24/0x30 [14196.102827] do_syscall_64+0xc1/0x440 [14196.102834] entry_SYSCALL_64_after_hwframe+0x49/0xae [14196.102840] RIP: 0033:0x7fa749552332 [14196.102848] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14196.102853] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14196.102857] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14196.102860] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14196.102863] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14196.102865] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14196.102867] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14196.500854] mdc_close+0x53b/0xd70 [mdc] [14196.503606] lmv_close+0x18d/0x490 [lmv] [14196.510428] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14196.512348] ll_md_real_close+0xa5/0x2a0 [lustre] [14196.514451] ll_md_close+0x274/0x800 [lustre] [14196.516499] ll_file_release+0x128/0xc50 [lustre] [14196.521636] __fput+0x10d/0x440 [14196.525208] ____fput+0x12/0x20 [14196.526506] task_work_run+0xc6/0x110 [14196.528528] exit_to_usermode_loop+0x1dd/0x1f0 [14196.531262] do_syscall_64+0x3d6/0x440 [14196.532624] entry_SYSCALL_64_after_hwframe+0x49/0xae [14196.537614] RIP: 0033:0x7fa749552cd8 [14196.540621] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14196.551984] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14196.558256] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14196.561351] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14196.564552] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14196.567186] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14196.570617] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14200.058055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14200.061070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14200.078340] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14200.083652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14200.087168] RIP: 0010:native_safe_halt+0x2e/0x50 [14200.089092] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14200.095814] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14200.098745] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14200.101586] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14200.104362] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14200.107315] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14200.110093] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14200.112957] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14200.115931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14200.118093] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14200.120809] Call Trace: [14200.121642] [14200.122388] ? show_regs.cold.9+0x22/0x2f [14200.123837] ? watchdog_timer_fn.cold.6+0x62/0x10a [14200.125947] ? __hrtimer_run_queues+0x1c2/0x4d0 [14200.127415] ? watchdog+0x40/0x40 [14200.128824] ? hrtimer_interrupt+0x128/0x2c0 [14200.130260] ? smp_apic_timer_interrupt+0xa6/0x270 [14200.132890] ? apic_timer_interrupt+0xf/0x20 [14200.134528] [14200.135372] ? native_safe_halt+0x2e/0x50 [14200.137682] kvm_wait+0xcd/0xe0 [14200.138786] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14200.142139] do_raw_spin_lock+0x177/0x190 [14200.145076] _raw_spin_lock+0x21/0x30 [14200.147263] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14200.149627] obd_mod_free+0x46/0x140 [ptlrpc] [14200.151698] mdc_commit_open+0x91/0xb0 [mdc] [14200.153498] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14200.156009] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14200.158467] after_reply+0xe2c/0x1780 [ptlrpc] [14200.160519] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14200.162718] ? __next_timer_interrupt+0x160/0x160 [14200.164037] ptlrpcd+0xa82/0xe40 [ptlrpc] [14200.166051] ? do_wait_intr+0xf0/0xf0 [14200.168085] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14200.170661] kthread+0x1d1/0x200 [14200.172016] ? set_kthread_struct+0x70/0x70 [14200.173919] ret_from_fork+0x1f/0x30 [14200.801373] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12283s! [14200.811481] Showing busy workqueues and worker pools: [14200.813035] workqueue events: flags=0x0 [14200.817730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14200.821813] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14200.828369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14200.831612] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14200.837583] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.841844] pending: kfree_rcu_monitor [14200.845833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14200.853855] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14200.863757] workqueue events_long: flags=0x0 [14200.864771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.873107] pending: lu_objects_flush [obdclass] [14200.874461] workqueue events_unbound: flags=0x2 [14200.875501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14200.878758] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14200.880925] workqueue events_power_efficient: flags=0x80 [14200.884556] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.889646] pending: check_lifetime [14200.891310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.894197] pending: neigh_periodic_work [14200.902581] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.907348] pending: neigh_periodic_work [14200.909784] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.914083] pending: do_cache_clean [14200.915472] workqueue mm_percpu_wq: flags=0x8 [14200.917092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.926625] pending: vmstat_update [14200.927581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.934121] pending: vmstat_update [14200.935107] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14200.941762] pending: vmstat_update [14200.947260] workqueue writeback: flags=0x4a [14200.952544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14200.954216] pending: wb_workfn, wb_workfn [14200.963598] workqueue xprtiod: flags=0xa [14200.966429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14200.971660] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14200.975871] workqueue ipv6_addrconf: flags=0x40008 [14200.978175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14200.981485] pending: addrconf_verify_work [14200.982902] workqueue knbd0-recv: flags=0x1a [14200.985572] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14200.987848] in-flight: 651:recv_work [14200.990821] workqueue ptlrpc_pinger: flags=0x2 [14200.998800] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14201.001759] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14201.005789] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12283s workers=4 idle: 37 31846 28897 [14201.008617] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14190s workers=2 idle: 653 [14212.060944] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14212.066401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14212.102982] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14212.110343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14212.115534] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14212.119361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14212.122994] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14212.132146] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14212.135710] RAX: 0000000000001013 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14212.138653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14212.142043] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14212.145127] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14212.148266] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14212.151324] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14212.155166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14212.157808] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14212.160871] Call Trace: [14212.162090] [14212.162993] ? show_regs.cold.9+0x22/0x2f [14212.164895] ? watchdog_timer_fn.cold.6+0x62/0x10a [14212.166868] ? __hrtimer_run_queues+0x1c2/0x4d0 [14212.169024] ? watchdog+0x40/0x40 [14212.170389] ? hrtimer_interrupt+0x128/0x2c0 [14212.172347] ? smp_apic_timer_interrupt+0xa6/0x270 [14212.174514] ? apic_timer_interrupt+0xf/0x20 [14212.176104] [14212.177163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14212.180146] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14212.182377] do_raw_spin_lock+0x177/0x190 [14212.184023] _raw_spin_lock+0x21/0x30 [14212.185869] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14212.188284] process_one_work+0x2c8/0x700 [14212.190034] worker_thread+0x296/0x6e0 [14212.191175] ? rescuer_thread+0x570/0x570 [14212.192943] kthread+0x1d1/0x200 [14212.194316] ? set_kthread_struct+0x70/0x70 [14212.196480] ret_from_fork+0x1f/0x30 [14213.957022] rcu: INFO: rcu_sched self-detected stall on CPU [14213.960035] rcu: 2-...!: (7020763 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [14213.964635] (t=12300268 jiffies g=258701 q=4887) [14213.966621] rcu: rcu_sched kthread starved for 12297063 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [14213.970584] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14213.977004] rcu: RCU grace-period kthread stack dump: [14213.979200] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14213.984813] Call Trace: [14213.985563] __schedule+0x351/0xcb0 [14213.989552] schedule+0xc0/0x180 [14213.991029] schedule_timeout+0xb4/0x190 [14213.992209] ? do_raw_spin_unlock+0x75/0x190 [14213.995829] ? __next_timer_interrupt+0x160/0x160 [14213.997031] ? prepare_to_swait_event+0xf3/0x140 [14213.998210] rcu_gp_kthread+0x8dd/0x15b0 [14214.001818] ? force_qs_rnp+0x2c0/0x2c0 [14214.004965] kthread+0x1d1/0x200 [14214.007150] ? set_kthread_struct+0x70/0x70 [14214.013603] ret_from_fork+0x1f/0x30 [14214.015145] rcu: Stack dump where RCU GP kthread last ran: [14214.017101] Sending NMI from CPU 2 to CPUs 3: [14214.018845] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [14214.026004] NMI backtrace for cpu 2 [14214.027898] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14214.037432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14214.041918] Call Trace: [14214.042635] [14214.043753] ? dump_stack+0xbb/0x10e [14214.049028] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14214.054617] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14214.058208] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14214.061169] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14214.062644] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14214.064632] ? rcu_dump_cpu_stacks+0x11b/0x171 [14214.065900] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14214.070734] ? update_process_times+0x6d/0xc0 [14214.072492] ? tick_sched_handle.isra.18+0x27/0x90 [14214.074780] ? tick_sched_timer+0x53/0xb0 [14214.076922] ? __hrtimer_run_queues+0x1c2/0x4d0 [14214.081596] ? tick_sched_handle.isra.18+0x90/0x90 [14214.087768] ? hrtimer_interrupt+0x128/0x2c0 [14214.089302] ? smp_apic_timer_interrupt+0xa6/0x270 [14214.090725] ? apic_timer_interrupt+0xf/0x20 [14214.093073] [14214.094046] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14214.096517] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14214.098639] ? do_raw_spin_lock+0x177/0x190 [14214.101322] ? _raw_spin_lock+0x21/0x30 [14214.102697] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14214.107181] ? obd_mod_free+0x46/0x140 [ptlrpc] [14214.112523] ? mdc_commit_open+0x91/0xb0 [mdc] [14214.114879] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14214.116696] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14214.119342] ? after_reply+0xe2c/0x1780 [ptlrpc] [14214.128196] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14214.132449] ? __next_timer_interrupt+0x160/0x160 [14214.136262] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14214.139265] ? do_wait_intr+0xf0/0xf0 [14214.140519] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14214.147046] ? kthread+0x1d1/0x200 [14214.148739] ? set_kthread_struct+0x70/0x70 [14214.151368] ? ret_from_fork+0x1f/0x30 [14228.054016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14228.056506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14228.079705] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14228.085445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14228.090055] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14228.092999] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14228.102869] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14228.106812] RAX: 0000000000005386 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14228.110331] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14228.114120] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14228.117520] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14228.121106] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14228.124584] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14228.129682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14228.132306] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14228.135873] Call Trace: [14228.137037] [14228.138177] ? show_regs.cold.9+0x22/0x2f [14228.140175] ? watchdog_timer_fn.cold.6+0x62/0x10a [14228.142979] ? __hrtimer_run_queues+0x1c2/0x4d0 [14228.145097] ? watchdog+0x40/0x40 [14228.146852] ? hrtimer_interrupt+0x128/0x2c0 [14228.148970] ? smp_apic_timer_interrupt+0xa6/0x270 [14228.151626] ? apic_timer_interrupt+0xf/0x20 [14228.153582] [14228.154580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14228.157225] do_raw_spin_lock+0x177/0x190 [14228.159131] _raw_spin_lock+0x21/0x30 [14228.161003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14228.163758] ? do_raw_spin_unlock+0x75/0x190 [14228.165611] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14228.168873] ? do_wait_intr+0xf0/0xf0 [14228.170885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14228.173864] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14228.176681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14228.179758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14228.182209] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14228.185099] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14228.187271] mdc_intent_lock+0x353/0x7e0 [mdc] [14228.190533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14228.193350] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14228.196531] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14228.199288] lmv_intent_open+0x338/0x1150 [lmv] [14228.202103] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14228.204703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14228.206771] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14228.209218] ll_intent_lock+0xc5/0xaa0 [lustre] [14228.211850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14228.214773] ll_lookup_it+0xe0e/0x2270 [lustre] [14228.217199] ? do_raw_spin_unlock+0x75/0x190 [14228.219166] ? _raw_spin_unlock+0x12/0x30 [14228.221057] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14228.223979] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14228.226204] ll_atomic_open+0x36a/0xd70 [lustre] [14228.228993] lookup_open+0x356/0xb40 [14228.231123] do_last+0x313/0xfc0 [14228.232938] ? nd_jump_root+0xe5/0x160 [14228.234880] ? path_init+0x437/0x520 [14228.236799] path_openat+0xf7/0x500 [14228.238734] ? do_raw_spin_unlock+0x75/0x190 [14228.241146] do_filp_open+0x99/0x140 [14228.242734] ? getname_flags+0x6e/0x330 [14228.245900] ? __check_object_size+0xff/0x256 [14228.248312] ? do_raw_spin_unlock+0x75/0x190 [14228.251484] ? _raw_spin_unlock+0x12/0x30 [14228.253559] do_sys_openat2+0x2b4/0x410 [14228.255256] do_sys_open+0x73/0xa0 [14228.257394] __x64_sys_openat+0x24/0x30 [14228.259589] do_syscall_64+0xc1/0x440 [14228.261288] entry_SYSCALL_64_after_hwframe+0x49/0xae [14228.264607] RIP: 0033:0x7fa749552332 [14228.266615] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14228.276243] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14228.279618] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14228.283431] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14228.287455] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14228.290993] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14228.294444] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14231.008189] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12313s! [14231.013095] Showing busy workqueues and worker pools: [14231.015754] workqueue events: flags=0x0 [14231.017158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14231.025038] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14231.038803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14231.043359] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14231.047761] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.055329] pending: kfree_rcu_monitor [14231.062100] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14231.066474] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14231.075127] workqueue events_long: flags=0x0 [14231.078104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.082247] pending: lu_objects_flush [obdclass] [14231.086199] workqueue events_unbound: flags=0x2 [14231.090366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14231.096815] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14231.104909] workqueue events_power_efficient: flags=0x80 [14231.109847] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.119078] pending: check_lifetime [14231.122337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.129761] pending: neigh_periodic_work [14231.135102] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.139228] pending: neigh_periodic_work [14231.142276] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.147766] pending: do_cache_clean [14231.149637] workqueue mm_percpu_wq: flags=0x8 [14231.151423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.155780] pending: vmstat_update [14231.156936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.161866] pending: vmstat_update [14231.163282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14231.165723] pending: vmstat_update [14231.167524] workqueue writeback: flags=0x4a [14231.169715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14231.173005] pending: wb_workfn, wb_workfn [14231.176882] workqueue xprtiod: flags=0xa [14231.178741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14231.182651] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14231.184657] workqueue ipv6_addrconf: flags=0x40008 [14231.186834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14231.189896] pending: addrconf_verify_work [14231.192684] workqueue knbd0-recv: flags=0x1a [14231.194703] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14231.197900] in-flight: 651:recv_work [14231.200128] workqueue ptlrpc_pinger: flags=0x2 [14231.203962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14231.206929] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14231.209660] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12314s workers=4 idle: 37 31846 28897 [14231.214363] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14221s workers=2 idle: 653 [14240.057065] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14240.058228] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14240.060824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14240.060899] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14240.060907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14240.060913] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14240.061187] RIP: 0010:native_safe_halt+0x2e/0x50 [14240.061198] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14240.061202] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14240.061206] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14240.061209] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14240.061211] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14240.061213] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14240.061215] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14240.061219] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14240.061222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14240.061225] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14240.061242] Call Trace: [14240.061249] [14240.061254] ? show_regs.cold.9+0x22/0x2f [14240.061263] ? watchdog_timer_fn.cold.6+0x62/0x10a [14240.061280] ? __hrtimer_run_queues+0x1c2/0x4d0 [14240.061286] ? watchdog+0x40/0x40 [14240.061290] ? hrtimer_interrupt+0x128/0x2c0 [14240.061294] ? smp_apic_timer_interrupt+0xa6/0x270 [14240.061301] ? apic_timer_interrupt+0xf/0x20 [14240.061306] [14240.061309] ? native_safe_halt+0x2e/0x50 [14240.061314] kvm_wait+0xcd/0xe0 [14240.061321] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14240.061328] do_raw_spin_lock+0x177/0x190 [14240.061333] _raw_spin_lock+0x21/0x30 [14240.061337] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14240.061547] process_one_work+0x2c8/0x700 [14240.061554] worker_thread+0x296/0x6e0 [14240.061559] ? rescuer_thread+0x570/0x570 [14240.061562] kthread+0x1d1/0x200 [14240.061567] ? set_kthread_struct+0x70/0x70 [14240.061570] ret_from_fork+0x1f/0x30 [14240.061703] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14240.061709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14240.061765] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14240.061769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14240.064557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14240.064634] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14240.064643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14240.064647] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14240.064661] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14240.064665] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14240.064669] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14240.064672] RDX: 0000000000003103 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14240.064674] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14240.064676] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14240.064679] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14240.064687] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14240.064690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14240.064692] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14240.064700] Call Trace: [14240.064705] [14240.064711] ? show_regs.cold.9+0x22/0x2f [14240.064719] ? watchdog_timer_fn.cold.6+0x62/0x10a [14240.064725] ? __hrtimer_run_queues+0x1c2/0x4d0 [14240.064730] ? watchdog+0x40/0x40 [14240.064733] ? hrtimer_interrupt+0x128/0x2c0 [14240.064737] ? smp_apic_timer_interrupt+0xa6/0x270 [14240.064745] ? apic_timer_interrupt+0xf/0x20 [14240.064749] [14240.064751] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14240.064755] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14240.064759] do_raw_spin_lock+0x177/0x190 [14240.064764] _raw_spin_lock+0x21/0x30 [14240.064769] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14240.065032] obd_mod_free+0x46/0x140 [ptlrpc] [14240.065466] mdc_commit_open+0x91/0xb0 [mdc] [14240.065503] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14240.065711] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14240.065917] after_reply+0xe2c/0x1780 [ptlrpc] [14240.066111] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14240.066414] ? __next_timer_interrupt+0x160/0x160 [14240.066424] ptlrpcd+0xa82/0xe40 [ptlrpc] [14240.067002] ? do_wait_intr+0xf0/0xf0 [14240.067013] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14240.067217] kthread+0x1d1/0x200 [14240.067224] ? set_kthread_struct+0x70/0x70 [14240.067341] ret_from_fork+0x1f/0x30 [14240.356310] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14240.358729] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14240.367702] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14240.371640] RAX: 0000000000005c7c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14240.377142] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14240.384096] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14240.388135] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14240.391859] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14240.395190] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14240.400551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14240.404944] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14240.408972] Call Trace: [14240.410064] [14240.410885] ? show_regs.cold.9+0x22/0x2f [14240.413190] ? watchdog_timer_fn.cold.6+0x62/0x10a [14240.414919] ? __hrtimer_run_queues+0x1c2/0x4d0 [14240.418645] ? watchdog+0x40/0x40 [14240.420077] ? hrtimer_interrupt+0x128/0x2c0 [14240.422788] ? smp_apic_timer_interrupt+0xa6/0x270 [14240.424588] ? apic_timer_interrupt+0xf/0x20 [14240.426188] [14240.427060] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14240.429088] do_raw_spin_lock+0x177/0x190 [14240.430726] _raw_spin_lock+0x21/0x30 [14240.432215] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14240.434518] ? do_raw_spin_unlock+0x75/0x190 [14240.436121] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14240.438382] ? do_wait_intr+0xf0/0xf0 [14240.439721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14240.441934] mdc_close+0x53b/0xd70 [mdc] [14240.444016] lmv_close+0x18d/0x490 [lmv] [14240.447098] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14240.450102] ll_md_real_close+0xa5/0x2a0 [lustre] [14240.452892] ll_md_close+0x274/0x800 [lustre] [14240.456384] ll_file_release+0x128/0xc50 [lustre] [14240.459752] __fput+0x10d/0x440 [14240.462738] ____fput+0x12/0x20 [14240.464923] task_work_run+0xc6/0x110 [14240.467082] exit_to_usermode_loop+0x1dd/0x1f0 [14240.471370] do_syscall_64+0x3d6/0x440 [14240.474286] entry_SYSCALL_64_after_hwframe+0x49/0xae [14240.477666] RIP: 0033:0x7fa749552cd8 [14240.480072] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14240.494915] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14240.499014] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14240.507361] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14240.511375] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14240.516291] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14240.520732] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14256.054062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14256.056810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14256.078101] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14256.082156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14256.086560] RIP: 0010:native_safe_halt+0x2e/0x50 [14256.088887] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14256.097364] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14256.101268] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14256.104960] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14256.108267] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14256.111198] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14256.116270] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14256.119796] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14256.123679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14256.130075] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14256.136448] Call Trace: [14256.137538] [14256.138617] ? show_regs.cold.9+0x22/0x2f [14256.140540] ? watchdog_timer_fn.cold.6+0x62/0x10a [14256.144914] ? __hrtimer_run_queues+0x1c2/0x4d0 [14256.147208] ? watchdog+0x40/0x40 [14256.148924] ? hrtimer_interrupt+0x128/0x2c0 [14256.150969] ? smp_apic_timer_interrupt+0xa6/0x270 [14256.153344] ? apic_timer_interrupt+0xf/0x20 [14256.155245] [14256.156187] ? native_safe_halt+0x2e/0x50 [14256.157978] kvm_wait+0xcd/0xe0 [14256.159521] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14256.161699] do_raw_spin_lock+0x177/0x190 [14256.163759] _raw_spin_lock+0x21/0x30 [14256.165413] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14256.168187] ? do_raw_spin_unlock+0x75/0x190 [14256.170207] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14256.172779] ? do_wait_intr+0xf0/0xf0 [14256.174636] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14256.177528] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14256.180081] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14256.182734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14256.185539] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14256.187977] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14256.190362] mdc_intent_lock+0x353/0x7e0 [mdc] [14256.192277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14256.195155] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14256.197673] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14256.200321] lmv_intent_open+0x338/0x1150 [lmv] [14256.202226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14256.204382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14256.206795] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14256.208829] ll_intent_lock+0xc5/0xaa0 [lustre] [14256.211216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14256.213818] ll_lookup_it+0xe0e/0x2270 [lustre] [14256.215880] ? do_raw_spin_unlock+0x75/0x190 [14256.217913] ? _raw_spin_unlock+0x12/0x30 [14256.219740] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14256.222161] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14256.224577] ll_atomic_open+0x36a/0xd70 [lustre] [14256.227517] lookup_open+0x356/0xb40 [14256.229294] do_last+0x313/0xfc0 [14256.230877] ? nd_jump_root+0xe5/0x160 [14256.232529] ? path_init+0x437/0x520 [14256.234269] path_openat+0xf7/0x500 [14256.235693] ? do_raw_spin_unlock+0x75/0x190 [14256.237762] do_filp_open+0x99/0x140 [14256.239347] ? getname_flags+0x6e/0x330 [14256.240972] ? __check_object_size+0xff/0x256 [14256.243134] ? do_raw_spin_unlock+0x75/0x190 [14256.245266] ? _raw_spin_unlock+0x12/0x30 [14256.247429] do_sys_openat2+0x2b4/0x410 [14256.249041] do_sys_open+0x73/0xa0 [14256.250695] __x64_sys_openat+0x24/0x30 [14256.252486] do_syscall_64+0xc1/0x440 [14256.254518] entry_SYSCALL_64_after_hwframe+0x49/0xae [14256.256776] RIP: 0033:0x7fa749552332 [14256.258597] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14256.267254] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14256.270797] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14256.274538] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14256.278291] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14256.280910] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14256.284664] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14261.729477] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12344s! [14261.735341] Showing busy workqueues and worker pools: [14261.740713] workqueue events: flags=0x0 [14261.742643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14261.747574] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14261.754757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14261.763101] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14261.768269] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.771114] pending: kfree_rcu_monitor [14261.772438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14261.775701] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14261.779701] workqueue events_long: flags=0x0 [14261.782132] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.785525] pending: lu_objects_flush [obdclass] [14261.787687] workqueue events_unbound: flags=0x2 [14261.792082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14261.795322] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14261.800112] workqueue events_power_efficient: flags=0x80 [14261.801865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.807528] pending: check_lifetime [14261.812690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.820561] pending: neigh_periodic_work [14261.821955] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.827509] pending: neigh_periodic_work [14261.828771] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.833816] pending: do_cache_clean [14261.838099] workqueue mm_percpu_wq: flags=0x8 [14261.840656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.845914] pending: vmstat_update [14261.847130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.852256] pending: vmstat_update [14261.856479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14261.859448] pending: vmstat_update [14261.861494] workqueue writeback: flags=0x4a [14261.866262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14261.873152] pending: wb_workfn, wb_workfn [14261.875620] workqueue xprtiod: flags=0xa [14261.877677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14261.879479] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14261.884663] workqueue ipv6_addrconf: flags=0x40008 [14261.890694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14261.893674] pending: addrconf_verify_work [14261.895511] workqueue knbd0-recv: flags=0x1a [14261.897554] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14261.900681] in-flight: 651:recv_work [14261.902366] workqueue ptlrpc_pinger: flags=0x2 [14261.904290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14261.908061] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14261.912265] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12344s workers=4 idle: 37 31846 28897 [14261.915553] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14251s workers=2 idle: 653 [14268.057287] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14268.059697] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14268.059975] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14268.060275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14268.061688] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14268.061695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14268.061698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [14268.061715] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [14268.061719] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14268.061726] RAX: 0000000000005289 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14268.061729] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14268.061732] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14268.061734] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14268.061738] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14268.061742] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14268.061745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14268.061747] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14268.061755] Call Trace: [14268.061763] [14268.061768] ? show_regs.cold.9+0x22/0x2f [14268.062004] ? watchdog_timer_fn.cold.6+0x62/0x10a [14268.062010] ? __hrtimer_run_queues+0x1c2/0x4d0 [14268.062014] ? watchdog+0x40/0x40 [14268.062018] ? hrtimer_interrupt+0x128/0x2c0 [14268.062031] ? smp_apic_timer_interrupt+0xa6/0x270 [14268.062041] ? apic_timer_interrupt+0xf/0x20 [14268.062046] [14268.062050] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [14268.062057] do_raw_spin_lock+0x177/0x190 [14268.062062] _raw_spin_lock+0x21/0x30 [14268.062067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14268.063389] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [14268.063420] ? do_raw_spin_unlock+0x75/0x190 [14268.063425] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14268.063871] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14268.063876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14268.063882] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14268.064147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14268.064155] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14268.064159] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14268.064164] RAX: 0000000000005121 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14268.064166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14268.064168] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14268.064170] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14268.064173] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14268.064175] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14268.064179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14268.064181] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14268.064190] Call Trace: [14268.064193] [14268.063432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14268.064364] ? do_wait_intr+0xf0/0xf0 [14268.064373] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14268.064671] mdc_close+0x53b/0xd70 [mdc] [14268.064197] ? show_regs.cold.9+0x22/0x2f [14268.064714] ? watchdog_timer_fn.cold.6+0x62/0x10a [14268.064721] ? __hrtimer_run_queues+0x1c2/0x4d0 [14268.064725] ? watchdog+0x40/0x40 [14268.064712] lmv_close+0x18d/0x490 [lmv] [14268.064735] ? hrtimer_interrupt+0x128/0x2c0 [14268.064801] ? smp_apic_timer_interrupt+0xa6/0x270 [14268.064810] ? apic_timer_interrupt+0xf/0x20 [14268.064815] [14268.064818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14268.064822] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14268.064828] do_raw_spin_lock+0x177/0x190 [14268.064833] _raw_spin_lock+0x21/0x30 [14268.064842] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14268.065071] process_one_work+0x2c8/0x700 [14268.065079] worker_thread+0x296/0x6e0 [14268.065083] ? rescuer_thread+0x570/0x570 [14268.065087] kthread+0x1d1/0x200 [14268.065091] ? set_kthread_struct+0x70/0x70 [14268.065096] ret_from_fork+0x1f/0x30 [14268.064748] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14268.068642] Modules linked in: lustre(O) [14268.071943] ll_md_real_close+0xa5/0x2a0 [lustre] [14268.072091] ll_md_close+0x274/0x800 [lustre] [14268.072173] ll_file_release+0x128/0xc50 [lustre] [14268.072320] __fput+0x10d/0x440 [14268.072333] ____fput+0x12/0x20 [14268.072338] task_work_run+0xc6/0x110 [14268.072349] exit_to_usermode_loop+0x1dd/0x1f0 [14268.072358] do_syscall_64+0x3d6/0x440 [14268.072362] entry_SYSCALL_64_after_hwframe+0x49/0xae [14268.072390] RIP: 0033:0x7fa749552cd8 [14268.072400] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14268.072405] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14268.072413] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14268.072416] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14268.072419] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14268.072421] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14268.072426] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14268.325911] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14268.345962] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14268.350467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14268.357342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14268.361368] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14268.372415] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14268.376701] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14268.379219] RDX: 0000000000004c29 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14268.381518] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14268.384800] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14268.387716] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14268.389864] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14268.392193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14268.393855] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14268.396548] Call Trace: [14268.397479] [14268.398370] ? show_regs.cold.9+0x22/0x2f [14268.399544] ? watchdog_timer_fn.cold.6+0x62/0x10a [14268.400937] ? __hrtimer_run_queues+0x1c2/0x4d0 [14268.402398] ? watchdog+0x40/0x40 [14268.403529] ? hrtimer_interrupt+0x128/0x2c0 [14268.404652] ? smp_apic_timer_interrupt+0xa6/0x270 [14268.406276] ? apic_timer_interrupt+0xf/0x20 [14268.407424] [14268.408109] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14268.409837] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14268.411461] do_raw_spin_lock+0x177/0x190 [14268.412927] _raw_spin_lock+0x21/0x30 [14268.415589] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14268.418534] obd_mod_free+0x46/0x140 [ptlrpc] [14268.422331] mdc_commit_open+0x91/0xb0 [mdc] [14268.423348] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14268.434914] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14268.439445] after_reply+0xe2c/0x1780 [ptlrpc] [14268.444251] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14268.445772] ? __next_timer_interrupt+0x160/0x160 [14268.453096] ptlrpcd+0xa82/0xe40 [ptlrpc] [14268.459662] ? do_wait_intr+0xf0/0xf0 [14268.464716] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14268.466588] kthread+0x1d1/0x200 [14268.471953] ? set_kthread_struct+0x70/0x70 [14268.479334] ret_from_fork+0x1f/0x30 [14288.054374] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14288.056601] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14288.077304] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14288.088273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14288.095183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14288.101633] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14288.122466] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14288.135688] RAX: 00000000000053cc RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14288.149350] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14288.151911] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14288.158021] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14288.162277] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14288.165690] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14288.172186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14288.175253] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14288.177800] Call Trace: [14288.180257] [14288.181237] ? show_regs.cold.9+0x22/0x2f [14288.182722] ? watchdog_timer_fn.cold.6+0x62/0x10a [14288.194649] ? __hrtimer_run_queues+0x1c2/0x4d0 [14288.200481] ? watchdog+0x40/0x40 [14288.203836] ? hrtimer_interrupt+0x128/0x2c0 [14288.213045] ? smp_apic_timer_interrupt+0xa6/0x270 [14288.220561] ? apic_timer_interrupt+0xf/0x20 [14288.225482] [14288.228148] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14288.233183] do_raw_spin_lock+0x177/0x190 [14288.237156] _raw_spin_lock+0x21/0x30 [14288.239333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14288.242362] ? do_raw_spin_unlock+0x75/0x190 [14288.246036] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14288.255880] ? do_wait_intr+0xf0/0xf0 [14288.261027] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14288.267178] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14288.270431] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14288.274957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14288.280174] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14288.283649] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14288.285299] mdc_intent_lock+0x353/0x7e0 [mdc] [14288.296336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14288.301503] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14288.305537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14288.312099] lmv_intent_open+0x338/0x1150 [lmv] [14288.320461] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14288.325151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14288.328150] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14288.335513] ll_intent_lock+0xc5/0xaa0 [lustre] [14288.341730] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14288.347552] ll_lookup_it+0xe0e/0x2270 [lustre] [14288.352946] ? do_raw_spin_unlock+0x75/0x190 [14288.356872] ? _raw_spin_unlock+0x12/0x30 [14288.358941] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14288.365849] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14288.367606] ll_atomic_open+0x36a/0xd70 [lustre] [14288.377942] lookup_open+0x356/0xb40 [14288.379241] do_last+0x313/0xfc0 [14288.384733] ? nd_jump_root+0xe5/0x160 [14288.387883] ? path_init+0x437/0x520 [14288.393451] path_openat+0xf7/0x500 [14288.398506] ? do_raw_spin_unlock+0x75/0x190 [14288.402586] do_filp_open+0x99/0x140 [14288.408947] ? getname_flags+0x6e/0x330 [14288.411730] ? __check_object_size+0xff/0x256 [14288.416120] ? do_raw_spin_unlock+0x75/0x190 [14288.418217] ? _raw_spin_unlock+0x12/0x30 [14288.420647] do_sys_openat2+0x2b4/0x410 [14288.424022] do_sys_open+0x73/0xa0 [14288.429187] __x64_sys_openat+0x24/0x30 [14288.433962] do_syscall_64+0xc1/0x440 [14288.435291] entry_SYSCALL_64_after_hwframe+0x49/0xae [14288.443954] RIP: 0033:0x7fa749552332 [14288.446988] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14288.466700] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14288.476816] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14288.482394] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14288.490198] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14288.501631] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14288.513150] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14291.936100] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12374s! [14291.942895] Showing busy workqueues and worker pools: [14291.946104] workqueue events: flags=0x0 [14291.953727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14291.968662] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14291.979284] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14291.994670] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14292.010265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.012645] pending: kfree_rcu_monitor [14292.016209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14292.022373] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14292.030305] workqueue events_long: flags=0x0 [14292.042836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.051727] pending: lu_objects_flush [obdclass] [14292.061804] workqueue events_unbound: flags=0x2 [14292.063353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14292.078821] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14292.086274] workqueue events_power_efficient: flags=0x80 [14292.091340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.095974] pending: check_lifetime [14292.097284] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.111754] pending: neigh_periodic_work [14292.118644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.121228] pending: neigh_periodic_work [14292.128410] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.130575] pending: do_cache_clean [14292.138839] workqueue mm_percpu_wq: flags=0x8 [14292.140179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.152157] pending: vmstat_update [14292.155350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.164265] pending: vmstat_update [14292.166291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14292.172860] pending: vmstat_update [14292.178262] workqueue writeback: flags=0x4a [14292.192778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14292.197935] pending: wb_workfn, wb_workfn [14292.205430] workqueue xprtiod: flags=0xa [14292.206694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14292.222793] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14292.228405] workqueue ipv6_addrconf: flags=0x40008 [14292.238686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14292.242696] pending: addrconf_verify_work [14292.244046] workqueue knbd0-recv: flags=0x1a [14292.246364] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14292.249654] in-flight: 651:recv_work [14292.258347] workqueue ptlrpc_pinger: flags=0x2 [14292.267656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14292.276345] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14292.284982] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12375s workers=4 idle: 37 31846 28897 [14292.299857] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14282s workers=2 idle: 653 [14296.057750] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14296.058037] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14296.059012] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14296.059019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14296.059082] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14296.059089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14296.059092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14296.059105] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14296.059109] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14296.059113] RAX: 0000000000005909 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14296.059115] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14296.059118] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14296.059120] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14296.059122] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14296.059126] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14296.059128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14296.059131] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14296.059141] Call Trace: [14296.059146] [14296.059151] ? show_regs.cold.9+0x22/0x2f [14296.059161] ? watchdog_timer_fn.cold.6+0x62/0x10a [14296.059167] ? __hrtimer_run_queues+0x1c2/0x4d0 [14296.059171] ? watchdog+0x40/0x40 [14296.059175] ? hrtimer_interrupt+0x128/0x2c0 [14296.059178] ? smp_apic_timer_interrupt+0xa6/0x270 [14296.059186] ? apic_timer_interrupt+0xf/0x20 [14296.059191] [14296.059194] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14296.059200] do_raw_spin_lock+0x177/0x190 [14296.059205] _raw_spin_lock+0x21/0x30 [14296.059849] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14296.060110] ? do_raw_spin_unlock+0x75/0x190 [14296.060117] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14296.060341] ? do_wait_intr+0xf0/0xf0 [14296.060348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14296.060553] mdc_close+0x53b/0xd70 [mdc] [14296.060593] lmv_close+0x18d/0x490 [lmv] [14296.060668] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14296.060733] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14296.060738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14296.060623] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14296.060937] ll_md_real_close+0xa5/0x2a0 [lustre] [14296.061021] ll_md_close+0x274/0x800 [lustre] [14296.061103] ll_file_release+0x128/0xc50 [lustre] [14296.061185] __fput+0x10d/0x440 [14296.061193] ____fput+0x12/0x20 [14296.061198] task_work_run+0xc6/0x110 [14296.061205] exit_to_usermode_loop+0x1dd/0x1f0 [14296.061212] do_syscall_64+0x3d6/0x440 [14296.061216] entry_SYSCALL_64_after_hwframe+0x49/0xae [14296.061223] RIP: 0033:0x7fa749552cd8 [14296.061228] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14296.061232] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14296.061236] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14296.061239] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14296.061241] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14296.061243] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14296.061245] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14296.060743] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14296.068128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14296.068856] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14296.068866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14296.068869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14296.068883] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14296.068887] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14296.068892] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14296.068894] RDX: 0000000000000493 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14296.068896] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14296.068899] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14296.068901] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14296.068904] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14296.068907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14296.068909] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14296.068918] Call Trace: [14296.068924] [14296.068930] ? show_regs.cold.9+0x22/0x2f [14296.068939] ? watchdog_timer_fn.cold.6+0x62/0x10a [14296.068945] ? __hrtimer_run_queues+0x1c2/0x4d0 [14296.068949] ? watchdog+0x40/0x40 [14296.068954] ? hrtimer_interrupt+0x128/0x2c0 [14296.068958] ? smp_apic_timer_interrupt+0xa6/0x270 [14296.068966] ? apic_timer_interrupt+0xf/0x20 [14296.068971] [14296.068974] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14296.068978] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14296.068983] do_raw_spin_lock+0x177/0x190 [14296.068987] _raw_spin_lock+0x21/0x30 [14296.068994] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14296.072610] [14296.072617] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14296.072632] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14296.072636] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14296.072641] RAX: 0000000000007d90 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14296.072643] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14296.072645] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14296.072647] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14296.072649] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14296.072652] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14296.072655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14296.072658] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14296.072667] Call Trace: [14296.072672] [14296.072677] ? show_regs.cold.9+0x22/0x2f [14296.072686] ? watchdog_timer_fn.cold.6+0x62/0x10a [14296.072693] ? __hrtimer_run_queues+0x1c2/0x4d0 [14296.072697] ? watchdog+0x40/0x40 [14296.072701] ? hrtimer_interrupt+0x128/0x2c0 [14296.072706] ? smp_apic_timer_interrupt+0xa6/0x270 [14296.072714] ? apic_timer_interrupt+0xf/0x20 [14296.072718] [14296.072722] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14296.072727] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14296.072733] do_raw_spin_lock+0x177/0x190 [14296.072738] _raw_spin_lock+0x21/0x30 [14296.072742] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14296.109742] obd_mod_free+0x46/0x140 [ptlrpc] [14296.110010] mdc_commit_open+0x91/0xb0 [mdc] [14296.110048] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14296.110464] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14296.111004] after_reply+0xe2c/0x1780 [ptlrpc] [14296.111216] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14296.111494] ? __next_timer_interrupt+0x160/0x160 [14296.111504] ptlrpcd+0xa82/0xe40 [ptlrpc] [14296.111718] ? do_wait_intr+0xf0/0xf0 [14296.111729] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14296.111942] kthread+0x1d1/0x200 [14296.111948] ? set_kthread_struct+0x70/0x70 [14296.111952] ret_from_fork+0x1f/0x30 [14296.504168] process_one_work+0x2c8/0x700 [14296.506074] worker_thread+0x296/0x6e0 [14296.507401] ? rescuer_thread+0x570/0x570 [14296.509188] kthread+0x1d1/0x200 [14296.510859] ? set_kthread_struct+0x70/0x70 [14296.512598] ret_from_fork+0x1f/0x30 [14320.054463] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14320.063807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14320.095227] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14320.102012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14320.107364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14320.112700] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14320.121474] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14320.125444] RAX: 0000000000007c9f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14320.128396] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14320.131009] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14320.134624] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14320.137135] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14320.138752] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14320.142293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14320.144234] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14320.147952] Call Trace: [14320.148495] [14320.149463] ? show_regs.cold.9+0x22/0x2f [14320.151045] ? watchdog_timer_fn.cold.6+0x62/0x10a [14320.153606] ? __hrtimer_run_queues+0x1c2/0x4d0 [14320.159016] ? watchdog+0x40/0x40 [14320.160500] ? hrtimer_interrupt+0x128/0x2c0 [14320.162212] ? smp_apic_timer_interrupt+0xa6/0x270 [14320.163961] ? apic_timer_interrupt+0xf/0x20 [14320.165289] [14320.166238] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14320.167994] do_raw_spin_lock+0x177/0x190 [14320.169961] _raw_spin_lock+0x21/0x30 [14320.171353] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14320.173735] ? do_raw_spin_unlock+0x75/0x190 [14320.175099] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14320.179284] ? do_wait_intr+0xf0/0xf0 [14320.182003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14320.194709] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14320.198095] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14320.206299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.213582] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14320.222353] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14320.227205] mdc_intent_lock+0x353/0x7e0 [mdc] [14320.230816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.235453] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14320.238866] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14320.241154] lmv_intent_open+0x338/0x1150 [lmv] [14320.242714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.244465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.246331] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14320.247714] ll_intent_lock+0xc5/0xaa0 [lustre] [14320.252030] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.255088] ll_lookup_it+0xe0e/0x2270 [lustre] [14320.257222] ? do_raw_spin_unlock+0x75/0x190 [14320.258586] ? _raw_spin_unlock+0x12/0x30 [14320.259925] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14320.264136] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14320.265918] ll_atomic_open+0x36a/0xd70 [lustre] [14320.268099] lookup_open+0x356/0xb40 [14320.268970] do_last+0x313/0xfc0 [14320.269739] ? nd_jump_root+0xe5/0x160 [14320.270656] ? path_init+0x437/0x520 [14320.272496] path_openat+0xf7/0x500 [14320.274268] ? do_raw_spin_unlock+0x75/0x190 [14320.276211] do_filp_open+0x99/0x140 [14320.280052] ? getname_flags+0x6e/0x330 [14320.283032] ? __check_object_size+0xff/0x256 [14320.285026] ? do_raw_spin_unlock+0x75/0x190 [14320.287166] ? _raw_spin_unlock+0x12/0x30 [14320.288981] do_sys_openat2+0x2b4/0x410 [14320.290279] do_sys_open+0x73/0xa0 [14320.292252] __x64_sys_openat+0x24/0x30 [14320.294579] do_syscall_64+0xc1/0x440 [14320.296300] entry_SYSCALL_64_after_hwframe+0x49/0xae [14320.302482] RIP: 0033:0x7fa749552332 [14320.305057] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14320.320011] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14320.323234] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14320.327238] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14320.329976] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14320.339396] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14320.347335] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14322.656897] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12405s! [14322.663635] Showing busy workqueues and worker pools: [14322.669832] workqueue events: flags=0x0 [14322.672276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14322.680904] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14322.689184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14322.694169] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14322.701867] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.705898] pending: kfree_rcu_monitor [14322.710352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14322.718039] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14322.724511] workqueue events_long: flags=0x0 [14322.727279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.732641] pending: lu_objects_flush [obdclass] [14322.739270] workqueue events_unbound: flags=0x2 [14322.741761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14322.744139] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14322.749423] workqueue events_power_efficient: flags=0x80 [14322.751545] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.754236] pending: check_lifetime [14322.757624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.761414] pending: neigh_periodic_work [14322.763678] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.767058] pending: neigh_periodic_work [14322.775594] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.782477] pending: do_cache_clean [14322.783929] workqueue mm_percpu_wq: flags=0x8 [14322.785614] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.792587] pending: vmstat_update [14322.794558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.797326] pending: vmstat_update [14322.799082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14322.801885] pending: vmstat_update [14322.803536] workqueue writeback: flags=0x4a [14322.805364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14322.807983] pending: wb_workfn, wb_workfn [14322.809829] workqueue xprtiod: flags=0xa [14322.811500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14322.814150] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14322.816829] workqueue ipv6_addrconf: flags=0x40008 [14322.817862] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14322.821769] pending: addrconf_verify_work [14322.823742] workqueue knbd0-recv: flags=0x1a [14322.825670] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14322.828777] in-flight: 651:recv_work [14322.831184] workqueue ptlrpc_pinger: flags=0x2 [14322.833265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14322.835975] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14322.840196] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12405s workers=4 idle: 37 31846 28897 [14322.844230] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14312s workers=2 idle: 653 [14324.059885] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14324.060139] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14324.060404] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14324.060410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14324.060481] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14324.060486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14324.060489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14324.060501] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14324.060505] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14324.060510] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14324.060512] RDX: 0000000000001e23 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14324.060514] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14324.060516] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14324.060518] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14324.060522] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14324.060525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14324.060527] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14324.060535] Call Trace: [14324.060540] [14324.060546] ? show_regs.cold.9+0x22/0x2f [14324.060556] ? watchdog_timer_fn.cold.6+0x62/0x10a [14324.060562] ? __hrtimer_run_queues+0x1c2/0x4d0 [14324.060566] ? watchdog+0x40/0x40 [14324.060570] ? hrtimer_interrupt+0x128/0x2c0 [14324.060575] ? smp_apic_timer_interrupt+0xa6/0x270 [14324.060583] ? apic_timer_interrupt+0xf/0x20 [14324.060588] [14324.060591] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14324.060596] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14324.060601] do_raw_spin_lock+0x177/0x190 [14324.060606] _raw_spin_lock+0x21/0x30 [14324.060612] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14324.060881] obd_mod_free+0x46/0x140 [ptlrpc] [14324.061095] mdc_commit_open+0x91/0xb0 [mdc] [14324.061133] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14324.061588] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14324.061785] after_reply+0xe2c/0x1780 [ptlrpc] [14324.063521] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14324.063745] ? __next_timer_interrupt+0x160/0x160 [14324.063754] ptlrpcd+0xa82/0xe40 [ptlrpc] [14324.065241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [14324.063952] ? do_wait_intr+0xf0/0xf0 [14324.065874] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14324.066130] kthread+0x1d1/0x200 [14324.066137] ? set_kthread_struct+0x70/0x70 [14324.066141] ret_from_fork+0x1f/0x30 [14324.068182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14324.068236] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14324.068241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14324.068244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14324.068255] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14324.068258] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14324.068261] RAX: 0000000000007f53 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14324.068262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14324.068264] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14324.068265] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14324.068267] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14324.068270] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14324.068272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14324.068275] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14324.068282] Call Trace: [14324.068287] [14324.068291] ? show_regs.cold.9+0x22/0x2f [14324.068300] ? watchdog_timer_fn.cold.6+0x62/0x10a [14324.068306] ? __hrtimer_run_queues+0x1c2/0x4d0 [14324.068310] ? watchdog+0x40/0x40 [14324.068314] ? hrtimer_interrupt+0x128/0x2c0 [14324.068318] ? smp_apic_timer_interrupt+0xa6/0x270 [14324.068326] ? apic_timer_interrupt+0xf/0x20 [14324.068330] [14324.068333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14324.068339] do_raw_spin_lock+0x177/0x190 [14324.068343] _raw_spin_lock+0x21/0x30 [14324.068348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14324.071787] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [14324.092260] ? do_raw_spin_unlock+0x75/0x190 [14324.092281] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14324.092553] ? do_wait_intr+0xf0/0xf0 [14324.092561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14324.092751] mdc_close+0x53b/0xd70 [mdc] [14324.101284] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14324.101329] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14324.101336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14324.101342] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14324.105702] lmv_close+0x18d/0x490 [lmv] [14324.105742] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14324.105961] ll_md_real_close+0xa5/0x2a0 [lustre] [14324.106041] ll_md_close+0x274/0x800 [lustre] [14324.106136] ll_file_release+0x128/0xc50 [lustre] [14324.106216] __fput+0x10d/0x440 [14324.106225] ____fput+0x12/0x20 [14324.106230] task_work_run+0xc6/0x110 [14324.106238] exit_to_usermode_loop+0x1dd/0x1f0 [14324.106245] do_syscall_64+0x3d6/0x440 [14324.106250] entry_SYSCALL_64_after_hwframe+0x49/0xae [14324.106259] RIP: 0033:0x7fa749552cd8 [14324.106266] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14324.106270] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14324.106275] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14324.106278] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14324.106280] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14324.106282] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14324.106284] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14324.484138] [14324.484926] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14324.489991] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14324.506310] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14324.514054] RAX: 0000000000007f3f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14324.519738] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14324.526115] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14324.530234] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14324.537862] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14324.544930] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14324.553233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14324.558259] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14324.567166] Call Trace: [14324.569866] [14324.571492] ? show_regs.cold.9+0x22/0x2f [14324.576585] ? watchdog_timer_fn.cold.6+0x62/0x10a [14324.580478] ? __hrtimer_run_queues+0x1c2/0x4d0 [14324.585475] ? watchdog+0x40/0x40 [14324.588214] ? hrtimer_interrupt+0x128/0x2c0 [14324.593828] ? smp_apic_timer_interrupt+0xa6/0x270 [14324.598454] ? apic_timer_interrupt+0xf/0x20 [14324.601521] [14324.604133] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14324.609666] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14324.617447] do_raw_spin_lock+0x177/0x190 [14324.622139] _raw_spin_lock+0x21/0x30 [14324.625689] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14324.628940] process_one_work+0x2c8/0x700 [14324.632001] worker_thread+0x296/0x6e0 [14324.633274] ? rescuer_thread+0x570/0x570 [14324.635079] kthread+0x1d1/0x200 [14324.637579] ? set_kthread_struct+0x70/0x70 [14324.639537] ret_from_fork+0x1f/0x30 [14348.054011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14348.056014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14348.075714] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14348.079985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14348.083947] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [14348.086844] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [14348.096121] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14348.099253] RAX: 00000000000031d4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14348.102226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14348.105651] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14348.108894] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14348.111620] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14348.114316] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14348.117368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14348.120171] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14348.122984] Call Trace: [14348.124252] [14348.124995] ? show_regs.cold.9+0x22/0x2f [14348.126658] ? watchdog_timer_fn.cold.6+0x62/0x10a [14348.128050] ? __hrtimer_run_queues+0x1c2/0x4d0 [14348.130033] ? watchdog+0x40/0x40 [14348.131391] ? hrtimer_interrupt+0x128/0x2c0 [14348.132826] ? smp_apic_timer_interrupt+0xa6/0x270 [14348.134812] ? apic_timer_interrupt+0xf/0x20 [14348.136077] [14348.137089] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [14348.139270] do_raw_spin_lock+0x177/0x190 [14348.141353] _raw_spin_lock+0x21/0x30 [14348.143173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14348.146258] ? do_raw_spin_unlock+0x75/0x190 [14348.148606] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14348.151087] ? do_wait_intr+0xf0/0xf0 [14348.152452] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14348.155378] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14348.158140] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14348.161101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.164034] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14348.166851] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14348.169004] mdc_intent_lock+0x353/0x7e0 [mdc] [14348.171139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.173816] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14348.176456] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14348.178853] lmv_intent_open+0x338/0x1150 [lmv] [14348.181183] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.183613] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.185724] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14348.187694] ll_intent_lock+0xc5/0xaa0 [lustre] [14348.190058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.192976] ll_lookup_it+0xe0e/0x2270 [lustre] [14348.195611] ? do_raw_spin_unlock+0x75/0x190 [14348.198147] ? _raw_spin_unlock+0x12/0x30 [14348.200205] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14348.202723] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14348.205471] ll_atomic_open+0x36a/0xd70 [lustre] [14348.208242] lookup_open+0x356/0xb40 [14348.209902] do_last+0x313/0xfc0 [14348.211707] ? nd_jump_root+0xe5/0x160 [14348.213317] ? path_init+0x437/0x520 [14348.215428] path_openat+0xf7/0x500 [14348.217097] ? do_raw_spin_unlock+0x75/0x190 [14348.219519] do_filp_open+0x99/0x140 [14348.221366] ? getname_flags+0x6e/0x330 [14348.223395] ? __check_object_size+0xff/0x256 [14348.225471] ? do_raw_spin_unlock+0x75/0x190 [14348.227852] ? _raw_spin_unlock+0x12/0x30 [14348.230135] do_sys_openat2+0x2b4/0x410 [14348.231992] do_sys_open+0x73/0xa0 [14348.233328] __x64_sys_openat+0x24/0x30 [14348.234964] do_syscall_64+0xc1/0x440 [14348.236338] entry_SYSCALL_64_after_hwframe+0x49/0xae [14348.239184] RIP: 0033:0x7fa749552332 [14348.240678] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14348.250137] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14348.253033] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14348.256782] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14348.260492] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14348.264645] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14348.269031] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14352.058035] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14352.060341] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14352.061144] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14352.061231] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14352.061299] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14352.061306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14352.061308] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14352.061323] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14352.061327] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14352.061332] RAX: 0000000000005a6c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14352.061334] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14352.061336] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14352.061338] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14352.061340] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14352.061343] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14352.061346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14352.061348] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14352.061358] Call Trace: [14352.061363] [14352.061369] ? show_regs.cold.9+0x22/0x2f [14352.061378] ? watchdog_timer_fn.cold.6+0x62/0x10a [14352.061385] ? __hrtimer_run_queues+0x1c2/0x4d0 [14352.061389] ? watchdog+0x40/0x40 [14352.061393] ? hrtimer_interrupt+0x128/0x2c0 [14352.061397] ? smp_apic_timer_interrupt+0xa6/0x270 [14352.061404] ? apic_timer_interrupt+0xf/0x20 [14352.061409] [14352.061412] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14352.061419] do_raw_spin_lock+0x177/0x190 [14352.061424] _raw_spin_lock+0x21/0x30 [14352.061429] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14352.062598] ? do_raw_spin_unlock+0x75/0x190 [14352.062605] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14352.062901] ? do_wait_intr+0xf0/0xf0 [14352.062910] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14352.063163] Modules linked in: [14352.063125] mdc_close+0x53b/0xd70 [mdc] [14352.063166] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [14352.063185] lmv_close+0x18d/0x490 [lmv] [14352.063205] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14352.063238] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14352.063244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14352.063247] RIP: 0010:native_safe_halt+0x2e/0x50 [14352.063256] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14352.063259] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14352.063220] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14352.063327] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14352.063334] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14352.063338] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14352.063341] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14352.063343] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14352.063346] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14352.063349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14352.063352] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14352.063358] Call Trace: [14352.063361] [14352.063337] ll_md_real_close+0xa5/0x2a0 [lustre] [14352.063364] ? show_regs.cold.9+0x22/0x2f [14352.063422] ? watchdog_timer_fn.cold.6+0x62/0x10a [14352.063431] ? __hrtimer_run_queues+0x1c2/0x4d0 [14352.063436] ? watchdog+0x40/0x40 [14352.063440] ? hrtimer_interrupt+0x128/0x2c0 [14352.063443] ? smp_apic_timer_interrupt+0xa6/0x270 [14352.063449] ? apic_timer_interrupt+0xf/0x20 [14352.063454] [14352.063456] ? native_safe_halt+0x2e/0x50 [14352.063460] kvm_wait+0xcd/0xe0 [14352.063466] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14352.063428] ll_md_close+0x274/0x800 [lustre] [14352.063473] do_raw_spin_lock+0x177/0x190 [14352.063478] _raw_spin_lock+0x21/0x30 [14352.063596] ll_file_release+0x128/0xc50 [lustre] [14352.063482] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14352.065873] __fput+0x10d/0x440 [14352.065888] ____fput+0x12/0x20 [14352.065893] task_work_run+0xc6/0x110 [14352.065901] exit_to_usermode_loop+0x1dd/0x1f0 [14352.065909] do_syscall_64+0x3d6/0x440 [14352.065914] entry_SYSCALL_64_after_hwframe+0x49/0xae [14352.065923] RIP: 0033:0x7fa749552cd8 [14352.065931] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14352.065936] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14352.065941] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14352.065944] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14352.065946] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14352.065948] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14352.065951] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14352.070117] Modules linked in: lustre(O) [14352.076738] obd_mod_free+0x46/0x140 [ptlrpc] [14352.077090] mdc_commit_open+0x91/0xb0 [mdc] [14352.077128] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14352.116073] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14352.116143] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14352.116150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14352.116155] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14352.116463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14352.116474] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14352.116478] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14352.116482] RAX: 0000000000007020 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14352.116485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14352.116488] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14352.116490] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14352.116493] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14352.116496] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14352.116499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14352.116501] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14352.116682] Call Trace: [14352.116688] [14352.116694] ? show_regs.cold.9+0x22/0x2f [14352.116704] ? watchdog_timer_fn.cold.6+0x62/0x10a [14352.116710] ? __hrtimer_run_queues+0x1c2/0x4d0 [14352.116714] ? watchdog+0x40/0x40 [14352.116719] ? hrtimer_interrupt+0x128/0x2c0 [14352.116723] ? smp_apic_timer_interrupt+0xa6/0x270 [14352.116732] ? apic_timer_interrupt+0xf/0x20 [14352.116738] [14352.116741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14352.116746] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14352.116752] do_raw_spin_lock+0x177/0x190 [14352.116756] _raw_spin_lock+0x21/0x30 [14352.116762] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14352.116983] process_one_work+0x2c8/0x700 [14352.116990] worker_thread+0x296/0x6e0 [14352.116995] ? rescuer_thread+0x570/0x570 [14352.116998] kthread+0x1d1/0x200 [14352.117003] ? set_kthread_struct+0x70/0x70 [14352.117006] ret_from_fork+0x1f/0x30 [14352.541512] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14352.544156] after_reply+0xe2c/0x1780 [ptlrpc] [14352.546702] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14352.549371] ? __next_timer_interrupt+0x160/0x160 [14352.551643] ptlrpcd+0xa82/0xe40 [ptlrpc] [14352.553880] ? do_wait_intr+0xf0/0xf0 [14352.555797] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14352.558271] kthread+0x1d1/0x200 [14352.560069] ? set_kthread_struct+0x70/0x70 [14352.561874] ret_from_fork+0x1f/0x30 [14352.865030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12435s! [14352.868820] Showing busy workqueues and worker pools: [14352.870271] workqueue events: flags=0x0 [14352.871480] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14352.884477] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14352.902168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14352.905717] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14352.910685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14352.917783] pending: kfree_rcu_monitor [14352.921796] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14352.933305] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14352.941304] workqueue events_long: flags=0x0 [14352.944846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14352.950897] pending: lu_objects_flush [obdclass] [14352.956206] workqueue events_unbound: flags=0x2 [14352.977222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14352.982047] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14352.987818] workqueue events_power_efficient: flags=0x80 [14352.990318] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14352.993197] pending: check_lifetime [14352.994837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14353.002694] pending: neigh_periodic_work [14353.006287] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14353.010505] pending: neigh_periodic_work [14353.013631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14353.015628] pending: do_cache_clean [14353.019143] workqueue mm_percpu_wq: flags=0x8 [14353.021381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14353.023711] pending: vmstat_update [14353.027168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14353.031343] pending: vmstat_update [14353.032803] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14353.035819] pending: vmstat_update [14353.036911] workqueue writeback: flags=0x4a [14353.040055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14353.043779] pending: wb_workfn, wb_workfn [14353.045219] workqueue xprtiod: flags=0xa [14353.046371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14353.052729] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14353.057873] workqueue ipv6_addrconf: flags=0x40008 [14353.059743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14353.063588] pending: addrconf_verify_work [14353.065201] workqueue knbd0-recv: flags=0x1a [14353.066213] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14353.070268] in-flight: 651:recv_work [14353.071221] workqueue ptlrpc_pinger: flags=0x2 [14353.074521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14353.076983] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14353.079733] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12435s workers=4 idle: 37 31846 28897 [14353.082911] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14342s workers=2 idle: 653 [14380.054026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14380.057081] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14380.057825] Modules linked in: [14380.058021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14380.058027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14380.058109] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14380.058115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14380.058118] RIP: 0010:native_safe_halt+0x2e/0x50 [14380.058132] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14380.058136] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14380.058142] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14380.058144] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14380.058146] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14380.058149] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14380.058152] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14380.058155] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14380.058159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14380.058161] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14380.058169] Call Trace: [14380.058175] [14380.058179] ? show_regs.cold.9+0x22/0x2f [14380.058190] ? watchdog_timer_fn.cold.6+0x62/0x10a [14380.058196] ? __hrtimer_run_queues+0x1c2/0x4d0 [14380.058201] ? watchdog+0x40/0x40 [14380.058205] ? hrtimer_interrupt+0x128/0x2c0 [14380.058209] ? smp_apic_timer_interrupt+0xa6/0x270 [14380.058216] ? apic_timer_interrupt+0xf/0x20 [14380.058221] [14380.058224] ? native_safe_halt+0x2e/0x50 [14380.058228] kvm_wait+0xcd/0xe0 [14380.058235] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14380.058243] do_raw_spin_lock+0x177/0x190 [14380.058247] _raw_spin_lock+0x21/0x30 [14380.058253] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14380.058513] obd_mod_free+0x46/0x140 [ptlrpc] [14380.058722] mdc_commit_open+0x91/0xb0 [mdc] [14380.058758] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14380.059023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14380.059030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14380.059102] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14380.059111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14380.059114] RIP: 0010:native_safe_halt+0x2e/0x50 [14380.059123] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14380.059127] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14380.059132] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14380.059135] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14380.059137] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14380.059139] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14380.059142] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14380.059145] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14380.059148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14380.059151] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14380.059159] Call Trace: [14380.059162] [14380.059165] ? show_regs.cold.9+0x22/0x2f [14380.059113] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14380.059298] ? watchdog_timer_fn.cold.6+0x62/0x10a [14380.059305] ? __hrtimer_run_queues+0x1c2/0x4d0 [14380.059309] ? watchdog+0x40/0x40 [14380.059313] ? hrtimer_interrupt+0x128/0x2c0 [14380.059318] ? smp_apic_timer_interrupt+0xa6/0x270 [14380.059324] ? apic_timer_interrupt+0xf/0x20 [14380.059333] [14380.059337] ? native_safe_halt+0x2e/0x50 [14380.059342] kvm_wait+0xcd/0xe0 [14380.059349] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14380.059355] do_raw_spin_lock+0x177/0x190 [14380.059361] _raw_spin_lock+0x21/0x30 [14380.059332] after_reply+0xe2c/0x1780 [ptlrpc] [14380.059365] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14380.059903] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14380.060317] ? __next_timer_interrupt+0x160/0x160 [14380.060326] ptlrpcd+0xa82/0xe40 [ptlrpc] [14380.060612] ? do_wait_intr+0xf0/0xf0 [14380.060829] ? do_raw_spin_unlock+0x75/0x190 [14380.060839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14380.061185] ? do_wait_intr+0xf0/0xf0 [14380.060621] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14380.061450] kthread+0x1d1/0x200 [14380.061457] ? set_kthread_struct+0x70/0x70 [14380.061461] ret_from_fork+0x1f/0x30 [14380.061192] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14380.061651] mdc_close+0x53b/0xd70 [mdc] [14380.061698] lmv_close+0x18d/0x490 [lmv] [14380.061735] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14380.061870] ll_md_real_close+0xa5/0x2a0 [lustre] [14380.062135] ll_md_close+0x274/0x800 [lustre] [14380.062534] ll_file_release+0x128/0xc50 [lustre] [14380.062617] __fput+0x10d/0x440 [14380.062628] ____fput+0x12/0x20 [14380.062704] task_work_run+0xc6/0x110 [14380.062714] exit_to_usermode_loop+0x1dd/0x1f0 [14380.062721] do_syscall_64+0x3d6/0x440 [14380.062727] entry_SYSCALL_64_after_hwframe+0x49/0xae [14380.062733] RIP: 0033:0x7fa749552cd8 [14380.062740] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14380.062744] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14380.062750] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14380.062752] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14380.062754] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14380.062756] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14380.062758] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14380.066642] Modules linked in: [14380.068103] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14380.068179] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14380.068187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14380.068189] RIP: 0010:native_safe_halt+0x2e/0x50 [14380.068202] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14380.068206] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14380.068211] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14380.068213] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14380.068216] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14380.068218] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14380.068220] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14380.068224] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14380.068227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14380.068229] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14380.068238] Call Trace: [14380.068243] [14380.068248] ? show_regs.cold.9+0x22/0x2f [14380.068259] ? watchdog_timer_fn.cold.6+0x62/0x10a [14380.068265] ? __hrtimer_run_queues+0x1c2/0x4d0 [14380.068270] ? watchdog+0x40/0x40 [14380.068274] ? hrtimer_interrupt+0x128/0x2c0 [14380.068279] ? smp_apic_timer_interrupt+0xa6/0x270 [14380.068285] ? apic_timer_interrupt+0xf/0x20 [14380.068290] [14380.068293] ? native_safe_halt+0x2e/0x50 [14380.068297] kvm_wait+0xcd/0xe0 [14380.068304] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14380.068313] do_raw_spin_lock+0x177/0x190 [14380.068318] _raw_spin_lock+0x21/0x30 [14380.068323] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14380.068601] ? do_raw_spin_unlock+0x75/0x190 [14380.068607] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14380.068974] ? do_wait_intr+0xf0/0xf0 [14380.068983] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14380.069204] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14380.069418] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14380.069651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.069960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14380.070009] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14380.070039] mdc_intent_lock+0x353/0x7e0 [mdc] [14380.070079] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.070176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14380.076922] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [14380.094128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14380.094183] lmv_intent_open+0x338/0x1150 [lmv] [14380.094220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.094338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.094421] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14380.094444] ll_intent_lock+0xc5/0xaa0 [lustre] [14380.105387] obdecho(O) [14380.109197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.109324] ll_lookup_it+0xe0e/0x2270 [lustre] [14380.109412] ? do_raw_spin_unlock+0x75/0x190 [14380.109421] ? _raw_spin_unlock+0x12/0x30 [14380.109429] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14380.109463] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14380.109485] ll_atomic_open+0x36a/0xd70 [lustre] [14380.109570] lookup_open+0x356/0xb40 [14380.109580] do_last+0x313/0xfc0 [14380.109584] ? nd_jump_root+0xe5/0x160 [14380.109590] ? path_init+0x437/0x520 [14380.109594] path_openat+0xf7/0x500 [14380.109599] ? do_raw_spin_unlock+0x75/0x190 [14380.109604] do_filp_open+0x99/0x140 [14380.109611] ? getname_flags+0x6e/0x330 [14380.109616] ? __check_object_size+0xff/0x256 [14380.109623] ? do_raw_spin_unlock+0x75/0x190 [14380.109627] ? _raw_spin_unlock+0x12/0x30 [14380.109631] do_sys_openat2+0x2b4/0x410 [14380.109635] do_sys_open+0x73/0xa0 [14380.109639] __x64_sys_openat+0x24/0x30 [14380.109642] do_syscall_64+0xc1/0x440 [14380.109650] entry_SYSCALL_64_after_hwframe+0x49/0xae [14380.109726] RIP: 0033:0x7fa749552332 [14380.109735] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14380.109739] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14380.109744] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14380.109746] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14380.109748] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14380.109750] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14380.109752] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14380.470702] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14380.479681] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14380.482308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14380.485131] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14380.486869] RIP: 0010:native_safe_halt+0x2e/0x50 [14380.488566] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14380.494749] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14380.496569] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14380.498402] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14380.500875] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14380.503440] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14380.506217] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14380.509094] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14380.512178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14380.514428] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14380.517026] Call Trace: [14380.518149] [14380.519015] ? show_regs.cold.9+0x22/0x2f [14380.520328] ? watchdog_timer_fn.cold.6+0x62/0x10a [14380.521977] ? __hrtimer_run_queues+0x1c2/0x4d0 [14380.523682] ? watchdog+0x40/0x40 [14380.524990] ? hrtimer_interrupt+0x128/0x2c0 [14380.526705] ? smp_apic_timer_interrupt+0xa6/0x270 [14380.528460] ? apic_timer_interrupt+0xf/0x20 [14380.530113] [14380.530968] ? native_safe_halt+0x2e/0x50 [14380.532537] kvm_wait+0xcd/0xe0 [14380.533905] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14380.535898] do_raw_spin_lock+0x177/0x190 [14380.537454] _raw_spin_lock+0x21/0x30 [14380.539025] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14380.541133] process_one_work+0x2c8/0x700 [14380.542716] worker_thread+0x296/0x6e0 [14380.545224] ? rescuer_thread+0x570/0x570 [14380.547008] kthread+0x1d1/0x200 [14380.548297] ? set_kthread_struct+0x70/0x70 [14380.552482] ret_from_fork+0x1f/0x30 [14383.585041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12466s! [14383.589538] Showing busy workqueues and worker pools: [14383.592442] workqueue events: flags=0x0 [14383.594972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14383.601686] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14383.614143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14383.618687] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14383.624074] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.629076] pending: kfree_rcu_monitor [14383.631414] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14383.634738] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14383.640136] workqueue events_long: flags=0x0 [14383.642758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.646644] pending: lu_objects_flush [obdclass] [14383.649607] workqueue events_unbound: flags=0x2 [14383.652710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14383.656499] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14383.661226] workqueue events_power_efficient: flags=0x80 [14383.663670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.666915] pending: check_lifetime [14383.668266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.671108] pending: neigh_periodic_work [14383.673154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.675968] pending: neigh_periodic_work [14383.677707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.680929] pending: do_cache_clean [14383.682705] workqueue mm_percpu_wq: flags=0x8 [14383.684556] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.687231] pending: vmstat_update [14383.688492] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.691156] pending: vmstat_update [14383.692339] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14383.695014] pending: vmstat_update [14383.696887] workqueue writeback: flags=0x4a [14383.698351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14383.702591] pending: wb_workfn, wb_workfn [14383.705361] workqueue xprtiod: flags=0xa [14383.707600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14383.711308] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14383.716026] workqueue ipv6_addrconf: flags=0x40008 [14383.718597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14383.722264] pending: addrconf_verify_work [14383.724167] workqueue knbd0-recv: flags=0x1a [14383.726192] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14383.728456] in-flight: 651:recv_work [14383.730602] workqueue ptlrpc_pinger: flags=0x2 [14383.733597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14383.737124] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14383.740348] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12466s workers=4 idle: 37 31846 28897 [14383.744455] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14373s workers=2 idle: 653 [14393.960252] rcu: INFO: rcu_sched self-detected stall on CPU [14393.963364] rcu: 2-...!: (7157455 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [14393.967851] (t=12480271 jiffies g=258701 q=4913) [14393.970165] rcu: rcu_sched kthread starved for 12477066 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [14393.975325] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14393.978243] rcu: RCU grace-period kthread stack dump: [14393.979680] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14393.982413] Call Trace: [14393.983063] __schedule+0x351/0xcb0 [14393.984084] schedule+0xc0/0x180 [14393.985190] schedule_timeout+0xb4/0x190 [14393.986319] ? do_raw_spin_unlock+0x75/0x190 [14393.987625] ? __next_timer_interrupt+0x160/0x160 [14393.988842] ? prepare_to_swait_event+0xf3/0x140 [14393.991121] rcu_gp_kthread+0x8dd/0x15b0 [14393.992338] ? force_qs_rnp+0x2c0/0x2c0 [14393.997194] kthread+0x1d1/0x200 [14393.998494] ? set_kthread_struct+0x70/0x70 [14394.002011] ret_from_fork+0x1f/0x30 [14394.003528] rcu: Stack dump where RCU GP kthread last ran: [14394.006290] Sending NMI from CPU 2 to CPUs 1: [14394.011932] NMI backtrace for cpu 1 [14394.011938] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14394.011940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14394.011942] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14394.011944] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14394.011946] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14394.011948] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 [14394.011950] RAX: 00000000000015fd RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14394.011951] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14394.011952] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14394.011953] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14394.011954] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14394.011955] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14394.011957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14394.011958] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14394.011959] Call Trace: [14394.011959] [14394.011960] ? show_regs.cold.9+0x22/0x2f [14394.011961] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14394.011962] ? nmi_cpu_backtrace_handler+0x15/0x40 [14394.011963] ? nmi_handle+0xbd/0x210 [14394.011964] ? default_do_nmi+0x4b/0x1b0 [14394.011964] ? do_nmi+0x280/0x380 [14394.011965] ? end_repeat_nmi+0x16/0x1f [14394.011966] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14394.011967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14394.011968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14394.011969] [14394.011970] do_raw_spin_lock+0x177/0x190 [14394.011971] _raw_spin_lock+0x21/0x30 [14394.011972] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14394.011972] process_one_work+0x2c8/0x700 [14394.011973] worker_thread+0x296/0x6e0 [14394.011974] ? rescuer_thread+0x570/0x570 [14394.011975] kthread+0x1d1/0x200 [14394.011976] ? set_kthread_struct+0x70/0x70 [14394.011977] ret_from_fork+0x1f/0x30 [14394.013442] NMI backtrace for cpu 2 [14394.227045] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14394.240742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14394.251967] Call Trace: [14394.252837] [14394.253495] ? dump_stack+0xbb/0x10e [14394.256857] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14394.262022] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14394.263632] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14394.276524] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14394.280689] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14394.284371] ? rcu_dump_cpu_stacks+0x11b/0x171 [14394.288786] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14394.291242] ? update_process_times+0x6d/0xc0 [14394.297235] ? tick_sched_handle.isra.18+0x27/0x90 [14394.302366] ? tick_sched_timer+0x53/0xb0 [14394.305838] ? __hrtimer_run_queues+0x1c2/0x4d0 [14394.312984] ? tick_sched_handle.isra.18+0x90/0x90 [14394.317376] ? hrtimer_interrupt+0x128/0x2c0 [14394.321419] ? smp_apic_timer_interrupt+0xa6/0x270 [14394.331579] ? apic_timer_interrupt+0xf/0x20 [14394.334440] [14394.344271] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14394.352944] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14394.364317] ? do_raw_spin_lock+0x177/0x190 [14394.370053] ? _raw_spin_lock+0x21/0x30 [14394.374092] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14394.378437] ? obd_mod_free+0x46/0x140 [ptlrpc] [14394.387915] ? mdc_commit_open+0x91/0xb0 [mdc] [14394.399000] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14394.402965] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14394.405541] ? after_reply+0xe2c/0x1780 [ptlrpc] [14394.417897] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14394.421367] ? __next_timer_interrupt+0x160/0x160 [14394.423868] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14394.432338] ? do_wait_intr+0xf0/0xf0 [14394.434512] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14394.439500] ? kthread+0x1d1/0x200 [14394.441334] ? set_kthread_struct+0x70/0x70 [14394.447854] ? ret_from_fork+0x1f/0x30 [14408.054048] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14408.056735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14408.059320] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14408.072637] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14408.072650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14408.072653] RIP: 0010:native_safe_halt+0x2e/0x50 [14408.072665] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14408.072675] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14408.072681] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14408.072683] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14408.072686] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14408.072688] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14408.072691] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14408.072694] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14408.072698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14408.072700] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14408.072708] Call Trace: [14408.072713] [14408.072718] ? show_regs.cold.9+0x22/0x2f [14408.072727] ? watchdog_timer_fn.cold.6+0x62/0x10a [14408.072734] ? __hrtimer_run_queues+0x1c2/0x4d0 [14408.072739] ? watchdog+0x40/0x40 [14408.072743] ? hrtimer_interrupt+0x128/0x2c0 [14408.072748] ? smp_apic_timer_interrupt+0xa6/0x270 [14408.072760] ? apic_timer_interrupt+0xf/0x20 [14408.072764] [14408.072768] ? native_safe_halt+0x2e/0x50 [14408.072772] kvm_wait+0xcd/0xe0 [14408.072783] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14408.072793] do_raw_spin_lock+0x177/0x190 [14408.072798] _raw_spin_lock+0x21/0x30 [14408.072802] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14408.073072] ? do_raw_spin_unlock+0x75/0x190 [14408.073078] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14408.073272] ? do_wait_intr+0xf0/0xf0 [14408.073280] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14408.073530] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14408.073948] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14408.074157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14408.074274] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14408.074372] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14408.074405] mdc_intent_lock+0x353/0x7e0 [mdc] [14408.074435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14408.074517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14408.077259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14408.077345] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14408.077352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14408.077355] RIP: 0010:native_safe_halt+0x2e/0x50 [14408.077366] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14408.077370] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14408.077374] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14408.077377] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14408.077379] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14408.077384] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14408.077386] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14408.077390] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14408.077392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14408.077395] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14408.077402] Call Trace: [14408.077407] [14408.077412] ? show_regs.cold.9+0x22/0x2f [14408.077421] ? watchdog_timer_fn.cold.6+0x62/0x10a [14408.077427] ? __hrtimer_run_queues+0x1c2/0x4d0 [14408.077431] ? watchdog+0x40/0x40 [14408.077435] ? hrtimer_interrupt+0x128/0x2c0 [14408.077439] ? smp_apic_timer_interrupt+0xa6/0x270 [14408.077446] ? apic_timer_interrupt+0xf/0x20 [14408.077451] [14408.077454] ? native_safe_halt+0x2e/0x50 [14408.077459] kvm_wait+0xcd/0xe0 [14408.077467] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14408.077473] do_raw_spin_lock+0x177/0x190 [14408.077476] _raw_spin_lock+0x21/0x30 [14408.077479] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14408.077838] ? do_raw_spin_unlock+0x75/0x190 [14408.077843] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14408.078053] ? do_wait_intr+0xf0/0xf0 [14408.078061] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14408.078321] mdc_close+0x53b/0xd70 [mdc] [14408.078363] lmv_close+0x18d/0x490 [lmv] [14408.078399] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14408.078564] ll_md_real_close+0xa5/0x2a0 [lustre] [14408.078745] ll_md_close+0x274/0x800 [lustre] [14408.078835] ll_file_release+0x128/0xc50 [lustre] [14408.078922] __fput+0x10d/0x440 [14408.078931] ____fput+0x12/0x20 [14408.078935] task_work_run+0xc6/0x110 [14408.078942] exit_to_usermode_loop+0x1dd/0x1f0 [14408.078953] do_syscall_64+0x3d6/0x440 [14408.078958] entry_SYSCALL_64_after_hwframe+0x49/0xae [14408.078965] RIP: 0033:0x7fa749552cd8 [14408.078973] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14408.078977] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14408.078982] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14408.078984] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14408.078987] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14408.078989] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14408.078992] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14408.315983] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14408.319251] lmv_intent_open+0x338/0x1150 [lmv] [14408.321693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14408.323952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14408.327080] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14408.328966] ll_intent_lock+0xc5/0xaa0 [lustre] [14408.331020] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14408.333641] ll_lookup_it+0xe0e/0x2270 [lustre] [14408.335733] ? do_raw_spin_unlock+0x75/0x190 [14408.337939] ? _raw_spin_unlock+0x12/0x30 [14408.339757] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14408.342167] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14408.344432] ll_atomic_open+0x36a/0xd70 [lustre] [14408.346660] lookup_open+0x356/0xb40 [14408.347956] do_last+0x313/0xfc0 [14408.349514] ? nd_jump_root+0xe5/0x160 [14408.350989] ? path_init+0x437/0x520 [14408.352824] path_openat+0xf7/0x500 [14408.354261] ? do_raw_spin_unlock+0x75/0x190 [14408.356422] do_filp_open+0x99/0x140 [14408.357926] ? getname_flags+0x6e/0x330 [14408.359477] ? __check_object_size+0xff/0x256 [14408.361170] ? do_raw_spin_unlock+0x75/0x190 [14408.362606] ? _raw_spin_unlock+0x12/0x30 [14408.364126] do_sys_openat2+0x2b4/0x410 [14408.365467] do_sys_open+0x73/0xa0 [14408.366660] __x64_sys_openat+0x24/0x30 [14408.368122] do_syscall_64+0xc1/0x440 [14408.369463] entry_SYSCALL_64_after_hwframe+0x49/0xae [14408.371404] RIP: 0033:0x7fa749552332 [14408.372835] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14408.381658] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14408.385169] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14408.388385] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14408.391760] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14408.395015] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14408.397106] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14413.792041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12496s! [14413.798758] Showing busy workqueues and worker pools: [14413.805750] workqueue events: flags=0x0 [14413.807043] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14413.813161] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14413.819954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14413.823540] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14413.827539] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.830612] pending: kfree_rcu_monitor [14413.832429] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14413.835518] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14413.839836] workqueue events_long: flags=0x0 [14413.841723] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.844622] pending: lu_objects_flush [obdclass] [14413.847178] workqueue events_unbound: flags=0x2 [14413.848993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14413.852025] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14413.856035] workqueue events_power_efficient: flags=0x80 [14413.858382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.863551] pending: check_lifetime [14413.865029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.867536] pending: neigh_periodic_work [14413.868850] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.871335] pending: neigh_periodic_work [14413.872792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.875067] pending: do_cache_clean [14413.876563] workqueue mm_percpu_wq: flags=0x8 [14413.877812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.880353] pending: vmstat_update [14413.881639] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.883942] pending: vmstat_update [14413.885390] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.887553] pending: vmstat_update [14413.888671] workqueue writeback: flags=0x4a [14413.889908] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14413.891918] pending: wb_workfn, wb_workfn [14413.893433] workqueue xprtiod: flags=0xa [14413.894669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14413.896648] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14413.899361] workqueue ipv6_addrconf: flags=0x40008 [14413.900748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14413.902911] pending: addrconf_verify_work [14413.904494] workqueue knbd0-recv: flags=0x1a [14413.905831] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14413.907929] in-flight: 651:recv_work [14413.909446] workqueue ptlrpc_pinger: flags=0x2 [14413.910741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14413.912760] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14413.915297] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12496s workers=4 idle: 37 31846 28897 [14413.917761] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14403s workers=2 idle: 653 [14420.057548] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14420.059794] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14420.060419] Modules linked in: [14420.064116] Modules linked in: [14420.064117] lustre(O) osp(O) ofd(O) [14420.066256] lustre(O) [14420.067717] lod(O) [14420.069531] osp(O) [14420.070852] mdt(O) [14420.071882] ofd(O) [14420.072761] mdd(O) [14420.074228] lod(O) [14420.075043] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14420.075103] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14420.075110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14420.075116] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14420.076998] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14420.077063] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14420.077069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14420.077072] RIP: 0010:native_safe_halt+0x2e/0x50 [14420.077143] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14420.077147] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14420.077152] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14420.077154] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14420.077157] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14420.077160] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14420.077162] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14420.077165] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14420.077168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14420.077170] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14420.077178] Call Trace: [14420.077184] [14420.077189] ? show_regs.cold.9+0x22/0x2f [14420.077198] ? watchdog_timer_fn.cold.6+0x62/0x10a [14420.077204] ? __hrtimer_run_queues+0x1c2/0x4d0 [14420.077207] ? watchdog+0x40/0x40 [14420.077211] ? hrtimer_interrupt+0x128/0x2c0 [14420.077215] ? smp_apic_timer_interrupt+0xa6/0x270 [14420.077221] ? apic_timer_interrupt+0xf/0x20 [14420.077227] [14420.077229] ? native_safe_halt+0x2e/0x50 [14420.077233] kvm_wait+0xcd/0xe0 [14420.077241] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14420.077248] do_raw_spin_lock+0x177/0x190 [14420.077252] _raw_spin_lock+0x21/0x30 [14420.077257] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14420.091645] [14420.091654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14420.091671] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14420.091676] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14420.091681] RAX: 000000000000760b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14420.091683] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14420.091685] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14420.091687] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14420.091689] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14420.091693] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14420.091696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14420.091698] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14420.091707] Call Trace: [14420.091712] [14420.091718] ? show_regs.cold.9+0x22/0x2f [14420.091727] ? watchdog_timer_fn.cold.6+0x62/0x10a [14420.091732] ? __hrtimer_run_queues+0x1c2/0x4d0 [14420.091736] ? watchdog+0x40/0x40 [14420.091741] ? hrtimer_interrupt+0x128/0x2c0 [14420.091745] ? smp_apic_timer_interrupt+0xa6/0x270 [14420.091752] ? apic_timer_interrupt+0xf/0x20 [14420.091756] [14420.091759] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14420.091763] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14420.091769] do_raw_spin_lock+0x177/0x190 [14420.091774] _raw_spin_lock+0x21/0x30 [14420.091778] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14420.092035] process_one_work+0x2c8/0x700 [14420.092042] worker_thread+0x296/0x6e0 [14420.092046] ? rescuer_thread+0x570/0x570 [14420.092049] kthread+0x1d1/0x200 [14420.092054] ? set_kthread_struct+0x70/0x70 [14420.092057] ret_from_fork+0x1f/0x30 [14420.313037] obd_mod_free+0x46/0x140 [ptlrpc] [14420.314844] mdc_commit_open+0x91/0xb0 [mdc] [14420.316774] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14420.318812] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14420.321891] after_reply+0xe2c/0x1780 [ptlrpc] [14420.324181] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14420.326068] ? __next_timer_interrupt+0x160/0x160 [14420.327777] ptlrpcd+0xa82/0xe40 [ptlrpc] [14420.329437] ? do_wait_intr+0xf0/0xf0 [14420.330693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14420.332323] kthread+0x1d1/0x200 [14420.333203] ? set_kthread_struct+0x70/0x70 [14420.334639] ret_from_fork+0x1f/0x30 [14436.059008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14436.062660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14436.082936] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14436.088222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14436.091363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14436.094243] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14436.101992] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14436.105667] RAX: 0000000000001b6f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14436.108966] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14436.112459] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14436.115941] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14436.119570] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14436.123255] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14436.127128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14436.129944] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14436.133419] Call Trace: [14436.134706] [14436.135787] ? show_regs.cold.9+0x22/0x2f [14436.138097] ? watchdog_timer_fn.cold.6+0x62/0x10a [14436.140709] ? __hrtimer_run_queues+0x1c2/0x4d0 [14436.142680] ? watchdog+0x40/0x40 [14436.144339] ? hrtimer_interrupt+0x128/0x2c0 [14436.146461] ? smp_apic_timer_interrupt+0xa6/0x270 [14436.148994] ? apic_timer_interrupt+0xf/0x20 [14436.151390] [14436.152229] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14436.155053] do_raw_spin_lock+0x177/0x190 [14436.157102] _raw_spin_lock+0x21/0x30 [14436.158942] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14436.162252] ? do_raw_spin_unlock+0x75/0x190 [14436.164436] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14436.167657] ? do_wait_intr+0xf0/0xf0 [14436.169669] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14436.172609] mdc_close+0x53b/0xd70 [mdc] [14436.174978] lmv_close+0x18d/0x490 [lmv] [14436.177197] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14436.180585] ll_md_real_close+0xa5/0x2a0 [lustre] [14436.183460] ll_md_close+0x274/0x800 [lustre] [14436.185571] ll_file_release+0x128/0xc50 [lustre] [14436.187981] __fput+0x10d/0x440 [14436.189555] ____fput+0x12/0x20 [14436.191362] task_work_run+0xc6/0x110 [14436.193372] exit_to_usermode_loop+0x1dd/0x1f0 [14436.195531] do_syscall_64+0x3d6/0x440 [14436.197768] entry_SYSCALL_64_after_hwframe+0x49/0xae [14436.200722] RIP: 0033:0x7fa749552cd8 [14436.202705] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14436.212951] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14436.217026] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14436.220489] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14436.223528] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14436.226567] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14436.230823] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14440.056472] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14440.062748] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14440.121903] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14440.139045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14440.146028] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14440.155984] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14440.167113] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14440.176495] RAX: 0000000000001e86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14440.191510] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14440.201991] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14440.213632] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14440.218504] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14440.224954] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14440.234855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14440.237968] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14440.241067] Call Trace: [14440.243095] [14440.244138] ? show_regs.cold.9+0x22/0x2f [14440.246560] ? watchdog_timer_fn.cold.6+0x62/0x10a [14440.250640] ? __hrtimer_run_queues+0x1c2/0x4d0 [14440.257120] ? watchdog+0x40/0x40 [14440.259423] ? hrtimer_interrupt+0x128/0x2c0 [14440.264913] ? smp_apic_timer_interrupt+0xa6/0x270 [14440.268421] ? apic_timer_interrupt+0xf/0x20 [14440.272971] [14440.276497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14440.279402] do_raw_spin_lock+0x177/0x190 [14440.281087] _raw_spin_lock+0x21/0x30 [14440.282369] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14440.286551] ? do_raw_spin_unlock+0x75/0x190 [14440.289572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14440.293151] ? do_wait_intr+0xf0/0xf0 [14440.295826] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14440.298934] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14440.302575] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14440.306974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14440.310961] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14440.314622] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14440.317152] mdc_intent_lock+0x353/0x7e0 [mdc] [14440.321961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14440.325543] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14440.329107] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14440.332810] lmv_intent_open+0x338/0x1150 [lmv] [14440.335501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14440.338403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14440.341560] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14440.343981] ll_intent_lock+0xc5/0xaa0 [lustre] [14440.346628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14440.349915] ll_lookup_it+0xe0e/0x2270 [lustre] [14440.352142] ? do_raw_spin_unlock+0x75/0x190 [14440.354518] ? _raw_spin_unlock+0x12/0x30 [14440.356669] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14440.359427] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14440.361087] ll_atomic_open+0x36a/0xd70 [lustre] [14440.362552] lookup_open+0x356/0xb40 [14440.364690] do_last+0x313/0xfc0 [14440.366162] ? nd_jump_root+0xe5/0x160 [14440.368329] ? path_init+0x437/0x520 [14440.370140] path_openat+0xf7/0x500 [14440.372371] ? do_raw_spin_unlock+0x75/0x190 [14440.374674] do_filp_open+0x99/0x140 [14440.376874] ? getname_flags+0x6e/0x330 [14440.378973] ? __check_object_size+0xff/0x256 [14440.380973] ? do_raw_spin_unlock+0x75/0x190 [14440.382179] ? _raw_spin_unlock+0x12/0x30 [14440.383982] do_sys_openat2+0x2b4/0x410 [14440.385963] do_sys_open+0x73/0xa0 [14440.387974] __x64_sys_openat+0x24/0x30 [14440.389954] do_syscall_64+0xc1/0x440 [14440.392869] entry_SYSCALL_64_after_hwframe+0x49/0xae [14440.395668] RIP: 0033:0x7fa749552332 [14440.397932] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14440.406787] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14440.410548] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14440.414139] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14440.417668] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14440.421526] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14440.424576] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14444.002728] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12526s! [14444.010394] Showing busy workqueues and worker pools: [14444.013316] workqueue events: flags=0x0 [14444.017358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14444.020780] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14444.030236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14444.033637] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14444.048093] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.054517] pending: kfree_rcu_monitor [14444.063495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14444.068471] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14444.072456] workqueue events_long: flags=0x0 [14444.076361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.080665] pending: lu_objects_flush [obdclass] [14444.083891] workqueue events_unbound: flags=0x2 [14444.087117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14444.090286] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14444.098961] workqueue events_power_efficient: flags=0x80 [14444.102440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.108423] pending: check_lifetime [14444.110491] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.116780] pending: neigh_periodic_work [14444.121830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.128545] pending: neigh_periodic_work [14444.133620] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.146407] pending: do_cache_clean [14444.151643] workqueue mm_percpu_wq: flags=0x8 [14444.160036] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.171633] pending: vmstat_update [14444.174241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.181246] pending: vmstat_update [14444.182457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.185533] pending: vmstat_update [14444.188811] workqueue writeback: flags=0x4a [14444.197527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14444.201369] pending: wb_workfn, wb_workfn [14444.206879] workqueue xprtiod: flags=0xa [14444.209527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14444.212845] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14444.216068] workqueue ipv6_addrconf: flags=0x40008 [14444.218033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14444.221056] pending: addrconf_verify_work [14444.223423] workqueue knbd0-recv: flags=0x1a [14444.225447] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14444.228386] in-flight: 651:recv_work [14444.229456] workqueue ptlrpc_pinger: flags=0x2 [14444.231613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14444.233672] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14444.236978] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12527s workers=4 idle: 37 31846 28897 [14444.240838] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14434s workers=2 idle: 653 [14448.057958] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14448.059122] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14448.062763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14448.064295] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14448.064302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14448.064309] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14448.065082] RIP: 0010:native_safe_halt+0x2e/0x50 [14448.065095] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14448.065099] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14448.065104] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14448.065106] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14448.065108] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14448.065110] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14448.065112] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14448.065116] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14448.065118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14448.065121] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14448.065129] Call Trace: [14448.065134] [14448.065140] ? show_regs.cold.9+0x22/0x2f [14448.065154] ? watchdog_timer_fn.cold.6+0x62/0x10a [14448.065162] ? __hrtimer_run_queues+0x1c2/0x4d0 [14448.065167] ? watchdog+0x40/0x40 [14448.065172] ? hrtimer_interrupt+0x128/0x2c0 [14448.065176] ? smp_apic_timer_interrupt+0xa6/0x270 [14448.065183] ? apic_timer_interrupt+0xf/0x20 [14448.065188] [14448.065191] ? native_safe_halt+0x2e/0x50 [14448.065200] kvm_wait+0xcd/0xe0 [14448.065209] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14448.065216] do_raw_spin_lock+0x177/0x190 [14448.065221] _raw_spin_lock+0x21/0x30 [14448.065225] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14448.065469] process_one_work+0x2c8/0x700 [14448.065478] worker_thread+0x296/0x6e0 [14448.065483] ? rescuer_thread+0x570/0x570 [14448.065493] kthread+0x1d1/0x200 [14448.065498] ? set_kthread_struct+0x70/0x70 [14448.065501] ret_from_fork+0x1f/0x30 [14448.192885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14448.219684] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14448.225632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14448.230575] RIP: 0010:native_safe_halt+0x2e/0x50 [14448.233208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14448.243433] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14448.247478] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14448.250681] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14448.254723] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14448.258739] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14448.262467] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14448.266224] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14448.270617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14448.273408] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14448.277482] Call Trace: [14448.278723] [14448.279853] ? show_regs.cold.9+0x22/0x2f [14448.282281] ? watchdog_timer_fn.cold.6+0x62/0x10a [14448.284931] ? __hrtimer_run_queues+0x1c2/0x4d0 [14448.287302] ? watchdog+0x40/0x40 [14448.288670] ? hrtimer_interrupt+0x128/0x2c0 [14448.290978] ? smp_apic_timer_interrupt+0xa6/0x270 [14448.293515] ? apic_timer_interrupt+0xf/0x20 [14448.295768] [14448.296736] ? native_safe_halt+0x2e/0x50 [14448.298934] kvm_wait+0xcd/0xe0 [14448.300553] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14448.303681] do_raw_spin_lock+0x177/0x190 [14448.305519] _raw_spin_lock+0x21/0x30 [14448.307807] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14448.310833] obd_mod_free+0x46/0x140 [ptlrpc] [14448.313711] mdc_commit_open+0x91/0xb0 [mdc] [14448.315791] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14448.318650] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14448.322178] after_reply+0xe2c/0x1780 [ptlrpc] [14448.324463] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14448.327351] ? __next_timer_interrupt+0x160/0x160 [14448.329722] ptlrpcd+0xa82/0xe40 [ptlrpc] [14448.332316] ? do_wait_intr+0xf0/0xf0 [14448.334294] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14448.336985] kthread+0x1d1/0x200 [14448.338631] ? set_kthread_struct+0x70/0x70 [14448.340507] ret_from_fork+0x1f/0x30 [14464.063471] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14464.071636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14464.114050] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14464.120587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14464.125123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14464.128242] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14464.137260] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14464.141621] RAX: 0000000000007598 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14464.146358] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14464.149250] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14464.154370] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14464.168465] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14464.171168] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14464.181509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14464.191114] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14464.197893] Call Trace: [14464.200527] [14464.202936] ? show_regs.cold.9+0x22/0x2f [14464.206454] ? watchdog_timer_fn.cold.6+0x62/0x10a [14464.209969] ? __hrtimer_run_queues+0x1c2/0x4d0 [14464.215604] ? watchdog+0x40/0x40 [14464.228685] ? hrtimer_interrupt+0x128/0x2c0 [14464.233923] ? smp_apic_timer_interrupt+0xa6/0x270 [14464.240212] ? apic_timer_interrupt+0xf/0x20 [14464.246070] [14464.247947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14464.252247] do_raw_spin_lock+0x177/0x190 [14464.257036] _raw_spin_lock+0x21/0x30 [14464.260280] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14464.266648] ? do_raw_spin_unlock+0x75/0x190 [14464.271582] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14464.277665] ? do_wait_intr+0xf0/0xf0 [14464.280154] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14464.295818] mdc_close+0x53b/0xd70 [mdc] [14464.297442] lmv_close+0x18d/0x490 [lmv] [14464.301045] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14464.306831] ll_md_real_close+0xa5/0x2a0 [lustre] [14464.315662] ll_md_close+0x274/0x800 [lustre] [14464.320916] ll_file_release+0x128/0xc50 [lustre] [14464.329553] __fput+0x10d/0x440 [14464.334366] ____fput+0x12/0x20 [14464.336716] task_work_run+0xc6/0x110 [14464.343993] exit_to_usermode_loop+0x1dd/0x1f0 [14464.350932] do_syscall_64+0x3d6/0x440 [14464.353605] entry_SYSCALL_64_after_hwframe+0x49/0xae [14464.359417] RIP: 0033:0x7fa749552cd8 [14464.361521] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14464.377447] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14464.388213] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14464.396541] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14464.409878] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14464.415419] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14464.417686] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14472.058160] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14472.062023] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14472.115540] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14472.123521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14472.140382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14472.152983] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14472.176430] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14472.179253] RAX: 000000000000698c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14472.182061] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14472.184936] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14472.187503] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14472.190242] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14472.193100] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14472.196822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14472.198966] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14472.201434] Call Trace: [14472.202594] [14472.203369] ? show_regs.cold.9+0x22/0x2f [14472.205091] ? watchdog_timer_fn.cold.6+0x62/0x10a [14472.207097] ? __hrtimer_run_queues+0x1c2/0x4d0 [14472.209119] ? watchdog+0x40/0x40 [14472.210620] ? hrtimer_interrupt+0x128/0x2c0 [14472.212598] ? smp_apic_timer_interrupt+0xa6/0x270 [14472.214876] ? apic_timer_interrupt+0xf/0x20 [14472.217336] [14472.219320] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14472.221997] do_raw_spin_lock+0x177/0x190 [14472.224895] _raw_spin_lock+0x21/0x30 [14472.227024] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14472.239146] ? do_raw_spin_unlock+0x75/0x190 [14472.241048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14472.250553] ? do_wait_intr+0xf0/0xf0 [14472.253066] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14472.256350] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14472.263685] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14472.266865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14472.271323] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14472.274074] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14472.277218] mdc_intent_lock+0x353/0x7e0 [mdc] [14472.281014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14472.284526] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14472.287527] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14472.290999] lmv_intent_open+0x338/0x1150 [lmv] [14472.292985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14472.296087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14472.300950] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14472.305224] ll_intent_lock+0xc5/0xaa0 [lustre] [14472.309550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14472.320819] ll_lookup_it+0xe0e/0x2270 [lustre] [14472.324371] ? do_raw_spin_unlock+0x75/0x190 [14472.329511] ? _raw_spin_unlock+0x12/0x30 [14472.332896] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14472.338405] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14472.342562] ll_atomic_open+0x36a/0xd70 [lustre] [14472.347042] lookup_open+0x356/0xb40 [14472.355775] do_last+0x313/0xfc0 [14472.356919] ? nd_jump_root+0xe5/0x160 [14472.358272] ? path_init+0x437/0x520 [14472.365021] path_openat+0xf7/0x500 [14472.369221] ? do_raw_spin_unlock+0x75/0x190 [14472.373366] do_filp_open+0x99/0x140 [14472.374626] ? getname_flags+0x6e/0x330 [14472.377838] ? __check_object_size+0xff/0x256 [14472.381152] ? do_raw_spin_unlock+0x75/0x190 [14472.384578] ? _raw_spin_unlock+0x12/0x30 [14472.390860] do_sys_openat2+0x2b4/0x410 [14472.395946] do_sys_open+0x73/0xa0 [14472.399986] __x64_sys_openat+0x24/0x30 [14472.407760] do_syscall_64+0xc1/0x440 [14472.412010] entry_SYSCALL_64_after_hwframe+0x49/0xae [14472.423978] RIP: 0033:0x7fa749552332 [14472.430477] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14472.449140] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14472.458326] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14472.470912] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14472.477523] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14472.482845] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14472.486814] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14474.723474] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12557s! [14474.731673] Showing busy workqueues and worker pools: [14474.738547] workqueue events: flags=0x0 [14474.741766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14474.766025] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14474.781587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14474.790718] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14474.799789] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.807511] pending: kfree_rcu_monitor [14474.811425] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14474.819879] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14474.827450] workqueue events_long: flags=0x0 [14474.829793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.833510] pending: lu_objects_flush [obdclass] [14474.839051] workqueue events_unbound: flags=0x2 [14474.843114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14474.849226] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14474.856890] workqueue events_power_efficient: flags=0x80 [14474.869010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.882676] pending: check_lifetime [14474.892894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.900708] pending: neigh_periodic_work [14474.905616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.913808] pending: neigh_periodic_work [14474.916914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.929250] pending: do_cache_clean [14474.933518] workqueue mm_percpu_wq: flags=0x8 [14474.935099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.952440] pending: vmstat_update [14474.957419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.966960] pending: vmstat_update [14474.974040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.991202] pending: vmstat_update [14474.997088] workqueue writeback: flags=0x4a [14475.006170] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14475.014686] pending: wb_workfn, wb_workfn [14475.016119] workqueue xprtiod: flags=0xa [14475.028633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14475.039708] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14475.045468] workqueue ipv6_addrconf: flags=0x40008 [14475.047284] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14475.068413] pending: addrconf_verify_work [14475.077789] workqueue knbd0-recv: flags=0x1a [14475.081818] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14475.086590] in-flight: 651:recv_work [14475.094066] workqueue ptlrpc_pinger: flags=0x2 [14475.100441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14475.108532] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14475.117369] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12557s workers=4 idle: 37 31846 28897 [14475.127993] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14465s workers=2 idle: 653 [14476.062496] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14476.063615] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14476.066988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14476.067068] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14476.067076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14476.067082] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14476.067617] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14476.067628] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14476.067632] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14476.067637] RAX: 0000000000007985 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14476.067639] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14476.067641] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14476.067643] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14476.067645] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14476.067648] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14476.067651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14476.067653] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14476.067661] Call Trace: [14476.067667] [14476.067672] ? show_regs.cold.9+0x22/0x2f [14476.067681] ? watchdog_timer_fn.cold.6+0x62/0x10a [14476.067687] ? __hrtimer_run_queues+0x1c2/0x4d0 [14476.067691] ? watchdog+0x40/0x40 [14476.067695] ? hrtimer_interrupt+0x128/0x2c0 [14476.067699] ? smp_apic_timer_interrupt+0xa6/0x270 [14476.067706] ? apic_timer_interrupt+0xf/0x20 [14476.067710] [14476.067713] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14476.067718] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14476.067975] do_raw_spin_lock+0x177/0x190 [14476.067980] _raw_spin_lock+0x21/0x30 [14476.067985] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14476.069224] process_one_work+0x2c8/0x700 [14476.069234] worker_thread+0x296/0x6e0 [14476.069239] ? rescuer_thread+0x570/0x570 [14476.069242] kthread+0x1d1/0x200 [14476.069247] ? set_kthread_struct+0x70/0x70 [14476.069251] ret_from_fork+0x1f/0x30 [14476.296833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14476.364748] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14476.375556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14476.395460] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14476.403986] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14476.432830] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14476.435597] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14476.443060] RDX: 00000000000045c7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14476.449395] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14476.455560] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14476.462898] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14476.468057] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14476.474990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14476.482523] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14476.491402] Call Trace: [14476.493614] [14476.496619] ? show_regs.cold.9+0x22/0x2f [14476.503068] ? watchdog_timer_fn.cold.6+0x62/0x10a [14476.513775] ? __hrtimer_run_queues+0x1c2/0x4d0 [14476.517839] ? watchdog+0x40/0x40 [14476.520826] ? hrtimer_interrupt+0x128/0x2c0 [14476.523893] ? smp_apic_timer_interrupt+0xa6/0x270 [14476.527464] ? apic_timer_interrupt+0xf/0x20 [14476.533319] [14476.536453] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14476.543844] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14476.548498] do_raw_spin_lock+0x177/0x190 [14476.553591] _raw_spin_lock+0x21/0x30 [14476.558638] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14476.566158] obd_mod_free+0x46/0x140 [ptlrpc] [14476.571355] mdc_commit_open+0x91/0xb0 [mdc] [14476.575562] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14476.581139] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14476.583862] after_reply+0xe2c/0x1780 [ptlrpc] [14476.585870] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14476.589340] ? __next_timer_interrupt+0x160/0x160 [14476.595652] ptlrpcd+0xa82/0xe40 [ptlrpc] [14476.603007] ? do_wait_intr+0xf0/0xf0 [14476.611653] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14476.620620] kthread+0x1d1/0x200 [14476.627853] ? set_kthread_struct+0x70/0x70 [14476.635673] ret_from_fork+0x1f/0x30 [14492.059035] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14492.063634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14492.085580] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14492.091905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14492.097716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14492.099727] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14492.109839] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14492.112970] RAX: 00000000000011c3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14492.116740] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14492.120254] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14492.124213] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14492.128258] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14492.132020] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14492.134960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14492.137959] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14492.141709] Call Trace: [14492.143598] [14492.144736] ? show_regs.cold.9+0x22/0x2f [14492.146950] ? watchdog_timer_fn.cold.6+0x62/0x10a [14492.149988] ? __hrtimer_run_queues+0x1c2/0x4d0 [14492.152591] ? watchdog+0x40/0x40 [14492.154591] ? hrtimer_interrupt+0x128/0x2c0 [14492.156666] ? smp_apic_timer_interrupt+0xa6/0x270 [14492.159060] ? apic_timer_interrupt+0xf/0x20 [14492.162821] [14492.163796] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14492.170022] do_raw_spin_lock+0x177/0x190 [14492.172144] _raw_spin_lock+0x21/0x30 [14492.174626] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14492.177911] ? do_raw_spin_unlock+0x75/0x190 [14492.180078] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14492.182788] ? do_wait_intr+0xf0/0xf0 [14492.184764] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14492.188111] mdc_close+0x53b/0xd70 [mdc] [14492.190113] lmv_close+0x18d/0x490 [lmv] [14492.192112] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14492.194730] ll_md_real_close+0xa5/0x2a0 [lustre] [14492.197048] ll_md_close+0x274/0x800 [lustre] [14492.198733] ll_file_release+0x128/0xc50 [lustre] [14492.201610] __fput+0x10d/0x440 [14492.203597] ____fput+0x12/0x20 [14492.205084] task_work_run+0xc6/0x110 [14492.206835] exit_to_usermode_loop+0x1dd/0x1f0 [14492.209567] do_syscall_64+0x3d6/0x440 [14492.211615] entry_SYSCALL_64_after_hwframe+0x49/0xae [14492.214074] RIP: 0033:0x7fa749552cd8 [14492.215976] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14492.225378] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14492.229060] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14492.232304] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14492.236228] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14492.240470] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14492.245228] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14500.054071] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14500.057472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14500.094886] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14500.103129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14500.108093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14500.112144] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14500.121904] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14500.125165] RAX: 0000000000007a91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14500.128995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14500.132086] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14500.135784] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14500.140025] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14500.143806] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14500.152461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14500.158021] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14500.166058] Call Trace: [14500.168435] [14500.169218] ? show_regs.cold.9+0x22/0x2f [14500.170556] ? watchdog_timer_fn.cold.6+0x62/0x10a [14500.176909] ? __hrtimer_run_queues+0x1c2/0x4d0 [14500.181059] ? watchdog+0x40/0x40 [14500.184945] ? hrtimer_interrupt+0x128/0x2c0 [14500.189692] ? smp_apic_timer_interrupt+0xa6/0x270 [14500.193162] ? apic_timer_interrupt+0xf/0x20 [14500.196586] [14500.198120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14500.201558] do_raw_spin_lock+0x177/0x190 [14500.207095] _raw_spin_lock+0x21/0x30 [14500.209669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14500.215946] ? do_raw_spin_unlock+0x75/0x190 [14500.218061] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14500.225265] ? do_wait_intr+0xf0/0xf0 [14500.227896] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14500.237550] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14500.245619] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14500.257143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14500.265040] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14500.271943] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14500.276267] mdc_intent_lock+0x353/0x7e0 [mdc] [14500.279856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14500.283888] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14500.289057] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14500.293280] lmv_intent_open+0x338/0x1150 [lmv] [14500.303154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14500.306021] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14500.312240] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14500.317777] ll_intent_lock+0xc5/0xaa0 [lustre] [14500.322055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14500.324456] ll_lookup_it+0xe0e/0x2270 [lustre] [14500.330917] ? do_raw_spin_unlock+0x75/0x190 [14500.332863] ? _raw_spin_unlock+0x12/0x30 [14500.336253] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14500.340896] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14500.345057] ll_atomic_open+0x36a/0xd70 [lustre] [14500.352859] lookup_open+0x356/0xb40 [14500.354102] do_last+0x313/0xfc0 [14500.355163] ? nd_jump_root+0xe5/0x160 [14500.366175] ? path_init+0x437/0x520 [14500.368533] path_openat+0xf7/0x500 [14500.373773] ? do_raw_spin_unlock+0x75/0x190 [14500.376324] do_filp_open+0x99/0x140 [14500.382428] ? getname_flags+0x6e/0x330 [14500.384921] ? __check_object_size+0xff/0x256 [14500.389845] ? do_raw_spin_unlock+0x75/0x190 [14500.392975] ? _raw_spin_unlock+0x12/0x30 [14500.395231] do_sys_openat2+0x2b4/0x410 [14500.397218] do_sys_open+0x73/0xa0 [14500.398994] __x64_sys_openat+0x24/0x30 [14500.400856] do_syscall_64+0xc1/0x440 [14500.405084] entry_SYSCALL_64_after_hwframe+0x49/0xae [14500.407108] RIP: 0033:0x7fa749552332 [14500.409919] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14500.427058] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14500.432631] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14500.435816] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14500.439422] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14500.443245] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14500.448640] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14504.058209] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14504.062165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14504.063240] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14504.096721] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14504.096736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14504.096742] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14504.097102] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14504.097113] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14504.097118] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14504.097122] RAX: 0000000000007780 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14504.097125] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14504.097127] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14504.097129] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14504.097132] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14504.097136] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14504.097138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14504.097140] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14504.097148] Call Trace: [14504.097153] [14504.097159] ? show_regs.cold.9+0x22/0x2f [14504.097167] ? watchdog_timer_fn.cold.6+0x62/0x10a [14504.097173] ? __hrtimer_run_queues+0x1c2/0x4d0 [14504.097178] ? watchdog+0x40/0x40 [14504.097182] ? hrtimer_interrupt+0x128/0x2c0 [14504.097186] ? smp_apic_timer_interrupt+0xa6/0x270 [14504.097193] ? apic_timer_interrupt+0xf/0x20 [14504.097197] [14504.097199] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14504.097204] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14504.097208] do_raw_spin_lock+0x177/0x190 [14504.097213] _raw_spin_lock+0x21/0x30 [14504.097218] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14504.097421] process_one_work+0x2c8/0x700 [14504.097428] worker_thread+0x296/0x6e0 [14504.097433] ? rescuer_thread+0x570/0x570 [14504.097437] kthread+0x1d1/0x200 [14504.097441] ? set_kthread_struct+0x70/0x70 [14504.097444] ret_from_fork+0x1f/0x30 [14504.290208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14504.319092] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14504.325707] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14504.339412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14504.345730] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14504.358518] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14504.363479] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14504.366947] RDX: 00000000000065df RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14504.370427] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14504.374738] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14504.378292] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14504.385254] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14504.395707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14504.402215] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14504.410958] Call Trace: [14504.412427] [14504.418783] ? show_regs.cold.9+0x22/0x2f [14504.424941] ? watchdog_timer_fn.cold.6+0x62/0x10a [14504.426764] ? __hrtimer_run_queues+0x1c2/0x4d0 [14504.434962] ? watchdog+0x40/0x40 [14504.440526] ? hrtimer_interrupt+0x128/0x2c0 [14504.444472] ? smp_apic_timer_interrupt+0xa6/0x270 [14504.454130] ? apic_timer_interrupt+0xf/0x20 [14504.459233] [14504.460348] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14504.466277] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14504.470430] do_raw_spin_lock+0x177/0x190 [14504.481048] _raw_spin_lock+0x21/0x30 [14504.483382] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14504.493943] obd_mod_free+0x46/0x140 [ptlrpc] [14504.503867] mdc_commit_open+0x91/0xb0 [mdc] [14504.508543] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14504.521317] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14504.535167] after_reply+0xe2c/0x1780 [ptlrpc] [14504.547221] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14504.557152] ? __next_timer_interrupt+0x160/0x160 [14504.567229] ptlrpcd+0xa82/0xe40 [ptlrpc] [14504.577244] ? do_wait_intr+0xf0/0xf0 [14504.578623] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14504.592127] kthread+0x1d1/0x200 [14504.599204] ? set_kthread_struct+0x70/0x70 [14504.600823] ret_from_fork+0x1f/0x30 [14505.444058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12588s! [14505.450108] Showing busy workqueues and worker pools: [14505.452462] workqueue events: flags=0x0 [14505.454973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14505.457663] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14505.462783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14505.465830] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14505.469747] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.472412] pending: kfree_rcu_monitor [14505.475013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14505.478800] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14505.483177] workqueue events_long: flags=0x0 [14505.485165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.488112] pending: lu_objects_flush [obdclass] [14505.492300] workqueue events_unbound: flags=0x2 [14505.495058] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14505.498562] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14505.509189] workqueue events_power_efficient: flags=0x80 [14505.511967] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.518422] pending: check_lifetime [14505.521132] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.526518] pending: neigh_periodic_work [14505.529961] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.534858] pending: neigh_periodic_work [14505.538914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.542574] pending: do_cache_clean [14505.547757] workqueue mm_percpu_wq: flags=0x8 [14505.548981] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.551854] pending: vmstat_update [14505.555463] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.559173] pending: vmstat_update [14505.563366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.566524] pending: vmstat_update [14505.569359] workqueue writeback: flags=0x4a [14505.572494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14505.575985] pending: wb_workfn, wb_workfn [14505.578322] workqueue xprtiod: flags=0xa [14505.581617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14505.585898] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14505.592914] workqueue ipv6_addrconf: flags=0x40008 [14505.595302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14505.598047] pending: addrconf_verify_work [14505.600099] workqueue knbd0-recv: flags=0x1a [14505.601968] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14505.604284] in-flight: 651:recv_work [14505.605553] workqueue ptlrpc_pinger: flags=0x2 [14505.607470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14505.612662] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14505.618143] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12588s workers=4 idle: 37 31846 28897 [14505.629259] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14495s workers=2 idle: 653 [14520.059032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14520.062214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14520.082012] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14520.087159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14520.091215] RIP: 0010:native_safe_halt+0x2e/0x50 [14520.093637] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14520.101802] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14520.105072] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14520.108167] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14520.111542] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14520.114674] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14520.118065] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14520.120896] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14520.124498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14520.126672] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14520.130107] Call Trace: [14520.131185] [14520.132162] ? show_regs.cold.9+0x22/0x2f [14520.133886] ? watchdog_timer_fn.cold.6+0x62/0x10a [14520.135920] ? __hrtimer_run_queues+0x1c2/0x4d0 [14520.137878] ? watchdog+0x40/0x40 [14520.139254] ? hrtimer_interrupt+0x128/0x2c0 [14520.141175] ? smp_apic_timer_interrupt+0xa6/0x270 [14520.143802] ? apic_timer_interrupt+0xf/0x20 [14520.145834] [14520.146717] ? native_safe_halt+0x2e/0x50 [14520.148287] kvm_wait+0xcd/0xe0 [14520.149826] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14520.154720] do_raw_spin_lock+0x177/0x190 [14520.156479] _raw_spin_lock+0x21/0x30 [14520.157923] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14520.160174] ? do_raw_spin_unlock+0x75/0x190 [14520.162017] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14520.164309] ? do_wait_intr+0xf0/0xf0 [14520.166557] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14520.169477] mdc_close+0x53b/0xd70 [mdc] [14520.171207] lmv_close+0x18d/0x490 [lmv] [14520.173043] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14520.175782] ll_md_real_close+0xa5/0x2a0 [lustre] [14520.177870] ll_md_close+0x274/0x800 [lustre] [14520.179726] ll_file_release+0x128/0xc50 [lustre] [14520.182008] __fput+0x10d/0x440 [14520.183320] ____fput+0x12/0x20 [14520.184642] task_work_run+0xc6/0x110 [14520.186380] exit_to_usermode_loop+0x1dd/0x1f0 [14520.188304] do_syscall_64+0x3d6/0x440 [14520.190005] entry_SYSCALL_64_after_hwframe+0x49/0xae [14520.191815] RIP: 0033:0x7fa749552cd8 [14520.194081] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14520.201284] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14520.205236] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14520.208319] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14520.211238] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14520.214053] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14520.216774] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14532.057371] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14532.058230] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14532.059264] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14532.061732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14532.061809] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14532.061815] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14532.061818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14532.061832] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14532.061837] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14532.061841] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14532.061844] RDX: 0000000000003992 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14532.061847] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14532.061849] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14532.061851] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14532.061854] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14532.061857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14532.061860] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14532.061868] Call Trace: [14532.061872] [14532.061878] ? show_regs.cold.9+0x22/0x2f [14532.061885] ? watchdog_timer_fn.cold.6+0x62/0x10a [14532.061891] ? __hrtimer_run_queues+0x1c2/0x4d0 [14532.061895] ? watchdog+0x40/0x40 [14532.061899] ? hrtimer_interrupt+0x128/0x2c0 [14532.061904] ? smp_apic_timer_interrupt+0xa6/0x270 [14532.061911] ? apic_timer_interrupt+0xf/0x20 [14532.061916] [14532.061919] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14532.061923] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14532.061928] do_raw_spin_lock+0x177/0x190 [14532.061932] _raw_spin_lock+0x21/0x30 [14532.061937] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14532.064276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [14532.064200] obd_mod_free+0x46/0x140 [ptlrpc] [14532.064349] sha512_generic [14532.064359] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14532.064365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14532.064769] mdc_commit_open+0x91/0xb0 [mdc] [14532.064805] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14532.065168] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14532.064371] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14532.065575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14532.065588] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14532.065592] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14532.065595] RAX: 0000000000001b09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14532.065598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14532.065600] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14532.065603] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14532.065605] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14532.065609] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14532.065612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14532.065614] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14532.065622] Call Trace: [14532.065626] [14532.065629] ? show_regs.cold.9+0x22/0x2f [14532.065637] ? watchdog_timer_fn.cold.6+0x62/0x10a [14532.065648] ? __hrtimer_run_queues+0x1c2/0x4d0 [14532.065655] ? watchdog+0x40/0x40 [14532.065660] ? hrtimer_interrupt+0x128/0x2c0 [14532.065664] ? smp_apic_timer_interrupt+0xa6/0x270 [14532.065672] ? apic_timer_interrupt+0xf/0x20 [14532.065677] [14532.065679] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14532.065684] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14532.065688] do_raw_spin_lock+0x177/0x190 [14532.065693] _raw_spin_lock+0x21/0x30 [14532.065698] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14532.066039] process_one_work+0x2c8/0x700 [14532.066045] worker_thread+0x296/0x6e0 [14532.066049] ? rescuer_thread+0x570/0x570 [14532.066052] kthread+0x1d1/0x200 [14532.066056] ? set_kthread_struct+0x70/0x70 [14532.066059] ret_from_fork+0x1f/0x30 [14532.065650] after_reply+0xe2c/0x1780 [ptlrpc] [14532.070455] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14532.070550] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14532.070558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14532.070561] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14532.070574] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14532.070579] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14532.070583] RAX: 0000000000007319 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14532.070586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14532.070588] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14532.070591] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14532.070594] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14532.070597] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14532.070600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14532.070603] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14532.070611] Call Trace: [14532.070615] [14532.070620] ? show_regs.cold.9+0x22/0x2f [14532.070629] ? watchdog_timer_fn.cold.6+0x62/0x10a [14532.070635] ? __hrtimer_run_queues+0x1c2/0x4d0 [14532.070639] ? watchdog+0x40/0x40 [14532.070643] ? hrtimer_interrupt+0x128/0x2c0 [14532.070647] ? smp_apic_timer_interrupt+0xa6/0x270 [14532.070655] ? apic_timer_interrupt+0xf/0x20 [14532.070659] [14532.070662] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14532.070669] do_raw_spin_lock+0x177/0x190 [14532.070673] _raw_spin_lock+0x21/0x30 [14532.070678] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14532.071347] ? do_raw_spin_unlock+0x75/0x190 [14532.071353] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14532.071688] ? do_wait_intr+0xf0/0xf0 [14532.071696] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14532.072394] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14532.072610] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14532.072956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.073368] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14532.077275] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14532.100641] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14532.100696] mdc_intent_lock+0x353/0x7e0 [mdc] [14532.100733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.100848] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14532.101424] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14532.101457] lmv_intent_open+0x338/0x1150 [lmv] [14532.101488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.101580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.101663] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14532.101685] ll_intent_lock+0xc5/0xaa0 [lustre] [14532.101771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.101852] ll_lookup_it+0xe0e/0x2270 [lustre] [14532.101922] ? do_raw_spin_unlock+0x75/0x190 [14532.101929] ? _raw_spin_unlock+0x12/0x30 [14532.101935] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14532.101959] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14532.101973] ll_atomic_open+0x36a/0xd70 [lustre] [14532.102045] lookup_open+0x356/0xb40 [14532.102055] do_last+0x313/0xfc0 [14532.102060] ? nd_jump_root+0xe5/0x160 [14532.102065] ? path_init+0x437/0x520 [14532.102068] path_openat+0xf7/0x500 [14532.102072] ? do_raw_spin_unlock+0x75/0x190 [14532.102076] do_filp_open+0x99/0x140 [14532.102080] ? getname_flags+0x6e/0x330 [14532.102084] ? __check_object_size+0xff/0x256 [14532.102091] ? do_raw_spin_unlock+0x75/0x190 [14532.102093] ? _raw_spin_unlock+0x12/0x30 [14532.102096] do_sys_openat2+0x2b4/0x410 [14532.102099] do_sys_open+0x73/0xa0 [14532.102102] __x64_sys_openat+0x24/0x30 [14532.102104] do_syscall_64+0xc1/0x440 [14532.102109] entry_SYSCALL_64_after_hwframe+0x49/0xae [14532.102114] RIP: 0033:0x7fa749552332 [14532.104656] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14532.104661] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14532.104667] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14532.104669] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14532.104672] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14532.104674] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14532.104676] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14532.987262] ? __next_timer_interrupt+0x160/0x160 [14532.989131] ptlrpcd+0xa82/0xe40 [ptlrpc] [14532.999968] ? do_wait_intr+0xf0/0xf0 [14533.003859] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14533.009189] kthread+0x1d1/0x200 [14533.023886] ? set_kthread_struct+0x70/0x70 [14533.035762] ret_from_fork+0x1f/0x30 [14535.648160] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12618s! [14535.653741] Showing busy workqueues and worker pools: [14535.657131] workqueue events: flags=0x0 [14535.660122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14535.664822] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14535.673887] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14535.678726] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14535.684774] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.689535] pending: kfree_rcu_monitor [14535.692120] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14535.697406] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14535.703673] workqueue events_long: flags=0x0 [14535.706650] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.711092] pending: lu_objects_flush [obdclass] [14535.715550] workqueue events_unbound: flags=0x2 [14535.718544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14535.722858] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14535.729100] workqueue events_power_efficient: flags=0x80 [14535.733390] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.737947] pending: check_lifetime [14535.741335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.745697] pending: neigh_periodic_work [14535.750516] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.755222] pending: neigh_periodic_work [14535.758483] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.764279] pending: do_cache_clean [14535.767033] workqueue mm_percpu_wq: flags=0x8 [14535.771074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.775554] pending: vmstat_update [14535.778530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.783937] pending: vmstat_update [14535.786669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14535.791971] pending: vmstat_update [14535.794682] workqueue writeback: flags=0x4a [14535.798018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14535.802209] pending: wb_workfn, wb_workfn [14535.804915] workqueue xprtiod: flags=0xa [14535.807247] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14535.811190] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14535.816069] workqueue ipv6_addrconf: flags=0x40008 [14535.818963] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14535.824577] pending: addrconf_verify_work [14535.827689] workqueue knbd0-recv: flags=0x1a [14535.831786] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14535.834858] in-flight: 651:recv_work [14535.837195] workqueue ptlrpc_pinger: flags=0x2 [14535.840019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14535.843469] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14535.847113] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12618s workers=4 idle: 37 31846 28897 [14535.852220] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14525s workers=2 idle: 653 [14548.059362] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14548.067394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14548.091649] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14548.096399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14548.100192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14548.103705] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14548.122647] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14548.126958] RAX: 0000000000000d0c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14548.133758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14548.136200] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14548.146453] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14548.152476] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14548.156768] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14548.161300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14548.166690] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14548.179678] Call Trace: [14548.180294] [14548.180739] ? show_regs.cold.9+0x22/0x2f [14548.183350] ? watchdog_timer_fn.cold.6+0x62/0x10a [14548.187634] ? __hrtimer_run_queues+0x1c2/0x4d0 [14548.190649] ? watchdog+0x40/0x40 [14548.194949] ? hrtimer_interrupt+0x128/0x2c0 [14548.199146] ? smp_apic_timer_interrupt+0xa6/0x270 [14548.204147] ? apic_timer_interrupt+0xf/0x20 [14548.206808] [14548.209070] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14548.214260] do_raw_spin_lock+0x177/0x190 [14548.218055] _raw_spin_lock+0x21/0x30 [14548.222733] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14548.228844] ? do_raw_spin_unlock+0x75/0x190 [14548.231308] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14548.233780] ? do_wait_intr+0xf0/0xf0 [14548.236894] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14548.240681] mdc_close+0x53b/0xd70 [mdc] [14548.243134] lmv_close+0x18d/0x490 [lmv] [14548.249601] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14548.254775] ll_md_real_close+0xa5/0x2a0 [lustre] [14548.259651] ll_md_close+0x274/0x800 [lustre] [14548.267004] ll_file_release+0x128/0xc50 [lustre] [14548.269651] __fput+0x10d/0x440 [14548.272762] ____fput+0x12/0x20 [14548.275165] task_work_run+0xc6/0x110 [14548.276992] exit_to_usermode_loop+0x1dd/0x1f0 [14548.279221] do_syscall_64+0x3d6/0x440 [14548.280988] entry_SYSCALL_64_after_hwframe+0x49/0xae [14548.283435] RIP: 0033:0x7fa749552cd8 [14548.285419] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14548.301741] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14548.308663] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14548.312195] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14548.320888] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14548.324075] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14548.326891] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14560.054594] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14560.058036] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14560.061980] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14560.061995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14560.062094] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14560.062100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14560.062107] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14560.063971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14560.063989] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14560.063993] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14560.064047] RAX: 0000000000007bc9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14560.064050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14560.064052] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14560.064054] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14560.064057] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14560.064060] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14560.064063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14560.064065] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14560.064073] Call Trace: [14560.064079] [14560.064085] ? show_regs.cold.9+0x22/0x2f [14560.064094] ? watchdog_timer_fn.cold.6+0x62/0x10a [14560.064100] ? __hrtimer_run_queues+0x1c2/0x4d0 [14560.064105] ? watchdog+0x40/0x40 [14560.064109] ? hrtimer_interrupt+0x128/0x2c0 [14560.064113] ? smp_apic_timer_interrupt+0xa6/0x270 [14560.064121] ? apic_timer_interrupt+0xf/0x20 [14560.064125] [14560.064128] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14560.064132] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14560.064143] do_raw_spin_lock+0x177/0x190 [14560.064148] _raw_spin_lock+0x21/0x30 [14560.064152] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14560.065297] process_one_work+0x2c8/0x700 [14560.065378] worker_thread+0x296/0x6e0 [14560.065384] ? rescuer_thread+0x570/0x570 [14560.065465] kthread+0x1d1/0x200 [14560.065471] ? set_kthread_struct+0x70/0x70 [14560.065475] ret_from_fork+0x1f/0x30 [14560.067686] Modules linked in: lustre(O) [14560.070006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14560.070141] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14560.070148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14560.070150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14560.070164] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14560.070168] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14560.070172] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14560.070175] RDX: 00000000000056aa RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14560.070177] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14560.070179] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14560.070182] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14560.070189] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14560.070192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14560.070194] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14560.070200] Call Trace: [14560.070205] [14560.070210] ? show_regs.cold.9+0x22/0x2f [14560.070220] ? watchdog_timer_fn.cold.6+0x62/0x10a [14560.070227] ? __hrtimer_run_queues+0x1c2/0x4d0 [14560.070230] ? watchdog+0x40/0x40 [14560.070235] ? hrtimer_interrupt+0x128/0x2c0 [14560.070238] ? smp_apic_timer_interrupt+0xa6/0x270 [14560.070246] ? apic_timer_interrupt+0xf/0x20 [14560.070250] [14560.070252] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14560.070256] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14560.070260] do_raw_spin_lock+0x177/0x190 [14560.070264] _raw_spin_lock+0x21/0x30 [14560.070270] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14560.070618] obd_mod_free+0x46/0x140 [ptlrpc] [14560.070851] mdc_commit_open+0x91/0xb0 [mdc] [14560.070944] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14560.071262] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14560.071643] after_reply+0xe2c/0x1780 [ptlrpc] [14560.071789] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14560.072101] ? __next_timer_interrupt+0x160/0x160 [14560.072110] ptlrpcd+0xa82/0xe40 [ptlrpc] [14560.072314] ? do_wait_intr+0xf0/0xf0 [14560.072322] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14560.072591] kthread+0x1d1/0x200 [14560.072595] ? set_kthread_struct+0x70/0x70 [14560.072597] ret_from_fork+0x1f/0x30 [14560.411584] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14560.448843] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14560.460366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14560.469689] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14560.473014] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14560.489152] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14560.494057] RAX: 0000000000004313 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14560.496595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14560.503716] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14560.510018] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14560.516024] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14560.518786] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14560.531601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14560.541400] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14560.550343] Call Trace: [14560.555587] [14560.556090] ? show_regs.cold.9+0x22/0x2f [14560.559528] ? watchdog_timer_fn.cold.6+0x62/0x10a [14560.563822] ? __hrtimer_run_queues+0x1c2/0x4d0 [14560.565444] ? watchdog+0x40/0x40 [14560.566483] ? hrtimer_interrupt+0x128/0x2c0 [14560.572550] ? smp_apic_timer_interrupt+0xa6/0x270 [14560.574007] ? apic_timer_interrupt+0xf/0x20 [14560.575626] [14560.576326] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14560.577901] do_raw_spin_lock+0x177/0x190 [14560.579531] _raw_spin_lock+0x21/0x30 [14560.581385] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14560.583343] ? do_raw_spin_unlock+0x75/0x190 [14560.584640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14560.586366] ? do_wait_intr+0xf0/0xf0 [14560.587875] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14560.589813] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14560.592137] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14560.594144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.595898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14560.597377] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14560.598818] mdc_intent_lock+0x353/0x7e0 [mdc] [14560.600078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.601989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14560.603813] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14560.605279] lmv_intent_open+0x338/0x1150 [lmv] [14560.606366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.607859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.609316] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14560.610332] ll_intent_lock+0xc5/0xaa0 [lustre] [14560.611973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.613272] ll_lookup_it+0xe0e/0x2270 [lustre] [14560.614359] ? do_raw_spin_unlock+0x75/0x190 [14560.615896] ? _raw_spin_unlock+0x12/0x30 [14560.616978] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14560.618414] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14560.619746] ll_atomic_open+0x36a/0xd70 [lustre] [14560.621167] lookup_open+0x356/0xb40 [14560.622117] do_last+0x313/0xfc0 [14560.623182] ? nd_jump_root+0xe5/0x160 [14560.624149] ? path_init+0x437/0x520 [14560.625178] path_openat+0xf7/0x500 [14560.626097] ? do_raw_spin_unlock+0x75/0x190 [14560.627246] do_filp_open+0x99/0x140 [14560.628173] ? getname_flags+0x6e/0x330 [14560.629368] ? __check_object_size+0xff/0x256 [14560.630817] ? do_raw_spin_unlock+0x75/0x190 [14560.631984] ? _raw_spin_unlock+0x12/0x30 [14560.633734] do_sys_openat2+0x2b4/0x410 [14560.634721] do_sys_open+0x73/0xa0 [14560.635600] __x64_sys_openat+0x24/0x30 [14560.636654] do_syscall_64+0xc1/0x440 [14560.637697] entry_SYSCALL_64_after_hwframe+0x49/0xae [14560.639061] RIP: 0033:0x7fa749552332 [14560.639991] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14560.644812] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14560.646914] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14560.648978] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14560.651016] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14560.653035] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14560.655189] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14565.858361] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12648s! [14565.864863] Showing busy workqueues and worker pools: [14565.868017] workqueue events: flags=0x0 [14565.873099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14565.881359] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14565.890231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14565.897374] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14565.908359] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14565.920535] pending: kfree_rcu_monitor [14565.929745] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14565.942349] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14565.961458] workqueue events_long: flags=0x0 [14565.963413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14565.971762] pending: lu_objects_flush [obdclass] [14565.977504] workqueue events_unbound: flags=0x2 [14565.981029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14565.984271] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14565.987769] workqueue events_power_efficient: flags=0x80 [14565.995505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14566.005437] pending: check_lifetime [14566.010196] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14566.015391] pending: neigh_periodic_work [14566.020919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14566.025343] pending: neigh_periodic_work [14566.027989] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14566.037964] pending: do_cache_clean [14566.040097] workqueue mm_percpu_wq: flags=0x8 [14566.049308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14566.055012] pending: vmstat_update [14566.058212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14566.065586] pending: vmstat_update [14566.068366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14566.077237] pending: vmstat_update [14566.081433] workqueue writeback: flags=0x4a [14566.084121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14566.091358] pending: wb_workfn, wb_workfn [14566.094436] workqueue xprtiod: flags=0xa [14566.098150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14566.105124] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14566.117136] workqueue ipv6_addrconf: flags=0x40008 [14566.123897] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14566.128819] pending: addrconf_verify_work [14566.131606] workqueue knbd0-recv: flags=0x1a [14566.134361] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14566.142194] in-flight: 651:recv_work [14566.144197] workqueue ptlrpc_pinger: flags=0x2 [14566.149644] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14566.151613] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14566.161048] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12649s workers=4 idle: 37 31846 28897 [14566.164912] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14556s workers=2 idle: 653 [14573.963514] rcu: INFO: rcu_sched self-detected stall on CPU [14573.968183] rcu: 2-...!: (7268314 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [14573.982812] (t=12660274 jiffies g=258701 q=4979) [14573.987318] rcu: rcu_sched kthread starved for 12657069 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [14574.002736] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14574.011112] rcu: RCU grace-period kthread stack dump: [14574.014243] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14574.026747] Call Trace: [14574.030742] __schedule+0x351/0xcb0 [14574.036610] schedule+0xc0/0x180 [14574.039150] schedule_timeout+0xb4/0x190 [14574.044587] ? do_raw_spin_unlock+0x75/0x190 [14574.048667] ? __next_timer_interrupt+0x160/0x160 [14574.055046] ? prepare_to_swait_event+0xf3/0x140 [14574.059050] rcu_gp_kthread+0x8dd/0x15b0 [14574.061322] ? force_qs_rnp+0x2c0/0x2c0 [14574.065552] kthread+0x1d1/0x200 [14574.067245] ? set_kthread_struct+0x70/0x70 [14574.070631] ret_from_fork+0x1f/0x30 [14574.072972] rcu: Stack dump where RCU GP kthread last ran: [14574.083251] Sending NMI from CPU 2 to CPUs 0: [14574.095163] NMI backtrace for cpu 0 [14574.095172] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14574.095175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14574.095176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14574.095178] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14574.095181] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [14574.095183] RAX: 0000000000005308 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14574.095185] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14574.095186] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14574.095188] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14574.095189] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14574.095191] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14574.095192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14574.095193] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14574.095194] Call Trace: [14574.095195] [14574.095196] ? show_regs.cold.9+0x22/0x2f [14574.095197] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14574.095198] ? nmi_cpu_backtrace_handler+0x15/0x40 [14574.095199] ? nmi_handle+0xbd/0x210 [14574.095201] ? default_do_nmi+0x4b/0x1b0 [14574.095202] ? do_nmi+0x280/0x380 [14574.095203] ? end_repeat_nmi+0x16/0x1f [14574.095203] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14574.095205] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14574.095206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14574.095206] [14574.095207] do_raw_spin_lock+0x177/0x190 [14574.095208] _raw_spin_lock+0x21/0x30 [14574.095209] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14574.095210] ? do_raw_spin_unlock+0x75/0x190 [14574.095211] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14574.095212] ? do_wait_intr+0xf0/0xf0 [14574.095214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14574.095215] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14574.095216] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14574.095217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14574.095218] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14574.095219] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14574.095220] mdc_intent_lock+0x353/0x7e0 [mdc] [14574.095221] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14574.095223] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14574.095224] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14574.095225] lmv_intent_open+0x338/0x1150 [lmv] [14574.095226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14574.095227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14574.095228] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14574.095229] ll_intent_lock+0xc5/0xaa0 [lustre] [14574.095230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14574.095231] ll_lookup_it+0xe0e/0x2270 [lustre] [14574.095232] ? do_raw_spin_unlock+0x75/0x190 [14574.095233] ? _raw_spin_unlock+0x12/0x30 [14574.095234] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14574.095235] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14574.095237] ll_atomic_open+0x36a/0xd70 [lustre] [14574.095237] lookup_open+0x356/0xb40 [14574.095238] do_last+0x313/0xfc0 [14574.095239] ? nd_jump_root+0xe5/0x160 [14574.095240] ? path_init+0x437/0x520 [14574.095241] path_openat+0xf7/0x500 [14574.095242] ? do_raw_spin_unlock+0x75/0x190 [14574.095243] do_filp_open+0x99/0x140 [14574.095244] ? getname_flags+0x6e/0x330 [14574.095246] ? __check_object_size+0xff/0x256 [14574.095247] ? do_raw_spin_unlock+0x75/0x190 [14574.095248] ? _raw_spin_unlock+0x12/0x30 [14574.095249] do_sys_openat2+0x2b4/0x410 [14574.095250] do_sys_open+0x73/0xa0 [14574.095250] __x64_sys_openat+0x24/0x30 [14574.095251] do_syscall_64+0xc1/0x440 [14574.095252] entry_SYSCALL_64_after_hwframe+0x49/0xae [14574.095253] RIP: 0033:0x7fa749552332 [14574.095254] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14574.095256] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14574.095258] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14574.095271] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14574.095272] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14574.095274] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14574.095274] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14574.103951] NMI backtrace for cpu 2 [14574.399164] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14574.405697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14574.409790] Call Trace: [14574.410927] [14574.411868] ? dump_stack+0xbb/0x10e [14574.413334] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14574.416023] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14574.418281] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14574.420837] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14574.423067] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14574.425923] ? rcu_dump_cpu_stacks+0x11b/0x171 [14574.428010] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14574.430177] ? update_process_times+0x6d/0xc0 [14574.432975] ? tick_sched_handle.isra.18+0x27/0x90 [14574.435180] ? tick_sched_timer+0x53/0xb0 [14574.437127] ? __hrtimer_run_queues+0x1c2/0x4d0 [14574.439332] ? tick_sched_handle.isra.18+0x90/0x90 [14574.441790] ? hrtimer_interrupt+0x128/0x2c0 [14574.443883] ? smp_apic_timer_interrupt+0xa6/0x270 [14574.451339] ? apic_timer_interrupt+0xf/0x20 [14574.455849] [14574.458448] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14574.461270] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14574.464063] ? do_raw_spin_lock+0x177/0x190 [14574.466135] ? _raw_spin_lock+0x21/0x30 [14574.468059] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14574.470893] ? obd_mod_free+0x46/0x140 [ptlrpc] [14574.473362] ? mdc_commit_open+0x91/0xb0 [mdc] [14574.476938] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14574.480254] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14574.486228] ? after_reply+0xe2c/0x1780 [ptlrpc] [14574.487953] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14574.499291] ? __next_timer_interrupt+0x160/0x160 [14574.507981] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14574.511148] ? do_wait_intr+0xf0/0xf0 [14574.516518] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14574.523280] ? kthread+0x1d1/0x200 [14574.525863] ? set_kthread_struct+0x70/0x70 [14574.528064] ? ret_from_fork+0x1f/0x30 [14576.059045] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14576.062538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14576.083773] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14576.089926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14576.094957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14576.097891] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14576.106090] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14576.109199] RAX: 0000000000004687 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14576.112723] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14576.115776] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14576.119279] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14576.122788] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14576.126093] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14576.130154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14576.132893] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14576.136214] Call Trace: [14576.137151] [14576.138249] ? show_regs.cold.9+0x22/0x2f [14576.140682] ? watchdog_timer_fn.cold.6+0x62/0x10a [14576.142580] ? __hrtimer_run_queues+0x1c2/0x4d0 [14576.145200] ? watchdog+0x40/0x40 [14576.146779] ? hrtimer_interrupt+0x128/0x2c0 [14576.148819] ? smp_apic_timer_interrupt+0xa6/0x270 [14576.151458] ? apic_timer_interrupt+0xf/0x20 [14576.153532] [14576.154526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14576.157011] do_raw_spin_lock+0x177/0x190 [14576.158958] _raw_spin_lock+0x21/0x30 [14576.160650] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14576.163252] ? do_raw_spin_unlock+0x75/0x190 [14576.165262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14576.167853] ? do_wait_intr+0xf0/0xf0 [14576.169535] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14576.172096] mdc_close+0x53b/0xd70 [mdc] [14576.174249] lmv_close+0x18d/0x490 [lmv] [14576.176113] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14576.179003] ll_md_real_close+0xa5/0x2a0 [lustre] [14576.181136] ll_md_close+0x274/0x800 [lustre] [14576.183592] ll_file_release+0x128/0xc50 [lustre] [14576.186083] __fput+0x10d/0x440 [14576.187554] ____fput+0x12/0x20 [14576.189141] task_work_run+0xc6/0x110 [14576.191082] exit_to_usermode_loop+0x1dd/0x1f0 [14576.193287] do_syscall_64+0x3d6/0x440 [14576.195517] entry_SYSCALL_64_after_hwframe+0x49/0xae [14576.197843] RIP: 0033:0x7fa749552cd8 [14576.200090] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14576.208961] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14576.212154] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14576.216074] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14576.219517] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14576.223054] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14576.226827] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14588.057017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14588.062276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14588.099043] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14588.106301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14588.111066] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14588.114922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14588.117625] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14588.143043] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14588.151317] RAX: 0000000000003c82 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14588.159086] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14588.168841] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14588.172906] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14588.187576] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14588.199084] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14588.209126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14588.220108] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14588.230092] Call Trace: [14588.231701] [14588.236277] ? show_regs.cold.9+0x22/0x2f [14588.243830] ? watchdog_timer_fn.cold.6+0x62/0x10a [14588.245281] ? __hrtimer_run_queues+0x1c2/0x4d0 [14588.251624] ? watchdog+0x40/0x40 [14588.261874] ? hrtimer_interrupt+0x128/0x2c0 [14588.265298] ? smp_apic_timer_interrupt+0xa6/0x270 [14588.271715] ? apic_timer_interrupt+0xf/0x20 [14588.274873] [14588.275508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14588.288699] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14588.291483] do_raw_spin_lock+0x177/0x190 [14588.306541] _raw_spin_lock+0x21/0x30 [14588.308591] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14588.316686] process_one_work+0x2c8/0x700 [14588.320662] worker_thread+0x296/0x6e0 [14588.324638] ? rescuer_thread+0x570/0x570 [14588.327340] kthread+0x1d1/0x200 [14588.330706] ? set_kthread_struct+0x70/0x70 [14588.336105] ret_from_fork+0x1f/0x30 [14596.576129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12679s! [14596.580398] Showing busy workqueues and worker pools: [14596.582981] workqueue events: flags=0x0 [14596.584971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14596.588186] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14596.593502] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14596.597249] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14596.601629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.604817] pending: kfree_rcu_monitor [14596.607288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14596.610612] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14596.614855] workqueue events_long: flags=0x0 [14596.616824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.619821] pending: lu_objects_flush [obdclass] [14596.622678] workqueue events_unbound: flags=0x2 [14596.624569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14596.627790] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14596.632182] workqueue events_power_efficient: flags=0x80 [14596.635354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.638410] pending: check_lifetime [14596.640175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.643750] pending: neigh_periodic_work [14596.645791] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.648704] pending: neigh_periodic_work [14596.650597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.654176] pending: do_cache_clean [14596.655673] workqueue mm_percpu_wq: flags=0x8 [14596.658103] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.660919] pending: vmstat_update [14596.662643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.665507] pending: vmstat_update [14596.667140] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14596.670163] pending: vmstat_update [14596.672153] workqueue writeback: flags=0x4a [14596.674586] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14596.677605] pending: wb_workfn, wb_workfn [14596.679416] workqueue xprtiod: flags=0xa [14596.681163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14596.683855] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14596.687433] workqueue ipv6_addrconf: flags=0x40008 [14596.689558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14596.692550] pending: addrconf_verify_work [14596.694828] workqueue knbd0-recv: flags=0x1a [14596.696977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14596.699856] in-flight: 651:recv_work [14596.701467] workqueue ptlrpc_pinger: flags=0x2 [14596.703820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14596.706486] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14596.709439] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12679s workers=4 idle: 37 31846 28897 [14596.713140] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14586s workers=2 idle: 653 [14600.058014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14600.062748] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14600.095888] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14600.104482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14600.110016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14600.113783] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14600.126136] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14600.131617] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14600.136476] RDX: 00000000000066d2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14600.141123] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14600.146903] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14600.151811] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14600.155801] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14600.161492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14600.164846] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14600.169724] Call Trace: [14600.171341] [14600.172565] ? show_regs.cold.9+0x22/0x2f [14600.174734] ? watchdog_timer_fn.cold.6+0x62/0x10a [14600.177858] ? __hrtimer_run_queues+0x1c2/0x4d0 [14600.181027] ? watchdog+0x40/0x40 [14600.182591] ? hrtimer_interrupt+0x128/0x2c0 [14600.184194] ? smp_apic_timer_interrupt+0xa6/0x270 [14600.188792] ? apic_timer_interrupt+0xf/0x20 [14600.191935] [14600.193665] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14600.197138] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14600.199987] do_raw_spin_lock+0x177/0x190 [14600.202797] _raw_spin_lock+0x21/0x30 [14600.204642] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14600.207526] obd_mod_free+0x46/0x140 [ptlrpc] [14600.209820] mdc_commit_open+0x91/0xb0 [mdc] [14600.212522] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14600.215871] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14600.219528] after_reply+0xe2c/0x1780 [ptlrpc] [14600.223380] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14600.226384] ? __next_timer_interrupt+0x160/0x160 [14600.229115] ptlrpcd+0xa82/0xe40 [ptlrpc] [14600.233606] ? do_wait_intr+0xf0/0xf0 [14600.235653] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14600.239532] kthread+0x1d1/0x200 [14600.241659] ? set_kthread_struct+0x70/0x70 [14600.244742] ret_from_fork+0x1f/0x30 [14604.062770] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14604.068036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14604.087220] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14604.091719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14604.095445] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14604.098294] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14604.110380] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14604.113677] RAX: 0000000000007285 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14604.116401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14604.119451] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14604.122259] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14604.125006] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14604.127518] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14604.130659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14604.132734] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14604.135510] Call Trace: [14604.136402] [14604.137438] ? show_regs.cold.9+0x22/0x2f [14604.139227] ? watchdog_timer_fn.cold.6+0x62/0x10a [14604.141184] ? __hrtimer_run_queues+0x1c2/0x4d0 [14604.142964] ? watchdog+0x40/0x40 [14604.144620] ? hrtimer_interrupt+0x128/0x2c0 [14604.146436] ? smp_apic_timer_interrupt+0xa6/0x270 [14604.148495] ? apic_timer_interrupt+0xf/0x20 [14604.150299] [14604.151140] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14604.153411] do_raw_spin_lock+0x177/0x190 [14604.155094] _raw_spin_lock+0x21/0x30 [14604.156322] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14604.158319] ? do_raw_spin_unlock+0x75/0x190 [14604.159819] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14604.161781] ? do_wait_intr+0xf0/0xf0 [14604.163179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14604.165421] mdc_close+0x53b/0xd70 [mdc] [14604.167011] lmv_close+0x18d/0x490 [lmv] [14604.168444] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14604.171211] ll_md_real_close+0xa5/0x2a0 [lustre] [14604.173037] ll_md_close+0x274/0x800 [lustre] [14604.174932] ll_file_release+0x128/0xc50 [lustre] [14604.176800] __fput+0x10d/0x440 [14604.178125] ____fput+0x12/0x20 [14604.179306] task_work_run+0xc6/0x110 [14604.180759] exit_to_usermode_loop+0x1dd/0x1f0 [14604.182762] do_syscall_64+0x3d6/0x440 [14604.188687] entry_SYSCALL_64_after_hwframe+0x49/0xae [14604.194849] RIP: 0033:0x7fa749552cd8 [14604.196093] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14604.204953] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14604.208133] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14604.211327] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14604.217781] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14604.224915] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14604.231482] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14616.057025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14616.061051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14616.083340] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14616.088739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14616.092668] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14616.096566] RIP: 0010:native_safe_halt+0x2e/0x50 [14616.098943] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14616.107165] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14616.110051] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14616.113151] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14616.117393] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14616.120722] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14616.123717] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14616.126832] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14616.130552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14616.133591] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14616.137123] Call Trace: [14616.138189] [14616.139130] ? show_regs.cold.9+0x22/0x2f [14616.140897] ? watchdog_timer_fn.cold.6+0x62/0x10a [14616.143100] ? __hrtimer_run_queues+0x1c2/0x4d0 [14616.144851] ? watchdog+0x40/0x40 [14616.147534] ? hrtimer_interrupt+0x128/0x2c0 [14616.149759] ? smp_apic_timer_interrupt+0xa6/0x270 [14616.153179] ? apic_timer_interrupt+0xf/0x20 [14616.155054] [14616.156220] ? native_safe_halt+0x2e/0x50 [14616.157798] kvm_wait+0xcd/0xe0 [14616.159178] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14616.161799] do_raw_spin_lock+0x177/0x190 [14616.163495] _raw_spin_lock+0x21/0x30 [14616.165332] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14616.169166] process_one_work+0x2c8/0x700 [14616.170861] worker_thread+0x296/0x6e0 [14616.172520] ? rescuer_thread+0x570/0x570 [14616.174308] kthread+0x1d1/0x200 [14616.175912] ? set_kthread_struct+0x70/0x70 [14616.178381] ret_from_fork+0x1f/0x30 [14624.054656] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14624.059364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14624.105021] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14624.112547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14624.125453] RIP: 0010:native_safe_halt+0x2e/0x50 [14624.130661] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14624.149039] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14624.156717] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14624.162158] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14624.167966] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14624.174236] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14624.181711] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14624.188372] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14624.196029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14624.200546] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14624.207947] Call Trace: [14624.211551] [14624.212648] ? show_regs.cold.9+0x22/0x2f [14624.216221] ? watchdog_timer_fn.cold.6+0x62/0x10a [14624.219410] ? __hrtimer_run_queues+0x1c2/0x4d0 [14624.221328] ? watchdog+0x40/0x40 [14624.223010] ? hrtimer_interrupt+0x128/0x2c0 [14624.225002] ? smp_apic_timer_interrupt+0xa6/0x270 [14624.227351] ? apic_timer_interrupt+0xf/0x20 [14624.229484] [14624.234385] ? native_safe_halt+0x2e/0x50 [14624.239492] kvm_wait+0xcd/0xe0 [14624.245479] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14624.248309] do_raw_spin_lock+0x177/0x190 [14624.254380] _raw_spin_lock+0x21/0x30 [14624.256271] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14624.266447] ? do_raw_spin_unlock+0x75/0x190 [14624.272864] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14624.281937] ? do_wait_intr+0xf0/0xf0 [14624.284372] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14624.286111] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14624.292031] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14624.299871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14624.303137] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14624.304932] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14624.306397] mdc_intent_lock+0x353/0x7e0 [mdc] [14624.311681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14624.313362] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14624.322425] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14624.327843] lmv_intent_open+0x338/0x1150 [lmv] [14624.331639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14624.336274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14624.339350] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14624.341623] ll_intent_lock+0xc5/0xaa0 [lustre] [14624.346518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14624.351734] ll_lookup_it+0xe0e/0x2270 [lustre] [14624.354408] ? do_raw_spin_unlock+0x75/0x190 [14624.356486] ? _raw_spin_unlock+0x12/0x30 [14624.358398] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14624.361869] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14624.364295] ll_atomic_open+0x36a/0xd70 [lustre] [14624.366663] lookup_open+0x356/0xb40 [14624.368826] do_last+0x313/0xfc0 [14624.370389] ? nd_jump_root+0xe5/0x160 [14624.372107] ? path_init+0x437/0x520 [14624.373762] path_openat+0xf7/0x500 [14624.375170] ? do_raw_spin_unlock+0x75/0x190 [14624.377109] do_filp_open+0x99/0x140 [14624.378858] ? getname_flags+0x6e/0x330 [14624.380760] ? __check_object_size+0xff/0x256 [14624.382850] ? do_raw_spin_unlock+0x75/0x190 [14624.391020] ? _raw_spin_unlock+0x12/0x30 [14624.398815] do_sys_openat2+0x2b4/0x410 [14624.410054] do_sys_open+0x73/0xa0 [14624.415015] __x64_sys_openat+0x24/0x30 [14624.418924] do_syscall_64+0xc1/0x440 [14624.421397] entry_SYSCALL_64_after_hwframe+0x49/0xae [14624.425798] RIP: 0033:0x7fa749552332 [14624.427657] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14624.446421] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14624.450695] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14624.457402] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14624.470470] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14624.484333] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14624.491573] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14626.784175] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12709s! [14626.790520] Showing busy workqueues and worker pools: [14626.802830] workqueue events: flags=0x0 [14626.809142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14626.816989] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14626.829949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14626.845728] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14626.850930] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14626.854022] pending: kfree_rcu_monitor [14626.856495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14626.859936] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14626.867858] workqueue events_long: flags=0x0 [14626.872698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14626.876332] pending: lu_objects_flush [obdclass] [14626.880844] workqueue events_unbound: flags=0x2 [14626.883021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14626.887151] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14626.896690] workqueue events_power_efficient: flags=0x80 [14626.904237] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14626.912145] pending: check_lifetime [14626.916958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14626.931451] pending: neigh_periodic_work [14626.937129] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14626.946850] pending: neigh_periodic_work [14626.954108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14626.958359] pending: do_cache_clean [14626.966032] workqueue mm_percpu_wq: flags=0x8 [14626.976255] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14626.986127] pending: vmstat_update [14626.997648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14627.012905] pending: vmstat_update [14627.015863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14627.020837] pending: vmstat_update [14627.022971] workqueue writeback: flags=0x4a [14627.025705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14627.029676] pending: wb_workfn, wb_workfn [14627.032144] workqueue xprtiod: flags=0xa [14627.034260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14627.037890] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14627.046920] workqueue ipv6_addrconf: flags=0x40008 [14627.049263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14627.052956] pending: addrconf_verify_work [14627.055835] workqueue knbd0-recv: flags=0x1a [14627.058332] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14627.065779] in-flight: 651:recv_work [14627.068364] workqueue ptlrpc_pinger: flags=0x2 [14627.071493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14627.077141] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14627.086423] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12709s workers=4 idle: 37 31846 28897 [14627.095835] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14616s workers=2 idle: 653 [14628.058307] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14628.063777] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14628.087973] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14628.092525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14628.095474] RIP: 0010:native_safe_halt+0x2e/0x50 [14628.097413] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14628.107213] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14628.111195] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14628.114414] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14628.116543] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14628.122274] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14628.124803] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14628.128324] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14628.132405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14628.135418] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14628.142542] Call Trace: [14628.145083] [14628.145969] ? show_regs.cold.9+0x22/0x2f [14628.148184] ? watchdog_timer_fn.cold.6+0x62/0x10a [14628.150404] ? __hrtimer_run_queues+0x1c2/0x4d0 [14628.152846] ? watchdog+0x40/0x40 [14628.154411] ? hrtimer_interrupt+0x128/0x2c0 [14628.156269] ? smp_apic_timer_interrupt+0xa6/0x270 [14628.158692] ? apic_timer_interrupt+0xf/0x20 [14628.160707] [14628.161664] ? native_safe_halt+0x2e/0x50 [14628.163430] kvm_wait+0xcd/0xe0 [14628.165217] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14628.167329] do_raw_spin_lock+0x177/0x190 [14628.169265] _raw_spin_lock+0x21/0x30 [14628.171371] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14628.174038] obd_mod_free+0x46/0x140 [ptlrpc] [14628.176242] mdc_commit_open+0x91/0xb0 [mdc] [14628.178218] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14628.180569] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14628.183224] after_reply+0xe2c/0x1780 [ptlrpc] [14628.185599] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14628.187840] ? __next_timer_interrupt+0x160/0x160 [14628.190253] ptlrpcd+0xa82/0xe40 [ptlrpc] [14628.192768] ? do_wait_intr+0xf0/0xf0 [14628.194447] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14628.197106] kthread+0x1d1/0x200 [14628.198742] ? set_kthread_struct+0x70/0x70 [14628.200573] ret_from_fork+0x1f/0x30 [14632.059668] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14632.064454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14632.098347] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14632.106983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14632.111831] RIP: 0010:native_safe_halt+0x2e/0x50 [14632.114245] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14632.123361] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14632.127443] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14632.131863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14632.136900] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14632.149298] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14632.153128] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14632.156973] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14632.160987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14632.167115] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14632.170800] Call Trace: [14632.172839] [14632.173770] ? show_regs.cold.9+0x22/0x2f [14632.175778] ? watchdog_timer_fn.cold.6+0x62/0x10a [14632.177852] ? __hrtimer_run_queues+0x1c2/0x4d0 [14632.179806] ? watchdog+0x40/0x40 [14632.180713] ? hrtimer_interrupt+0x128/0x2c0 [14632.182564] ? smp_apic_timer_interrupt+0xa6/0x270 [14632.184313] ? apic_timer_interrupt+0xf/0x20 [14632.187923] [14632.189817] ? native_safe_halt+0x2e/0x50 [14632.193495] kvm_wait+0xcd/0xe0 [14632.195866] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14632.202763] do_raw_spin_lock+0x177/0x190 [14632.204311] _raw_spin_lock+0x21/0x30 [14632.205529] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14632.209275] ? do_raw_spin_unlock+0x75/0x190 [14632.210645] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14632.212983] ? do_wait_intr+0xf0/0xf0 [14632.217622] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14632.221644] mdc_close+0x53b/0xd70 [mdc] [14632.223952] lmv_close+0x18d/0x490 [lmv] [14632.225994] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14632.228324] ll_md_real_close+0xa5/0x2a0 [lustre] [14632.231242] ll_md_close+0x274/0x800 [lustre] [14632.233716] ll_file_release+0x128/0xc50 [lustre] [14632.237743] __fput+0x10d/0x440 [14632.239542] ____fput+0x12/0x20 [14632.242001] task_work_run+0xc6/0x110 [14632.244465] exit_to_usermode_loop+0x1dd/0x1f0 [14632.248048] do_syscall_64+0x3d6/0x440 [14632.251322] entry_SYSCALL_64_after_hwframe+0x49/0xae [14632.254590] RIP: 0033:0x7fa749552cd8 [14632.257019] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14632.269442] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14632.273912] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14632.280251] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14632.285957] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14632.290443] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14632.295480] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14644.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14644.065346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14644.088927] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14644.093906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14644.097577] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14644.105456] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14644.109187] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14644.133664] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14644.143200] RAX: 000000000000704b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14644.154620] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14644.160704] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14644.165393] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14644.170508] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14644.174628] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14644.183035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14644.185748] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14644.194630] Call Trace: [14644.196285] [14644.197977] ? show_regs.cold.9+0x22/0x2f [14644.199647] ? watchdog_timer_fn.cold.6+0x62/0x10a [14644.201383] ? __hrtimer_run_queues+0x1c2/0x4d0 [14644.205633] ? watchdog+0x40/0x40 [14644.206932] ? hrtimer_interrupt+0x128/0x2c0 [14644.208649] ? smp_apic_timer_interrupt+0xa6/0x270 [14644.221686] ? apic_timer_interrupt+0xf/0x20 [14644.227577] [14644.228676] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14644.232705] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14644.234152] do_raw_spin_lock+0x177/0x190 [14644.238564] _raw_spin_lock+0x21/0x30 [14644.247671] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14644.251141] process_one_work+0x2c8/0x700 [14644.257664] worker_thread+0x296/0x6e0 [14644.259646] ? rescuer_thread+0x570/0x570 [14644.263730] kthread+0x1d1/0x200 [14644.265216] ? set_kthread_struct+0x70/0x70 [14644.268780] ret_from_fork+0x1f/0x30 [14652.054477] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14652.063461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14652.095655] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14652.098195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14652.111952] RIP: 0010:native_safe_halt+0x2e/0x50 [14652.120801] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14652.132740] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14652.136461] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14652.141840] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14652.144777] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14652.149451] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14652.153592] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14652.158238] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14652.163563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14652.165401] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14652.167992] Call Trace: [14652.169408] [14652.170562] ? show_regs.cold.9+0x22/0x2f [14652.171873] ? watchdog_timer_fn.cold.6+0x62/0x10a [14652.173760] ? __hrtimer_run_queues+0x1c2/0x4d0 [14652.175527] ? watchdog+0x40/0x40 [14652.176939] ? hrtimer_interrupt+0x128/0x2c0 [14652.178470] ? smp_apic_timer_interrupt+0xa6/0x270 [14652.179734] ? apic_timer_interrupt+0xf/0x20 [14652.181085] [14652.181863] ? native_safe_halt+0x2e/0x50 [14652.183566] kvm_wait+0xcd/0xe0 [14652.184737] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14652.187242] do_raw_spin_lock+0x177/0x190 [14652.188762] _raw_spin_lock+0x21/0x30 [14652.190196] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14652.192693] ? do_raw_spin_unlock+0x75/0x190 [14652.193782] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14652.195422] ? do_wait_intr+0xf0/0xf0 [14652.196469] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14652.198237] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14652.199782] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14652.201555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14652.203311] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14652.205409] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14652.207151] mdc_intent_lock+0x353/0x7e0 [mdc] [14652.212995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14652.215731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14652.218848] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14652.221430] lmv_intent_open+0x338/0x1150 [lmv] [14652.223393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14652.225822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14652.228294] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14652.230958] ll_intent_lock+0xc5/0xaa0 [lustre] [14652.232980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14652.234859] ll_lookup_it+0xe0e/0x2270 [lustre] [14652.236764] ? do_raw_spin_unlock+0x75/0x190 [14652.238585] ? _raw_spin_unlock+0x12/0x30 [14652.239779] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14652.241974] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14652.243609] ll_atomic_open+0x36a/0xd70 [lustre] [14652.245200] lookup_open+0x356/0xb40 [14652.246649] do_last+0x313/0xfc0 [14652.247836] ? nd_jump_root+0xe5/0x160 [14652.249120] ? path_init+0x437/0x520 [14652.250234] path_openat+0xf7/0x500 [14652.251568] ? do_raw_spin_unlock+0x75/0x190 [14652.252808] do_filp_open+0x99/0x140 [14652.253779] ? getname_flags+0x6e/0x330 [14652.255438] ? __check_object_size+0xff/0x256 [14652.256501] ? do_raw_spin_unlock+0x75/0x190 [14652.257682] ? _raw_spin_unlock+0x12/0x30 [14652.260137] do_sys_openat2+0x2b4/0x410 [14652.261723] do_sys_open+0x73/0xa0 [14652.263133] __x64_sys_openat+0x24/0x30 [14652.264603] do_syscall_64+0xc1/0x440 [14652.265861] entry_SYSCALL_64_after_hwframe+0x49/0xae [14652.267854] RIP: 0033:0x7fa749552332 [14652.269610] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14652.275609] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14652.277769] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14652.279885] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14652.282060] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14652.284114] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14652.285809] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14656.058031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14656.063129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14656.128683] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14656.138625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14656.144091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14656.147844] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14656.156335] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14656.168269] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14656.176316] RDX: 00000000000079a6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14656.183456] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14656.194340] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14656.203373] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14656.207601] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14656.215551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14656.218369] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14656.221345] Call Trace: [14656.222998] [14656.223939] ? show_regs.cold.9+0x22/0x2f [14656.229941] ? watchdog_timer_fn.cold.6+0x62/0x10a [14656.238367] ? __hrtimer_run_queues+0x1c2/0x4d0 [14656.245428] ? watchdog+0x40/0x40 [14656.248566] ? hrtimer_interrupt+0x128/0x2c0 [14656.253286] ? smp_apic_timer_interrupt+0xa6/0x270 [14656.255795] ? apic_timer_interrupt+0xf/0x20 [14656.258952] [14656.259939] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14656.267411] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14656.270123] do_raw_spin_lock+0x177/0x190 [14656.274566] _raw_spin_lock+0x21/0x30 [14656.279213] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14656.289415] obd_mod_free+0x46/0x140 [ptlrpc] [14656.296406] mdc_commit_open+0x91/0xb0 [mdc] [14656.302388] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14656.311923] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14656.318484] after_reply+0xe2c/0x1780 [ptlrpc] [14656.322505] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14656.329227] ? __next_timer_interrupt+0x160/0x160 [14656.332148] ptlrpcd+0xa82/0xe40 [ptlrpc] [14656.334838] ? do_wait_intr+0xf0/0xf0 [14656.337842] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14656.341030] kthread+0x1d1/0x200 [14656.342198] ? set_kthread_struct+0x70/0x70 [14656.344514] ret_from_fork+0x1f/0x30 [14657.504790] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12740s! [14657.514030] Showing busy workqueues and worker pools: [14657.518583] workqueue events: flags=0x0 [14657.520410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14657.523282] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14657.528164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14657.531322] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14657.535015] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.538639] pending: kfree_rcu_monitor [14657.540163] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14657.542567] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14657.545894] workqueue events_long: flags=0x0 [14657.547620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.549697] pending: lu_objects_flush [obdclass] [14657.551802] workqueue events_unbound: flags=0x2 [14657.553193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14657.554883] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14657.557331] workqueue events_power_efficient: flags=0x80 [14657.559247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.562179] pending: check_lifetime [14657.563384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.565386] pending: neigh_periodic_work [14657.567385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.570512] pending: neigh_periodic_work [14657.572359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.575324] pending: do_cache_clean [14657.577266] workqueue mm_percpu_wq: flags=0x8 [14657.579626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.582824] pending: vmstat_update [14657.584410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.587401] pending: vmstat_update [14657.588913] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14657.593628] pending: vmstat_update [14657.596721] workqueue writeback: flags=0x4a [14657.598876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14657.601125] pending: wb_workfn, wb_workfn [14657.607884] workqueue xprtiod: flags=0xa [14657.609799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14657.613683] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14657.617317] workqueue ipv6_addrconf: flags=0x40008 [14657.621940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14657.624546] pending: addrconf_verify_work [14657.626433] workqueue knbd0-recv: flags=0x1a [14657.628633] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14657.635595] in-flight: 651:recv_work [14657.637037] workqueue ptlrpc_pinger: flags=0x2 [14657.641507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14657.647854] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14657.653328] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12740s workers=4 idle: 37 31846 28897 [14657.659155] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14647s workers=2 idle: 653 [14660.062945] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14660.067776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14660.094640] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14660.102645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14660.107582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14660.110119] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14660.123298] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14660.129857] RAX: 0000000000007d4a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14660.140297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14660.146746] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14660.153181] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14660.161338] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14660.170330] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14660.183178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14660.197785] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14660.210057] Call Trace: [14660.210857] [14660.211659] ? show_regs.cold.9+0x22/0x2f [14660.220842] ? watchdog_timer_fn.cold.6+0x62/0x10a [14660.222427] ? __hrtimer_run_queues+0x1c2/0x4d0 [14660.224007] ? watchdog+0x40/0x40 [14660.233525] ? hrtimer_interrupt+0x128/0x2c0 [14660.237177] ? smp_apic_timer_interrupt+0xa6/0x270 [14660.241452] ? apic_timer_interrupt+0xf/0x20 [14660.243761] [14660.252124] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14660.253910] do_raw_spin_lock+0x177/0x190 [14660.262916] _raw_spin_lock+0x21/0x30 [14660.264365] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14660.278143] ? do_raw_spin_unlock+0x75/0x190 [14660.279778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14660.287647] ? do_wait_intr+0xf0/0xf0 [14660.290611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14660.292434] mdc_close+0x53b/0xd70 [mdc] [14660.298127] lmv_close+0x18d/0x490 [lmv] [14660.299464] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14660.306277] ll_md_real_close+0xa5/0x2a0 [lustre] [14660.308213] ll_md_close+0x274/0x800 [lustre] [14660.313453] ll_file_release+0x128/0xc50 [lustre] [14660.315676] __fput+0x10d/0x440 [14660.317334] ____fput+0x12/0x20 [14660.320102] task_work_run+0xc6/0x110 [14660.321601] exit_to_usermode_loop+0x1dd/0x1f0 [14660.325238] do_syscall_64+0x3d6/0x440 [14660.326726] entry_SYSCALL_64_after_hwframe+0x49/0xae [14660.330543] RIP: 0033:0x7fa749552cd8 [14660.332078] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14660.345375] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14660.352209] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14660.359395] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14660.365129] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14660.371120] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14660.377235] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14672.057024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14672.060542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14672.094051] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14672.099922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14672.104620] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14672.107831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14672.111152] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14672.120628] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14672.124071] RAX: 0000000000000808 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14672.127878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14672.137003] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14672.145281] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14672.155000] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14672.161810] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14672.168261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14672.172971] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14672.180762] Call Trace: [14672.182729] [14672.188586] ? show_regs.cold.9+0x22/0x2f [14672.191386] ? watchdog_timer_fn.cold.6+0x62/0x10a [14672.192627] ? __hrtimer_run_queues+0x1c2/0x4d0 [14672.193935] ? watchdog+0x40/0x40 [14672.195973] ? hrtimer_interrupt+0x128/0x2c0 [14672.199074] ? smp_apic_timer_interrupt+0xa6/0x270 [14672.201543] ? apic_timer_interrupt+0xf/0x20 [14672.203049] [14672.204225] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14672.206495] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14672.208792] do_raw_spin_lock+0x177/0x190 [14672.210656] _raw_spin_lock+0x21/0x30 [14672.212018] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14672.215267] process_one_work+0x2c8/0x700 [14672.218788] worker_thread+0x296/0x6e0 [14672.220600] ? rescuer_thread+0x570/0x570 [14672.225752] kthread+0x1d1/0x200 [14672.228463] ? set_kthread_struct+0x70/0x70 [14672.230454] ret_from_fork+0x1f/0x30 [14684.054046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14684.058115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14684.065246] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14684.083384] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14684.083399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14684.083403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14684.083418] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14684.083423] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14684.083429] RAX: 00000000000034b1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14684.083432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14684.083435] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14684.083438] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14684.083440] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14684.083453] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14684.083730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14684.083733] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14684.083752] Call Trace: [14684.083758] [14684.083764] ? show_regs.cold.9+0x22/0x2f [14684.083782] ? watchdog_timer_fn.cold.6+0x62/0x10a [14684.083789] ? __hrtimer_run_queues+0x1c2/0x4d0 [14684.083794] ? watchdog+0x40/0x40 [14684.083799] ? hrtimer_interrupt+0x128/0x2c0 [14684.083803] ? smp_apic_timer_interrupt+0xa6/0x270 [14684.083811] ? apic_timer_interrupt+0xf/0x20 [14684.083816] [14684.083820] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14684.083828] do_raw_spin_lock+0x177/0x190 [14684.083834] _raw_spin_lock+0x21/0x30 [14684.083839] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14684.084131] ? do_raw_spin_unlock+0x75/0x190 [14684.084137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14684.084568] ? do_wait_intr+0xf0/0xf0 [14684.084578] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14684.084937] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14684.085411] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14684.085924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14684.086046] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14684.086086] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14684.086116] mdc_intent_lock+0x353/0x7e0 [mdc] [14684.086142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14684.086239] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14684.086779] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14684.086814] lmv_intent_open+0x338/0x1150 [lmv] [14684.086848] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14684.086942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14684.087029] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14684.087057] ll_intent_lock+0xc5/0xaa0 [lustre] [14684.087144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14684.087346] ll_lookup_it+0xe0e/0x2270 [lustre] [14684.087482] ? do_raw_spin_unlock+0x75/0x190 [14684.087489] ? _raw_spin_unlock+0x12/0x30 [14684.087496] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14684.087524] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14684.087544] ll_atomic_open+0x36a/0xd70 [lustre] [14684.087820] lookup_open+0x356/0xb40 [14684.087829] do_last+0x313/0xfc0 [14684.087834] ? nd_jump_root+0xe5/0x160 [14684.087838] ? path_init+0x437/0x520 [14684.087843] path_openat+0xf7/0x500 [14684.087848] ? do_raw_spin_unlock+0x75/0x190 [14684.087852] do_filp_open+0x99/0x140 [14684.087859] ? getname_flags+0x6e/0x330 [14684.087863] ? __check_object_size+0xff/0x256 [14684.087869] ? do_raw_spin_unlock+0x75/0x190 [14684.087873] ? _raw_spin_unlock+0x12/0x30 [14684.087877] do_sys_openat2+0x2b4/0x410 [14684.087881] do_sys_open+0x73/0xa0 [14684.087885] __x64_sys_openat+0x24/0x30 [14684.087889] do_syscall_64+0xc1/0x440 [14684.087894] entry_SYSCALL_64_after_hwframe+0x49/0xae [14684.087900] RIP: 0033:0x7fa749552332 [14684.087928] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14684.087932] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14684.087937] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14684.087939] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14684.087941] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14684.087943] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14684.087946] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14684.268028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14684.286562] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14684.291420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14684.296196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14684.298075] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14684.305141] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14684.308263] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14684.311451] RDX: 00000000000067a6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14684.314339] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14684.317121] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14684.320140] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14684.323201] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14684.326699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14684.329374] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14684.332722] Call Trace: [14684.333905] [14684.334773] ? show_regs.cold.9+0x22/0x2f [14684.336275] ? watchdog_timer_fn.cold.6+0x62/0x10a [14684.338326] ? __hrtimer_run_queues+0x1c2/0x4d0 [14684.340232] ? watchdog+0x40/0x40 [14684.341619] ? hrtimer_interrupt+0x128/0x2c0 [14684.343300] ? smp_apic_timer_interrupt+0xa6/0x270 [14684.345349] ? apic_timer_interrupt+0xf/0x20 [14684.347259] [14684.348007] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14684.350684] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14684.353294] do_raw_spin_lock+0x177/0x190 [14684.354880] _raw_spin_lock+0x21/0x30 [14684.356521] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14684.359111] obd_mod_free+0x46/0x140 [ptlrpc] [14684.361732] mdc_commit_open+0x91/0xb0 [mdc] [14684.364010] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14684.366165] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14684.368511] after_reply+0xe2c/0x1780 [ptlrpc] [14684.370442] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14684.373275] ? __next_timer_interrupt+0x160/0x160 [14684.375193] ptlrpcd+0xa82/0xe40 [ptlrpc] [14684.377273] ? do_wait_intr+0xf0/0xf0 [14684.379173] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14684.382086] kthread+0x1d1/0x200 [14684.383158] ? set_kthread_struct+0x70/0x70 [14684.384459] ret_from_fork+0x1f/0x30 [14687.717077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12770s! [14687.721437] Showing busy workqueues and worker pools: [14687.728281] workqueue events: flags=0x0 [14687.731850] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14687.735850] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14687.743115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14687.749110] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14687.757253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.762090] pending: kfree_rcu_monitor [14687.770186] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14687.775770] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14687.783267] workqueue events_long: flags=0x0 [14687.786782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.792744] pending: lu_objects_flush [obdclass] [14687.795718] workqueue events_unbound: flags=0x2 [14687.797361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14687.804789] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14687.817265] workqueue events_power_efficient: flags=0x80 [14687.822825] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.825834] pending: check_lifetime [14687.828032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.831797] pending: neigh_periodic_work [14687.833305] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.836461] pending: neigh_periodic_work [14687.838492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.841779] pending: do_cache_clean [14687.843530] workqueue mm_percpu_wq: flags=0x8 [14687.846548] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.850460] pending: vmstat_update [14687.852595] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.856060] pending: vmstat_update [14687.858570] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14687.862410] pending: vmstat_update [14687.865818] workqueue writeback: flags=0x4a [14687.872260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14687.882030] pending: wb_workfn, wb_workfn [14687.884319] workqueue xprtiod: flags=0xa [14687.891157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14687.897137] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14687.912744] workqueue ipv6_addrconf: flags=0x40008 [14687.916318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14687.935513] pending: addrconf_verify_work [14687.937126] workqueue knbd0-recv: flags=0x1a [14687.941321] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14687.957671] in-flight: 651:recv_work [14687.959202] workqueue ptlrpc_pinger: flags=0x2 [14687.966671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14687.980590] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14687.987521] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12770s workers=4 idle: 37 31846 28897 [14688.005721] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14677s workers=2 idle: 653 [14688.066077] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14688.075032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14688.140924] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14688.153536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14688.161188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14688.167085] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14688.184556] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14688.191606] RAX: 0000000000007714 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14688.195860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14688.201628] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14688.209262] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14688.217171] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14688.227776] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14688.234383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14688.237547] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14688.244885] Call Trace: [14688.246926] [14688.248528] ? show_regs.cold.9+0x22/0x2f [14688.250494] ? watchdog_timer_fn.cold.6+0x62/0x10a [14688.259737] ? __hrtimer_run_queues+0x1c2/0x4d0 [14688.263447] ? watchdog+0x40/0x40 [14688.265803] ? hrtimer_interrupt+0x128/0x2c0 [14688.271457] ? smp_apic_timer_interrupt+0xa6/0x270 [14688.273942] ? apic_timer_interrupt+0xf/0x20 [14688.279328] [14688.281491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14688.287890] do_raw_spin_lock+0x177/0x190 [14688.289249] _raw_spin_lock+0x21/0x30 [14688.292545] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14688.299712] ? do_raw_spin_unlock+0x75/0x190 [14688.305755] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14688.311117] ? do_wait_intr+0xf0/0xf0 [14688.315848] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14688.318360] mdc_close+0x53b/0xd70 [mdc] [14688.323471] lmv_close+0x18d/0x490 [lmv] [14688.325909] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14688.332115] ll_md_real_close+0xa5/0x2a0 [lustre] [14688.340573] ll_md_close+0x274/0x800 [lustre] [14688.345163] ll_file_release+0x128/0xc50 [lustre] [14688.349627] __fput+0x10d/0x440 [14688.352207] ____fput+0x12/0x20 [14688.355729] task_work_run+0xc6/0x110 [14688.357099] exit_to_usermode_loop+0x1dd/0x1f0 [14688.362935] do_syscall_64+0x3d6/0x440 [14688.367833] entry_SYSCALL_64_after_hwframe+0x49/0xae [14688.372262] RIP: 0033:0x7fa749552cd8 [14688.373586] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14688.394755] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14688.403171] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14688.410433] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14688.415792] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14688.418288] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14688.426797] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14700.059165] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14700.071672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14700.130784] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14700.138955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14700.144812] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14700.150700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14700.163574] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14700.180455] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14700.184507] RAX: 0000000000007c88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14700.193978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14700.205342] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14700.209360] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14700.213699] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14700.219392] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14700.225234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14700.228803] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14700.233426] Call Trace: [14700.235320] [14700.236233] ? show_regs.cold.9+0x22/0x2f [14700.237697] ? watchdog_timer_fn.cold.6+0x62/0x10a [14700.240083] ? __hrtimer_run_queues+0x1c2/0x4d0 [14700.247029] ? watchdog+0x40/0x40 [14700.250707] ? hrtimer_interrupt+0x128/0x2c0 [14700.256769] ? smp_apic_timer_interrupt+0xa6/0x270 [14700.259636] ? apic_timer_interrupt+0xf/0x20 [14700.264478] [14700.265679] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14700.270815] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14700.280834] do_raw_spin_lock+0x177/0x190 [14700.282057] _raw_spin_lock+0x21/0x30 [14700.286009] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14700.289823] process_one_work+0x2c8/0x700 [14700.291764] worker_thread+0x296/0x6e0 [14700.293353] ? rescuer_thread+0x570/0x570 [14700.295397] kthread+0x1d1/0x200 [14700.297333] ? set_kthread_struct+0x70/0x70 [14700.301107] ret_from_fork+0x1f/0x30 [14712.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14712.059077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14712.062788] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14712.098050] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14712.098065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14712.098068] RIP: 0010:native_safe_halt+0x2e/0x50 [14712.098082] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14712.098087] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14712.098093] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14712.098096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14712.098098] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14712.098100] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14712.098102] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14712.098106] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14712.098109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14712.098111] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14712.098119] Call Trace: [14712.098125] [14712.098131] ? show_regs.cold.9+0x22/0x2f [14712.098141] ? watchdog_timer_fn.cold.6+0x62/0x10a [14712.098147] ? __hrtimer_run_queues+0x1c2/0x4d0 [14712.098151] ? watchdog+0x40/0x40 [14712.098156] ? hrtimer_interrupt+0x128/0x2c0 [14712.098159] ? smp_apic_timer_interrupt+0xa6/0x270 [14712.098165] ? apic_timer_interrupt+0xf/0x20 [14712.098170] [14712.098172] ? native_safe_halt+0x2e/0x50 [14712.098176] kvm_wait+0xcd/0xe0 [14712.098183] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14712.098192] do_raw_spin_lock+0x177/0x190 [14712.098197] _raw_spin_lock+0x21/0x30 [14712.098201] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14712.100758] ? do_raw_spin_unlock+0x75/0x190 [14712.100768] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14712.108282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14712.108367] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14712.108374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14712.108377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14712.108391] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14712.108395] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14712.108399] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14712.108402] RDX: 0000000000000cda RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14712.108405] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14712.108407] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14712.108409] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14712.108412] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14712.108415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14712.108418] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14712.108426] Call Trace: [14712.108431] [14712.108436] ? show_regs.cold.9+0x22/0x2f [14712.108445] ? watchdog_timer_fn.cold.6+0x62/0x10a [14712.108452] ? __hrtimer_run_queues+0x1c2/0x4d0 [14712.108456] ? watchdog+0x40/0x40 [14712.108461] ? hrtimer_interrupt+0x128/0x2c0 [14712.108465] ? smp_apic_timer_interrupt+0xa6/0x270 [14712.108473] ? apic_timer_interrupt+0xf/0x20 [14712.108479] [14712.108483] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14712.108489] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14712.108494] do_raw_spin_lock+0x177/0x190 [14712.108499] _raw_spin_lock+0x21/0x30 [14712.108504] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14712.108782] obd_mod_free+0x46/0x140 [ptlrpc] [14712.108999] mdc_commit_open+0x91/0xb0 [mdc] [14712.109037] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14712.114799] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14712.125798] ? do_wait_intr+0xf0/0xf0 [14712.143432] after_reply+0xe2c/0x1780 [ptlrpc] [14712.153343] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14712.153609] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14712.153823] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14712.154051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.154165] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14712.154203] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14712.154233] mdc_intent_lock+0x353/0x7e0 [mdc] [14712.154275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.154365] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14712.154586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14712.154616] lmv_intent_open+0x338/0x1150 [lmv] [14712.154646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.154733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.154813] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14712.154832] ll_intent_lock+0xc5/0xaa0 [lustre] [14712.154915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.155003] ll_lookup_it+0xe0e/0x2270 [lustre] [14712.155091] ? do_raw_spin_unlock+0x75/0x190 [14712.155099] ? _raw_spin_unlock+0x12/0x30 [14712.155105] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14712.155132] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14712.155152] ll_atomic_open+0x36a/0xd70 [lustre] [14712.155241] lookup_open+0x356/0xb40 [14712.160288] do_last+0x313/0xfc0 [14712.169851] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14712.174870] ? __next_timer_interrupt+0x160/0x160 [14712.174881] ptlrpcd+0xa82/0xe40 [ptlrpc] [14712.175110] ? do_wait_intr+0xf0/0xf0 [14712.175119] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14712.175343] kthread+0x1d1/0x200 [14712.175349] ? set_kthread_struct+0x70/0x70 [14712.175352] ret_from_fork+0x1f/0x30 [14712.480366] ? nd_jump_root+0xe5/0x160 [14712.482189] ? path_init+0x437/0x520 [14712.484289] path_openat+0xf7/0x500 [14712.485633] ? do_raw_spin_unlock+0x75/0x190 [14712.487192] do_filp_open+0x99/0x140 [14712.490545] ? getname_flags+0x6e/0x330 [14712.492653] ? __check_object_size+0xff/0x256 [14712.497656] ? do_raw_spin_unlock+0x75/0x190 [14712.500401] ? _raw_spin_unlock+0x12/0x30 [14712.502956] do_sys_openat2+0x2b4/0x410 [14712.505480] do_sys_open+0x73/0xa0 [14712.506714] __x64_sys_openat+0x24/0x30 [14712.507971] do_syscall_64+0xc1/0x440 [14712.511750] entry_SYSCALL_64_after_hwframe+0x49/0xae [14712.517862] RIP: 0033:0x7fa749552332 [14712.521278] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14712.529716] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14712.532345] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14712.535687] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14712.538895] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14712.542685] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14712.546705] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14716.067778] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14716.080010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14716.126678] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14716.145581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14716.161320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14716.167863] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14716.193644] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14716.203335] RAX: 0000000000001103 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14716.211619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14716.217681] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14716.227337] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14716.241951] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14716.249792] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14716.257058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14716.265238] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14716.280483] Call Trace: [14716.281449] [14716.286756] ? show_regs.cold.9+0x22/0x2f [14716.293221] ? watchdog_timer_fn.cold.6+0x62/0x10a [14716.299708] ? __hrtimer_run_queues+0x1c2/0x4d0 [14716.306725] ? watchdog+0x40/0x40 [14716.310820] ? hrtimer_interrupt+0x128/0x2c0 [14716.315348] ? smp_apic_timer_interrupt+0xa6/0x270 [14716.325214] ? apic_timer_interrupt+0xf/0x20 [14716.328249] [14716.330348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14716.341141] do_raw_spin_lock+0x177/0x190 [14716.348500] _raw_spin_lock+0x21/0x30 [14716.353579] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14716.355711] ? do_raw_spin_unlock+0x75/0x190 [14716.360152] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14716.364268] ? do_wait_intr+0xf0/0xf0 [14716.366109] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14716.373589] mdc_close+0x53b/0xd70 [mdc] [14716.375086] lmv_close+0x18d/0x490 [lmv] [14716.379253] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14716.392894] ll_md_real_close+0xa5/0x2a0 [lustre] [14716.402659] ll_md_close+0x274/0x800 [lustre] [14716.407939] ll_file_release+0x128/0xc50 [lustre] [14716.409768] __fput+0x10d/0x440 [14716.431699] ____fput+0x12/0x20 [14716.437638] task_work_run+0xc6/0x110 [14716.442638] exit_to_usermode_loop+0x1dd/0x1f0 [14716.444264] do_syscall_64+0x3d6/0x440 [14716.456202] entry_SYSCALL_64_after_hwframe+0x49/0xae [14716.462244] RIP: 0033:0x7fa749552cd8 [14716.471332] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14716.505077] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14716.527197] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14716.541735] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14716.544347] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14716.567987] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14716.582924] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14718.432343] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12801s! [14718.438046] Showing busy workqueues and worker pools: [14718.440564] workqueue events: flags=0x0 [14718.442315] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14718.445396] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14718.450774] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14718.454364] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14718.458611] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.462132] pending: kfree_rcu_monitor [14718.464431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14718.467987] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14718.473552] workqueue events_long: flags=0x0 [14718.475987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.479570] pending: lu_objects_flush [obdclass] [14718.482412] workqueue events_unbound: flags=0x2 [14718.484825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14718.487963] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14718.492784] workqueue events_power_efficient: flags=0x80 [14718.495571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.498652] pending: check_lifetime [14718.500433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.502701] pending: neigh_periodic_work [14718.504826] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.507806] pending: neigh_periodic_work [14718.510307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.513773] pending: do_cache_clean [14718.515839] workqueue mm_percpu_wq: flags=0x8 [14718.518194] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.522440] pending: vmstat_update [14718.524664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.528351] pending: vmstat_update [14718.530666] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14718.535190] pending: vmstat_update [14718.536784] workqueue writeback: flags=0x4a [14718.539410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14718.543275] pending: wb_workfn, wb_workfn [14718.546201] workqueue xprtiod: flags=0xa [14718.548409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14718.552578] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14718.557845] workqueue ipv6_addrconf: flags=0x40008 [14718.560705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14718.565210] pending: addrconf_verify_work [14718.568209] workqueue knbd0-recv: flags=0x1a [14718.570573] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14718.574794] in-flight: 651:recv_work [14718.577727] workqueue ptlrpc_pinger: flags=0x2 [14718.579995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14718.583081] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14718.586223] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12801s workers=4 idle: 37 31846 28897 [14718.590522] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14708s workers=2 idle: 653 [14728.057155] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14728.065038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14728.110417] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14728.118321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14728.132960] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14728.135544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14728.146384] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14728.173206] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14728.177016] RAX: 000000000000457a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14728.180186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14728.188641] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14728.190753] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14728.202927] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14728.214106] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14728.225267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14728.231313] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14728.237151] Call Trace: [14728.243827] [14728.244928] ? show_regs.cold.9+0x22/0x2f [14728.248237] ? watchdog_timer_fn.cold.6+0x62/0x10a [14728.253271] ? __hrtimer_run_queues+0x1c2/0x4d0 [14728.262637] ? watchdog+0x40/0x40 [14728.269538] ? hrtimer_interrupt+0x128/0x2c0 [14728.277859] ? smp_apic_timer_interrupt+0xa6/0x270 [14728.288339] ? apic_timer_interrupt+0xf/0x20 [14728.295582] [14728.301540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14728.311474] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14728.315066] do_raw_spin_lock+0x177/0x190 [14728.321287] _raw_spin_lock+0x21/0x30 [14728.325157] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14728.331784] process_one_work+0x2c8/0x700 [14728.333605] worker_thread+0x296/0x6e0 [14728.349189] ? rescuer_thread+0x570/0x570 [14728.350861] kthread+0x1d1/0x200 [14728.353220] ? set_kthread_struct+0x70/0x70 [14728.358237] ret_from_fork+0x1f/0x30 [14740.058062] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14740.069454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14740.109544] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14740.128604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14740.137210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14740.143677] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14740.160672] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14740.166245] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14740.171012] RDX: 0000000000000b1e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14740.176217] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14740.182266] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14740.187422] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14740.193318] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14740.199675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14740.204404] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14740.207088] Call Trace: [14740.207872] [14740.208322] ? show_regs.cold.9+0x22/0x2f [14740.209258] ? watchdog_timer_fn.cold.6+0x62/0x10a [14740.212323] ? __hrtimer_run_queues+0x1c2/0x4d0 [14740.215466] ? watchdog+0x40/0x40 [14740.217856] ? hrtimer_interrupt+0x128/0x2c0 [14740.224609] ? smp_apic_timer_interrupt+0xa6/0x270 [14740.232713] ? apic_timer_interrupt+0xf/0x20 [14740.234090] [14740.234691] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14740.239967] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14740.245752] do_raw_spin_lock+0x177/0x190 [14740.248322] _raw_spin_lock+0x21/0x30 [14740.250226] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14740.252856] obd_mod_free+0x46/0x140 [ptlrpc] [14740.260856] mdc_commit_open+0x91/0xb0 [mdc] [14740.262824] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14740.275299] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14740.281267] after_reply+0xe2c/0x1780 [ptlrpc] [14740.286822] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14740.296602] ? __next_timer_interrupt+0x160/0x160 [14740.301802] ptlrpcd+0xa82/0xe40 [ptlrpc] [14740.307615] ? do_wait_intr+0xf0/0xf0 [14740.311437] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14740.316814] kthread+0x1d1/0x200 [14740.323074] ? set_kthread_struct+0x70/0x70 [14740.324756] ret_from_fork+0x1f/0x30 [14744.054163] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14744.060773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [14744.060798] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14744.060808] Modules linked in: [14744.068797] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14744.068866] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14744.068875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14744.068878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14744.068892] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14744.068896] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14744.068902] RAX: 0000000000002be1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14744.068904] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14744.068907] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14744.068909] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14744.068912] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14744.068916] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14744.068920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14744.068923] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14744.068932] Call Trace: [14744.068937] [14744.068943] ? show_regs.cold.9+0x22/0x2f [14744.068954] ? watchdog_timer_fn.cold.6+0x62/0x10a [14744.068960] ? __hrtimer_run_queues+0x1c2/0x4d0 [14744.068965] ? watchdog+0x40/0x40 [14744.068970] ? hrtimer_interrupt+0x128/0x2c0 [14744.068974] ? smp_apic_timer_interrupt+0xa6/0x270 [14744.068982] ? apic_timer_interrupt+0xf/0x20 [14744.068999] [14744.069003] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14744.069010] do_raw_spin_lock+0x177/0x190 [14744.069015] _raw_spin_lock+0x21/0x30 [14744.069021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14744.070855] ? do_raw_spin_unlock+0x75/0x190 [14744.070864] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14744.071222] ? do_wait_intr+0xf0/0xf0 [14744.071232] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14744.076179] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14744.076258] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14744.076266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14744.076269] RIP: 0010:native_safe_halt+0x2e/0x50 [14744.076282] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14744.076287] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14744.076291] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14744.076294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14744.076296] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14744.076298] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14744.076301] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14744.076304] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14744.076307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14744.076310] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14744.076320] Call Trace: [14744.076325] [14744.076331] ? show_regs.cold.9+0x22/0x2f [14744.076341] ? watchdog_timer_fn.cold.6+0x62/0x10a [14744.076348] ? __hrtimer_run_queues+0x1c2/0x4d0 [14744.076352] ? watchdog+0x40/0x40 [14744.076356] ? hrtimer_interrupt+0x128/0x2c0 [14744.076361] ? smp_apic_timer_interrupt+0xa6/0x270 [14744.076369] ? apic_timer_interrupt+0xf/0x20 [14744.076373] [14744.076377] ? native_safe_halt+0x2e/0x50 [14744.076381] kvm_wait+0xcd/0xe0 [14744.076389] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14744.076398] do_raw_spin_lock+0x177/0x190 [14744.076403] _raw_spin_lock+0x21/0x30 [14744.076407] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14744.077819] ? do_raw_spin_unlock+0x75/0x190 [14744.077829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14744.078095] ? do_wait_intr+0xf0/0xf0 [14744.078106] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14744.078315] mdc_close+0x53b/0xd70 [mdc] [14744.078353] lmv_close+0x18d/0x490 [lmv] [14744.078384] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14744.078506] ll_md_real_close+0xa5/0x2a0 [lustre] [14744.078827] ll_md_close+0x274/0x800 [lustre] [14744.079706] ll_file_release+0x128/0xc50 [lustre] [14744.079792] __fput+0x10d/0x440 [14744.079802] ____fput+0x12/0x20 [14744.079807] task_work_run+0xc6/0x110 [14744.079815] exit_to_usermode_loop+0x1dd/0x1f0 [14744.079823] do_syscall_64+0x3d6/0x440 [14744.079828] entry_SYSCALL_64_after_hwframe+0x49/0xae [14744.079837] RIP: 0033:0x7fa749552cd8 [14744.079846] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14744.079850] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14744.079855] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14744.079857] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14744.079859] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14744.079862] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14744.079864] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14744.597264] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14744.603196] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14744.605353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14744.613649] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14744.617810] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14744.623730] mdc_intent_lock+0x353/0x7e0 [mdc] [14744.627045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14744.639642] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14744.641895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14744.648432] lmv_intent_open+0x338/0x1150 [lmv] [14744.653296] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14744.656545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14744.659820] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14744.664660] ll_intent_lock+0xc5/0xaa0 [lustre] [14744.669774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14744.674226] ll_lookup_it+0xe0e/0x2270 [lustre] [14744.682051] ? do_raw_spin_unlock+0x75/0x190 [14744.685435] ? _raw_spin_unlock+0x12/0x30 [14744.687744] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14744.691914] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14744.700476] ll_atomic_open+0x36a/0xd70 [lustre] [14744.705426] lookup_open+0x356/0xb40 [14744.706761] do_last+0x313/0xfc0 [14744.713937] ? nd_jump_root+0xe5/0x160 [14744.716921] ? path_init+0x437/0x520 [14744.729237] path_openat+0xf7/0x500 [14744.730622] ? do_raw_spin_unlock+0x75/0x190 [14744.738276] do_filp_open+0x99/0x140 [14744.741239] ? getname_flags+0x6e/0x330 [14744.746781] ? __check_object_size+0xff/0x256 [14744.748198] ? do_raw_spin_unlock+0x75/0x190 [14744.752891] ? _raw_spin_unlock+0x12/0x30 [14744.755345] do_sys_openat2+0x2b4/0x410 [14744.760026] do_sys_open+0x73/0xa0 [14744.769005] __x64_sys_openat+0x24/0x30 [14744.770436] do_syscall_64+0xc1/0x440 [14744.779333] entry_SYSCALL_64_after_hwframe+0x49/0xae [14744.783820] RIP: 0033:0x7fa749552332 [14744.786950] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14744.804658] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14744.815143] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14744.826196] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14744.844167] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14744.859323] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14744.872467] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14748.643988] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12831s! [14748.655099] Showing busy workqueues and worker pools: [14748.659521] workqueue events: flags=0x0 [14748.667853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14748.670205] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14748.684450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14748.689419] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14748.700302] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.704410] pending: kfree_rcu_monitor [14748.710995] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14748.718611] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14748.732813] workqueue events_long: flags=0x0 [14748.737142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.742118] pending: lu_objects_flush [obdclass] [14748.747835] workqueue events_unbound: flags=0x2 [14748.752588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14748.756272] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14748.765778] workqueue events_power_efficient: flags=0x80 [14748.770187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.774312] pending: check_lifetime [14748.779097] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.789379] pending: neigh_periodic_work [14748.797104] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.802016] pending: neigh_periodic_work [14748.809009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.816236] pending: do_cache_clean [14748.819128] workqueue mm_percpu_wq: flags=0x8 [14748.822167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.831312] pending: vmstat_update [14748.832343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.845769] pending: vmstat_update [14748.850067] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14748.859197] pending: vmstat_update [14748.860603] workqueue writeback: flags=0x4a [14748.867864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14748.877193] pending: wb_workfn, wb_workfn [14748.882151] workqueue xprtiod: flags=0xa [14748.885203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14748.887148] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14748.900044] workqueue ipv6_addrconf: flags=0x40008 [14748.907082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14748.922156] pending: addrconf_verify_work [14748.931044] workqueue knbd0-recv: flags=0x1a [14748.933178] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14748.943752] in-flight: 651:recv_work [14748.946318] workqueue ptlrpc_pinger: flags=0x2 [14748.949296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14748.955054] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14748.959709] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12831s workers=4 idle: 37 31846 28897 [14748.968781] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14738s workers=2 idle: 653 [14753.972354] rcu: INFO: rcu_sched self-detected stall on CPU [14753.976192] rcu: 2-...!: (7375101 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [14753.992946] (t=12840283 jiffies g=258701 q=5059) [14753.996083] rcu: rcu_sched kthread starved for 12837078 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [14754.008080] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14754.019139] rcu: RCU grace-period kthread stack dump: [14754.023983] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14754.042003] Call Trace: [14754.045448] __schedule+0x351/0xcb0 [14754.049508] schedule+0xc0/0x180 [14754.052104] schedule_timeout+0xb4/0x190 [14754.055227] ? do_raw_spin_unlock+0x75/0x190 [14754.060474] ? __next_timer_interrupt+0x160/0x160 [14754.067495] ? prepare_to_swait_event+0xf3/0x140 [14754.069831] rcu_gp_kthread+0x8dd/0x15b0 [14754.072132] ? force_qs_rnp+0x2c0/0x2c0 [14754.076742] kthread+0x1d1/0x200 [14754.079121] ? set_kthread_struct+0x70/0x70 [14754.081399] ret_from_fork+0x1f/0x30 [14754.084204] rcu: Stack dump where RCU GP kthread last ran: [14754.088053] Sending NMI from CPU 2 to CPUs 3: [14754.092622] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [14754.093580] NMI backtrace for cpu 2 [14754.099006] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14754.106434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14754.113618] Call Trace: [14754.115080] [14754.116182] ? dump_stack+0xbb/0x10e [14754.117743] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14754.120623] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14754.126049] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14754.134081] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14754.139511] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14754.147202] ? rcu_dump_cpu_stacks+0x11b/0x171 [14754.155320] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14754.160997] ? update_process_times+0x6d/0xc0 [14754.165663] ? tick_sched_handle.isra.18+0x27/0x90 [14754.168368] ? tick_sched_timer+0x53/0xb0 [14754.173212] ? __hrtimer_run_queues+0x1c2/0x4d0 [14754.177960] ? tick_sched_handle.isra.18+0x90/0x90 [14754.181087] ? hrtimer_interrupt+0x128/0x2c0 [14754.183593] ? smp_apic_timer_interrupt+0xa6/0x270 [14754.186129] ? apic_timer_interrupt+0xf/0x20 [14754.189209] [14754.190003] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14754.193715] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14754.196409] ? do_raw_spin_lock+0x177/0x190 [14754.198698] ? _raw_spin_lock+0x21/0x30 [14754.201412] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14754.205551] ? obd_mod_free+0x46/0x140 [ptlrpc] [14754.208788] ? mdc_commit_open+0x91/0xb0 [mdc] [14754.210693] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14754.213529] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14754.217437] ? after_reply+0xe2c/0x1780 [ptlrpc] [14754.220541] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14754.226212] ? __next_timer_interrupt+0x160/0x160 [14754.230453] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14754.234357] ? do_wait_intr+0xf0/0xf0 [14754.236195] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14754.241731] ? kthread+0x1d1/0x200 [14754.242511] ? set_kthread_struct+0x70/0x70 [14754.243453] ? ret_from_fork+0x1f/0x30 [14756.060496] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14756.070995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14756.128694] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14756.140301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14756.153466] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14756.160988] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14756.163710] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14756.198182] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14756.209609] RAX: 0000000000002e89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14756.216967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14756.228100] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14756.238073] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14756.251993] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14756.267371] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14756.284411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14756.290954] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14756.299958] Call Trace: [14756.303741] [14756.305937] ? show_regs.cold.9+0x22/0x2f [14756.309245] ? watchdog_timer_fn.cold.6+0x62/0x10a [14756.317949] ? __hrtimer_run_queues+0x1c2/0x4d0 [14756.320781] ? watchdog+0x40/0x40 [14756.326611] ? hrtimer_interrupt+0x128/0x2c0 [14756.329593] ? smp_apic_timer_interrupt+0xa6/0x270 [14756.335039] ? apic_timer_interrupt+0xf/0x20 [14756.340093] [14756.354736] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14756.365708] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14756.370987] do_raw_spin_lock+0x177/0x190 [14756.374983] _raw_spin_lock+0x21/0x30 [14756.377138] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14756.383299] process_one_work+0x2c8/0x700 [14756.393067] worker_thread+0x296/0x6e0 [14756.395518] ? rescuer_thread+0x570/0x570 [14756.402907] kthread+0x1d1/0x200 [14756.405241] ? set_kthread_struct+0x70/0x70 [14756.407491] ret_from_fork+0x1f/0x30 [14776.058405] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14776.062447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14776.103131] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14776.110557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14776.120503] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14776.124242] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14776.149826] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14776.160799] RAX: 0000000000007790 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14776.175527] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14776.178113] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14776.186251] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14776.190328] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14776.194527] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14776.199945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14776.202576] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14776.208183] Call Trace: [14776.210272] [14776.211036] ? show_regs.cold.9+0x22/0x2f [14776.212508] ? watchdog_timer_fn.cold.6+0x62/0x10a [14776.217342] ? __hrtimer_run_queues+0x1c2/0x4d0 [14776.221202] ? watchdog+0x40/0x40 [14776.222511] ? hrtimer_interrupt+0x128/0x2c0 [14776.226712] ? smp_apic_timer_interrupt+0xa6/0x270 [14776.231298] ? apic_timer_interrupt+0xf/0x20 [14776.233740] [14776.234759] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14776.237807] do_raw_spin_lock+0x177/0x190 [14776.240953] _raw_spin_lock+0x21/0x30 [14776.242804] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14776.246977] ? do_raw_spin_unlock+0x75/0x190 [14776.250805] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14776.253327] ? do_wait_intr+0xf0/0xf0 [14776.257000] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14776.259639] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14776.263614] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14776.267940] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14776.271175] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14776.274597] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14776.276935] mdc_intent_lock+0x353/0x7e0 [mdc] [14776.281188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14776.285289] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14776.288896] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14776.292298] lmv_intent_open+0x338/0x1150 [lmv] [14776.295531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14776.299609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14776.306548] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14776.309489] ll_intent_lock+0xc5/0xaa0 [lustre] [14776.313505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14776.317427] ll_lookup_it+0xe0e/0x2270 [lustre] [14776.320443] ? do_raw_spin_unlock+0x75/0x190 [14776.323090] ? _raw_spin_unlock+0x12/0x30 [14776.324094] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14776.328513] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14776.332631] ll_atomic_open+0x36a/0xd70 [lustre] [14776.335420] lookup_open+0x356/0xb40 [14776.338262] do_last+0x313/0xfc0 [14776.340267] ? nd_jump_root+0xe5/0x160 [14776.343767] ? path_init+0x437/0x520 [14776.345119] path_openat+0xf7/0x500 [14776.348425] ? do_raw_spin_unlock+0x75/0x190 [14776.352977] do_filp_open+0x99/0x140 [14776.358188] ? getname_flags+0x6e/0x330 [14776.362066] ? __check_object_size+0xff/0x256 [14776.364332] ? do_raw_spin_unlock+0x75/0x190 [14776.367055] ? _raw_spin_unlock+0x12/0x30 [14776.369337] do_sys_openat2+0x2b4/0x410 [14776.372747] do_sys_open+0x73/0xa0 [14776.376909] __x64_sys_openat+0x24/0x30 [14776.378958] do_syscall_64+0xc1/0x440 [14776.382326] entry_SYSCALL_64_after_hwframe+0x49/0xae [14776.386114] RIP: 0033:0x7fa749552332 [14776.387669] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14776.412242] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14776.417397] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14776.421574] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14776.429347] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14776.433579] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14776.437515] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14779.360228] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12862s! [14779.367473] Showing busy workqueues and worker pools: [14779.373969] workqueue events: flags=0x0 [14779.377728] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14779.385523] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14779.399851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14779.406757] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14779.420260] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.428679] pending: kfree_rcu_monitor [14779.435577] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14779.440670] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14779.462675] workqueue events_long: flags=0x0 [14779.466290] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.474851] pending: lu_objects_flush [obdclass] [14779.479739] workqueue events_unbound: flags=0x2 [14779.482239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14779.487174] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14779.493277] workqueue events_power_efficient: flags=0x80 [14779.496117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.499548] pending: check_lifetime [14779.501048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.504463] pending: neigh_periodic_work [14779.507023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.509903] pending: neigh_periodic_work [14779.511795] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.515293] pending: do_cache_clean [14779.517206] workqueue mm_percpu_wq: flags=0x8 [14779.519677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.523749] pending: vmstat_update [14779.527297] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.531004] pending: vmstat_update [14779.532852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14779.535954] pending: vmstat_update [14779.537756] workqueue writeback: flags=0x4a [14779.540330] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14779.545150] pending: wb_workfn, wb_workfn [14779.547499] workqueue xprtiod: flags=0xa [14779.549864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14779.553303] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14779.558650] workqueue ipv6_addrconf: flags=0x40008 [14779.561513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14779.568533] pending: addrconf_verify_work [14779.574597] workqueue knbd0-recv: flags=0x1a [14779.577531] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14779.581727] in-flight: 651:recv_work [14779.586257] workqueue ptlrpc_pinger: flags=0x2 [14779.590525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14779.593522] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14779.596507] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12862s workers=4 idle: 37 31846 28897 [14779.601957] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14769s workers=2 idle: 653 [14780.058010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14780.062001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14780.062546] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14780.098632] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14780.098646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14780.098649] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14780.098662] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14780.098666] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14780.098672] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14780.098674] RDX: 0000000000002a24 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14780.098676] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14780.098678] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14780.098680] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14780.098683] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14780.098685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14780.098687] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14780.099669] Call Trace: [14780.099678] [14780.099684] ? show_regs.cold.9+0x22/0x2f [14780.099696] ? watchdog_timer_fn.cold.6+0x62/0x10a [14780.099701] ? __hrtimer_run_queues+0x1c2/0x4d0 [14780.099705] ? watchdog+0x40/0x40 [14780.099710] ? hrtimer_interrupt+0x128/0x2c0 [14780.099713] ? smp_apic_timer_interrupt+0xa6/0x270 [14780.099720] ? apic_timer_interrupt+0xf/0x20 [14780.099724] [14780.099727] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14780.099732] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14780.099736] do_raw_spin_lock+0x177/0x190 [14780.099740] _raw_spin_lock+0x21/0x30 [14780.099746] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14780.100047] obd_mod_free+0x46/0x140 [ptlrpc] [14780.100260] mdc_commit_open+0x91/0xb0 [mdc] [14780.100293] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14780.100487] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14780.100680] after_reply+0xe2c/0x1780 [ptlrpc] [14780.100886] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14780.101105] ? __next_timer_interrupt+0x160/0x160 [14780.101114] ptlrpcd+0xa82/0xe40 [ptlrpc] [14780.101318] ? do_wait_intr+0xf0/0xf0 [14780.101327] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14780.101528] kthread+0x1d1/0x200 [14780.101534] ? set_kthread_struct+0x70/0x70 [14780.101538] ret_from_fork+0x1f/0x30 [14780.420457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14780.493787] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14780.509655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14780.517368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14780.523660] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14780.541505] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14780.549519] RAX: 0000000000007f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14780.558207] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14780.564964] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14780.572128] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14780.579064] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14780.586569] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14780.592316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14780.600083] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14780.609370] Call Trace: [14780.611201] [14780.613371] ? show_regs.cold.9+0x22/0x2f [14780.619452] ? watchdog_timer_fn.cold.6+0x62/0x10a [14780.626043] ? __hrtimer_run_queues+0x1c2/0x4d0 [14780.628477] ? watchdog+0x40/0x40 [14780.633303] ? hrtimer_interrupt+0x128/0x2c0 [14780.637040] ? smp_apic_timer_interrupt+0xa6/0x270 [14780.644358] ? apic_timer_interrupt+0xf/0x20 [14780.648388] [14780.650034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14780.660751] do_raw_spin_lock+0x177/0x190 [14780.667098] _raw_spin_lock+0x21/0x30 [14780.672324] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14780.679186] ? do_raw_spin_unlock+0x75/0x190 [14780.687521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14780.691255] ? do_wait_intr+0xf0/0xf0 [14780.697092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14780.701207] mdc_close+0x53b/0xd70 [mdc] [14780.703371] lmv_close+0x18d/0x490 [lmv] [14780.705389] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14780.709071] ll_md_real_close+0xa5/0x2a0 [lustre] [14780.714112] ll_md_close+0x274/0x800 [lustre] [14780.720236] ll_file_release+0x128/0xc50 [lustre] [14780.724979] __fput+0x10d/0x440 [14780.726228] ____fput+0x12/0x20 [14780.729319] task_work_run+0xc6/0x110 [14780.730578] exit_to_usermode_loop+0x1dd/0x1f0 [14780.733147] do_syscall_64+0x3d6/0x440 [14780.734707] entry_SYSCALL_64_after_hwframe+0x49/0xae [14780.737913] RIP: 0033:0x7fa749552cd8 [14780.740522] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14780.761601] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14780.771443] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14780.781037] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14780.785946] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14780.796447] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14780.802591] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14784.057028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14784.062141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14784.121926] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14784.130317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14784.137902] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14784.143823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14784.152723] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14784.171917] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14784.176855] RAX: 0000000000006794 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14784.181355] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14784.187413] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14784.198098] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14784.202149] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14784.215091] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14784.217909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14784.226877] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14784.233008] Call Trace: [14784.239087] [14784.243031] ? show_regs.cold.9+0x22/0x2f [14784.248508] ? watchdog_timer_fn.cold.6+0x62/0x10a [14784.250158] ? __hrtimer_run_queues+0x1c2/0x4d0 [14784.255805] ? watchdog+0x40/0x40 [14784.257110] ? hrtimer_interrupt+0x128/0x2c0 [14784.262003] ? smp_apic_timer_interrupt+0xa6/0x270 [14784.263552] ? apic_timer_interrupt+0xf/0x20 [14784.264713] [14784.265869] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14784.268399] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14784.270927] do_raw_spin_lock+0x177/0x190 [14784.272812] _raw_spin_lock+0x21/0x30 [14784.273578] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14784.276134] process_one_work+0x2c8/0x700 [14784.277382] worker_thread+0x296/0x6e0 [14784.280246] ? rescuer_thread+0x570/0x570 [14784.283453] kthread+0x1d1/0x200 [14784.284368] ? set_kthread_struct+0x70/0x70 [14784.285657] ret_from_fork+0x1f/0x30 [14804.054031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14804.057134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14804.083347] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14804.094278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14804.098585] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14804.100738] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14804.107144] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14804.109786] RAX: 00000000000039c6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14804.111820] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14804.115527] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14804.117914] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14804.129025] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14804.133961] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14804.140260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14804.144538] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14804.149281] Call Trace: [14804.150021] [14804.150508] ? show_regs.cold.9+0x22/0x2f [14804.154050] ? watchdog_timer_fn.cold.6+0x62/0x10a [14804.155851] ? __hrtimer_run_queues+0x1c2/0x4d0 [14804.157294] ? watchdog+0x40/0x40 [14804.158307] ? hrtimer_interrupt+0x128/0x2c0 [14804.160101] ? smp_apic_timer_interrupt+0xa6/0x270 [14804.161552] ? apic_timer_interrupt+0xf/0x20 [14804.163157] [14804.164142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14804.166692] do_raw_spin_lock+0x177/0x190 [14804.168778] _raw_spin_lock+0x21/0x30 [14804.170353] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14804.172782] ? do_raw_spin_unlock+0x75/0x190 [14804.174727] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14804.176748] ? do_wait_intr+0xf0/0xf0 [14804.177865] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14804.179898] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14804.181897] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14804.184211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14804.186658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14804.188860] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14804.190322] mdc_intent_lock+0x353/0x7e0 [mdc] [14804.192052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14804.194258] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14804.196707] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14804.202169] lmv_intent_open+0x338/0x1150 [lmv] [14804.204222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14804.206882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14804.209167] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14804.211214] ll_intent_lock+0xc5/0xaa0 [lustre] [14804.213453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14804.216069] ll_lookup_it+0xe0e/0x2270 [lustre] [14804.219004] ? do_raw_spin_unlock+0x75/0x190 [14804.220906] ? _raw_spin_unlock+0x12/0x30 [14804.225298] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14804.228833] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14804.233909] ll_atomic_open+0x36a/0xd70 [lustre] [14804.237865] lookup_open+0x356/0xb40 [14804.240129] do_last+0x313/0xfc0 [14804.241415] ? nd_jump_root+0xe5/0x160 [14804.247323] ? path_init+0x437/0x520 [14804.249897] path_openat+0xf7/0x500 [14804.253499] ? do_raw_spin_unlock+0x75/0x190 [14804.255140] do_filp_open+0x99/0x140 [14804.258894] ? getname_flags+0x6e/0x330 [14804.260127] ? __check_object_size+0xff/0x256 [14804.261483] ? do_raw_spin_unlock+0x75/0x190 [14804.264248] ? _raw_spin_unlock+0x12/0x30 [14804.267572] do_sys_openat2+0x2b4/0x410 [14804.269909] do_sys_open+0x73/0xa0 [14804.271197] __x64_sys_openat+0x24/0x30 [14804.274427] do_syscall_64+0xc1/0x440 [14804.275679] entry_SYSCALL_64_after_hwframe+0x49/0xae [14804.277608] RIP: 0033:0x7fa749552332 [14804.279185] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14804.289846] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14804.292914] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14804.296006] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14804.299816] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14804.304297] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14804.308066] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14808.058015] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14808.062164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14808.063542] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14808.090562] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14808.090587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14808.090590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14808.090604] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14808.090609] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14808.090615] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14808.090618] RDX: 0000000000005377 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14808.090620] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14808.090623] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14808.090625] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14808.090629] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14808.090632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14808.090634] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14808.090643] Call Trace: [14808.090649] [14808.090655] ? show_regs.cold.9+0x22/0x2f [14808.090665] ? watchdog_timer_fn.cold.6+0x62/0x10a [14808.090672] ? __hrtimer_run_queues+0x1c2/0x4d0 [14808.090676] ? watchdog+0x40/0x40 [14808.090681] ? hrtimer_interrupt+0x128/0x2c0 [14808.090685] ? smp_apic_timer_interrupt+0xa6/0x270 [14808.090694] ? apic_timer_interrupt+0xf/0x20 [14808.090699] [14808.090702] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14808.090707] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14808.090712] do_raw_spin_lock+0x177/0x190 [14808.090717] _raw_spin_lock+0x21/0x30 [14808.090725] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14808.090998] obd_mod_free+0x46/0x140 [ptlrpc] [14808.099594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14808.099674] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14808.099682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14808.099685] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14808.099699] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14808.099704] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14808.099709] RAX: 0000000000005f83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14808.099712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14808.099714] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14808.099717] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14808.099719] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14808.099723] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14808.099726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14808.099729] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14808.099737] Call Trace: [14808.099742] [14808.099748] ? show_regs.cold.9+0x22/0x2f [14808.099756] ? watchdog_timer_fn.cold.6+0x62/0x10a [14808.099763] ? __hrtimer_run_queues+0x1c2/0x4d0 [14808.099766] ? watchdog+0x40/0x40 [14808.099770] ? hrtimer_interrupt+0x128/0x2c0 [14808.099774] ? smp_apic_timer_interrupt+0xa6/0x270 [14808.099783] ? apic_timer_interrupt+0xf/0x20 [14808.099788] [14808.099791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14808.099798] do_raw_spin_lock+0x177/0x190 [14808.099803] _raw_spin_lock+0x21/0x30 [14808.099810] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14808.107610] mdc_commit_open+0x91/0xb0 [mdc] [14808.111509] ? do_raw_spin_unlock+0x75/0x190 [14808.114067] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14808.126406] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14808.133062] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14808.140600] ? do_wait_intr+0xf0/0xf0 [14808.140623] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14808.140902] mdc_close+0x53b/0xd70 [mdc] [14808.140941] lmv_close+0x18d/0x490 [lmv] [14808.140966] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14808.141086] ll_md_real_close+0xa5/0x2a0 [lustre] [14808.145112] after_reply+0xe2c/0x1780 [ptlrpc] [14808.155586] ll_md_close+0x274/0x800 [lustre] [14808.162762] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14808.170846] ll_file_release+0x128/0xc50 [lustre] [14808.174626] ? __next_timer_interrupt+0x160/0x160 [14808.182011] __fput+0x10d/0x440 [14808.182034] ____fput+0x12/0x20 [14808.182039] task_work_run+0xc6/0x110 [14808.182046] exit_to_usermode_loop+0x1dd/0x1f0 [14808.182053] do_syscall_64+0x3d6/0x440 [14808.182057] entry_SYSCALL_64_after_hwframe+0x49/0xae [14808.182065] RIP: 0033:0x7fa749552cd8 [14808.182073] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14808.182076] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14808.182081] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14808.182084] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14808.182086] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14808.182088] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14808.182091] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14808.461755] ptlrpcd+0xa82/0xe40 [ptlrpc] [14808.464463] ? do_wait_intr+0xf0/0xf0 [14808.465665] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14808.470057] kthread+0x1d1/0x200 [14808.471192] ? set_kthread_struct+0x70/0x70 [14808.474794] ret_from_fork+0x1f/0x30 [14810.080199] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12892s! [14810.084851] Showing busy workqueues and worker pools: [14810.087699] workqueue events: flags=0x0 [14810.093341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14810.097843] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14810.106977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14810.110419] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14810.121570] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.128667] pending: kfree_rcu_monitor [14810.129985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14810.137755] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14810.146226] workqueue events_long: flags=0x0 [14810.149124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.156967] pending: lu_objects_flush [obdclass] [14810.161155] workqueue events_unbound: flags=0x2 [14810.163646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14810.167288] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14810.172089] workqueue events_power_efficient: flags=0x80 [14810.175802] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.180885] pending: check_lifetime [14810.184641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.189285] pending: neigh_periodic_work [14810.193027] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.197126] pending: neigh_periodic_work [14810.199969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.204914] pending: do_cache_clean [14810.208787] workqueue mm_percpu_wq: flags=0x8 [14810.216130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.226314] pending: vmstat_update [14810.232848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.240895] pending: vmstat_update [14810.249631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14810.257862] pending: vmstat_update [14810.262632] workqueue writeback: flags=0x4a [14810.270171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14810.283366] pending: wb_workfn, wb_workfn [14810.290286] workqueue xprtiod: flags=0xa [14810.296392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14810.299644] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14810.303217] workqueue ipv6_addrconf: flags=0x40008 [14810.305320] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14810.308437] pending: addrconf_verify_work [14810.310806] workqueue knbd0-recv: flags=0x1a [14810.312614] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14810.315462] in-flight: 651:recv_work [14810.317601] workqueue ptlrpc_pinger: flags=0x2 [14810.319506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14810.322065] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14810.326013] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12893s workers=4 idle: 37 31846 28897 [14810.335564] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14800s workers=2 idle: 653 [14812.060754] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14812.072211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14812.129055] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14812.141395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14812.168098] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14812.177980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14812.186190] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14812.224972] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14812.238474] RAX: 000000000000790a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14812.251702] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14812.269357] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14812.285052] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14812.295099] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14812.306977] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14812.322717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14812.324681] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14812.336877] Call Trace: [14812.337802] [14812.346416] ? show_regs.cold.9+0x22/0x2f [14812.350749] ? watchdog_timer_fn.cold.6+0x62/0x10a [14812.361261] ? __hrtimer_run_queues+0x1c2/0x4d0 [14812.366721] ? watchdog+0x40/0x40 [14812.376044] ? hrtimer_interrupt+0x128/0x2c0 [14812.377424] ? smp_apic_timer_interrupt+0xa6/0x270 [14812.388439] ? apic_timer_interrupt+0xf/0x20 [14812.395981] [14812.399307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14812.407305] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14812.416914] do_raw_spin_lock+0x177/0x190 [14812.422249] _raw_spin_lock+0x21/0x30 [14812.424076] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14812.425648] process_one_work+0x2c8/0x700 [14812.430616] worker_thread+0x296/0x6e0 [14812.435670] ? rescuer_thread+0x570/0x570 [14812.439474] kthread+0x1d1/0x200 [14812.446975] ? set_kthread_struct+0x70/0x70 [14812.454031] ret_from_fork+0x1f/0x30 [14836.054039] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14836.057350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.058024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14836.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14836.059032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.059133] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.059142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.059145] RIP: 0010:native_safe_halt+0x2e/0x50 [14836.059161] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14836.059166] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14836.059171] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14836.059174] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14836.059176] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14836.059179] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14836.059181] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14836.059188] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14836.059192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.059194] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14836.059203] Call Trace: [14836.059212] [14836.059218] ? show_regs.cold.9+0x22/0x2f [14836.059232] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.059238] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.059243] ? watchdog+0x40/0x40 [14836.059248] ? hrtimer_interrupt+0x128/0x2c0 [14836.059253] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.059260] ? apic_timer_interrupt+0xf/0x20 [14836.059273] [14836.059277] ? native_safe_halt+0x2e/0x50 [14836.059282] kvm_wait+0xcd/0xe0 [14836.059290] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14836.059299] do_raw_spin_lock+0x177/0x190 [14836.059303] _raw_spin_lock+0x21/0x30 [14836.059308] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14836.059669] ? do_raw_spin_unlock+0x75/0x190 [14836.059675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14836.059919] ? do_wait_intr+0xf0/0xf0 [14836.059927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14836.060250] mdc_close+0x53b/0xd70 [mdc] [14836.060286] lmv_close+0x18d/0x490 [lmv] [14836.060317] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14836.060434] ll_md_real_close+0xa5/0x2a0 [lustre] [14836.060515] ll_md_close+0x274/0x800 [lustre] [14836.060604] ll_file_release+0x128/0xc50 [lustre] [14836.060697] __fput+0x10d/0x440 [14836.060705] ____fput+0x12/0x20 [14836.060710] task_work_run+0xc6/0x110 [14836.060799] exit_to_usermode_loop+0x1dd/0x1f0 [14836.060807] do_syscall_64+0x3d6/0x440 [14836.060812] entry_SYSCALL_64_after_hwframe+0x49/0xae [14836.060821] RIP: 0033:0x7fa749552cd8 [14836.060828] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14836.060836] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14836.060845] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14836.060847] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14836.060850] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14836.060852] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14836.060853] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14836.075208] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.075223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.075227] RIP: 0010:native_safe_halt+0x2e/0x50 [14836.075240] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14836.075245] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14836.075250] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14836.075252] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [14836.075254] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14836.075256] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14836.075258] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14836.075261] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14836.075265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.075267] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14836.075282] Call Trace: [14836.075288] [14836.079744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.079827] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.079834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.079837] RIP: 0010:native_safe_halt+0x2e/0x50 [14836.079849] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14836.079853] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14836.079858] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14836.079860] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14836.079862] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14836.079864] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14836.079866] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14836.079869] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14836.079872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.079874] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14836.079880] Call Trace: [14836.079884] [14836.079889] ? show_regs.cold.9+0x22/0x2f [14836.079898] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.079904] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.079907] ? watchdog+0x40/0x40 [14836.079911] ? hrtimer_interrupt+0x128/0x2c0 [14836.079915] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.079921] ? apic_timer_interrupt+0xf/0x20 [14836.079926] [14836.079929] ? native_safe_halt+0x2e/0x50 [14836.079933] kvm_wait+0xcd/0xe0 [14836.079940] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14836.079946] do_raw_spin_lock+0x177/0x190 [14836.079950] _raw_spin_lock+0x21/0x30 [14836.079956] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14836.080221] obd_mod_free+0x46/0x140 [ptlrpc] [14836.080578] mdc_commit_open+0x91/0xb0 [mdc] [14836.080620] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14836.080858] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14836.081051] after_reply+0xe2c/0x1780 [ptlrpc] [14836.081257] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14836.081619] ? __next_timer_interrupt+0x160/0x160 [14836.081628] ptlrpcd+0xa82/0xe40 [ptlrpc] [14836.081884] ? do_wait_intr+0xf0/0xf0 [14836.081892] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14836.082100] kthread+0x1d1/0x200 [14836.082105] ? set_kthread_struct+0x70/0x70 [14836.082109] ret_from_fork+0x1f/0x30 [14836.382551] ? show_regs.cold.9+0x22/0x2f [14836.383745] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.385264] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.386724] ? watchdog+0x40/0x40 [14836.387786] ? hrtimer_interrupt+0x128/0x2c0 [14836.389201] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.390575] ? apic_timer_interrupt+0xf/0x20 [14836.392017] [14836.392578] ? native_safe_halt+0x2e/0x50 [14836.393752] kvm_wait+0xcd/0xe0 [14836.394625] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14836.396342] do_raw_spin_lock+0x177/0x190 [14836.397605] _raw_spin_lock+0x21/0x30 [14836.398770] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14836.400682] ? do_raw_spin_unlock+0x75/0x190 [14836.402278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14836.404072] ? do_wait_intr+0xf0/0xf0 [14836.405255] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14836.407087] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14836.408901] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14836.410787] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.412518] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14836.414309] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14836.415682] mdc_intent_lock+0x353/0x7e0 [mdc] [14836.417158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.418746] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14836.420612] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14836.422290] lmv_intent_open+0x338/0x1150 [lmv] [14836.423594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.426095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.428380] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14836.430173] ll_intent_lock+0xc5/0xaa0 [lustre] [14836.432583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.435538] ll_lookup_it+0xe0e/0x2270 [lustre] [14836.437242] ? do_raw_spin_unlock+0x75/0x190 [14836.438904] ? _raw_spin_unlock+0x12/0x30 [14836.440225] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14836.442092] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14836.443945] ll_atomic_open+0x36a/0xd70 [lustre] [14836.445695] lookup_open+0x356/0xb40 [14836.447885] do_last+0x313/0xfc0 [14836.448993] ? nd_jump_root+0xe5/0x160 [14836.450265] ? path_init+0x437/0x520 [14836.454030] path_openat+0xf7/0x500 [14836.456328] ? do_raw_spin_unlock+0x75/0x190 [14836.457856] do_filp_open+0x99/0x140 [14836.459264] ? getname_flags+0x6e/0x330 [14836.460647] ? __check_object_size+0xff/0x256 [14836.461838] ? do_raw_spin_unlock+0x75/0x190 [14836.466515] ? _raw_spin_unlock+0x12/0x30 [14836.468478] do_sys_openat2+0x2b4/0x410 [14836.472395] do_sys_open+0x73/0xa0 [14836.473594] __x64_sys_openat+0x24/0x30 [14836.476272] do_syscall_64+0xc1/0x440 [14836.477416] entry_SYSCALL_64_after_hwframe+0x49/0xae [14836.479142] RIP: 0033:0x7fa749552332 [14836.480394] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14836.486417] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14836.489312] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14836.493104] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14836.495245] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14836.497499] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14836.499954] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14840.057036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14840.060154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14840.089007] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14840.093539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14840.097410] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14840.099648] RIP: 0010:native_safe_halt+0x2e/0x50 [14840.102457] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14840.118964] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14840.124478] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14840.129332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14840.131806] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14840.141950] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14840.144613] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14840.149436] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14840.154096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14840.157905] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14840.161005] Call Trace: [14840.161684] [14840.163703] ? show_regs.cold.9+0x22/0x2f [14840.166150] ? watchdog_timer_fn.cold.6+0x62/0x10a [14840.170512] ? __hrtimer_run_queues+0x1c2/0x4d0 [14840.172925] ? watchdog+0x40/0x40 [14840.174664] ? hrtimer_interrupt+0x128/0x2c0 [14840.176552] ? smp_apic_timer_interrupt+0xa6/0x270 [14840.179082] ? apic_timer_interrupt+0xf/0x20 [14840.181040] [14840.181988] ? native_safe_halt+0x2e/0x50 [14840.183691] kvm_wait+0xcd/0xe0 [14840.189178] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14840.194223] do_raw_spin_lock+0x177/0x190 [14840.198652] _raw_spin_lock+0x21/0x30 [14840.200068] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14840.203901] process_one_work+0x2c8/0x700 [14840.205096] worker_thread+0x296/0x6e0 [14840.211202] ? rescuer_thread+0x570/0x570 [14840.213414] kthread+0x1d1/0x200 [14840.216192] ? set_kthread_struct+0x70/0x70 [14840.219234] ret_from_fork+0x1f/0x30 [14840.806515] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12923s! [14840.811533] Showing busy workqueues and worker pools: [14840.814460] workqueue events: flags=0x0 [14840.822166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14840.831121] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14840.844629] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14840.854095] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14840.859374] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.863431] pending: kfree_rcu_monitor [14840.866132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14840.869499] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14840.873552] workqueue events_long: flags=0x0 [14840.876566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.879811] pending: lu_objects_flush [obdclass] [14840.882050] workqueue events_unbound: flags=0x2 [14840.883672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14840.886736] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14840.890867] workqueue events_power_efficient: flags=0x80 [14840.893663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.897199] pending: check_lifetime [14840.899088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.902195] pending: neigh_periodic_work [14840.904432] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.908376] pending: neigh_periodic_work [14840.910422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.915866] pending: do_cache_clean [14840.916891] workqueue mm_percpu_wq: flags=0x8 [14840.917989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.921371] pending: vmstat_update [14840.923010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.925698] pending: vmstat_update [14840.931791] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14840.935432] pending: vmstat_update [14840.937281] workqueue writeback: flags=0x4a [14840.939134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14840.941818] pending: wb_workfn, wb_workfn [14840.944173] workqueue xprtiod: flags=0xa [14840.945826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14840.952510] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14840.956978] workqueue ipv6_addrconf: flags=0x40008 [14840.964804] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14840.969696] pending: addrconf_verify_work [14840.971462] workqueue knbd0-recv: flags=0x1a [14840.973299] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14840.976276] in-flight: 651:recv_work [14840.978412] workqueue ptlrpc_pinger: flags=0x2 [14840.981077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14840.984705] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14840.988451] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12923s workers=4 idle: 37 31846 28897 [14840.993690] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14830s workers=2 idle: 653 [14864.058316] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14864.064564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14864.066699] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14864.088156] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14864.088169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14864.088171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14864.088189] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14864.088193] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14864.088199] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14864.088202] RDX: 0000000000005294 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14864.088204] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14864.088206] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14864.088209] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14864.088212] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14864.088214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14864.088217] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14864.088521] Call Trace: [14864.088528] [14864.088533] ? show_regs.cold.9+0x22/0x2f [14864.088543] ? watchdog_timer_fn.cold.6+0x62/0x10a [14864.088549] ? __hrtimer_run_queues+0x1c2/0x4d0 [14864.088554] ? watchdog+0x40/0x40 [14864.088559] ? hrtimer_interrupt+0x128/0x2c0 [14864.088563] ? smp_apic_timer_interrupt+0xa6/0x270 [14864.088571] ? apic_timer_interrupt+0xf/0x20 [14864.088576] [14864.088579] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14864.088599] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14864.088605] do_raw_spin_lock+0x177/0x190 [14864.088610] _raw_spin_lock+0x21/0x30 [14864.088617] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14864.093983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14864.094067] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14864.094075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14864.094077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14864.094091] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14864.094095] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14864.094100] RAX: 0000000000007f19 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14864.094102] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14864.094105] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14864.094107] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14864.094109] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14864.094113] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14864.094116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14864.094118] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14864.094128] Call Trace: [14864.094133] [14864.094139] ? show_regs.cold.9+0x22/0x2f [14864.094148] ? watchdog_timer_fn.cold.6+0x62/0x10a [14864.094155] ? __hrtimer_run_queues+0x1c2/0x4d0 [14864.094159] ? watchdog+0x40/0x40 [14864.094163] ? hrtimer_interrupt+0x128/0x2c0 [14864.094167] ? smp_apic_timer_interrupt+0xa6/0x270 [14864.094176] ? apic_timer_interrupt+0xf/0x20 [14864.094181] [14864.094184] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14864.094191] do_raw_spin_lock+0x177/0x190 [14864.094197] _raw_spin_lock+0x21/0x30 [14864.094392] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14864.103487] obd_mod_free+0x46/0x140 [ptlrpc] [14864.103803] mdc_commit_open+0x91/0xb0 [mdc] [14864.103841] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14864.104083] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14864.104243] after_reply+0xe2c/0x1780 [ptlrpc] [14864.104507] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14864.104728] ? __next_timer_interrupt+0x160/0x160 [14864.104738] ptlrpcd+0xa82/0xe40 [ptlrpc] [14864.104932] ? do_wait_intr+0xf0/0xf0 [14864.104940] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14864.105187] kthread+0x1d1/0x200 [14864.105193] ? set_kthread_struct+0x70/0x70 [14864.105196] ret_from_fork+0x1f/0x30 [14864.373246] ? do_raw_spin_unlock+0x75/0x190 [14864.378223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14864.382148] ? do_wait_intr+0xf0/0xf0 [14864.385805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14864.391736] mdc_close+0x53b/0xd70 [mdc] [14864.395746] lmv_close+0x18d/0x490 [lmv] [14864.396968] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14864.400842] ll_md_real_close+0xa5/0x2a0 [lustre] [14864.403977] ll_md_close+0x274/0x800 [lustre] [14864.411069] ll_file_release+0x128/0xc50 [lustre] [14864.413185] __fput+0x10d/0x440 [14864.414216] ____fput+0x12/0x20 [14864.415150] task_work_run+0xc6/0x110 [14864.418767] exit_to_usermode_loop+0x1dd/0x1f0 [14864.421757] do_syscall_64+0x3d6/0x440 [14864.423288] entry_SYSCALL_64_after_hwframe+0x49/0xae [14864.425732] RIP: 0033:0x7fa749552cd8 [14864.427153] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14864.440967] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14864.444747] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14864.448990] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14864.453682] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14864.457740] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14864.461337] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14868.054444] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14868.057223] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [14868.058321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14868.058397] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14868.058404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14868.058407] RIP: 0010:native_safe_halt+0x2e/0x50 [14868.058420] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14868.058423] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14868.058428] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14868.058430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14868.058432] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14868.058434] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14868.058436] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14868.058439] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14868.058442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14868.058444] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14868.058452] Call Trace: [14868.058457] [14868.058462] ? show_regs.cold.9+0x22/0x2f [14868.058472] ? watchdog_timer_fn.cold.6+0x62/0x10a [14868.058477] ? __hrtimer_run_queues+0x1c2/0x4d0 [14868.058481] ? watchdog+0x40/0x40 [14868.058486] ? hrtimer_interrupt+0x128/0x2c0 [14868.058489] ? smp_apic_timer_interrupt+0xa6/0x270 [14868.058495] ? apic_timer_interrupt+0xf/0x20 [14868.058499] [14868.058502] ? native_safe_halt+0x2e/0x50 [14868.058506] kvm_wait+0xcd/0xe0 [14868.058514] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14868.058537] do_raw_spin_lock+0x177/0x190 [14868.058543] _raw_spin_lock+0x21/0x30 [14868.058548] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14868.059115] ? do_raw_spin_unlock+0x75/0x190 [14868.059121] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14868.064226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14868.064315] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14868.064322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14868.064329] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14868.067700] RIP: 0010:native_safe_halt+0x2e/0x50 [14868.067715] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14868.067719] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14868.067724] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14868.067726] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14868.067728] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14868.067730] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14868.067733] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14868.067736] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14868.067739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14868.067741] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14868.067750] Call Trace: [14868.067756] [14868.067761] ? show_regs.cold.9+0x22/0x2f [14868.067771] ? watchdog_timer_fn.cold.6+0x62/0x10a [14868.067777] ? __hrtimer_run_queues+0x1c2/0x4d0 [14868.067781] ? watchdog+0x40/0x40 [14868.067786] ? hrtimer_interrupt+0x128/0x2c0 [14868.067789] ? smp_apic_timer_interrupt+0xa6/0x270 [14868.067796] ? apic_timer_interrupt+0xf/0x20 [14868.067800] [14868.067803] ? native_safe_halt+0x2e/0x50 [14868.067807] kvm_wait+0xcd/0xe0 [14868.067815] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14868.067823] do_raw_spin_lock+0x177/0x190 [14868.067828] _raw_spin_lock+0x21/0x30 [14868.067832] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14868.068379] process_one_work+0x2c8/0x700 [14868.068393] worker_thread+0x296/0x6e0 [14868.068398] ? rescuer_thread+0x570/0x570 [14868.068402] kthread+0x1d1/0x200 [14868.068407] ? set_kthread_struct+0x70/0x70 [14868.068410] ret_from_fork+0x1f/0x30 [14868.588888] ? do_wait_intr+0xf0/0xf0 [14868.593480] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14868.595606] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14868.605425] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14868.612406] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14868.616035] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14868.620332] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14868.625292] mdc_intent_lock+0x353/0x7e0 [mdc] [14868.631244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14868.633928] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14868.639179] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14868.644486] lmv_intent_open+0x338/0x1150 [lmv] [14868.646694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14868.652332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14868.656321] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14868.658989] ll_intent_lock+0xc5/0xaa0 [lustre] [14868.663334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14868.668437] ll_lookup_it+0xe0e/0x2270 [lustre] [14868.675036] ? do_raw_spin_unlock+0x75/0x190 [14868.677527] ? _raw_spin_unlock+0x12/0x30 [14868.684966] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14868.688886] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14868.693969] ll_atomic_open+0x36a/0xd70 [lustre] [14868.700930] lookup_open+0x356/0xb40 [14868.705348] do_last+0x313/0xfc0 [14868.709887] ? nd_jump_root+0xe5/0x160 [14868.713331] ? path_init+0x437/0x520 [14868.721508] path_openat+0xf7/0x500 [14868.729020] ? do_raw_spin_unlock+0x75/0x190 [14868.733055] do_filp_open+0x99/0x140 [14868.735452] ? getname_flags+0x6e/0x330 [14868.745409] ? __check_object_size+0xff/0x256 [14868.749091] ? do_raw_spin_unlock+0x75/0x190 [14868.753495] ? _raw_spin_unlock+0x12/0x30 [14868.755433] do_sys_openat2+0x2b4/0x410 [14868.765449] do_sys_open+0x73/0xa0 [14868.768762] __x64_sys_openat+0x24/0x30 [14868.777042] do_syscall_64+0xc1/0x440 [14868.784858] entry_SYSCALL_64_after_hwframe+0x49/0xae [14868.792115] RIP: 0033:0x7fa749552332 [14868.797291] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14868.822708] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14868.829444] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14868.839508] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14868.855008] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14868.869490] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14868.879554] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14871.009036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12953s! [14871.012976] Showing busy workqueues and worker pools: [14871.015434] workqueue events: flags=0x0 [14871.017313] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14871.020051] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14871.026035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14871.028789] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14871.033405] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.036527] pending: kfree_rcu_monitor [14871.038207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14871.041699] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14871.045941] workqueue events_long: flags=0x0 [14871.048080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.051507] pending: lu_objects_flush [obdclass] [14871.054209] workqueue events_unbound: flags=0x2 [14871.056555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14871.059657] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14871.064039] workqueue events_power_efficient: flags=0x80 [14871.066744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.070597] pending: check_lifetime [14871.073658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.077006] pending: neigh_periodic_work [14871.078892] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.082055] pending: neigh_periodic_work [14871.084656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.088141] pending: do_cache_clean [14871.090617] workqueue mm_percpu_wq: flags=0x8 [14871.092663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.095721] pending: vmstat_update [14871.097903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.100775] pending: vmstat_update [14871.102665] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14871.106001] pending: vmstat_update [14871.107899] workqueue writeback: flags=0x4a [14871.109700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14871.112926] pending: wb_workfn, wb_workfn [14871.114958] workqueue xprtiod: flags=0xa [14871.117520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14871.120229] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14871.123812] workqueue ipv6_addrconf: flags=0x40008 [14871.126174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14871.129412] pending: addrconf_verify_work [14871.131402] workqueue knbd0-recv: flags=0x1a [14871.133467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14871.136098] in-flight: 651:recv_work [14871.138076] workqueue ptlrpc_pinger: flags=0x2 [14871.140335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14871.143399] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14871.146203] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12953s workers=4 idle: 37 31846 28897 [14871.150016] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14861s workers=2 idle: 653 [14892.058012] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [14892.059865] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [14892.062807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14892.062899] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14892.062907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14892.062910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14892.062925] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14892.062930] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14892.062934] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14892.062937] RDX: 0000000000001bb0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14892.062939] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14892.062942] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14892.062944] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14892.062948] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14892.062951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14892.062953] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14892.062961] Call Trace: [14892.062967] [14892.062972] ? show_regs.cold.9+0x22/0x2f [14892.062981] ? watchdog_timer_fn.cold.6+0x62/0x10a [14892.062987] ? __hrtimer_run_queues+0x1c2/0x4d0 [14892.062991] ? watchdog+0x40/0x40 [14892.062995] ? hrtimer_interrupt+0x128/0x2c0 [14892.062998] ? smp_apic_timer_interrupt+0xa6/0x270 [14892.063006] ? apic_timer_interrupt+0xf/0x20 [14892.063011] [14892.063014] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14892.063019] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14892.063024] do_raw_spin_lock+0x177/0x190 [14892.063028] _raw_spin_lock+0x21/0x30 [14892.063033] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14892.064996] obd_mod_free+0x46/0x140 [ptlrpc] [14892.065631] mdc_commit_open+0x91/0xb0 [mdc] [14892.065666] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14892.065874] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14892.066070] after_reply+0xe2c/0x1780 [ptlrpc] [14892.072787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14892.072875] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14892.072883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14892.072886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.072899] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14892.072904] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14892.072909] RAX: 0000000000002b91 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14892.072911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14892.072913] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14892.072916] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14892.072918] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14892.072921] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14892.072923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14892.072926] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14892.072934] Call Trace: [14892.072940] [14892.072947] ? show_regs.cold.9+0x22/0x2f [14892.072957] ? watchdog_timer_fn.cold.6+0x62/0x10a [14892.072963] ? __hrtimer_run_queues+0x1c2/0x4d0 [14892.072967] ? watchdog+0x40/0x40 [14892.072971] ? hrtimer_interrupt+0x128/0x2c0 [14892.072976] ? smp_apic_timer_interrupt+0xa6/0x270 [14892.072984] ? apic_timer_interrupt+0xf/0x20 [14892.072989] [14892.072992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.072999] do_raw_spin_lock+0x177/0x190 [14892.073004] _raw_spin_lock+0x21/0x30 [14892.073010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14892.073687] ? do_raw_spin_unlock+0x75/0x190 [14892.073694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14892.074004] ? do_wait_intr+0xf0/0xf0 [14892.074012] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14892.074235] mdc_close+0x53b/0xd70 [mdc] [14892.074277] lmv_close+0x18d/0x490 [lmv] [14892.074308] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14892.074479] ll_md_real_close+0xa5/0x2a0 [lustre] [14892.074668] ll_md_close+0x274/0x800 [lustre] [14892.074760] ll_file_release+0x128/0xc50 [lustre] [14892.124325] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14892.129335] __fput+0x10d/0x440 [14892.129357] ____fput+0x12/0x20 [14892.129362] task_work_run+0xc6/0x110 [14892.129370] exit_to_usermode_loop+0x1dd/0x1f0 [14892.129378] do_syscall_64+0x3d6/0x440 [14892.129383] entry_SYSCALL_64_after_hwframe+0x49/0xae [14892.129390] RIP: 0033:0x7fa749552cd8 [14892.129398] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14892.129402] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14892.129408] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14892.129411] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14892.129413] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14892.129415] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14892.129417] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14892.578389] ? __next_timer_interrupt+0x160/0x160 [14892.581703] ptlrpcd+0xa82/0xe40 [ptlrpc] [14892.583565] ? do_wait_intr+0xf0/0xf0 [14892.584909] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14892.598341] kthread+0x1d1/0x200 [14892.603240] ? set_kthread_struct+0x70/0x70 [14892.606208] ret_from_fork+0x1f/0x30 [14896.054014] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [14896.057059] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14896.058162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14896.058233] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14896.058239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14896.058242] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14896.058256] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14896.058263] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14896.058267] RAX: 0000000000005c94 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14896.058270] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14896.058272] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14896.058274] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14896.058277] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14896.058280] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14896.058282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14896.058285] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14896.058291] Call Trace: [14896.058295] [14896.058301] ? show_regs.cold.9+0x22/0x2f [14896.058313] ? watchdog_timer_fn.cold.6+0x62/0x10a [14896.058319] ? __hrtimer_run_queues+0x1c2/0x4d0 [14896.058322] ? watchdog+0x40/0x40 [14896.058327] ? hrtimer_interrupt+0x128/0x2c0 [14896.058330] ? smp_apic_timer_interrupt+0xa6/0x270 [14896.058341] ? apic_timer_interrupt+0xf/0x20 [14896.058346] [14896.058349] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14896.058355] do_raw_spin_lock+0x177/0x190 [14896.058360] _raw_spin_lock+0x21/0x30 [14896.058364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14896.062842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [14896.080228] ? do_raw_spin_unlock+0x75/0x190 [14896.086775] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14896.086818] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14896.086828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14896.086833] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14896.089716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14896.093784] [14896.093793] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14896.093815] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14896.093820] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14896.093825] RAX: 0000000000007dfe RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14896.093827] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14896.093830] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14896.101583] ? do_wait_intr+0xf0/0xf0 [14896.105142] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14896.107632] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14896.110809] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14896.110820] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14896.110823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14896.110826] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14896.110834] Call Trace: [14896.110952] [14896.113713] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14896.118126] ? show_regs.cold.9+0x22/0x2f [14896.121218] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14896.125077] ? watchdog_timer_fn.cold.6+0x62/0x10a [14896.125095] ? __hrtimer_run_queues+0x1c2/0x4d0 [14896.125100] ? watchdog+0x40/0x40 [14896.125106] ? hrtimer_interrupt+0x128/0x2c0 [14896.125110] ? smp_apic_timer_interrupt+0xa6/0x270 [14896.125119] ? apic_timer_interrupt+0xf/0x20 [14896.125123] [14896.125128] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14896.125134] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14896.125142] do_raw_spin_lock+0x177/0x190 [14896.125227] _raw_spin_lock+0x21/0x30 [14896.125233] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14896.128351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14896.128699] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14896.128804] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14896.128842] mdc_intent_lock+0x353/0x7e0 [mdc] [14896.128937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14896.129050] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14896.129506] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14896.129705] lmv_intent_open+0x338/0x1150 [lmv] [14896.129735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14896.135578] process_one_work+0x2c8/0x700 [14896.135598] worker_thread+0x296/0x6e0 [14896.135603] ? rescuer_thread+0x570/0x570 [14896.135721] kthread+0x1d1/0x200 [14896.135729] ? set_kthread_struct+0x70/0x70 [14896.135733] ret_from_fork+0x1f/0x30 [14896.389928] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14896.392021] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14896.393907] ll_intent_lock+0xc5/0xaa0 [lustre] [14896.395936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14896.397863] ll_lookup_it+0xe0e/0x2270 [lustre] [14896.399972] ? do_raw_spin_unlock+0x75/0x190 [14896.401998] ? _raw_spin_unlock+0x12/0x30 [14896.404279] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14896.406258] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14896.408231] ll_atomic_open+0x36a/0xd70 [lustre] [14896.410700] lookup_open+0x356/0xb40 [14896.412194] do_last+0x313/0xfc0 [14896.413541] ? nd_jump_root+0xe5/0x160 [14896.415146] ? path_init+0x437/0x520 [14896.416736] path_openat+0xf7/0x500 [14896.418193] ? do_raw_spin_unlock+0x75/0x190 [14896.420075] do_filp_open+0x99/0x140 [14896.421539] ? getname_flags+0x6e/0x330 [14896.423155] ? __check_object_size+0xff/0x256 [14896.424950] ? do_raw_spin_unlock+0x75/0x190 [14896.426343] ? _raw_spin_unlock+0x12/0x30 [14896.428289] do_sys_openat2+0x2b4/0x410 [14896.429674] do_sys_open+0x73/0xa0 [14896.431319] __x64_sys_openat+0x24/0x30 [14896.432755] do_syscall_64+0xc1/0x440 [14896.434146] entry_SYSCALL_64_after_hwframe+0x49/0xae [14896.436078] RIP: 0033:0x7fa749552332 [14896.437515] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14896.445729] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14896.448585] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14896.451301] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14896.454058] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14896.456976] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14896.459672] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14901.221312] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12984s! [14901.235473] Showing busy workqueues and worker pools: [14901.238239] workqueue events: flags=0x0 [14901.241432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14901.244531] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14901.250870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14901.254476] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14901.268367] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.280405] pending: kfree_rcu_monitor [14901.288737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14901.295290] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14901.305096] workqueue events_long: flags=0x0 [14901.311175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.320067] pending: lu_objects_flush [obdclass] [14901.327556] workqueue events_unbound: flags=0x2 [14901.331888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14901.338123] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14901.345401] workqueue events_power_efficient: flags=0x80 [14901.350336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.356429] pending: check_lifetime [14901.358843] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.367172] pending: neigh_periodic_work [14901.372303] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.376296] pending: neigh_periodic_work [14901.380920] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.392717] pending: do_cache_clean [14901.394564] workqueue mm_percpu_wq: flags=0x8 [14901.397538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.404319] pending: vmstat_update [14901.409637] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.414452] pending: vmstat_update [14901.419101] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14901.428233] pending: vmstat_update [14901.429639] workqueue writeback: flags=0x4a [14901.435640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14901.440455] pending: wb_workfn, wb_workfn [14901.443441] workqueue xprtiod: flags=0xa [14901.451724] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14901.457872] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14901.466425] workqueue ipv6_addrconf: flags=0x40008 [14901.470122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14901.473108] pending: addrconf_verify_work [14901.481083] workqueue knbd0-recv: flags=0x1a [14901.484287] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14901.489885] in-flight: 651:recv_work [14901.495583] workqueue ptlrpc_pinger: flags=0x2 [14901.500901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14901.506355] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14901.521083] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12984s workers=4 idle: 37 31846 28897 [14901.532800] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14891s workers=2 idle: 653 [14920.058113] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14920.060080] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14920.061978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14920.062074] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14920.066622] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14920.066828] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14920.066835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14920.066837] RIP: 0010:native_safe_halt+0x2e/0x50 [14920.066850] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14920.066858] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14920.066862] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14920.066864] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14920.066866] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14920.066868] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14920.066871] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14920.066874] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14920.066877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14920.066879] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14920.066886] Call Trace: [14920.066891] [14920.066896] ? show_regs.cold.9+0x22/0x2f [14920.066905] ? watchdog_timer_fn.cold.6+0x62/0x10a [14920.066910] ? __hrtimer_run_queues+0x1c2/0x4d0 [14920.066914] ? watchdog+0x40/0x40 [14920.066919] ? hrtimer_interrupt+0x128/0x2c0 [14920.066923] ? smp_apic_timer_interrupt+0xa6/0x270 [14920.066929] ? apic_timer_interrupt+0xf/0x20 [14920.066934] [14920.066937] ? native_safe_halt+0x2e/0x50 [14920.066940] kvm_wait+0xcd/0xe0 [14920.066947] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14920.066955] do_raw_spin_lock+0x177/0x190 [14920.066960] _raw_spin_lock+0x21/0x30 [14920.066964] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14920.067314] ? do_raw_spin_unlock+0x75/0x190 [14920.067320] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14920.067693] ? do_wait_intr+0xf0/0xf0 [14920.067701] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14920.068124] mdc_close+0x53b/0xd70 [mdc] [14920.068164] lmv_close+0x18d/0x490 [lmv] [14920.068194] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14920.068340] ll_md_real_close+0xa5/0x2a0 [lustre] [14920.068436] ll_md_close+0x274/0x800 [lustre] [14920.068519] ll_file_release+0x128/0xc50 [lustre] [14920.068626] __fput+0x10d/0x440 [14920.068636] ____fput+0x12/0x20 [14920.068640] task_work_run+0xc6/0x110 [14920.068649] exit_to_usermode_loop+0x1dd/0x1f0 [14920.068657] do_syscall_64+0x3d6/0x440 [14920.068796] entry_SYSCALL_64_after_hwframe+0x49/0xae [14920.068804] RIP: 0033:0x7fa749552cd8 [14920.068811] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14920.068816] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14920.068821] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14920.068824] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14920.068826] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14920.068829] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14920.068831] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14920.302302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14920.304787] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14920.306850] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14920.311758] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14920.314094] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14920.316152] RDX: 0000000000002fac RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14920.318891] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14920.321834] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14920.323807] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14920.325576] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14920.327975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14920.329784] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14920.332390] Call Trace: [14920.333689] [14920.334964] ? show_regs.cold.9+0x22/0x2f [14920.337082] ? watchdog_timer_fn.cold.6+0x62/0x10a [14920.339440] ? __hrtimer_run_queues+0x1c2/0x4d0 [14920.341637] ? watchdog+0x40/0x40 [14920.343019] ? hrtimer_interrupt+0x128/0x2c0 [14920.345531] ? smp_apic_timer_interrupt+0xa6/0x270 [14920.347803] ? apic_timer_interrupt+0xf/0x20 [14920.353325] [14920.356713] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14920.360009] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14920.364017] do_raw_spin_lock+0x177/0x190 [14920.365854] _raw_spin_lock+0x21/0x30 [14920.367387] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14920.369863] obd_mod_free+0x46/0x140 [ptlrpc] [14920.372145] mdc_commit_open+0x91/0xb0 [mdc] [14920.374452] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14920.376972] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14920.379836] after_reply+0xe2c/0x1780 [ptlrpc] [14920.382316] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14920.384881] ? __next_timer_interrupt+0x160/0x160 [14920.387483] ptlrpcd+0xa82/0xe40 [ptlrpc] [14920.389959] ? do_wait_intr+0xf0/0xf0 [14920.391742] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14920.394241] kthread+0x1d1/0x200 [14920.395903] ? set_kthread_struct+0x70/0x70 [14920.397178] ret_from_fork+0x1f/0x30 [14924.058608] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14924.064047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14924.104750] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14924.113150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14924.118248] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14924.121042] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14924.124356] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14924.133015] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14924.139130] RAX: 0000000000003285 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14924.143878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14924.150075] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14924.158671] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14924.168450] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14924.175998] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14924.184287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14924.189414] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14924.194564] Call Trace: [14924.201532] [14924.204079] ? show_regs.cold.9+0x22/0x2f [14924.211554] ? watchdog_timer_fn.cold.6+0x62/0x10a [14924.214243] ? __hrtimer_run_queues+0x1c2/0x4d0 [14924.221649] ? watchdog+0x40/0x40 [14924.224404] ? hrtimer_interrupt+0x128/0x2c0 [14924.227573] ? smp_apic_timer_interrupt+0xa6/0x270 [14924.233820] ? apic_timer_interrupt+0xf/0x20 [14924.239532] [14924.242570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14924.249473] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14924.253447] do_raw_spin_lock+0x177/0x190 [14924.258532] _raw_spin_lock+0x21/0x30 [14924.261245] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14924.267225] process_one_work+0x2c8/0x700 [14924.270059] worker_thread+0x296/0x6e0 [14924.272319] ? rescuer_thread+0x570/0x570 [14924.278510] kthread+0x1d1/0x200 [14924.282633] ? set_kthread_struct+0x70/0x70 [14924.286787] ret_from_fork+0x1f/0x30 [14928.054658] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14928.058584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14928.097246] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14928.108438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14928.114354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14928.117058] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14928.139306] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14928.144725] RAX: 0000000000003811 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14928.165011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14928.171527] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14928.183995] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14928.189837] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14928.194099] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14928.197329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14928.199977] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14928.203775] Call Trace: [14928.204963] [14928.205798] ? show_regs.cold.9+0x22/0x2f [14928.207618] ? watchdog_timer_fn.cold.6+0x62/0x10a [14928.209736] ? __hrtimer_run_queues+0x1c2/0x4d0 [14928.211823] ? watchdog+0x40/0x40 [14928.213340] ? hrtimer_interrupt+0x128/0x2c0 [14928.215445] ? smp_apic_timer_interrupt+0xa6/0x270 [14928.217410] ? apic_timer_interrupt+0xf/0x20 [14928.219509] [14928.223514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14928.232757] do_raw_spin_lock+0x177/0x190 [14928.237024] _raw_spin_lock+0x21/0x30 [14928.239332] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14928.243736] ? do_raw_spin_unlock+0x75/0x190 [14928.250284] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14928.258240] ? do_wait_intr+0xf0/0xf0 [14928.259632] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14928.267097] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14928.271979] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14928.276516] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14928.279537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14928.284750] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14928.287939] mdc_intent_lock+0x353/0x7e0 [mdc] [14928.292743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14928.297372] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14928.304024] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14928.307489] lmv_intent_open+0x338/0x1150 [lmv] [14928.310992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14928.314459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14928.320419] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14928.323343] ll_intent_lock+0xc5/0xaa0 [lustre] [14928.325305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14928.330004] ll_lookup_it+0xe0e/0x2270 [lustre] [14928.335335] ? do_raw_spin_unlock+0x75/0x190 [14928.338000] ? _raw_spin_unlock+0x12/0x30 [14928.340791] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14928.342499] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14928.346763] ll_atomic_open+0x36a/0xd70 [lustre] [14928.355248] lookup_open+0x356/0xb40 [14928.357479] do_last+0x313/0xfc0 [14928.360787] ? nd_jump_root+0xe5/0x160 [14928.363439] ? path_init+0x437/0x520 [14928.370388] path_openat+0xf7/0x500 [14928.372387] ? do_raw_spin_unlock+0x75/0x190 [14928.375508] do_filp_open+0x99/0x140 [14928.378508] ? getname_flags+0x6e/0x330 [14928.383970] ? __check_object_size+0xff/0x256 [14928.385239] ? do_raw_spin_unlock+0x75/0x190 [14928.392021] ? _raw_spin_unlock+0x12/0x30 [14928.396584] do_sys_openat2+0x2b4/0x410 [14928.399365] do_sys_open+0x73/0xa0 [14928.401322] __x64_sys_openat+0x24/0x30 [14928.406135] do_syscall_64+0xc1/0x440 [14928.408562] entry_SYSCALL_64_after_hwframe+0x49/0xae [14928.412299] RIP: 0033:0x7fa749552332 [14928.413579] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14928.428506] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14928.443866] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14928.449009] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14928.453372] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14928.465129] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14928.470361] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14931.938049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13014s! [14931.944819] Showing busy workqueues and worker pools: [14931.946985] workqueue events: flags=0x0 [14931.948793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14931.952720] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14931.958235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14931.961582] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14931.966813] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14931.969676] pending: kfree_rcu_monitor [14931.971686] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14931.975359] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14931.979844] workqueue events_long: flags=0x0 [14931.981592] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14931.984605] pending: lu_objects_flush [obdclass] [14931.986820] workqueue events_unbound: flags=0x2 [14931.989059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14931.991950] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14931.996631] workqueue events_power_efficient: flags=0x80 [14931.999698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14932.003719] pending: check_lifetime [14932.008244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14932.012365] pending: neigh_periodic_work [14932.016551] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14932.020969] pending: neigh_periodic_work [14932.023486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14932.027485] pending: do_cache_clean [14932.030356] workqueue mm_percpu_wq: flags=0x8 [14932.033345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14932.036564] pending: vmstat_update [14932.038571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14932.041975] pending: vmstat_update [14932.043957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14932.047804] pending: vmstat_update [14932.050484] workqueue writeback: flags=0x4a [14932.052893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14932.056381] pending: wb_workfn, wb_workfn [14932.059114] workqueue xprtiod: flags=0xa [14932.061348] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14932.065101] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14932.069714] workqueue ipv6_addrconf: flags=0x40008 [14932.072050] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14932.075639] pending: addrconf_verify_work [14932.077889] workqueue knbd0-recv: flags=0x1a [14932.080699] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14932.084590] in-flight: 651:recv_work [14932.086484] workqueue ptlrpc_pinger: flags=0x2 [14932.088340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14932.091589] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14932.094739] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13014s workers=4 idle: 37 31846 28897 [14932.099370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14921s workers=2 idle: 653 [14933.981819] rcu: INFO: rcu_sched self-detected stall on CPU [14933.988581] rcu: 2-...!: (7492412 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [14933.994304] (t=13020292 jiffies g=258701 q=5087) [14933.996527] rcu: rcu_sched kthread starved for 13017087 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [14933.999494] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14934.003661] rcu: RCU grace-period kthread stack dump: [14934.005248] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14934.008002] Call Trace: [14934.008757] __schedule+0x351/0xcb0 [14934.009820] schedule+0xc0/0x180 [14934.010800] schedule_timeout+0xb4/0x190 [14934.011879] ? do_raw_spin_unlock+0x75/0x190 [14934.012970] ? __next_timer_interrupt+0x160/0x160 [14934.014506] ? prepare_to_swait_event+0xf3/0x140 [14934.015796] rcu_gp_kthread+0x8dd/0x15b0 [14934.016764] ? force_qs_rnp+0x2c0/0x2c0 [14934.017849] kthread+0x1d1/0x200 [14934.018991] ? set_kthread_struct+0x70/0x70 [14934.020651] ret_from_fork+0x1f/0x30 [14934.021782] rcu: Stack dump where RCU GP kthread last ran: [14934.023967] Sending NMI from CPU 2 to CPUs 0: [14934.034806] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 9.088 msecs [14934.035050] NMI backtrace for cpu 0 [14934.035054] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14934.035056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14934.035058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14934.035060] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14934.035063] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 [14934.035065] RAX: 0000000000001f17 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14934.035067] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14934.035068] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14934.035070] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14934.035071] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14934.035073] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14934.035075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14934.035076] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14934.035078] Call Trace: [14934.035079] [14934.035080] ? show_regs.cold.9+0x22/0x2f [14934.035081] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14934.035111] ? nmi_cpu_backtrace_handler+0x15/0x40 [14934.035112] ? nmi_handle+0xbd/0x210 [14934.035114] ? default_do_nmi+0x4b/0x1b0 [14934.035115] ? do_nmi+0x280/0x380 [14934.035116] ? end_repeat_nmi+0x16/0x1f [14934.035117] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14934.035118] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14934.035119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14934.035120] [14934.035121] do_raw_spin_lock+0x177/0x190 [14934.035122] _raw_spin_lock+0x21/0x30 [14934.035123] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14934.035124] ? do_raw_spin_unlock+0x75/0x190 [14934.035126] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14934.035127] ? do_wait_intr+0xf0/0xf0 [14934.035128] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14934.035129] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14934.035130] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14934.035131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.035133] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14934.035134] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14934.035135] mdc_intent_lock+0x353/0x7e0 [mdc] [14934.035136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.035137] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14934.035138] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14934.035140] lmv_intent_open+0x338/0x1150 [lmv] [14934.035141] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.035142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.035143] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14934.035144] ll_intent_lock+0xc5/0xaa0 [lustre] [14934.035146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.035147] ll_lookup_it+0xe0e/0x2270 [lustre] [14934.035148] ? do_raw_spin_unlock+0x75/0x190 [14934.035149] ? _raw_spin_unlock+0x12/0x30 [14934.035150] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14934.035151] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14934.035152] ll_atomic_open+0x36a/0xd70 [lustre] [14934.035153] lookup_open+0x356/0xb40 [14934.035154] do_last+0x313/0xfc0 [14934.035155] ? nd_jump_root+0xe5/0x160 [14934.035156] ? path_init+0x437/0x520 [14934.035157] path_openat+0xf7/0x500 [14934.035158] ? do_raw_spin_unlock+0x75/0x190 [14934.035159] do_filp_open+0x99/0x140 [14934.035160] ? getname_flags+0x6e/0x330 [14934.035161] ? __check_object_size+0xff/0x256 [14934.035162] ? do_raw_spin_unlock+0x75/0x190 [14934.035163] ? _raw_spin_unlock+0x12/0x30 [14934.035164] do_sys_openat2+0x2b4/0x410 [14934.035165] do_sys_open+0x73/0xa0 [14934.035166] __x64_sys_openat+0x24/0x30 [14934.035167] do_syscall_64+0xc1/0x440 [14934.035168] entry_SYSCALL_64_after_hwframe+0x49/0xae [14934.035169] RIP: 0033:0x7fa749552332 [14934.035171] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14934.035172] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14934.035175] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14934.035176] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14934.035178] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14934.035179] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14934.035180] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14934.035187] NMI backtrace for cpu 2 [14934.268609] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14934.272249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14934.275287] Call Trace: [14934.275999] [14934.276872] ? dump_stack+0xbb/0x10e [14934.278070] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14934.279604] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14934.280881] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14934.283005] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14934.284640] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14934.285850] ? rcu_dump_cpu_stacks+0x11b/0x171 [14934.286820] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14934.287966] ? update_process_times+0x6d/0xc0 [14934.289293] ? tick_sched_handle.isra.18+0x27/0x90 [14934.291140] ? tick_sched_timer+0x53/0xb0 [14934.292071] ? __hrtimer_run_queues+0x1c2/0x4d0 [14934.293086] ? tick_sched_handle.isra.18+0x90/0x90 [14934.294251] ? hrtimer_interrupt+0x128/0x2c0 [14934.295690] ? smp_apic_timer_interrupt+0xa6/0x270 [14934.297018] ? apic_timer_interrupt+0xf/0x20 [14934.304315] [14934.306281] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14934.309955] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14934.314386] ? do_raw_spin_lock+0x177/0x190 [14934.315516] ? _raw_spin_lock+0x21/0x30 [14934.318802] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14934.321474] ? obd_mod_free+0x46/0x140 [ptlrpc] [14934.325920] ? mdc_commit_open+0x91/0xb0 [mdc] [14934.329888] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14934.335927] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14934.344458] ? after_reply+0xe2c/0x1780 [ptlrpc] [14934.346880] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14934.349476] ? __next_timer_interrupt+0x160/0x160 [14934.351705] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14934.355696] ? do_wait_intr+0xf0/0xf0 [14934.357666] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14934.360763] ? kthread+0x1d1/0x200 [14934.362522] ? set_kthread_struct+0x70/0x70 [14934.365907] ? ret_from_fork+0x1f/0x30 [14948.059082] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14948.066044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14948.115686] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14948.123215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14948.130005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14948.132415] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14948.144391] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14948.149334] RAX: 0000000000004484 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14948.158268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14948.166482] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14948.174255] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14948.178804] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14948.183232] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14948.187676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14948.190427] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14948.193892] Call Trace: [14948.195162] [14948.196085] ? show_regs.cold.9+0x22/0x2f [14948.200559] ? watchdog_timer_fn.cold.6+0x62/0x10a [14948.206736] ? __hrtimer_run_queues+0x1c2/0x4d0 [14948.209167] ? watchdog+0x40/0x40 [14948.212278] ? hrtimer_interrupt+0x128/0x2c0 [14948.214473] ? smp_apic_timer_interrupt+0xa6/0x270 [14948.216803] ? apic_timer_interrupt+0xf/0x20 [14948.220018] [14948.224251] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14948.227891] do_raw_spin_lock+0x177/0x190 [14948.230407] _raw_spin_lock+0x21/0x30 [14948.233694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14948.237368] ? do_raw_spin_unlock+0x75/0x190 [14948.239583] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14948.242470] ? do_wait_intr+0xf0/0xf0 [14948.244627] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14948.247428] mdc_close+0x53b/0xd70 [mdc] [14948.249988] lmv_close+0x18d/0x490 [lmv] [14948.252065] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14948.255389] ll_md_real_close+0xa5/0x2a0 [lustre] [14948.257402] ll_md_close+0x274/0x800 [lustre] [14948.259190] ll_file_release+0x128/0xc50 [lustre] [14948.261494] __fput+0x10d/0x440 [14948.262788] ____fput+0x12/0x20 [14948.264106] task_work_run+0xc6/0x110 [14948.265609] exit_to_usermode_loop+0x1dd/0x1f0 [14948.267424] do_syscall_64+0x3d6/0x440 [14948.271914] entry_SYSCALL_64_after_hwframe+0x49/0xae [14948.276105] RIP: 0033:0x7fa749552cd8 [14948.280191] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14948.294697] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14948.298228] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14948.306132] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14948.310634] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14948.313734] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14948.316800] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14952.058126] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14952.071279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14952.114990] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14952.124826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14952.131709] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14952.141520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14952.148733] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14952.161746] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14952.168159] RAX: 0000000000003108 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14952.172156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14952.175356] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14952.179377] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14952.183033] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14952.186370] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14952.189497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14952.191874] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14952.194594] Call Trace: [14952.195742] [14952.196597] ? show_regs.cold.9+0x22/0x2f [14952.198393] ? watchdog_timer_fn.cold.6+0x62/0x10a [14952.207404] ? __hrtimer_run_queues+0x1c2/0x4d0 [14952.211607] ? watchdog+0x40/0x40 [14952.215803] ? hrtimer_interrupt+0x128/0x2c0 [14952.219929] ? smp_apic_timer_interrupt+0xa6/0x270 [14952.224671] ? apic_timer_interrupt+0xf/0x20 [14952.229413] [14952.230566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14952.233613] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14952.236497] do_raw_spin_lock+0x177/0x190 [14952.239235] _raw_spin_lock+0x21/0x30 [14952.243308] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14952.254100] process_one_work+0x2c8/0x700 [14952.257989] worker_thread+0x296/0x6e0 [14952.263407] ? rescuer_thread+0x570/0x570 [14952.267579] kthread+0x1d1/0x200 [14952.268733] ? set_kthread_struct+0x70/0x70 [14952.269973] ret_from_fork+0x1f/0x30 [14960.057264] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14960.059708] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14960.064978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14960.065057] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14960.065066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14960.065068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14960.065084] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14960.065088] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14960.065093] RAX: 000000000000543f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14960.065095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14960.065098] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14960.065101] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14960.065103] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14960.065107] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14960.065110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14960.065113] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14960.065122] Call Trace: [14960.065128] [14960.065134] ? show_regs.cold.9+0x22/0x2f [14960.065144] ? watchdog_timer_fn.cold.6+0x62/0x10a [14960.065150] ? __hrtimer_run_queues+0x1c2/0x4d0 [14960.065155] ? watchdog+0x40/0x40 [14960.065160] ? hrtimer_interrupt+0x128/0x2c0 [14960.065164] ? smp_apic_timer_interrupt+0xa6/0x270 [14960.065172] ? apic_timer_interrupt+0xf/0x20 [14960.065178] [14960.065181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14960.065188] do_raw_spin_lock+0x177/0x190 [14960.065193] _raw_spin_lock+0x21/0x30 [14960.065198] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14960.065487] ? do_raw_spin_unlock+0x75/0x190 [14960.065494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14960.065688] ? do_wait_intr+0xf0/0xf0 [14960.065697] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14960.065898] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14960.074183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14960.103498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14960.104192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.105004] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14960.105046] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14960.105076] mdc_intent_lock+0x353/0x7e0 [mdc] [14960.105102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.105209] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14960.105484] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14960.105515] lmv_intent_open+0x338/0x1150 [lmv] [14960.105548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.105795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.105879] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14960.105904] ll_intent_lock+0xc5/0xaa0 [lustre] [14960.106009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.106097] ll_lookup_it+0xe0e/0x2270 [lustre] [14960.106188] ? do_raw_spin_unlock+0x75/0x190 [14960.106196] ? _raw_spin_unlock+0x12/0x30 [14960.106203] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14960.106232] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14960.106252] ll_atomic_open+0x36a/0xd70 [lustre] [14960.106447] lookup_open+0x356/0xb40 [14960.106456] do_last+0x313/0xfc0 [14960.106461] ? nd_jump_root+0xe5/0x160 [14960.106467] ? path_init+0x437/0x520 [14960.106472] path_openat+0xf7/0x500 [14960.106477] ? do_raw_spin_unlock+0x75/0x190 [14960.106482] do_filp_open+0x99/0x140 [14960.106488] ? getname_flags+0x6e/0x330 [14960.106492] ? __check_object_size+0xff/0x256 [14960.106500] ? do_raw_spin_unlock+0x75/0x190 [14960.106504] ? _raw_spin_unlock+0x12/0x30 [14960.106509] do_sys_openat2+0x2b4/0x410 [14960.106514] do_sys_open+0x73/0xa0 [14960.106518] __x64_sys_openat+0x24/0x30 [14960.106522] do_syscall_64+0xc1/0x440 [14960.106529] entry_SYSCALL_64_after_hwframe+0x49/0xae [14960.106534] RIP: 0033:0x7fa749552332 [14960.106542] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14960.106546] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14960.106551] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14960.106554] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14960.106556] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14960.106558] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14960.106561] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14960.497270] [14960.498412] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14960.504333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14960.510880] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14960.513808] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14960.522423] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14960.526387] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14960.529457] RDX: 0000000000004340 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14960.532819] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14960.537704] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14960.540021] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14960.546258] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14960.550862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14960.553426] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14960.556810] Call Trace: [14960.558029] [14960.559351] ? show_regs.cold.9+0x22/0x2f [14960.561671] ? watchdog_timer_fn.cold.6+0x62/0x10a [14960.564044] ? __hrtimer_run_queues+0x1c2/0x4d0 [14960.566179] ? watchdog+0x40/0x40 [14960.567998] ? hrtimer_interrupt+0x128/0x2c0 [14960.570239] ? smp_apic_timer_interrupt+0xa6/0x270 [14960.572786] ? apic_timer_interrupt+0xf/0x20 [14960.575134] [14960.576124] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14960.578845] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14960.581485] do_raw_spin_lock+0x177/0x190 [14960.583420] _raw_spin_lock+0x21/0x30 [14960.584953] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14960.587481] obd_mod_free+0x46/0x140 [ptlrpc] [14960.589998] mdc_commit_open+0x91/0xb0 [mdc] [14960.591923] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14960.594898] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14960.597837] after_reply+0xe2c/0x1780 [ptlrpc] [14960.600378] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14960.603350] ? __next_timer_interrupt+0x160/0x160 [14960.605821] ptlrpcd+0xa82/0xe40 [ptlrpc] [14960.607840] ? do_wait_intr+0xf0/0xf0 [14960.609584] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14960.612536] kthread+0x1d1/0x200 [14960.614256] ? set_kthread_struct+0x70/0x70 [14960.616351] ret_from_fork+0x1f/0x30 [14962.147090] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13044s! [14962.150645] Showing busy workqueues and worker pools: [14962.155333] workqueue events: flags=0x0 [14962.159030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14962.165431] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14962.171410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14962.174806] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14962.178956] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.182470] pending: kfree_rcu_monitor [14962.184905] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14962.187896] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14962.194726] workqueue events_long: flags=0x0 [14962.196966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.201007] pending: lu_objects_flush [obdclass] [14962.203898] workqueue events_unbound: flags=0x2 [14962.209419] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14962.212737] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14962.217395] workqueue events_power_efficient: flags=0x80 [14962.220503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.227574] pending: check_lifetime [14962.231819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.235453] pending: neigh_periodic_work [14962.240441] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.245001] pending: neigh_periodic_work [14962.250641] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.257334] pending: do_cache_clean [14962.260498] workqueue mm_percpu_wq: flags=0x8 [14962.262921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.267176] pending: vmstat_update [14962.273744] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.280977] pending: vmstat_update [14962.282329] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14962.286399] pending: vmstat_update [14962.289841] workqueue writeback: flags=0x4a [14962.294014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14962.297814] pending: wb_workfn, wb_workfn [14962.300823] workqueue xprtiod: flags=0xa [14962.302331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14962.309265] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14962.314356] workqueue ipv6_addrconf: flags=0x40008 [14962.316713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14962.319512] pending: addrconf_verify_work [14962.321335] workqueue knbd0-recv: flags=0x1a [14962.323540] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14962.326831] in-flight: 651:recv_work [14962.332398] workqueue ptlrpc_pinger: flags=0x2 [14962.334991] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14962.339217] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14962.342058] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13045s workers=4 idle: 37 31846 28897 [14962.345802] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14952s workers=2 idle: 653 [14976.059055] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [14976.062462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14976.086670] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14976.091482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14976.095105] RIP: 0010:native_safe_halt+0x2e/0x50 [14976.097004] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14976.105323] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14976.111744] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14976.119688] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [14976.123333] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [14976.127747] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [14976.130930] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [14976.134399] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [14976.137902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14976.140564] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [14976.143169] Call Trace: [14976.144037] [14976.145532] ? show_regs.cold.9+0x22/0x2f [14976.148467] ? watchdog_timer_fn.cold.6+0x62/0x10a [14976.150576] ? __hrtimer_run_queues+0x1c2/0x4d0 [14976.152682] ? watchdog+0x40/0x40 [14976.154117] ? hrtimer_interrupt+0x128/0x2c0 [14976.156085] ? smp_apic_timer_interrupt+0xa6/0x270 [14976.157978] ? apic_timer_interrupt+0xf/0x20 [14976.159706] [14976.160499] ? native_safe_halt+0x2e/0x50 [14976.162128] kvm_wait+0xcd/0xe0 [14976.163531] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14976.166161] do_raw_spin_lock+0x177/0x190 [14976.168329] _raw_spin_lock+0x21/0x30 [14976.170905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14976.173542] ? do_raw_spin_unlock+0x75/0x190 [14976.175235] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14976.177758] ? do_wait_intr+0xf0/0xf0 [14976.179445] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14976.182391] mdc_close+0x53b/0xd70 [mdc] [14976.184701] lmv_close+0x18d/0x490 [lmv] [14976.186974] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14976.189507] ll_md_real_close+0xa5/0x2a0 [lustre] [14976.191540] ll_md_close+0x274/0x800 [lustre] [14976.193515] ll_file_release+0x128/0xc50 [lustre] [14976.195668] __fput+0x10d/0x440 [14976.196985] ____fput+0x12/0x20 [14976.199160] task_work_run+0xc6/0x110 [14976.205220] exit_to_usermode_loop+0x1dd/0x1f0 [14976.208484] do_syscall_64+0x3d6/0x440 [14976.213323] entry_SYSCALL_64_after_hwframe+0x49/0xae [14976.217899] RIP: 0033:0x7fa749552cd8 [14976.222837] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14976.232740] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14976.237554] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [14976.242305] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [14976.247057] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [14976.252773] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [14976.256649] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14980.057064] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [14980.061822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14980.082690] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14980.087502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14980.091253] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14980.094861] RIP: 0010:native_safe_halt+0x2e/0x50 [14980.097681] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14980.104882] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14980.107627] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14980.110085] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [14980.112602] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [14980.115657] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [14980.118246] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [14980.120704] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [14980.123503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14980.125453] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [14980.127732] Call Trace: [14980.128523] [14980.129090] ? show_regs.cold.9+0x22/0x2f [14980.130411] ? watchdog_timer_fn.cold.6+0x62/0x10a [14980.131826] ? __hrtimer_run_queues+0x1c2/0x4d0 [14980.133494] ? watchdog+0x40/0x40 [14980.134561] ? hrtimer_interrupt+0x128/0x2c0 [14980.136143] ? smp_apic_timer_interrupt+0xa6/0x270 [14980.143259] ? apic_timer_interrupt+0xf/0x20 [14980.145389] [14980.146032] ? native_safe_halt+0x2e/0x50 [14980.147649] kvm_wait+0xcd/0xe0 [14980.148894] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14980.150913] do_raw_spin_lock+0x177/0x190 [14980.152557] _raw_spin_lock+0x21/0x30 [14980.153983] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14980.156067] process_one_work+0x2c8/0x700 [14980.157739] worker_thread+0x296/0x6e0 [14980.159062] ? rescuer_thread+0x570/0x570 [14980.160654] kthread+0x1d1/0x200 [14980.161830] ? set_kthread_struct+0x70/0x70 [14980.163476] ret_from_fork+0x1f/0x30 [14988.054736] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [14988.058927] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [14988.064075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [14988.076008] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14988.076194] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14988.076200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14988.076203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14988.076216] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14988.076220] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14988.076224] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [14988.076227] RDX: 0000000000003ab6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [14988.076229] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [14988.076232] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [14988.076234] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14988.076237] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [14988.076240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14988.076243] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [14988.076250] Call Trace: [14988.076254] [14988.076260] ? show_regs.cold.9+0x22/0x2f [14988.076267] ? watchdog_timer_fn.cold.6+0x62/0x10a [14988.076273] ? __hrtimer_run_queues+0x1c2/0x4d0 [14988.076277] ? watchdog+0x40/0x40 [14988.076281] ? hrtimer_interrupt+0x128/0x2c0 [14988.076285] ? smp_apic_timer_interrupt+0xa6/0x270 [14988.076292] ? apic_timer_interrupt+0xf/0x20 [14988.076302] [14988.076305] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14988.076310] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14988.076314] do_raw_spin_lock+0x177/0x190 [14988.076319] _raw_spin_lock+0x21/0x30 [14988.076324] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14988.076846] obd_mod_free+0x46/0x140 [ptlrpc] [14988.096405] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [14988.127891] mdc_commit_open+0x91/0xb0 [mdc] [14988.127937] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14988.137636] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14988.137681] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14988.137688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14988.137691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14988.137703] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14988.137708] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14988.137712] RAX: 0000000000007eb5 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [14988.137714] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [14988.137716] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [14988.137718] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [14988.137721] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [14988.137724] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [14988.137728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14988.137731] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [14988.137739] Call Trace: [14988.137745] [14988.137750] ? show_regs.cold.9+0x22/0x2f [14988.137760] ? watchdog_timer_fn.cold.6+0x62/0x10a [14988.137768] ? __hrtimer_run_queues+0x1c2/0x4d0 [14988.137772] ? watchdog+0x40/0x40 [14988.137777] ? hrtimer_interrupt+0x128/0x2c0 [14988.137781] ? smp_apic_timer_interrupt+0xa6/0x270 [14988.137790] ? apic_timer_interrupt+0xf/0x20 [14988.137795] [14988.137797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14988.137804] do_raw_spin_lock+0x177/0x190 [14988.137808] _raw_spin_lock+0x21/0x30 [14988.137813] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14988.146727] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14988.146987] after_reply+0xe2c/0x1780 [ptlrpc] [14988.151778] ? do_raw_spin_unlock+0x75/0x190 [14988.151797] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14988.169242] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14988.169535] ? __next_timer_interrupt+0x160/0x160 [14988.169550] ptlrpcd+0xa82/0xe40 [ptlrpc] [14988.169758] ? do_wait_intr+0xf0/0xf0 [14988.169767] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14988.170288] kthread+0x1d1/0x200 [14988.170298] ? set_kthread_struct+0x70/0x70 [14988.170302] ret_from_fork+0x1f/0x30 [14988.468424] ? do_wait_intr+0xf0/0xf0 [14988.473904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14988.476919] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14988.480682] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14988.484663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.487923] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14988.491632] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14988.493217] mdc_intent_lock+0x353/0x7e0 [mdc] [14988.499554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.510794] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14988.515310] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14988.518659] lmv_intent_open+0x338/0x1150 [lmv] [14988.522478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.528821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.535939] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14988.542428] ll_intent_lock+0xc5/0xaa0 [lustre] [14988.549477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.552429] ll_lookup_it+0xe0e/0x2270 [lustre] [14988.559820] ? do_raw_spin_unlock+0x75/0x190 [14988.562703] ? _raw_spin_unlock+0x12/0x30 [14988.571792] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14988.582017] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14988.587498] ll_atomic_open+0x36a/0xd70 [lustre] [14988.589536] lookup_open+0x356/0xb40 [14988.590899] do_last+0x313/0xfc0 [14988.592061] ? nd_jump_root+0xe5/0x160 [14988.594196] ? path_init+0x437/0x520 [14988.595653] path_openat+0xf7/0x500 [14988.597094] ? do_raw_spin_unlock+0x75/0x190 [14988.598823] do_filp_open+0x99/0x140 [14988.600340] ? getname_flags+0x6e/0x330 [14988.601924] ? __check_object_size+0xff/0x256 [14988.603748] ? do_raw_spin_unlock+0x75/0x190 [14988.605506] ? _raw_spin_unlock+0x12/0x30 [14988.607216] do_sys_openat2+0x2b4/0x410 [14988.608992] do_sys_open+0x73/0xa0 [14988.610446] __x64_sys_openat+0x24/0x30 [14988.611939] do_syscall_64+0xc1/0x440 [14988.613406] entry_SYSCALL_64_after_hwframe+0x49/0xae [14988.615055] RIP: 0033:0x7fa749552332 [14988.616604] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14988.624591] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14988.627587] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [14988.636693] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [14988.645315] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [14988.654403] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [14988.661016] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [14992.356459] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13075s! [14992.358951] Showing busy workqueues and worker pools: [14992.360126] workqueue events: flags=0x0 [14992.362138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14992.366537] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [14992.374544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14992.377840] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14992.388601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.392527] pending: kfree_rcu_monitor [14992.394040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14992.397052] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [14992.400910] workqueue events_long: flags=0x0 [14992.402618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.405151] pending: lu_objects_flush [obdclass] [14992.409432] workqueue events_unbound: flags=0x2 [14992.411156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [14992.413472] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14992.418582] workqueue events_power_efficient: flags=0x80 [14992.422842] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.426491] pending: check_lifetime [14992.434932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.440880] pending: neigh_periodic_work [14992.447051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.450318] pending: neigh_periodic_work [14992.452667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.456533] pending: do_cache_clean [14992.457931] workqueue mm_percpu_wq: flags=0x8 [14992.459585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.461998] pending: vmstat_update [14992.464300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.468524] pending: vmstat_update [14992.470270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14992.473143] pending: vmstat_update [14992.475163] workqueue writeback: flags=0x4a [14992.476981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14992.480156] pending: wb_workfn, wb_workfn [14992.483232] workqueue xprtiod: flags=0xa [14992.485309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14992.488489] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [14992.491961] workqueue ipv6_addrconf: flags=0x40008 [14992.493844] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14992.496369] pending: addrconf_verify_work [14992.498176] workqueue knbd0-recv: flags=0x1a [14992.499830] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14992.504004] in-flight: 651:recv_work [14992.505624] workqueue ptlrpc_pinger: flags=0x2 [14992.507386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14992.509686] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [14992.511919] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13075s workers=4 idle: 37 31846 28897 [14992.516837] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14982s workers=2 idle: 653 [15004.059617] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15004.063843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15004.094087] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15004.108480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15004.120747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15004.124235] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15004.129878] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15004.141793] RAX: 000000000000131a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15004.146500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15004.150567] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15004.154171] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15004.157702] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15004.161395] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15004.166347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15004.168338] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15004.171690] Call Trace: [15004.172963] [15004.176268] ? show_regs.cold.9+0x22/0x2f [15004.177810] ? watchdog_timer_fn.cold.6+0x62/0x10a [15004.180410] ? __hrtimer_run_queues+0x1c2/0x4d0 [15004.183074] ? watchdog+0x40/0x40 [15004.185083] ? hrtimer_interrupt+0x128/0x2c0 [15004.187187] ? smp_apic_timer_interrupt+0xa6/0x270 [15004.189718] ? apic_timer_interrupt+0xf/0x20 [15004.192337] [15004.193502] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15004.196230] do_raw_spin_lock+0x177/0x190 [15004.198157] _raw_spin_lock+0x21/0x30 [15004.201048] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15004.204072] ? do_raw_spin_unlock+0x75/0x190 [15004.206650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15004.209457] ? do_wait_intr+0xf0/0xf0 [15004.211003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15004.213918] mdc_close+0x53b/0xd70 [mdc] [15004.215644] lmv_close+0x18d/0x490 [lmv] [15004.217356] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15004.220366] ll_md_real_close+0xa5/0x2a0 [lustre] [15004.222763] ll_md_close+0x274/0x800 [lustre] [15004.225544] ll_file_release+0x128/0xc50 [lustre] [15004.228382] __fput+0x10d/0x440 [15004.230198] ____fput+0x12/0x20 [15004.232196] task_work_run+0xc6/0x110 [15004.234031] exit_to_usermode_loop+0x1dd/0x1f0 [15004.236386] do_syscall_64+0x3d6/0x440 [15004.238085] entry_SYSCALL_64_after_hwframe+0x49/0xae [15004.240879] RIP: 0033:0x7fa749552cd8 [15004.242712] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15004.251562] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15004.255328] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15004.259281] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15004.262722] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15004.266627] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15004.269328] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15008.057036] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15008.062124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15008.101070] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15008.106476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15008.110852] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15008.114884] RIP: 0010:native_safe_halt+0x2e/0x50 [15008.119655] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15008.131138] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15008.138139] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15008.140567] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15008.143361] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15008.146064] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15008.148737] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15008.152062] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15008.155402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15008.157675] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15008.164976] Call Trace: [15008.167552] [15008.168755] ? show_regs.cold.9+0x22/0x2f [15008.171553] ? watchdog_timer_fn.cold.6+0x62/0x10a [15008.173655] ? __hrtimer_run_queues+0x1c2/0x4d0 [15008.176352] ? watchdog+0x40/0x40 [15008.178041] ? hrtimer_interrupt+0x128/0x2c0 [15008.179777] ? smp_apic_timer_interrupt+0xa6/0x270 [15008.181964] ? apic_timer_interrupt+0xf/0x20 [15008.184289] [15008.185493] ? native_safe_halt+0x2e/0x50 [15008.187528] kvm_wait+0xcd/0xe0 [15008.189015] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15008.191367] do_raw_spin_lock+0x177/0x190 [15008.194181] _raw_spin_lock+0x21/0x30 [15008.196963] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15008.199026] process_one_work+0x2c8/0x700 [15008.203877] worker_thread+0x296/0x6e0 [15008.218368] ? rescuer_thread+0x570/0x570 [15008.222257] kthread+0x1d1/0x200 [15008.223740] ? set_kthread_struct+0x70/0x70 [15008.229738] ret_from_fork+0x1f/0x30 [15016.058190] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15016.061369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15016.093336] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15016.099113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15016.103538] RIP: 0010:native_safe_halt+0x2e/0x50 [15016.105856] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15016.116740] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15016.120630] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15016.123180] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15016.128074] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15016.133390] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15016.137783] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15016.140537] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15016.145015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15016.148143] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15016.152270] Call Trace: [15016.153282] [15016.154794] ? show_regs.cold.9+0x22/0x2f [15016.158423] ? watchdog_timer_fn.cold.6+0x62/0x10a [15016.161875] ? __hrtimer_run_queues+0x1c2/0x4d0 [15016.166846] ? watchdog+0x40/0x40 [15016.168840] ? hrtimer_interrupt+0x128/0x2c0 [15016.171698] ? smp_apic_timer_interrupt+0xa6/0x270 [15016.174444] ? apic_timer_interrupt+0xf/0x20 [15016.178067] [15016.180179] ? native_safe_halt+0x2e/0x50 [15016.182462] kvm_wait+0xcd/0xe0 [15016.184136] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15016.187057] do_raw_spin_lock+0x177/0x190 [15016.189253] _raw_spin_lock+0x21/0x30 [15016.191216] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15016.195471] obd_mod_free+0x46/0x140 [ptlrpc] [15016.198091] mdc_commit_open+0x91/0xb0 [mdc] [15016.200461] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15016.203849] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15016.210250] after_reply+0xe2c/0x1780 [ptlrpc] [15016.213973] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15016.217400] ? __next_timer_interrupt+0x160/0x160 [15016.221097] ptlrpcd+0xa82/0xe40 [ptlrpc] [15016.225115] ? do_wait_intr+0xf0/0xf0 [15016.227739] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15016.232101] kthread+0x1d1/0x200 [15016.234590] ? set_kthread_struct+0x70/0x70 [15016.237450] ret_from_fork+0x1f/0x30 [15020.054193] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15020.064254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15020.115685] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15020.127236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15020.136174] RIP: 0010:native_safe_halt+0x2e/0x50 [15020.141456] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15020.167036] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15020.175713] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15020.180083] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15020.183439] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15020.186840] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15020.190414] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15020.193771] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15020.198117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15020.201191] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15020.204641] Call Trace: [15020.206123] [15020.207081] ? show_regs.cold.9+0x22/0x2f [15020.212700] ? watchdog_timer_fn.cold.6+0x62/0x10a [15020.217928] ? __hrtimer_run_queues+0x1c2/0x4d0 [15020.223455] ? watchdog+0x40/0x40 [15020.227374] ? hrtimer_interrupt+0x128/0x2c0 [15020.230328] ? smp_apic_timer_interrupt+0xa6/0x270 [15020.233503] ? apic_timer_interrupt+0xf/0x20 [15020.236460] [15020.240219] ? native_safe_halt+0x2e/0x50 [15020.242250] kvm_wait+0xcd/0xe0 [15020.247719] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15020.251355] do_raw_spin_lock+0x177/0x190 [15020.257090] _raw_spin_lock+0x21/0x30 [15020.261438] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15020.266118] ? do_raw_spin_unlock+0x75/0x190 [15020.268377] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15020.280042] ? do_wait_intr+0xf0/0xf0 [15020.283849] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15020.295304] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15020.304241] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15020.309139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15020.313122] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15020.320196] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15020.323312] mdc_intent_lock+0x353/0x7e0 [mdc] [15020.332373] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15020.336654] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15020.342979] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15020.344481] lmv_intent_open+0x338/0x1150 [lmv] [15020.345660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15020.358339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15020.362542] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15020.363726] ll_intent_lock+0xc5/0xaa0 [lustre] [15020.364980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15020.374117] ll_lookup_it+0xe0e/0x2270 [lustre] [15020.379547] ? do_raw_spin_unlock+0x75/0x190 [15020.391545] ? _raw_spin_unlock+0x12/0x30 [15020.392962] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15020.409245] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15020.417823] ll_atomic_open+0x36a/0xd70 [lustre] [15020.419648] lookup_open+0x356/0xb40 [15020.433304] do_last+0x313/0xfc0 [15020.434438] ? nd_jump_root+0xe5/0x160 [15020.435629] ? path_init+0x437/0x520 [15020.436759] path_openat+0xf7/0x500 [15020.445323] ? do_raw_spin_unlock+0x75/0x190 [15020.448799] do_filp_open+0x99/0x140 [15020.450444] ? getname_flags+0x6e/0x330 [15020.452305] ? __check_object_size+0xff/0x256 [15020.455002] ? do_raw_spin_unlock+0x75/0x190 [15020.457415] ? _raw_spin_unlock+0x12/0x30 [15020.460091] do_sys_openat2+0x2b4/0x410 [15020.461517] do_sys_open+0x73/0xa0 [15020.467042] __x64_sys_openat+0x24/0x30 [15020.469420] do_syscall_64+0xc1/0x440 [15020.478959] entry_SYSCALL_64_after_hwframe+0x49/0xae [15020.486705] RIP: 0033:0x7fa749552332 [15020.490025] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15020.521512] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15020.530282] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15020.538344] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15020.544764] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15020.548699] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15020.558045] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15022.560831] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13105s! [15022.565989] Showing busy workqueues and worker pools: [15022.568377] workqueue events: flags=0x0 [15022.571324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15022.574175] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15022.580120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15022.583334] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15022.593379] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.598325] pending: kfree_rcu_monitor [15022.604415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15022.611524] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15022.625222] workqueue events_long: flags=0x0 [15022.626736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.639244] pending: lu_objects_flush [obdclass] [15022.646095] workqueue events_unbound: flags=0x2 [15022.647636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15022.655022] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15022.676329] workqueue events_power_efficient: flags=0x80 [15022.690458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.701399] pending: check_lifetime [15022.707493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.713330] pending: neigh_periodic_work [15022.716400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.719155] pending: neigh_periodic_work [15022.722397] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.728299] pending: do_cache_clean [15022.731351] workqueue mm_percpu_wq: flags=0x8 [15022.736070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.744379] pending: vmstat_update [15022.750715] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.768469] pending: vmstat_update [15022.770341] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15022.773878] pending: vmstat_update [15022.775261] workqueue writeback: flags=0x4a [15022.776852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15022.780034] pending: wb_workfn, wb_workfn [15022.783043] workqueue xprtiod: flags=0xa [15022.789364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15022.800392] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15022.815624] workqueue ipv6_addrconf: flags=0x40008 [15022.817100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15022.825041] pending: addrconf_verify_work [15022.828698] workqueue knbd0-recv: flags=0x1a [15022.831096] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15022.834376] in-flight: 651:recv_work [15022.837474] workqueue ptlrpc_pinger: flags=0x2 [15022.840530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15022.845513] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15022.851808] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13105s workers=4 idle: 37 31846 28897 [15022.858958] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15012s workers=2 idle: 653 [15032.064671] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15032.068372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15032.123778] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15032.129515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15032.138131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15032.143537] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15032.158333] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15032.167056] RAX: 0000000000006a86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15032.171529] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15032.177904] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15032.183629] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15032.189403] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15032.196414] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15032.203916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15032.206888] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15032.212039] Call Trace: [15032.213204] [15032.219349] ? show_regs.cold.9+0x22/0x2f [15032.223330] ? watchdog_timer_fn.cold.6+0x62/0x10a [15032.230783] ? __hrtimer_run_queues+0x1c2/0x4d0 [15032.235923] ? watchdog+0x40/0x40 [15032.238262] ? hrtimer_interrupt+0x128/0x2c0 [15032.241069] ? smp_apic_timer_interrupt+0xa6/0x270 [15032.244063] ? apic_timer_interrupt+0xf/0x20 [15032.248120] [15032.249012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15032.252843] do_raw_spin_lock+0x177/0x190 [15032.256294] _raw_spin_lock+0x21/0x30 [15032.258023] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15032.263299] ? do_raw_spin_unlock+0x75/0x190 [15032.270163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15032.276154] ? do_wait_intr+0xf0/0xf0 [15032.278010] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15032.285168] mdc_close+0x53b/0xd70 [mdc] [15032.292080] lmv_close+0x18d/0x490 [lmv] [15032.293587] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15032.300006] ll_md_real_close+0xa5/0x2a0 [lustre] [15032.302727] ll_md_close+0x274/0x800 [lustre] [15032.306621] ll_file_release+0x128/0xc50 [lustre] [15032.309620] __fput+0x10d/0x440 [15032.315014] ____fput+0x12/0x20 [15032.318735] task_work_run+0xc6/0x110 [15032.327404] exit_to_usermode_loop+0x1dd/0x1f0 [15032.331612] do_syscall_64+0x3d6/0x440 [15032.336128] entry_SYSCALL_64_after_hwframe+0x49/0xae [15032.339382] RIP: 0033:0x7fa749552cd8 [15032.345176] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15032.359990] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15032.373975] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15032.381322] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15032.389909] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15032.403208] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15032.418023] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15036.057139] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15036.063709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15036.116147] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15036.126012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15036.135899] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15036.142840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15036.146701] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15036.165028] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15036.171850] RAX: 000000000000790e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15036.180843] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15036.186418] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15036.191291] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15036.198088] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15036.204136] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15036.211143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15036.218288] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15036.226910] Call Trace: [15036.227609] [15036.228135] ? show_regs.cold.9+0x22/0x2f [15036.233422] ? watchdog_timer_fn.cold.6+0x62/0x10a [15036.235767] ? __hrtimer_run_queues+0x1c2/0x4d0 [15036.238142] ? watchdog+0x40/0x40 [15036.241437] ? hrtimer_interrupt+0x128/0x2c0 [15036.245067] ? smp_apic_timer_interrupt+0xa6/0x270 [15036.247592] ? apic_timer_interrupt+0xf/0x20 [15036.252420] [15036.253355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15036.258619] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15036.262018] do_raw_spin_lock+0x177/0x190 [15036.265125] _raw_spin_lock+0x21/0x30 [15036.270036] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15036.277202] process_one_work+0x2c8/0x700 [15036.279660] worker_thread+0x296/0x6e0 [15036.284458] ? rescuer_thread+0x570/0x570 [15036.289089] kthread+0x1d1/0x200 [15036.292233] ? set_kthread_struct+0x70/0x70 [15036.293710] ret_from_fork+0x1f/0x30 [15044.058057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15044.063111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15044.096504] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15044.101416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15044.107358] RIP: 0010:native_safe_halt+0x2e/0x50 [15044.111879] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15044.124030] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15044.128684] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15044.135727] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15044.141011] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15044.147750] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15044.151409] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15044.155511] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15044.161654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15044.166678] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15044.170591] Call Trace: [15044.173037] [15044.173570] ? show_regs.cold.9+0x22/0x2f [15044.175589] ? watchdog_timer_fn.cold.6+0x62/0x10a [15044.178390] ? __hrtimer_run_queues+0x1c2/0x4d0 [15044.181916] ? watchdog+0x40/0x40 [15044.184076] ? hrtimer_interrupt+0x128/0x2c0 [15044.185671] ? smp_apic_timer_interrupt+0xa6/0x270 [15044.188017] ? apic_timer_interrupt+0xf/0x20 [15044.190415] [15044.192071] ? native_safe_halt+0x2e/0x50 [15044.193827] kvm_wait+0xcd/0xe0 [15044.196991] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15044.200772] do_raw_spin_lock+0x177/0x190 [15044.202895] _raw_spin_lock+0x21/0x30 [15044.206555] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15044.212681] obd_mod_free+0x46/0x140 [ptlrpc] [15044.222769] mdc_commit_open+0x91/0xb0 [mdc] [15044.225515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15044.227993] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15044.231877] after_reply+0xe2c/0x1780 [ptlrpc] [15044.234776] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15044.237340] ? __next_timer_interrupt+0x160/0x160 [15044.238987] ptlrpcd+0xa82/0xe40 [ptlrpc] [15044.241144] ? do_wait_intr+0xf0/0xf0 [15044.242858] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15044.245347] kthread+0x1d1/0x200 [15044.246702] ? set_kthread_struct+0x70/0x70 [15044.248303] ret_from_fork+0x1f/0x30 [15048.054057] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15048.057073] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15048.079309] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15048.085018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15048.090236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15048.093847] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15048.103026] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15048.105892] RAX: 0000000000002fd3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15048.108214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15048.110319] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15048.112697] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15048.115133] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15048.117500] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15048.120174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15048.122149] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15048.124543] Call Trace: [15048.125824] [15048.126862] ? show_regs.cold.9+0x22/0x2f [15048.129176] ? watchdog_timer_fn.cold.6+0x62/0x10a [15048.131600] ? __hrtimer_run_queues+0x1c2/0x4d0 [15048.134578] ? watchdog+0x40/0x40 [15048.136405] ? hrtimer_interrupt+0x128/0x2c0 [15048.140129] ? smp_apic_timer_interrupt+0xa6/0x270 [15048.141994] ? apic_timer_interrupt+0xf/0x20 [15048.143530] [15048.144393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15048.146480] do_raw_spin_lock+0x177/0x190 [15048.148050] _raw_spin_lock+0x21/0x30 [15048.149387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15048.151539] ? do_raw_spin_unlock+0x75/0x190 [15048.158338] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15048.161517] ? do_wait_intr+0xf0/0xf0 [15048.163119] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15048.165422] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15048.168327] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15048.175980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15048.181636] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15048.185702] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15048.193353] mdc_intent_lock+0x353/0x7e0 [mdc] [15048.195328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15048.197956] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15048.201446] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15048.204054] lmv_intent_open+0x338/0x1150 [lmv] [15048.206979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15048.212545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15048.217476] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15048.221438] ll_intent_lock+0xc5/0xaa0 [lustre] [15048.225660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15048.229255] ll_lookup_it+0xe0e/0x2270 [lustre] [15048.232464] ? do_raw_spin_unlock+0x75/0x190 [15048.235155] ? _raw_spin_unlock+0x12/0x30 [15048.237933] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15048.240469] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15048.243704] ll_atomic_open+0x36a/0xd70 [lustre] [15048.249510] lookup_open+0x356/0xb40 [15048.254805] do_last+0x313/0xfc0 [15048.260464] ? nd_jump_root+0xe5/0x160 [15048.264821] ? path_init+0x437/0x520 [15048.267661] path_openat+0xf7/0x500 [15048.269433] ? do_raw_spin_unlock+0x75/0x190 [15048.272494] do_filp_open+0x99/0x140 [15048.275048] ? getname_flags+0x6e/0x330 [15048.276667] ? __check_object_size+0xff/0x256 [15048.282527] ? do_raw_spin_unlock+0x75/0x190 [15048.285525] ? _raw_spin_unlock+0x12/0x30 [15048.287047] do_sys_openat2+0x2b4/0x410 [15048.288229] do_sys_open+0x73/0xa0 [15048.289282] __x64_sys_openat+0x24/0x30 [15048.290811] do_syscall_64+0xc1/0x440 [15048.292380] entry_SYSCALL_64_after_hwframe+0x49/0xae [15048.294546] RIP: 0033:0x7fa749552332 [15048.296165] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15048.305321] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15048.308606] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15048.317534] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15048.323635] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15048.331639] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15048.337390] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15053.282616] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13136s! [15053.286451] Showing busy workqueues and worker pools: [15053.288557] workqueue events: flags=0x0 [15053.291246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15053.297455] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15053.307772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15053.311703] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15053.319189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.323309] pending: kfree_rcu_monitor [15053.324793] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15053.327832] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15053.332251] workqueue events_long: flags=0x0 [15053.333865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.336149] pending: lu_objects_flush [obdclass] [15053.338589] workqueue events_unbound: flags=0x2 [15053.342090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15053.344895] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15053.348858] workqueue events_power_efficient: flags=0x80 [15053.351777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.353678] pending: check_lifetime [15053.359162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.361261] pending: neigh_periodic_work [15053.362970] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.365376] pending: neigh_periodic_work [15053.367717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.370155] pending: do_cache_clean [15053.371686] workqueue mm_percpu_wq: flags=0x8 [15053.373632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.377447] pending: vmstat_update [15053.378864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.381586] pending: vmstat_update [15053.382529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15053.384180] pending: vmstat_update [15053.386351] workqueue writeback: flags=0x4a [15053.390190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15053.392785] pending: wb_workfn, wb_workfn [15053.394229] workqueue xprtiod: flags=0xa [15053.396878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15053.400320] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15053.406809] workqueue ipv6_addrconf: flags=0x40008 [15053.408212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15053.412163] pending: addrconf_verify_work [15053.414950] workqueue knbd0-recv: flags=0x1a [15053.417804] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15053.422294] in-flight: 651:recv_work [15053.423879] workqueue ptlrpc_pinger: flags=0x2 [15053.426158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15053.429356] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15053.432098] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13136s workers=4 idle: 37 31846 28897 [15053.436709] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15043s workers=2 idle: 653 [15060.060572] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15060.065810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15060.105320] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15060.114938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15060.120886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15060.125423] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15060.139899] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15060.150331] RAX: 0000000000007dec RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15060.154665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15060.160440] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15060.166093] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15060.175487] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15060.181993] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15060.189097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15060.193413] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15060.200291] Call Trace: [15060.203717] [15060.205047] ? show_regs.cold.9+0x22/0x2f [15060.209722] ? watchdog_timer_fn.cold.6+0x62/0x10a [15060.213836] ? __hrtimer_run_queues+0x1c2/0x4d0 [15060.218304] ? watchdog+0x40/0x40 [15060.220613] ? hrtimer_interrupt+0x128/0x2c0 [15060.222676] ? smp_apic_timer_interrupt+0xa6/0x270 [15060.227150] ? apic_timer_interrupt+0xf/0x20 [15060.228932] [15060.229767] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15060.231935] do_raw_spin_lock+0x177/0x190 [15060.233430] _raw_spin_lock+0x21/0x30 [15060.234934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15060.236840] ? do_raw_spin_unlock+0x75/0x190 [15060.238398] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15060.240580] ? do_wait_intr+0xf0/0xf0 [15060.241918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15060.244151] mdc_close+0x53b/0xd70 [mdc] [15060.245785] lmv_close+0x18d/0x490 [lmv] [15060.247619] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15060.250077] ll_md_real_close+0xa5/0x2a0 [lustre] [15060.252064] ll_md_close+0x274/0x800 [lustre] [15060.254000] ll_file_release+0x128/0xc50 [lustre] [15060.256100] __fput+0x10d/0x440 [15060.257388] ____fput+0x12/0x20 [15060.261426] task_work_run+0xc6/0x110 [15060.265742] exit_to_usermode_loop+0x1dd/0x1f0 [15060.268104] do_syscall_64+0x3d6/0x440 [15060.271633] entry_SYSCALL_64_after_hwframe+0x49/0xae [15060.275767] RIP: 0033:0x7fa749552cd8 [15060.279439] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15060.294732] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15060.301548] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15060.305483] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15060.312380] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15060.316508] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15060.325399] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15064.062798] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15064.066967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15064.106235] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15064.124503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15064.133670] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15064.140300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15064.143333] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15064.167220] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15064.170429] RAX: 000000000000785a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15064.173132] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15064.175945] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15064.184205] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15064.187452] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15064.192787] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15064.196474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15064.198566] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15064.201969] Call Trace: [15064.203064] [15064.204219] ? show_regs.cold.9+0x22/0x2f [15064.206382] ? watchdog_timer_fn.cold.6+0x62/0x10a [15064.208323] ? __hrtimer_run_queues+0x1c2/0x4d0 [15064.209546] ? watchdog+0x40/0x40 [15064.212942] ? hrtimer_interrupt+0x128/0x2c0 [15064.214984] ? smp_apic_timer_interrupt+0xa6/0x270 [15064.221317] ? apic_timer_interrupt+0xf/0x20 [15064.224727] [15064.225761] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15064.231109] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15064.232754] do_raw_spin_lock+0x177/0x190 [15064.235778] _raw_spin_lock+0x21/0x30 [15064.237539] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15064.242185] process_one_work+0x2c8/0x700 [15064.245583] worker_thread+0x296/0x6e0 [15064.247530] ? rescuer_thread+0x570/0x570 [15064.249611] kthread+0x1d1/0x200 [15064.253372] ? set_kthread_struct+0x70/0x70 [15064.257183] ret_from_fork+0x1f/0x30 [15072.059450] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15072.066228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15072.101226] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15072.106989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15072.112224] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15072.116898] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15072.125825] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15072.129164] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15072.132855] RDX: 0000000000006a90 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15072.136500] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15072.140964] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15072.145201] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15072.150354] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15072.156093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15072.160258] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15072.165293] Call Trace: [15072.167082] [15072.168249] ? show_regs.cold.9+0x22/0x2f [15072.170262] ? watchdog_timer_fn.cold.6+0x62/0x10a [15072.173219] ? __hrtimer_run_queues+0x1c2/0x4d0 [15072.176923] ? watchdog+0x40/0x40 [15072.187127] ? hrtimer_interrupt+0x128/0x2c0 [15072.190902] ? smp_apic_timer_interrupt+0xa6/0x270 [15072.193358] ? apic_timer_interrupt+0xf/0x20 [15072.196017] [15072.197550] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15072.200336] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15072.203390] do_raw_spin_lock+0x177/0x190 [15072.204753] _raw_spin_lock+0x21/0x30 [15072.207187] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15072.210524] obd_mod_free+0x46/0x140 [ptlrpc] [15072.214941] mdc_commit_open+0x91/0xb0 [mdc] [15072.217100] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15072.220242] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15072.225019] after_reply+0xe2c/0x1780 [ptlrpc] [15072.227797] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15072.230499] ? __next_timer_interrupt+0x160/0x160 [15072.233189] ptlrpcd+0xa82/0xe40 [ptlrpc] [15072.236032] ? do_wait_intr+0xf0/0xf0 [15072.237998] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15072.241127] kthread+0x1d1/0x200 [15072.242409] ? set_kthread_struct+0x70/0x70 [15072.245194] ret_from_fork+0x1f/0x30 [15080.054013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15080.059113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15080.087112] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15080.104483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15080.115688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15080.119904] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15080.133431] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15080.138904] RAX: 00000000000001a3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15080.144210] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15080.150497] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15080.159247] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15080.166064] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15080.175253] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15080.186407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15080.191124] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15080.202433] Call Trace: [15080.204580] [15080.206527] ? show_regs.cold.9+0x22/0x2f [15080.211469] ? watchdog_timer_fn.cold.6+0x62/0x10a [15080.218710] ? __hrtimer_run_queues+0x1c2/0x4d0 [15080.224935] ? watchdog+0x40/0x40 [15080.229138] ? hrtimer_interrupt+0x128/0x2c0 [15080.233243] ? smp_apic_timer_interrupt+0xa6/0x270 [15080.238496] ? apic_timer_interrupt+0xf/0x20 [15080.240860] [15080.242952] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15080.251409] do_raw_spin_lock+0x177/0x190 [15080.254291] _raw_spin_lock+0x21/0x30 [15080.259415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15080.263949] ? do_raw_spin_unlock+0x75/0x190 [15080.267789] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15080.270122] ? do_wait_intr+0xf0/0xf0 [15080.271935] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15080.277851] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15080.283087] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15080.289384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15080.293797] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15080.298683] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15080.302442] mdc_intent_lock+0x353/0x7e0 [mdc] [15080.311122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15080.313240] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15080.318475] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15080.328399] lmv_intent_open+0x338/0x1150 [lmv] [15080.330152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15080.335269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15080.339722] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15080.343015] ll_intent_lock+0xc5/0xaa0 [lustre] [15080.347117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15080.353903] ll_lookup_it+0xe0e/0x2270 [lustre] [15080.360062] ? do_raw_spin_unlock+0x75/0x190 [15080.368247] ? _raw_spin_unlock+0x12/0x30 [15080.377060] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15080.384436] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15080.387956] ll_atomic_open+0x36a/0xd70 [lustre] [15080.389740] lookup_open+0x356/0xb40 [15080.394112] do_last+0x313/0xfc0 [15080.397803] ? nd_jump_root+0xe5/0x160 [15080.400375] ? path_init+0x437/0x520 [15080.406616] path_openat+0xf7/0x500 [15080.408075] ? do_raw_spin_unlock+0x75/0x190 [15080.410896] do_filp_open+0x99/0x140 [15080.413753] ? getname_flags+0x6e/0x330 [15080.416276] ? __check_object_size+0xff/0x256 [15080.420679] ? do_raw_spin_unlock+0x75/0x190 [15080.422200] ? _raw_spin_unlock+0x12/0x30 [15080.427070] do_sys_openat2+0x2b4/0x410 [15080.430015] do_sys_open+0x73/0xa0 [15080.433190] __x64_sys_openat+0x24/0x30 [15080.437821] do_syscall_64+0xc1/0x440 [15080.441667] entry_SYSCALL_64_after_hwframe+0x49/0xae [15080.447765] RIP: 0033:0x7fa749552332 [15080.450399] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15080.469248] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15080.491328] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15080.504349] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15080.506953] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15080.527202] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15080.534514] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15083.489172] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13166s! [15083.492822] Showing busy workqueues and worker pools: [15083.494617] workqueue events: flags=0x0 [15083.497605] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15083.501968] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15083.509219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15083.513147] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15083.518545] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.522618] pending: kfree_rcu_monitor [15083.525324] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15083.530026] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15083.534283] workqueue events_long: flags=0x0 [15083.536275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.539253] pending: lu_objects_flush [obdclass] [15083.540594] workqueue events_unbound: flags=0x2 [15083.541658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15083.545162] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15083.550789] workqueue events_power_efficient: flags=0x80 [15083.552817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.557695] pending: check_lifetime [15083.560331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.564231] pending: neigh_periodic_work [15083.568288] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.571344] pending: neigh_periodic_work [15083.573229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.576800] pending: do_cache_clean [15083.578808] workqueue mm_percpu_wq: flags=0x8 [15083.582747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.589365] pending: vmstat_update [15083.591274] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.594249] pending: vmstat_update [15083.595415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15083.597878] pending: vmstat_update [15083.599456] workqueue writeback: flags=0x4a [15083.601290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15083.605248] pending: wb_workfn, wb_workfn [15083.607472] workqueue xprtiod: flags=0xa [15083.610395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15083.613059] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15083.617194] workqueue ipv6_addrconf: flags=0x40008 [15083.619898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15083.621695] pending: addrconf_verify_work [15083.622985] workqueue knbd0-recv: flags=0x1a [15083.624375] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15083.627082] in-flight: 651:recv_work [15083.628806] workqueue ptlrpc_pinger: flags=0x2 [15083.631275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15083.635017] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15083.638643] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13166s workers=4 idle: 37 31846 28897 [15083.642273] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15073s workers=2 idle: 653 [15088.063924] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15088.068848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15088.092413] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15088.102803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15088.112750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15088.123477] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15088.147581] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15088.159403] RAX: 0000000000002886 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15088.168905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15088.179309] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15088.182935] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15088.192015] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15088.197906] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15088.203070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15088.206155] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15088.209502] Call Trace: [15088.211222] [15088.213027] ? show_regs.cold.9+0x22/0x2f [15088.217118] ? watchdog_timer_fn.cold.6+0x62/0x10a [15088.219885] ? __hrtimer_run_queues+0x1c2/0x4d0 [15088.226971] ? watchdog+0x40/0x40 [15088.232525] ? hrtimer_interrupt+0x128/0x2c0 [15088.239741] ? smp_apic_timer_interrupt+0xa6/0x270 [15088.249049] ? apic_timer_interrupt+0xf/0x20 [15088.252327] [15088.259269] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15088.267391] do_raw_spin_lock+0x177/0x190 [15088.272056] _raw_spin_lock+0x21/0x30 [15088.273383] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15088.277923] ? do_raw_spin_unlock+0x75/0x190 [15088.282340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15088.293151] ? do_wait_intr+0xf0/0xf0 [15088.296052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15088.298347] mdc_close+0x53b/0xd70 [mdc] [15088.300583] lmv_close+0x18d/0x490 [lmv] [15088.302403] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15088.306061] ll_md_real_close+0xa5/0x2a0 [lustre] [15088.309193] ll_md_close+0x274/0x800 [lustre] [15088.312018] ll_file_release+0x128/0xc50 [lustre] [15088.314084] __fput+0x10d/0x440 [15088.315832] ____fput+0x12/0x20 [15088.317199] task_work_run+0xc6/0x110 [15088.319142] exit_to_usermode_loop+0x1dd/0x1f0 [15088.322153] do_syscall_64+0x3d6/0x440 [15088.325447] entry_SYSCALL_64_after_hwframe+0x49/0xae [15088.329802] RIP: 0033:0x7fa749552cd8 [15088.332199] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15088.348960] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15088.353171] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15088.360196] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15088.365879] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15088.370959] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15088.373972] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15092.057083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15092.073746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15092.121083] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15092.127337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15092.134117] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15092.137799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15092.141451] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15092.162116] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15092.171545] RAX: 000000000000228a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15092.178414] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15092.183749] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15092.187745] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15092.191263] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15092.194992] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15092.199019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15092.201736] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15092.206631] Call Trace: [15092.207894] [15092.208925] ? show_regs.cold.9+0x22/0x2f [15092.210415] ? watchdog_timer_fn.cold.6+0x62/0x10a [15092.213144] ? __hrtimer_run_queues+0x1c2/0x4d0 [15092.215056] ? watchdog+0x40/0x40 [15092.216545] ? hrtimer_interrupt+0x128/0x2c0 [15092.218239] ? smp_apic_timer_interrupt+0xa6/0x270 [15092.220334] ? apic_timer_interrupt+0xf/0x20 [15092.222070] [15092.223006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15092.225154] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15092.227461] do_raw_spin_lock+0x177/0x190 [15092.229676] _raw_spin_lock+0x21/0x30 [15092.231047] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15092.233581] process_one_work+0x2c8/0x700 [15092.235108] worker_thread+0x296/0x6e0 [15092.236766] ? rescuer_thread+0x570/0x570 [15092.238499] kthread+0x1d1/0x200 [15092.240545] ? set_kthread_struct+0x70/0x70 [15092.242096] ret_from_fork+0x1f/0x30 [15100.058024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15100.062315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15100.092529] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15100.097915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15100.102621] RIP: 0010:native_safe_halt+0x2e/0x50 [15100.105753] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15100.116288] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15100.120800] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15100.124331] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15100.128439] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15100.132024] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15100.135587] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15100.139350] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15100.143793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15100.147044] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15100.151496] Call Trace: [15100.153042] [15100.154199] ? show_regs.cold.9+0x22/0x2f [15100.155564] ? watchdog_timer_fn.cold.6+0x62/0x10a [15100.157316] ? __hrtimer_run_queues+0x1c2/0x4d0 [15100.159533] ? watchdog+0x40/0x40 [15100.161295] ? hrtimer_interrupt+0x128/0x2c0 [15100.163515] ? smp_apic_timer_interrupt+0xa6/0x270 [15100.166046] ? apic_timer_interrupt+0xf/0x20 [15100.167987] [15100.169107] ? native_safe_halt+0x2e/0x50 [15100.170866] kvm_wait+0xcd/0xe0 [15100.172333] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15100.174712] do_raw_spin_lock+0x177/0x190 [15100.176886] _raw_spin_lock+0x21/0x30 [15100.178745] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15100.181498] obd_mod_free+0x46/0x140 [ptlrpc] [15100.183751] mdc_commit_open+0x91/0xb0 [mdc] [15100.185963] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15100.189362] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15100.192266] after_reply+0xe2c/0x1780 [ptlrpc] [15100.195085] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15100.197429] ? __next_timer_interrupt+0x160/0x160 [15100.199835] ptlrpcd+0xa82/0xe40 [ptlrpc] [15100.202958] ? do_wait_intr+0xf0/0xf0 [15100.204609] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15100.208419] kthread+0x1d1/0x200 [15100.209923] ? set_kthread_struct+0x70/0x70 [15100.212065] ret_from_fork+0x1f/0x30 [15108.054032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15108.056646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15108.080994] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15108.087604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15108.091601] RIP: 0010:native_safe_halt+0x2e/0x50 [15108.094288] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15108.104179] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15108.109133] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15108.112442] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15108.114655] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15108.119456] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15108.123809] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15108.128798] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15108.133265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15108.135807] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15108.139029] Call Trace: [15108.139970] [15108.140605] ? show_regs.cold.9+0x22/0x2f [15108.142173] ? watchdog_timer_fn.cold.6+0x62/0x10a [15108.143568] ? __hrtimer_run_queues+0x1c2/0x4d0 [15108.145342] ? watchdog+0x40/0x40 [15108.146808] ? hrtimer_interrupt+0x128/0x2c0 [15108.148399] ? smp_apic_timer_interrupt+0xa6/0x270 [15108.149915] ? apic_timer_interrupt+0xf/0x20 [15108.151302] [15108.152078] ? native_safe_halt+0x2e/0x50 [15108.153590] kvm_wait+0xcd/0xe0 [15108.155896] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15108.157673] do_raw_spin_lock+0x177/0x190 [15108.159033] _raw_spin_lock+0x21/0x30 [15108.160340] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15108.168794] ? do_raw_spin_unlock+0x75/0x190 [15108.170286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15108.173364] ? do_wait_intr+0xf0/0xf0 [15108.175003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15108.177836] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15108.180341] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15108.183229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15108.185298] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15108.187107] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15108.188644] mdc_intent_lock+0x353/0x7e0 [mdc] [15108.193437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15108.197607] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15108.199936] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15108.201549] lmv_intent_open+0x338/0x1150 [lmv] [15108.203620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15108.205678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15108.207631] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15108.209678] ll_intent_lock+0xc5/0xaa0 [lustre] [15108.211836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15108.214414] ll_lookup_it+0xe0e/0x2270 [lustre] [15108.216727] ? do_raw_spin_unlock+0x75/0x190 [15108.218665] ? _raw_spin_unlock+0x12/0x30 [15108.220785] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15108.222983] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15108.224972] ll_atomic_open+0x36a/0xd70 [lustre] [15108.226718] lookup_open+0x356/0xb40 [15108.227925] do_last+0x313/0xfc0 [15108.228987] ? nd_jump_root+0xe5/0x160 [15108.231767] ? path_init+0x437/0x520 [15108.233358] path_openat+0xf7/0x500 [15108.234479] ? do_raw_spin_unlock+0x75/0x190 [15108.235928] do_filp_open+0x99/0x140 [15108.237259] ? getname_flags+0x6e/0x330 [15108.238544] ? __check_object_size+0xff/0x256 [15108.239894] ? do_raw_spin_unlock+0x75/0x190 [15108.241629] ? _raw_spin_unlock+0x12/0x30 [15108.244950] do_sys_openat2+0x2b4/0x410 [15108.246543] do_sys_open+0x73/0xa0 [15108.252462] __x64_sys_openat+0x24/0x30 [15108.253930] do_syscall_64+0xc1/0x440 [15108.256221] entry_SYSCALL_64_after_hwframe+0x49/0xae [15108.258939] RIP: 0033:0x7fa749552332 [15108.260292] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15108.267656] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15108.271554] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15108.276900] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15108.280629] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15108.282905] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15108.287128] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15113.701059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13196s! [15113.713082] Showing busy workqueues and worker pools: [15113.717863] workqueue events: flags=0x0 [15113.720192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15113.725338] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15113.738898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15113.746152] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15113.750563] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.753393] pending: kfree_rcu_monitor [15113.755234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15113.758403] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15113.762718] workqueue events_long: flags=0x0 [15113.764319] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.770276] pending: lu_objects_flush [obdclass] [15113.772211] workqueue events_unbound: flags=0x2 [15113.781209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15113.793333] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15113.800549] workqueue events_power_efficient: flags=0x80 [15113.810350] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.814052] pending: check_lifetime [15113.823769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.828388] pending: neigh_periodic_work [15113.830039] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.835193] pending: neigh_periodic_work [15113.838469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.842739] pending: do_cache_clean [15113.846698] workqueue mm_percpu_wq: flags=0x8 [15113.850625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.857495] pending: vmstat_update [15113.861170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.863649] pending: vmstat_update [15113.865038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15113.869694] pending: vmstat_update [15113.871196] workqueue writeback: flags=0x4a [15113.872822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15113.875192] pending: wb_workfn, wb_workfn [15113.879862] workqueue xprtiod: flags=0xa [15113.886412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15113.890851] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15113.893377] workqueue ipv6_addrconf: flags=0x40008 [15113.895797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15113.900577] pending: addrconf_verify_work [15113.902267] workqueue knbd0-recv: flags=0x1a [15113.903911] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15113.907509] in-flight: 651:recv_work [15113.909382] workqueue ptlrpc_pinger: flags=0x2 [15113.914534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15113.918563] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15113.922718] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13196s workers=4 idle: 37 31846 28897 [15113.931450] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15103s workers=2 idle: 653 [15113.985377] rcu: INFO: rcu_sched self-detected stall on CPU [15113.992024] rcu: 2-...!: (7609836 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [15114.003686] (t=13200296 jiffies g=258701 q=5104) [15114.009152] rcu: rcu_sched kthread starved for 13197091 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [15114.017199] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15114.024745] rcu: RCU grace-period kthread stack dump: [15114.030862] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15114.049853] Call Trace: [15114.051554] __schedule+0x351/0xcb0 [15114.055761] schedule+0xc0/0x180 [15114.059521] schedule_timeout+0xb4/0x190 [15114.062787] ? do_raw_spin_unlock+0x75/0x190 [15114.067400] ? __next_timer_interrupt+0x160/0x160 [15114.071126] ? prepare_to_swait_event+0xf3/0x140 [15114.074048] rcu_gp_kthread+0x8dd/0x15b0 [15114.076310] ? force_qs_rnp+0x2c0/0x2c0 [15114.078461] kthread+0x1d1/0x200 [15114.081595] ? set_kthread_struct+0x70/0x70 [15114.084559] ret_from_fork+0x1f/0x30 [15114.089340] rcu: Stack dump where RCU GP kthread last ran: [15114.095279] Sending NMI from CPU 2 to CPUs 3: [15114.100027] NMI backtrace for cpu 3 [15114.100034] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15114.100036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15114.100038] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15114.100039] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15114.100041] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 [15114.100044] RAX: 0000000000007f03 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15114.100122] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15114.100124] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15114.100125] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15114.100126] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15114.100128] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15114.100129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15114.100130] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15114.100131] Call Trace: [15114.100132] [15114.100133] ? show_regs.cold.9+0x22/0x2f [15114.100134] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15114.100135] ? nmi_cpu_backtrace_handler+0x15/0x40 [15114.100136] ? nmi_handle+0xbd/0x210 [15114.100137] ? default_do_nmi+0x4b/0x1b0 [15114.100138] ? do_nmi+0x280/0x380 [15114.100139] ? end_repeat_nmi+0x16/0x1f [15114.100140] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15114.100141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15114.100142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15114.100143] [15114.100143] do_raw_spin_lock+0x177/0x190 [15114.100144] _raw_spin_lock+0x21/0x30 [15114.100145] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15114.100146] ? do_raw_spin_unlock+0x75/0x190 [15114.100148] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15114.100149] ? do_wait_intr+0xf0/0xf0 [15114.100150] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15114.100151] mdc_close+0x53b/0xd70 [mdc] [15114.100152] lmv_close+0x18d/0x490 [lmv] [15114.100153] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15114.100154] ll_md_real_close+0xa5/0x2a0 [lustre] [15114.100161] ll_md_close+0x274/0x800 [lustre] [15114.100162] ll_file_release+0x128/0xc50 [lustre] [15114.100163] __fput+0x10d/0x440 [15114.100164] ____fput+0x12/0x20 [15114.100165] task_work_run+0xc6/0x110 [15114.100166] exit_to_usermode_loop+0x1dd/0x1f0 [15114.100167] do_syscall_64+0x3d6/0x440 [15114.100168] entry_SYSCALL_64_after_hwframe+0x49/0xae [15114.100169] RIP: 0033:0x7fa749552cd8 [15114.100170] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15114.100173] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15114.100175] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15114.100176] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15114.100178] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15114.100179] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15114.100180] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15114.101074] NMI backtrace for cpu 2 [15114.257399] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15114.268190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15114.276398] Call Trace: [15114.279403] [15114.281998] ? dump_stack+0xbb/0x10e [15114.283867] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15114.285747] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15114.292667] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15114.300026] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15114.303182] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15114.309446] ? rcu_dump_cpu_stacks+0x11b/0x171 [15114.311119] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15114.316336] ? update_process_times+0x6d/0xc0 [15114.321404] ? tick_sched_handle.isra.18+0x27/0x90 [15114.327884] ? tick_sched_timer+0x53/0xb0 [15114.330530] ? __hrtimer_run_queues+0x1c2/0x4d0 [15114.334986] ? tick_sched_handle.isra.18+0x90/0x90 [15114.339364] ? hrtimer_interrupt+0x128/0x2c0 [15114.350414] ? smp_apic_timer_interrupt+0xa6/0x270 [15114.352098] ? apic_timer_interrupt+0xf/0x20 [15114.353581] [15114.366333] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15114.368323] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15114.377068] ? do_raw_spin_lock+0x177/0x190 [15114.382031] ? _raw_spin_lock+0x21/0x30 [15114.387509] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15114.395100] ? obd_mod_free+0x46/0x140 [ptlrpc] [15114.398391] ? mdc_commit_open+0x91/0xb0 [mdc] [15114.400916] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15114.406397] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15114.414166] ? after_reply+0xe2c/0x1780 [ptlrpc] [15114.419608] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15114.425551] ? __next_timer_interrupt+0x160/0x160 [15114.429598] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15114.434861] ? do_wait_intr+0xf0/0xf0 [15114.437156] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15114.443358] ? kthread+0x1d1/0x200 [15114.447786] ? set_kthread_struct+0x70/0x70 [15114.451277] ? ret_from_fork+0x1f/0x30 [15120.057027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15120.063347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15120.102657] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15120.114348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15120.126622] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15120.131850] RIP: 0010:native_safe_halt+0x2e/0x50 [15120.133569] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15120.149774] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15120.155146] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15120.161977] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15120.164337] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15120.174446] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15120.182375] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15120.191690] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15120.204183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15120.208402] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15120.213166] Call Trace: [15120.214663] [15120.215655] ? show_regs.cold.9+0x22/0x2f [15120.218609] ? watchdog_timer_fn.cold.6+0x62/0x10a [15120.225393] ? __hrtimer_run_queues+0x1c2/0x4d0 [15120.228404] ? watchdog+0x40/0x40 [15120.230065] ? hrtimer_interrupt+0x128/0x2c0 [15120.232205] ? smp_apic_timer_interrupt+0xa6/0x270 [15120.234584] ? apic_timer_interrupt+0xf/0x20 [15120.236515] [15120.237531] ? native_safe_halt+0x2e/0x50 [15120.239293] kvm_wait+0xcd/0xe0 [15120.242389] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15120.247561] do_raw_spin_lock+0x177/0x190 [15120.249462] _raw_spin_lock+0x21/0x30 [15120.253609] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15120.257634] process_one_work+0x2c8/0x700 [15120.258829] worker_thread+0x296/0x6e0 [15120.261635] ? rescuer_thread+0x570/0x570 [15120.263396] kthread+0x1d1/0x200 [15120.264797] ? set_kthread_struct+0x70/0x70 [15120.266838] ret_from_fork+0x1f/0x30 [15140.055954] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15140.062074] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15140.064725] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15140.064741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15140.064820] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15140.064826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15140.064829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15140.064844] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15140.064849] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15140.064854] RAX: 0000000000007b08 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15140.064857] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15140.064859] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15140.064861] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15140.064863] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15140.064866] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15140.064869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15140.064871] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15140.064880] Call Trace: [15140.064885] [15140.064891] ? show_regs.cold.9+0x22/0x2f [15140.064902] ? watchdog_timer_fn.cold.6+0x62/0x10a [15140.064909] ? __hrtimer_run_queues+0x1c2/0x4d0 [15140.064914] ? watchdog+0x40/0x40 [15140.064918] ? hrtimer_interrupt+0x128/0x2c0 [15140.064923] ? smp_apic_timer_interrupt+0xa6/0x270 [15140.064931] ? apic_timer_interrupt+0xf/0x20 [15140.064935] [15140.064938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15140.064946] do_raw_spin_lock+0x177/0x190 [15140.064950] _raw_spin_lock+0x21/0x30 [15140.064955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15140.065220] ? do_raw_spin_unlock+0x75/0x190 [15140.065226] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15140.065418] ? do_wait_intr+0xf0/0xf0 [15140.065425] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15140.065782] mdc_close+0x53b/0xd70 [mdc] [15140.065818] lmv_close+0x18d/0x490 [lmv] [15140.065848] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15140.065956] ll_md_real_close+0xa5/0x2a0 [lustre] [15140.066035] ll_md_close+0x274/0x800 [lustre] [15140.066116] ll_file_release+0x128/0xc50 [lustre] [15140.066196] __fput+0x10d/0x440 [15140.066204] ____fput+0x12/0x20 [15140.066209] task_work_run+0xc6/0x110 [15140.066215] exit_to_usermode_loop+0x1dd/0x1f0 [15140.066223] do_syscall_64+0x3d6/0x440 [15140.066227] entry_SYSCALL_64_after_hwframe+0x49/0xae [15140.066234] RIP: 0033:0x7fa749552cd8 [15140.066240] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15140.066244] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15140.066249] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15140.066251] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15140.066253] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15140.066255] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15140.066257] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15140.083610] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15140.083682] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15140.083691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15140.083693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15140.083706] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15140.083710] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15140.083715] RAX: 0000000000000483 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15140.083717] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15140.083720] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15140.083722] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15140.083724] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15140.083727] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15140.083730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15140.083732] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15140.083741] Call Trace: [15140.083746] [15140.083751] ? show_regs.cold.9+0x22/0x2f [15140.083761] ? watchdog_timer_fn.cold.6+0x62/0x10a [15140.083777] ? __hrtimer_run_queues+0x1c2/0x4d0 [15140.083782] ? watchdog+0x40/0x40 [15140.083786] ? hrtimer_interrupt+0x128/0x2c0 [15140.083791] ? smp_apic_timer_interrupt+0xa6/0x270 [15140.083799] ? apic_timer_interrupt+0xf/0x20 [15140.083804] [15140.083807] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15140.083813] do_raw_spin_lock+0x177/0x190 [15140.083818] _raw_spin_lock+0x21/0x30 [15140.083823] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15140.095900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [15140.108055] ? do_raw_spin_unlock+0x75/0x190 [15140.108076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15140.108347] ? do_wait_intr+0xf0/0xf0 [15140.108511] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15140.108711] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15140.168004] lfsck(O) obdecho(O) mgc(O) [15140.177010] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15140.177300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.177436] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15140.177479] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15140.188275] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul [15140.194612] mdc_intent_lock+0x353/0x7e0 [mdc] [15140.194666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.215340] crc32_pclmul [15140.222008] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15140.226402] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata [15140.230031] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15140.230083] lmv_intent_open+0x338/0x1150 [lmv] [15140.230116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.230234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.230318] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15140.230341] ll_intent_lock+0xc5/0xaa0 [lustre] [15140.231165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.231244] ll_lookup_it+0xe0e/0x2270 [lustre] [15140.231331] ? do_raw_spin_unlock+0x75/0x190 [15140.231339] ? _raw_spin_unlock+0x12/0x30 [15140.231345] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15140.242796] dm_mirror [15140.249729] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15140.259327] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15140.259349] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15140.259356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15140.259359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15140.259372] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15140.259377] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15140.259381] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15140.259384] RDX: 0000000000003368 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15140.259386] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15140.259399] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15140.259402] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15140.259405] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15140.259408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15140.259410] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15140.259418] Call Trace: [15140.259424] [15140.259430] ? show_regs.cold.9+0x22/0x2f [15140.259440] ? watchdog_timer_fn.cold.6+0x62/0x10a [15140.259447] ? __hrtimer_run_queues+0x1c2/0x4d0 [15140.259451] ? watchdog+0x40/0x40 [15140.259455] ? hrtimer_interrupt+0x128/0x2c0 [15140.259459] ? smp_apic_timer_interrupt+0xa6/0x270 [15140.259468] ? apic_timer_interrupt+0xf/0x20 [15140.259473] [15140.259476] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15140.259481] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15140.259486] do_raw_spin_lock+0x177/0x190 [15140.259491] _raw_spin_lock+0x21/0x30 [15140.259498] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15140.260900] obd_mod_free+0x46/0x140 [ptlrpc] [15140.261096] mdc_commit_open+0x91/0xb0 [mdc] [15140.261132] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15140.261324] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15140.271062] ll_atomic_open+0x36a/0xd70 [lustre] [15140.271200] lookup_open+0x356/0xb40 [15140.271211] do_last+0x313/0xfc0 [15140.271216] ? nd_jump_root+0xe5/0x160 [15140.271222] ? path_init+0x437/0x520 [15140.271226] path_openat+0xf7/0x500 [15140.271231] ? do_raw_spin_unlock+0x75/0x190 [15140.271239] do_filp_open+0x99/0x140 [15140.271246] ? getname_flags+0x6e/0x330 [15140.271251] ? __check_object_size+0xff/0x256 [15140.271259] ? do_raw_spin_unlock+0x75/0x190 [15140.271263] ? _raw_spin_unlock+0x12/0x30 [15140.271269] do_sys_openat2+0x2b4/0x410 [15140.271274] do_sys_open+0x73/0xa0 [15140.271279] __x64_sys_openat+0x24/0x30 [15140.271283] do_syscall_64+0xc1/0x440 [15140.271290] entry_SYSCALL_64_after_hwframe+0x49/0xae [15140.271296] RIP: 0033:0x7fa749552332 [15140.271303] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15140.271308] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15140.271313] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15140.271316] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15140.271318] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15140.271321] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15140.271323] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15141.055758] after_reply+0xe2c/0x1780 [ptlrpc] [15141.061099] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15141.065125] ? __next_timer_interrupt+0x160/0x160 [15141.070019] ptlrpcd+0xa82/0xe40 [ptlrpc] [15141.072522] ? do_wait_intr+0xf0/0xf0 [15141.074968] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15141.078411] kthread+0x1d1/0x200 [15141.080185] ? set_kthread_struct+0x70/0x70 [15141.081810] ret_from_fork+0x1f/0x30 [15144.420098] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13227s! [15144.433020] Showing busy workqueues and worker pools: [15144.439282] workqueue events: flags=0x0 [15144.448095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15144.460346] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15144.476078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15144.488912] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15144.498759] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.510814] pending: kfree_rcu_monitor [15144.512217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15144.525227] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15144.531940] workqueue events_long: flags=0x0 [15144.537126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.544893] pending: lu_objects_flush [obdclass] [15144.548246] workqueue events_unbound: flags=0x2 [15144.551112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15144.562597] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15144.576716] workqueue events_power_efficient: flags=0x80 [15144.582299] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.590133] pending: check_lifetime [15144.592917] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.597892] pending: neigh_periodic_work [15144.608516] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.624873] pending: neigh_periodic_work [15144.630025] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.638271] pending: do_cache_clean [15144.647671] workqueue mm_percpu_wq: flags=0x8 [15144.656924] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.669240] pending: vmstat_update [15144.675614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.683554] pending: vmstat_update [15144.685029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15144.690589] pending: vmstat_update [15144.699835] workqueue writeback: flags=0x4a [15144.702511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15144.704255] pending: wb_workfn, wb_workfn [15144.722199] workqueue xprtiod: flags=0xa [15144.728447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15144.752874] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15144.778701] workqueue ipv6_addrconf: flags=0x40008 [15144.781225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15144.784708] pending: addrconf_verify_work [15144.786951] workqueue knbd0-recv: flags=0x1a [15144.789087] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15144.792119] in-flight: 651:recv_work [15144.795048] workqueue ptlrpc_pinger: flags=0x2 [15144.799152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15144.804993] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15144.812246] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13227s workers=4 idle: 37 31846 28897 [15144.820070] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15134s workers=2 idle: 653 [15148.060897] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15148.066221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15148.091781] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15148.096836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15148.100939] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15148.104036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15148.106991] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15148.120064] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15148.126808] RAX: 0000000000007a8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15148.131919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15148.136367] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15148.140050] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15148.145081] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15148.149009] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15148.153749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15148.157173] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15148.161994] Call Trace: [15148.164290] [15148.165717] ? show_regs.cold.9+0x22/0x2f [15148.168790] ? watchdog_timer_fn.cold.6+0x62/0x10a [15148.172038] ? __hrtimer_run_queues+0x1c2/0x4d0 [15148.178612] ? watchdog+0x40/0x40 [15148.182176] ? hrtimer_interrupt+0x128/0x2c0 [15148.186168] ? smp_apic_timer_interrupt+0xa6/0x270 [15148.189759] ? apic_timer_interrupt+0xf/0x20 [15148.191190] [15148.192718] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15148.195727] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15148.199060] do_raw_spin_lock+0x177/0x190 [15148.200401] _raw_spin_lock+0x21/0x30 [15148.203302] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15148.207094] process_one_work+0x2c8/0x700 [15148.209575] worker_thread+0x296/0x6e0 [15148.212184] ? rescuer_thread+0x570/0x570 [15148.213687] kthread+0x1d1/0x200 [15148.216167] ? set_kthread_struct+0x70/0x70 [15148.218806] ret_from_fork+0x1f/0x30 [15168.059027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15168.060854] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15168.062934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15168.063014] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15168.063021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15168.063025] RIP: 0010:native_safe_halt+0x2e/0x50 [15168.063038] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15168.063042] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15168.063047] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15168.063049] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15168.063052] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15168.069821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [15168.110421] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15168.110431] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15168.110435] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15168.110438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15168.110441] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15168.110449] Call Trace: [15168.110455] [15168.110461] ? show_regs.cold.9+0x22/0x2f [15168.110473] ? watchdog_timer_fn.cold.6+0x62/0x10a [15168.110479] ? __hrtimer_run_queues+0x1c2/0x4d0 [15168.110484] ? watchdog+0x40/0x40 [15168.110488] ? hrtimer_interrupt+0x128/0x2c0 [15168.110492] ? smp_apic_timer_interrupt+0xa6/0x270 [15168.110501] ? apic_timer_interrupt+0xf/0x20 [15168.110506] [15168.110509] ? native_safe_halt+0x2e/0x50 [15168.110514] kvm_wait+0xcd/0xe0 [15168.110521] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15168.110530] do_raw_spin_lock+0x177/0x190 [15168.110535] _raw_spin_lock+0x21/0x30 [15168.110539] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15168.120933] sb_edac [15168.125944] ? do_raw_spin_unlock+0x75/0x190 [15168.128677] rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15168.128711] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15168.128719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15168.128722] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15168.128736] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15168.128740] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15168.128746] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15168.128748] RDX: 0000000000000aaa RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15168.128751] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15168.128753] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15168.128755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15168.128759] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15168.128762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15168.128764] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15168.128773] Call Trace: [15168.128779] [15168.128783] ? show_regs.cold.9+0x22/0x2f [15168.128793] ? watchdog_timer_fn.cold.6+0x62/0x10a [15168.128799] ? __hrtimer_run_queues+0x1c2/0x4d0 [15168.128804] ? watchdog+0x40/0x40 [15168.128808] ? hrtimer_interrupt+0x128/0x2c0 [15168.128811] ? smp_apic_timer_interrupt+0xa6/0x270 [15168.128819] ? apic_timer_interrupt+0xf/0x20 [15168.128824] [15168.128827] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15168.128832] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15168.128836] do_raw_spin_lock+0x177/0x190 [15168.128841] _raw_spin_lock+0x21/0x30 [15168.128848] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15168.129185] obd_mod_free+0x46/0x140 [ptlrpc] [15168.129462] mdc_commit_open+0x91/0xb0 [mdc] [15168.129498] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15168.129746] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15168.129941] after_reply+0xe2c/0x1780 [ptlrpc] [15168.130210] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15168.130476] ? __next_timer_interrupt+0x160/0x160 [15168.130486] ptlrpcd+0xa82/0xe40 [ptlrpc] [15168.130703] ? do_wait_intr+0xf0/0xf0 [15168.130713] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15168.130918] kthread+0x1d1/0x200 [15168.130924] ? set_kthread_struct+0x70/0x70 [15168.130928] ret_from_fork+0x1f/0x30 [15168.381028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15168.383545] ? do_wait_intr+0xf0/0xf0 [15168.385475] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15168.388469] mdc_close+0x53b/0xd70 [mdc] [15168.390711] lmv_close+0x18d/0x490 [lmv] [15168.392167] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15168.394940] ll_md_real_close+0xa5/0x2a0 [lustre] [15168.397953] ll_md_close+0x274/0x800 [lustre] [15168.400921] ll_file_release+0x128/0xc50 [lustre] [15168.403426] __fput+0x10d/0x440 [15168.404879] ____fput+0x12/0x20 [15168.406530] task_work_run+0xc6/0x110 [15168.408392] exit_to_usermode_loop+0x1dd/0x1f0 [15168.410876] do_syscall_64+0x3d6/0x440 [15168.413369] entry_SYSCALL_64_after_hwframe+0x49/0xae [15168.415139] RIP: 0033:0x7fa749552cd8 [15168.416761] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15168.425544] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15168.429339] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15168.432706] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15168.436618] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15168.440751] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15168.444470] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15172.054008] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15172.057170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15172.095736] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15172.105357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15172.113497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15172.118140] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15172.132359] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15172.136556] RAX: 000000000000648c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15172.144175] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15172.151464] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15172.155859] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15172.167738] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15172.172897] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15172.177923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15172.184383] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15172.191345] Call Trace: [15172.193821] [15172.195802] ? show_regs.cold.9+0x22/0x2f [15172.198365] ? watchdog_timer_fn.cold.6+0x62/0x10a [15172.200819] ? __hrtimer_run_queues+0x1c2/0x4d0 [15172.204576] ? watchdog+0x40/0x40 [15172.205670] ? hrtimer_interrupt+0x128/0x2c0 [15172.211719] ? smp_apic_timer_interrupt+0xa6/0x270 [15172.214075] ? apic_timer_interrupt+0xf/0x20 [15172.215518] [15172.217047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15172.219017] do_raw_spin_lock+0x177/0x190 [15172.220834] _raw_spin_lock+0x21/0x30 [15172.222801] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15172.225599] ? do_raw_spin_unlock+0x75/0x190 [15172.228081] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15172.230925] ? do_wait_intr+0xf0/0xf0 [15172.236808] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15172.243870] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15172.245813] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15172.249260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15172.258776] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15172.265175] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15172.268738] mdc_intent_lock+0x353/0x7e0 [mdc] [15172.272804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15172.275747] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15172.279033] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15172.281757] lmv_intent_open+0x338/0x1150 [lmv] [15172.285654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15172.291006] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15172.296344] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15172.299363] ll_intent_lock+0xc5/0xaa0 [lustre] [15172.302051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15172.305040] ll_lookup_it+0xe0e/0x2270 [lustre] [15172.306416] ? do_raw_spin_unlock+0x75/0x190 [15172.308131] ? _raw_spin_unlock+0x12/0x30 [15172.310677] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15172.314209] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15172.317749] ll_atomic_open+0x36a/0xd70 [lustre] [15172.322668] lookup_open+0x356/0xb40 [15172.326030] do_last+0x313/0xfc0 [15172.328158] ? nd_jump_root+0xe5/0x160 [15172.329718] ? path_init+0x437/0x520 [15172.332300] path_openat+0xf7/0x500 [15172.333906] ? do_raw_spin_unlock+0x75/0x190 [15172.338015] do_filp_open+0x99/0x140 [15172.340094] ? getname_flags+0x6e/0x330 [15172.342946] ? __check_object_size+0xff/0x256 [15172.349655] ? do_raw_spin_unlock+0x75/0x190 [15172.352294] ? _raw_spin_unlock+0x12/0x30 [15172.354864] do_sys_openat2+0x2b4/0x410 [15172.358433] do_sys_open+0x73/0xa0 [15172.360849] __x64_sys_openat+0x24/0x30 [15172.366304] do_syscall_64+0xc1/0x440 [15172.369411] entry_SYSCALL_64_after_hwframe+0x49/0xae [15172.371444] RIP: 0033:0x7fa749552332 [15172.372687] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15172.382232] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15172.385675] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15172.389288] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15172.392784] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15172.396641] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15172.402268] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15175.139076] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13257s! [15175.155221] Showing busy workqueues and worker pools: [15175.159727] workqueue events: flags=0x0 [15175.167041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15175.172728] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15175.186186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15175.190021] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15175.196429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.199755] pending: kfree_rcu_monitor [15175.201767] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15175.203754] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15175.217056] workqueue events_long: flags=0x0 [15175.219646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.223417] pending: lu_objects_flush [obdclass] [15175.226104] workqueue events_unbound: flags=0x2 [15175.228129] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15175.232360] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15175.237141] workqueue events_power_efficient: flags=0x80 [15175.241651] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.245266] pending: check_lifetime [15175.247132] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.250052] pending: neigh_periodic_work [15175.251784] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.254752] pending: neigh_periodic_work [15175.256703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.259492] pending: do_cache_clean [15175.261283] workqueue mm_percpu_wq: flags=0x8 [15175.263228] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.266422] pending: vmstat_update [15175.268527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.271455] pending: vmstat_update [15175.272945] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15175.275662] pending: vmstat_update [15175.277142] workqueue writeback: flags=0x4a [15175.278830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15175.281809] pending: wb_workfn, wb_workfn [15175.284083] workqueue xprtiod: flags=0xa [15175.286378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15175.289507] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15175.293440] workqueue ipv6_addrconf: flags=0x40008 [15175.295664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15175.299105] pending: addrconf_verify_work [15175.301455] workqueue knbd0-recv: flags=0x1a [15175.303389] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15175.306602] in-flight: 651:recv_work [15175.308369] workqueue ptlrpc_pinger: flags=0x2 [15175.310223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15175.312847] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15175.315685] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13258s workers=4 idle: 37 31846 28897 [15175.320020] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15165s workers=2 idle: 653 [15176.057465] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15176.061648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15176.098538] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15176.106909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15176.110106] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15176.115528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15176.122515] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15176.132839] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15176.137419] RAX: 0000000000007595 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15176.140675] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15176.144352] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15176.147538] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15176.152287] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15176.155493] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15176.160438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15176.165228] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15176.171604] Call Trace: [15176.172620] [15176.173730] ? show_regs.cold.9+0x22/0x2f [15176.178754] ? watchdog_timer_fn.cold.6+0x62/0x10a [15176.180736] ? __hrtimer_run_queues+0x1c2/0x4d0 [15176.184320] ? watchdog+0x40/0x40 [15176.187183] ? hrtimer_interrupt+0x128/0x2c0 [15176.191522] ? smp_apic_timer_interrupt+0xa6/0x270 [15176.197301] ? apic_timer_interrupt+0xf/0x20 [15176.198644] [15176.201338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15176.204038] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15176.205471] do_raw_spin_lock+0x177/0x190 [15176.208500] _raw_spin_lock+0x21/0x30 [15176.209607] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15176.211938] process_one_work+0x2c8/0x700 [15176.214060] worker_thread+0x296/0x6e0 [15176.215332] ? rescuer_thread+0x570/0x570 [15176.216310] kthread+0x1d1/0x200 [15176.217120] ? set_kthread_struct+0x70/0x70 [15176.218100] ret_from_fork+0x1f/0x30 [15196.058038] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15196.059026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15196.062223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [15196.067767] Modules linked in: [15196.067771] mdt(O) mdd(O) [15196.070858] lustre(O) [15196.072567] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15196.072632] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15196.072638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15196.072641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15196.072655] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15196.072660] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15196.072664] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15196.072667] RDX: 00000000000024a0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15196.072669] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15196.072672] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15196.072674] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15196.072677] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15196.072680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15196.072683] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15196.072692] Call Trace: [15196.072698] [15196.072703] ? show_regs.cold.9+0x22/0x2f [15196.072713] ? watchdog_timer_fn.cold.6+0x62/0x10a [15196.072719] ? __hrtimer_run_queues+0x1c2/0x4d0 [15196.072724] ? watchdog+0x40/0x40 [15196.072728] ? hrtimer_interrupt+0x128/0x2c0 [15196.072732] ? smp_apic_timer_interrupt+0xa6/0x270 [15196.072740] ? apic_timer_interrupt+0xf/0x20 [15196.072745] [15196.072748] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15196.072754] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15196.072759] do_raw_spin_lock+0x177/0x190 [15196.072764] _raw_spin_lock+0x21/0x30 [15196.072769] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15196.074603] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15196.074668] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15196.074676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15196.074678] RIP: 0010:native_safe_halt+0x2e/0x50 [15196.074688] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15196.074693] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15196.074697] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15196.074699] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15196.074700] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15196.074702] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15196.074704] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15196.074707] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15196.074709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15196.074711] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15196.074719] Call Trace: [15196.074724] [15196.074728] ? show_regs.cold.9+0x22/0x2f [15196.074736] ? watchdog_timer_fn.cold.6+0x62/0x10a [15196.074742] ? __hrtimer_run_queues+0x1c2/0x4d0 [15196.074745] ? watchdog+0x40/0x40 [15196.074749] ? hrtimer_interrupt+0x128/0x2c0 [15196.074753] ? smp_apic_timer_interrupt+0xa6/0x270 [15196.074759] ? apic_timer_interrupt+0xf/0x20 [15196.074764] [15196.074767] ? native_safe_halt+0x2e/0x50 [15196.074771] kvm_wait+0xcd/0xe0 [15196.074779] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15196.074797] do_raw_spin_lock+0x177/0x190 [15196.074802] _raw_spin_lock+0x21/0x30 [15196.074807] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15196.075122] ? do_raw_spin_unlock+0x75/0x190 [15196.075128] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15196.076773] obd_mod_free+0x46/0x140 [ptlrpc] [15196.077320] mdc_commit_open+0x91/0xb0 [mdc] [15196.100665] ? do_wait_intr+0xf0/0xf0 [15196.110341] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15196.117438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15196.120870] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15196.121247] after_reply+0xe2c/0x1780 [ptlrpc] [15196.121461] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15196.122455] ? __next_timer_interrupt+0x160/0x160 [15196.122467] ptlrpcd+0xa82/0xe40 [ptlrpc] [15196.123271] ? do_wait_intr+0xf0/0xf0 [15196.123280] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15196.123484] kthread+0x1d1/0x200 [15196.134329] mdc_close+0x53b/0xd70 [mdc] [15196.134381] lmv_close+0x18d/0x490 [lmv] [15196.134411] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15196.145944] ? set_kthread_struct+0x70/0x70 [15196.145956] ret_from_fork+0x1f/0x30 [15196.374215] ll_md_real_close+0xa5/0x2a0 [lustre] [15196.376333] ll_md_close+0x274/0x800 [lustre] [15196.378756] ll_file_release+0x128/0xc50 [lustre] [15196.380442] __fput+0x10d/0x440 [15196.381635] ____fput+0x12/0x20 [15196.384527] task_work_run+0xc6/0x110 [15196.385472] exit_to_usermode_loop+0x1dd/0x1f0 [15196.390927] do_syscall_64+0x3d6/0x440 [15196.395132] entry_SYSCALL_64_after_hwframe+0x49/0xae [15196.397879] RIP: 0033:0x7fa749552cd8 [15196.399722] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15196.410670] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15196.413709] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15196.419228] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15196.425890] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15196.430184] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15196.433821] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15200.054984] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15200.057958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15200.073156] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15200.076565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15200.079337] RIP: 0010:native_safe_halt+0x2e/0x50 [15200.080619] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15200.086077] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15200.088103] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15200.090095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15200.092200] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15200.094695] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15200.097181] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15200.099958] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15200.101849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15200.103264] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15200.105124] Call Trace: [15200.105879] [15200.106379] ? show_regs.cold.9+0x22/0x2f [15200.107683] ? watchdog_timer_fn.cold.6+0x62/0x10a [15200.109030] ? __hrtimer_run_queues+0x1c2/0x4d0 [15200.110212] ? watchdog+0x40/0x40 [15200.111108] ? hrtimer_interrupt+0x128/0x2c0 [15200.112253] ? smp_apic_timer_interrupt+0xa6/0x270 [15200.113636] ? apic_timer_interrupt+0xf/0x20 [15200.115216] [15200.115946] ? native_safe_halt+0x2e/0x50 [15200.117105] kvm_wait+0xcd/0xe0 [15200.117996] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15200.119404] do_raw_spin_lock+0x177/0x190 [15200.120551] _raw_spin_lock+0x21/0x30 [15200.122395] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15200.123885] ? do_raw_spin_unlock+0x75/0x190 [15200.125591] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15200.127235] ? do_wait_intr+0xf0/0xf0 [15200.128504] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15200.130128] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15200.131814] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15200.133957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15200.135527] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15200.137034] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15200.139978] mdc_intent_lock+0x353/0x7e0 [mdc] [15200.142044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15200.145266] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15200.147899] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15200.149221] lmv_intent_open+0x338/0x1150 [lmv] [15200.150983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15200.152960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15200.154774] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15200.155839] ll_intent_lock+0xc5/0xaa0 [lustre] [15200.156965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15200.159682] ll_lookup_it+0xe0e/0x2270 [lustre] [15200.161369] ? do_raw_spin_unlock+0x75/0x190 [15200.162929] ? _raw_spin_unlock+0x12/0x30 [15200.164375] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15200.166411] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15200.168175] ll_atomic_open+0x36a/0xd70 [lustre] [15200.169789] lookup_open+0x356/0xb40 [15200.170887] do_last+0x313/0xfc0 [15200.172056] ? nd_jump_root+0xe5/0x160 [15200.173281] ? path_init+0x437/0x520 [15200.174421] path_openat+0xf7/0x500 [15200.175531] ? do_raw_spin_unlock+0x75/0x190 [15200.177107] do_filp_open+0x99/0x140 [15200.178901] ? getname_flags+0x6e/0x330 [15200.180933] ? __check_object_size+0xff/0x256 [15200.183547] ? do_raw_spin_unlock+0x75/0x190 [15200.186541] ? _raw_spin_unlock+0x12/0x30 [15200.188906] do_sys_openat2+0x2b4/0x410 [15200.191398] do_sys_open+0x73/0xa0 [15200.193785] __x64_sys_openat+0x24/0x30 [15200.194861] do_syscall_64+0xc1/0x440 [15200.196118] entry_SYSCALL_64_after_hwframe+0x49/0xae [15200.197931] RIP: 0033:0x7fa749552332 [15200.199360] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15200.206299] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15200.208982] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15200.211508] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15200.214181] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15200.218210] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15200.220584] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15204.058087] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15204.062092] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15204.118761] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15204.135844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15204.147750] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15204.152449] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15204.160145] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15204.180595] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15204.192744] RAX: 0000000000007f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15204.196706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15204.202231] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15204.211390] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15204.213892] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15204.218178] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15204.222806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15204.229110] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15204.236612] Call Trace: [15204.238947] [15204.240237] ? show_regs.cold.9+0x22/0x2f [15204.242791] ? watchdog_timer_fn.cold.6+0x62/0x10a [15204.246164] ? __hrtimer_run_queues+0x1c2/0x4d0 [15204.249161] ? watchdog+0x40/0x40 [15204.250691] ? hrtimer_interrupt+0x128/0x2c0 [15204.253573] ? smp_apic_timer_interrupt+0xa6/0x270 [15204.256774] ? apic_timer_interrupt+0xf/0x20 [15204.262140] [15204.264118] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15204.270342] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15204.273224] do_raw_spin_lock+0x177/0x190 [15204.275926] _raw_spin_lock+0x21/0x30 [15204.278240] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15204.288601] process_one_work+0x2c8/0x700 [15204.290638] worker_thread+0x296/0x6e0 [15204.292587] ? rescuer_thread+0x570/0x570 [15204.296295] kthread+0x1d1/0x200 [15204.298466] ? set_kthread_struct+0x70/0x70 [15204.301282] ret_from_fork+0x1f/0x30 [15205.345032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13288s! [15205.349293] Showing busy workqueues and worker pools: [15205.351908] workqueue events: flags=0x0 [15205.353963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15205.357594] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15205.364382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15205.367245] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15205.372458] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.375618] pending: kfree_rcu_monitor [15205.376945] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15205.380538] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15205.385109] workqueue events_long: flags=0x0 [15205.387605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.390693] pending: lu_objects_flush [obdclass] [15205.393700] workqueue events_unbound: flags=0x2 [15205.396014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15205.399361] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15205.404275] workqueue events_power_efficient: flags=0x80 [15205.406934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.410453] pending: check_lifetime [15205.412659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.417149] pending: neigh_periodic_work [15205.419629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.423255] pending: neigh_periodic_work [15205.425637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.428673] pending: do_cache_clean [15205.431054] workqueue mm_percpu_wq: flags=0x8 [15205.433055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.436277] pending: vmstat_update [15205.438263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.441824] pending: vmstat_update [15205.443570] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15205.446466] pending: vmstat_update [15205.448092] workqueue writeback: flags=0x4a [15205.449973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15205.452852] pending: wb_workfn, wb_workfn [15205.454478] workqueue xprtiod: flags=0xa [15205.455940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15205.458805] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15205.461550] workqueue ipv6_addrconf: flags=0x40008 [15205.463116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15205.465476] pending: addrconf_verify_work [15205.467657] workqueue knbd0-recv: flags=0x1a [15205.469559] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15205.472236] in-flight: 651:recv_work [15205.473589] workqueue ptlrpc_pinger: flags=0x2 [15205.475037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15205.477062] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15205.479765] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13288s workers=4 idle: 37 31846 28897 [15205.483609] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15195s workers=2 idle: 653 [15224.058015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15224.059023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15224.062104] Modules linked in: lustre(O) [15224.065190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15224.065256] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15224.065263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15224.065266] RIP: 0010:native_safe_halt+0x2e/0x50 [15224.065283] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15224.065287] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15224.065290] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15224.065303] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15224.065306] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15224.065307] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15224.065309] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15224.065312] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15224.065314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15224.065316] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15224.065323] Call Trace: [15224.065328] [15224.065335] ? show_regs.cold.9+0x22/0x2f [15224.065344] ? watchdog_timer_fn.cold.6+0x62/0x10a [15224.065350] ? __hrtimer_run_queues+0x1c2/0x4d0 [15224.065353] ? watchdog+0x40/0x40 [15224.065356] ? hrtimer_interrupt+0x128/0x2c0 [15224.065358] ? smp_apic_timer_interrupt+0xa6/0x270 [15224.065365] ? apic_timer_interrupt+0xf/0x20 [15224.065368] [15224.065370] ? native_safe_halt+0x2e/0x50 [15224.065372] kvm_wait+0xcd/0xe0 [15224.065380] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15224.065386] do_raw_spin_lock+0x177/0x190 [15224.065389] _raw_spin_lock+0x21/0x30 [15224.065392] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15224.067548] osp(O) [15224.120336] ? do_raw_spin_unlock+0x75/0x190 [15224.126465] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15224.126519] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15224.126526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15224.126528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15224.126541] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15224.126547] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 [15224.131565] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15224.135210] ORIG_RAX: ffffffffffffff13 [15224.144715] ? do_wait_intr+0xf0/0xf0 [15224.149361] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15224.154540] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15224.158978] RDX: 000000000000061c RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15224.158993] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15224.158996] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15224.158998] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15224.159002] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15224.163416] mdc_close+0x53b/0xd70 [mdc] [15224.168173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15224.172029] lmv_close+0x18d/0x490 [lmv] [15224.176879] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15224.183425] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15224.187850] Call Trace: [15224.190161] ll_md_real_close+0xa5/0x2a0 [lustre] [15224.191301] [15224.193700] ll_md_close+0x274/0x800 [lustre] [15224.196778] ? show_regs.cold.9+0x22/0x2f [15224.198383] ll_file_release+0x128/0xc50 [lustre] [15224.199738] ? watchdog_timer_fn.cold.6+0x62/0x10a [15224.201272] __fput+0x10d/0x440 [15224.203560] ? __hrtimer_run_queues+0x1c2/0x4d0 [15224.209641] ____fput+0x12/0x20 [15224.209676] task_work_run+0xc6/0x110 [15224.209689] exit_to_usermode_loop+0x1dd/0x1f0 [15224.209699] do_syscall_64+0x3d6/0x440 [15224.209705] entry_SYSCALL_64_after_hwframe+0x49/0xae [15224.209717] RIP: 0033:0x7fa749552cd8 [15224.209729] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15224.209734] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15224.209740] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15224.209743] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15224.209746] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15224.209748] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15224.209751] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15224.389760] ? watchdog+0x40/0x40 [15224.390822] ? hrtimer_interrupt+0x128/0x2c0 [15224.393187] ? smp_apic_timer_interrupt+0xa6/0x270 [15224.395828] ? apic_timer_interrupt+0xf/0x20 [15224.398670] [15224.400739] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15224.404528] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15224.408309] do_raw_spin_lock+0x177/0x190 [15224.410066] _raw_spin_lock+0x21/0x30 [15224.411846] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15224.413780] obd_mod_free+0x46/0x140 [ptlrpc] [15224.415705] mdc_commit_open+0x91/0xb0 [mdc] [15224.418054] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15224.420870] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15224.424506] after_reply+0xe2c/0x1780 [ptlrpc] [15224.426717] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15224.430074] ? __next_timer_interrupt+0x160/0x160 [15224.433096] ptlrpcd+0xa82/0xe40 [ptlrpc] [15224.436318] ? do_wait_intr+0xf0/0xf0 [15224.438132] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15224.440782] kthread+0x1d1/0x200 [15224.442398] ? set_kthread_struct+0x70/0x70 [15224.444539] ret_from_fork+0x1f/0x30 [15232.054579] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15232.057037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15232.061106] Modules linked in: lustre(O) [15232.065094] Modules linked in: lustre(O) osp(O) ofd(O) [15232.066468] osp(O) [15232.066470] lod(O) mdt(O) [15232.068579] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [15232.069464] mdd(O) [15232.070897] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15232.070914] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15232.070921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15232.070925] RIP: 0010:native_safe_halt+0x2e/0x50 [15232.070936] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15232.070940] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15232.070944] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15232.070946] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15232.070948] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15232.070952] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15232.086185] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15232.086250] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15232.087260] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15232.090352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15232.095257] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15232.095267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15232.095270] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15232.095280] Call Trace: [15232.098292] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15232.100260] [15232.107836] [15232.111454] ? show_regs.cold.9+0x22/0x2f [15232.111474] ? watchdog_timer_fn.cold.6+0x62/0x10a [15232.111481] ? __hrtimer_run_queues+0x1c2/0x4d0 [15232.111485] ? watchdog+0x40/0x40 [15232.114370] RIP: 0010:native_safe_halt+0x2e/0x50 [15232.117460] ? hrtimer_interrupt+0x128/0x2c0 [15232.120376] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15232.123516] ? smp_apic_timer_interrupt+0xa6/0x270 [15232.141024] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 [15232.146408] ? apic_timer_interrupt+0xf/0x20 [15232.146437] [15232.146442] ? native_safe_halt+0x2e/0x50 [15232.146447] kvm_wait+0xcd/0xe0 [15232.146459] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15232.146474] do_raw_spin_lock+0x177/0x190 [15232.146479] _raw_spin_lock+0x21/0x30 [15232.146487] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15232.146807] ? do_raw_spin_unlock+0x75/0x190 [15232.146817] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15232.147121] ? do_wait_intr+0xf0/0xf0 [15232.147130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15232.147352] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15232.147630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15232.147855] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15232.148197] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15232.148236] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15232.148272] mdc_intent_lock+0x353/0x7e0 [mdc] [15232.148300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15232.148405] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15232.148796] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15232.148834] lmv_intent_open+0x338/0x1150 [lmv] [15232.148936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15232.149036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15232.149124] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15232.149147] ll_intent_lock+0xc5/0xaa0 [lustre] [15232.149247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15232.149335] ll_lookup_it+0xe0e/0x2270 [lustre] [15232.149502] ? do_raw_spin_unlock+0x75/0x190 [15232.149509] ? _raw_spin_unlock+0x12/0x30 [15232.152566] ORIG_RAX: ffffffffffffff13 [15232.152572] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15232.152633] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15232.152636] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15232.152638] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15232.152641] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15232.152644] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15232.152647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15232.152650] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15232.152657] Call Trace: [15232.152668] [15232.152673] ? show_regs.cold.9+0x22/0x2f [15232.152685] ? watchdog_timer_fn.cold.6+0x62/0x10a [15232.152690] ? __hrtimer_run_queues+0x1c2/0x4d0 [15232.152694] ? watchdog+0x40/0x40 [15232.152702] ? hrtimer_interrupt+0x128/0x2c0 [15232.152707] ? smp_apic_timer_interrupt+0xa6/0x270 [15232.152715] ? apic_timer_interrupt+0xf/0x20 [15232.152720] [15232.152727] ? native_safe_halt+0x2e/0x50 [15232.152732] kvm_wait+0xcd/0xe0 [15232.152739] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15232.152748] do_raw_spin_lock+0x177/0x190 [15232.152753] _raw_spin_lock+0x21/0x30 [15232.152757] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15232.153129] process_one_work+0x2c8/0x700 [15232.153137] worker_thread+0x296/0x6e0 [15232.153142] ? rescuer_thread+0x570/0x570 [15232.153146] kthread+0x1d1/0x200 [15232.153161] ? set_kthread_struct+0x70/0x70 [15232.153165] ret_from_fork+0x1f/0x30 [15232.331650] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15232.334430] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15232.336765] ll_atomic_open+0x36a/0xd70 [lustre] [15232.340179] lookup_open+0x356/0xb40 [15232.341616] do_last+0x313/0xfc0 [15232.343488] ? nd_jump_root+0xe5/0x160 [15232.346048] ? path_init+0x437/0x520 [15232.349157] path_openat+0xf7/0x500 [15232.352162] ? do_raw_spin_unlock+0x75/0x190 [15232.356112] do_filp_open+0x99/0x140 [15232.359535] ? getname_flags+0x6e/0x330 [15232.362375] ? __check_object_size+0xff/0x256 [15232.363736] ? do_raw_spin_unlock+0x75/0x190 [15232.366257] ? _raw_spin_unlock+0x12/0x30 [15232.368482] do_sys_openat2+0x2b4/0x410 [15232.370707] do_sys_open+0x73/0xa0 [15232.372684] __x64_sys_openat+0x24/0x30 [15232.376078] do_syscall_64+0xc1/0x440 [15232.378941] entry_SYSCALL_64_after_hwframe+0x49/0xae [15232.384485] RIP: 0033:0x7fa749552332 [15232.385737] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15232.397815] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15232.401807] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15232.416554] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15232.428261] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15232.436383] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15232.443521] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15235.553670] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13318s! [15235.556466] Showing busy workqueues and worker pools: [15235.562157] workqueue events: flags=0x0 [15235.566846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15235.568831] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15235.579690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15235.586769] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15235.591815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.596506] pending: kfree_rcu_monitor [15235.600282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15235.603119] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15235.610733] workqueue events_long: flags=0x0 [15235.612634] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.616311] pending: lu_objects_flush [obdclass] [15235.621727] workqueue events_unbound: flags=0x2 [15235.623130] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15235.625110] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15235.627625] workqueue events_power_efficient: flags=0x80 [15235.629606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.631528] pending: check_lifetime [15235.632747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.634786] pending: neigh_periodic_work [15235.636081] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.637929] pending: neigh_periodic_work [15235.639159] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.640955] pending: do_cache_clean [15235.641956] workqueue mm_percpu_wq: flags=0x8 [15235.643167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.645065] pending: vmstat_update [15235.646038] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.647981] pending: vmstat_update [15235.649038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15235.651021] pending: vmstat_update [15235.652033] workqueue writeback: flags=0x4a [15235.653180] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15235.654884] pending: wb_workfn, wb_workfn [15235.656079] workqueue xprtiod: flags=0xa [15235.657193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15235.658888] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15235.661017] workqueue ipv6_addrconf: flags=0x40008 [15235.662753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15235.665526] pending: addrconf_verify_work [15235.667179] workqueue knbd0-recv: flags=0x1a [15235.669180] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15235.671199] in-flight: 651:recv_work [15235.672772] workqueue ptlrpc_pinger: flags=0x2 [15235.674612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15235.677048] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15235.679168] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13318s workers=4 idle: 37 31846 28897 [15235.682517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15225s workers=2 idle: 653 [15252.058606] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15252.059260] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15252.063088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [15252.068371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15252.068503] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15252.068511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15252.068515] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15252.068539] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15252.068544] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15252.068548] RAX: 0000000000002ebd RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15252.068551] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15252.068554] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15252.068556] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15252.068559] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15252.068563] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15252.068565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15252.068568] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15252.068885] Call Trace: [15252.074818] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15252.074892] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15252.074901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15252.074903] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15252.074915] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15252.074919] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15252.074923] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15252.074926] RDX: 0000000000007322 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15252.074929] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15252.074931] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15252.074933] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15252.074936] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15252.074939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15252.074942] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15252.074951] Call Trace: [15252.074998] [15252.075005] ? show_regs.cold.9+0x22/0x2f [15252.075015] ? watchdog_timer_fn.cold.6+0x62/0x10a [15252.075021] ? __hrtimer_run_queues+0x1c2/0x4d0 [15252.075025] ? watchdog+0x40/0x40 [15252.075029] ? hrtimer_interrupt+0x128/0x2c0 [15252.075033] ? smp_apic_timer_interrupt+0xa6/0x270 [15252.075040] ? apic_timer_interrupt+0xf/0x20 [15252.075045] [15252.075048] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15252.075052] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15252.075057] do_raw_spin_lock+0x177/0x190 [15252.075062] _raw_spin_lock+0x21/0x30 [15252.075068] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15252.075398] obd_mod_free+0x46/0x140 [ptlrpc] [15252.075595] mdc_commit_open+0x91/0xb0 [mdc] [15252.075633] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15252.075841] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15252.119842] [15252.119855] ? show_regs.cold.9+0x22/0x2f [15252.124169] after_reply+0xe2c/0x1780 [ptlrpc] [15252.124462] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15252.124662] ? __next_timer_interrupt+0x160/0x160 [15252.124672] ptlrpcd+0xa82/0xe40 [ptlrpc] [15252.124891] ? do_wait_intr+0xf0/0xf0 [15252.124902] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15252.125115] kthread+0x1d1/0x200 [15252.125122] ? set_kthread_struct+0x70/0x70 [15252.125126] ret_from_fork+0x1f/0x30 [15252.370110] ? watchdog_timer_fn.cold.6+0x62/0x10a [15252.372330] ? __hrtimer_run_queues+0x1c2/0x4d0 [15252.375368] ? watchdog+0x40/0x40 [15252.376977] ? hrtimer_interrupt+0x128/0x2c0 [15252.378587] ? smp_apic_timer_interrupt+0xa6/0x270 [15252.380300] ? apic_timer_interrupt+0xf/0x20 [15252.382095] [15252.382952] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15252.384858] do_raw_spin_lock+0x177/0x190 [15252.386284] _raw_spin_lock+0x21/0x30 [15252.388139] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15252.390292] ? do_raw_spin_unlock+0x75/0x190 [15252.391896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15252.393998] ? do_wait_intr+0xf0/0xf0 [15252.395349] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15252.397424] mdc_close+0x53b/0xd70 [mdc] [15252.398973] lmv_close+0x18d/0x490 [lmv] [15252.400755] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15252.402865] ll_md_real_close+0xa5/0x2a0 [lustre] [15252.404776] ll_md_close+0x274/0x800 [lustre] [15252.407822] ll_file_release+0x128/0xc50 [lustre] [15252.409938] __fput+0x10d/0x440 [15252.411095] ____fput+0x12/0x20 [15252.412247] task_work_run+0xc6/0x110 [15252.413671] exit_to_usermode_loop+0x1dd/0x1f0 [15252.415258] do_syscall_64+0x3d6/0x440 [15252.416734] entry_SYSCALL_64_after_hwframe+0x49/0xae [15252.418422] RIP: 0033:0x7fa749552cd8 [15252.419862] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15252.428580] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15252.438872] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15252.447104] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15252.456268] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15252.460912] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15252.464388] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15260.054033] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15260.057020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15260.058862] Modules linked in: [15260.063019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15260.063118] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15260.063129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15260.063136] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15260.063866] RIP: 0010:native_safe_halt+0x2e/0x50 [15260.063878] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15260.063882] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15260.063887] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15260.063890] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15260.063892] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15260.063894] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15260.063896] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15260.063900] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15260.063902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15260.063904] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15260.063913] Call Trace: [15260.063919] [15260.063924] ? show_regs.cold.9+0x22/0x2f [15260.063934] ? watchdog_timer_fn.cold.6+0x62/0x10a [15260.063942] ? __hrtimer_run_queues+0x1c2/0x4d0 [15260.063946] ? watchdog+0x40/0x40 [15260.063951] ? hrtimer_interrupt+0x128/0x2c0 [15260.063954] ? smp_apic_timer_interrupt+0xa6/0x270 [15260.063962] ? apic_timer_interrupt+0xf/0x20 [15260.063966] [15260.063969] ? native_safe_halt+0x2e/0x50 [15260.063973] kvm_wait+0xcd/0xe0 [15260.063981] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15260.063990] do_raw_spin_lock+0x177/0x190 [15260.063995] _raw_spin_lock+0x21/0x30 [15260.063999] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15260.066170] process_one_work+0x2c8/0x700 [15260.066185] worker_thread+0x296/0x6e0 [15260.068459] ? rescuer_thread+0x570/0x570 [15260.074103] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15260.074183] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15260.074203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15260.074207] RIP: 0010:native_safe_halt+0x2e/0x50 [15260.074221] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15260.074225] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15260.074230] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15260.074232] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15260.074234] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15260.074237] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15260.074239] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15260.074243] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15260.074246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15260.074248] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15260.074258] Call Trace: [15260.074264] [15260.074270] ? show_regs.cold.9+0x22/0x2f [15260.074280] ? watchdog_timer_fn.cold.6+0x62/0x10a [15260.074286] ? __hrtimer_run_queues+0x1c2/0x4d0 [15260.074292] ? watchdog+0x40/0x40 [15260.074296] ? hrtimer_interrupt+0x128/0x2c0 [15260.074300] ? smp_apic_timer_interrupt+0xa6/0x270 [15260.074306] ? apic_timer_interrupt+0xf/0x20 [15260.074310] [15260.074313] ? native_safe_halt+0x2e/0x50 [15260.074317] kvm_wait+0xcd/0xe0 [15260.074325] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15260.074335] do_raw_spin_lock+0x177/0x190 [15260.074340] _raw_spin_lock+0x21/0x30 [15260.074345] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15260.076332] ? do_raw_spin_unlock+0x75/0x190 [15260.076343] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15260.077315] ? do_wait_intr+0xf0/0xf0 [15260.077325] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15260.078622] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15260.078938] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15260.079162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15260.080018] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15260.080073] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15260.080106] mdc_intent_lock+0x353/0x7e0 [mdc] [15260.080138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15260.080891] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15260.119239] kthread+0x1d1/0x200 [15260.119260] ? set_kthread_struct+0x70/0x70 [15260.119265] ret_from_fork+0x1f/0x30 [15260.519717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15260.521475] lmv_intent_open+0x338/0x1150 [lmv] [15260.525162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15260.528795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15260.536582] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15260.541588] ll_intent_lock+0xc5/0xaa0 [lustre] [15260.547730] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15260.554568] ll_lookup_it+0xe0e/0x2270 [lustre] [15260.558053] ? do_raw_spin_unlock+0x75/0x190 [15260.560086] ? _raw_spin_unlock+0x12/0x30 [15260.565150] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15260.575334] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15260.578806] ll_atomic_open+0x36a/0xd70 [lustre] [15260.580512] lookup_open+0x356/0xb40 [15260.596710] do_last+0x313/0xfc0 [15260.601164] ? nd_jump_root+0xe5/0x160 [15260.603087] ? path_init+0x437/0x520 [15260.609131] path_openat+0xf7/0x500 [15260.611054] ? do_raw_spin_unlock+0x75/0x190 [15260.617003] do_filp_open+0x99/0x140 [15260.624677] ? getname_flags+0x6e/0x330 [15260.630907] ? __check_object_size+0xff/0x256 [15260.635073] ? do_raw_spin_unlock+0x75/0x190 [15260.641151] ? _raw_spin_unlock+0x12/0x30 [15260.648469] do_sys_openat2+0x2b4/0x410 [15260.649462] do_sys_open+0x73/0xa0 [15260.650259] __x64_sys_openat+0x24/0x30 [15260.653775] do_syscall_64+0xc1/0x440 [15260.662310] entry_SYSCALL_64_after_hwframe+0x49/0xae [15260.676275] RIP: 0033:0x7fa749552332 [15260.677516] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15260.700663] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15260.705123] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15260.716582] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15260.719985] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15260.725668] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15260.730534] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15265.763068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13348s! [15265.767274] Showing busy workqueues and worker pools: [15265.769088] workqueue events: flags=0x0 [15265.773353] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15265.778966] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15265.787270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15265.792212] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15265.799190] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.802844] pending: kfree_rcu_monitor [15265.806147] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15265.811104] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15265.820618] workqueue events_long: flags=0x0 [15265.823783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.827047] pending: lu_objects_flush [obdclass] [15265.829681] workqueue events_unbound: flags=0x2 [15265.831370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15265.838576] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15265.845556] workqueue events_power_efficient: flags=0x80 [15265.850175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.855731] pending: check_lifetime [15265.857859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.861034] pending: neigh_periodic_work [15265.864017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.866367] pending: neigh_periodic_work [15265.870787] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.876745] pending: do_cache_clean [15265.879452] workqueue mm_percpu_wq: flags=0x8 [15265.881675] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.885671] pending: vmstat_update [15265.889140] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.893213] pending: vmstat_update [15265.895401] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15265.900483] pending: vmstat_update [15265.901871] workqueue writeback: flags=0x4a [15265.903357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15265.908013] pending: wb_workfn, wb_workfn [15265.910465] workqueue xprtiod: flags=0xa [15265.913367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15265.920221] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15265.925664] workqueue ipv6_addrconf: flags=0x40008 [15265.928162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15265.929744] pending: addrconf_verify_work [15265.931598] workqueue knbd0-recv: flags=0x1a [15265.932922] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15265.936403] in-flight: 651:recv_work [15265.939118] workqueue ptlrpc_pinger: flags=0x2 [15265.943849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15265.946960] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15265.951356] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13348s workers=4 idle: 37 31846 28897 [15265.956166] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15255s workers=2 idle: 653 [15280.058219] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15280.060766] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15280.064757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15280.064864] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15280.064956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15280.064959] RIP: 0010:native_safe_halt+0x2e/0x50 [15280.064975] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15280.064980] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15280.064985] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15280.064987] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15280.064990] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15280.064992] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15280.064995] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15280.064998] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15280.065001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15280.065004] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15280.065011] Call Trace: [15280.065020] [15280.065077] ? show_regs.cold.9+0x22/0x2f [15280.065088] ? watchdog_timer_fn.cold.6+0x62/0x10a [15280.065095] ? __hrtimer_run_queues+0x1c2/0x4d0 [15280.065099] ? watchdog+0x40/0x40 [15280.065104] ? hrtimer_interrupt+0x128/0x2c0 [15280.065108] ? smp_apic_timer_interrupt+0xa6/0x270 [15280.065114] ? apic_timer_interrupt+0xf/0x20 [15280.065119] [15280.065122] ? native_safe_halt+0x2e/0x50 [15280.065126] kvm_wait+0xcd/0xe0 [15280.065137] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15280.065153] do_raw_spin_lock+0x177/0x190 [15280.065157] _raw_spin_lock+0x21/0x30 [15280.065162] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15280.065579] obd_mod_free+0x46/0x140 [ptlrpc] [15280.069970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [15280.102670] mdc_commit_open+0x91/0xb0 [mdc] [15280.102720] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15280.103025] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15280.103470] after_reply+0xe2c/0x1780 [ptlrpc] [15280.103804] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15280.104367] ? __next_timer_interrupt+0x160/0x160 [15280.104379] ptlrpcd+0xa82/0xe40 [ptlrpc] [15280.104648] ? do_wait_intr+0xf0/0xf0 [15280.104662] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15280.106945] kthread+0x1d1/0x200 [15280.106958] ? set_kthread_struct+0x70/0x70 [15280.106963] ret_from_fork+0x1f/0x30 [15280.227747] sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15280.234584] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15280.238223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15280.240596] RIP: 0010:native_safe_halt+0x2e/0x50 [15280.242141] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15280.247601] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15280.249751] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15280.251686] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15280.253687] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15280.255553] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15280.257450] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15280.259434] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15280.261643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15280.263130] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15280.265117] Call Trace: [15280.265790] [15280.266330] ? show_regs.cold.9+0x22/0x2f [15280.267428] ? watchdog_timer_fn.cold.6+0x62/0x10a [15280.268762] ? __hrtimer_run_queues+0x1c2/0x4d0 [15280.270184] ? watchdog+0x40/0x40 [15280.271446] ? hrtimer_interrupt+0x128/0x2c0 [15280.273179] ? smp_apic_timer_interrupt+0xa6/0x270 [15280.274972] ? apic_timer_interrupt+0xf/0x20 [15280.276683] [15280.277556] ? native_safe_halt+0x2e/0x50 [15280.278915] kvm_wait+0xcd/0xe0 [15280.280204] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15280.282241] do_raw_spin_lock+0x177/0x190 [15280.283723] _raw_spin_lock+0x21/0x30 [15280.285163] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15280.287333] ? do_raw_spin_unlock+0x75/0x190 [15280.288877] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15280.290973] ? do_wait_intr+0xf0/0xf0 [15280.292443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15280.294531] mdc_close+0x53b/0xd70 [mdc] [15280.295988] lmv_close+0x18d/0x490 [lmv] [15280.297554] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15280.299741] ll_md_real_close+0xa5/0x2a0 [lustre] [15280.301627] ll_md_close+0x274/0x800 [lustre] [15280.303384] ll_file_release+0x128/0xc50 [lustre] [15280.305227] __fput+0x10d/0x440 [15280.306447] ____fput+0x12/0x20 [15280.307407] task_work_run+0xc6/0x110 [15280.308725] exit_to_usermode_loop+0x1dd/0x1f0 [15280.310375] do_syscall_64+0x3d6/0x440 [15280.311591] entry_SYSCALL_64_after_hwframe+0x49/0xae [15280.315929] RIP: 0033:0x7fa749552cd8 [15280.318735] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15280.329406] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15280.332403] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15280.335115] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15280.337816] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15280.342184] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15280.347598] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15288.057022] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15288.060323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15288.083049] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15288.091367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15288.096043] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15288.100323] RIP: 0010:native_safe_halt+0x2e/0x50 [15288.102326] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15288.111475] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15288.114777] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15288.119114] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15288.123305] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15288.126720] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15288.130395] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15288.133641] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15288.137464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15288.141437] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15288.144856] Call Trace: [15288.145836] [15288.146814] ? show_regs.cold.9+0x22/0x2f [15288.150969] ? watchdog_timer_fn.cold.6+0x62/0x10a [15288.153573] ? __hrtimer_run_queues+0x1c2/0x4d0 [15288.155417] ? watchdog+0x40/0x40 [15288.156788] ? hrtimer_interrupt+0x128/0x2c0 [15288.158238] ? smp_apic_timer_interrupt+0xa6/0x270 [15288.159809] ? apic_timer_interrupt+0xf/0x20 [15288.162083] [15288.162986] ? native_safe_halt+0x2e/0x50 [15288.166466] kvm_wait+0xcd/0xe0 [15288.168528] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15288.170716] do_raw_spin_lock+0x177/0x190 [15288.175005] _raw_spin_lock+0x21/0x30 [15288.176542] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15288.179356] process_one_work+0x2c8/0x700 [15288.181862] worker_thread+0x296/0x6e0 [15288.184453] ? rescuer_thread+0x570/0x570 [15288.186605] kthread+0x1d1/0x200 [15288.189437] ? set_kthread_struct+0x70/0x70 [15288.191545] ret_from_fork+0x1f/0x30 [15292.055662] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15292.061904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15292.084029] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15292.098603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15292.104070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15292.106918] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15292.124464] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15292.130662] RAX: 0000000000003a8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15292.137480] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15292.146758] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15292.151599] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15292.157657] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15292.162016] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15292.166693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15292.171409] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15292.177602] Call Trace: [15292.179879] [15292.180675] ? show_regs.cold.9+0x22/0x2f [15292.182729] ? watchdog_timer_fn.cold.6+0x62/0x10a [15292.185304] ? __hrtimer_run_queues+0x1c2/0x4d0 [15292.187729] ? watchdog+0x40/0x40 [15292.189829] ? hrtimer_interrupt+0x128/0x2c0 [15292.193200] ? smp_apic_timer_interrupt+0xa6/0x270 [15292.196650] ? apic_timer_interrupt+0xf/0x20 [15292.199385] [15292.200391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15292.202890] do_raw_spin_lock+0x177/0x190 [15292.205040] _raw_spin_lock+0x21/0x30 [15292.206059] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15292.208666] ? do_raw_spin_unlock+0x75/0x190 [15292.210748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15292.213263] ? do_wait_intr+0xf0/0xf0 [15292.215080] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15292.217633] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15292.220038] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15292.223508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15292.226153] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15292.228704] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15292.230372] mdc_intent_lock+0x353/0x7e0 [mdc] [15292.232617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15292.235132] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15292.237959] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15292.239724] lmv_intent_open+0x338/0x1150 [lmv] [15292.241638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15292.247775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15292.251756] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15292.254539] ll_intent_lock+0xc5/0xaa0 [lustre] [15292.256536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15292.259186] ll_lookup_it+0xe0e/0x2270 [lustre] [15292.261684] ? do_raw_spin_unlock+0x75/0x190 [15292.263630] ? _raw_spin_unlock+0x12/0x30 [15292.265572] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15292.267927] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15292.270196] ll_atomic_open+0x36a/0xd70 [lustre] [15292.273402] lookup_open+0x356/0xb40 [15292.275172] do_last+0x313/0xfc0 [15292.276759] ? nd_jump_root+0xe5/0x160 [15292.282227] ? path_init+0x437/0x520 [15292.286942] path_openat+0xf7/0x500 [15292.288880] ? do_raw_spin_unlock+0x75/0x190 [15292.291107] do_filp_open+0x99/0x140 [15292.293162] ? getname_flags+0x6e/0x330 [15292.294980] ? __check_object_size+0xff/0x256 [15292.297381] ? do_raw_spin_unlock+0x75/0x190 [15292.299179] ? _raw_spin_unlock+0x12/0x30 [15292.301246] do_sys_openat2+0x2b4/0x410 [15292.303013] do_sys_open+0x73/0xa0 [15292.304578] __x64_sys_openat+0x24/0x30 [15292.306241] do_syscall_64+0xc1/0x440 [15292.308386] entry_SYSCALL_64_after_hwframe+0x49/0xae [15292.310599] RIP: 0033:0x7fa749552332 [15292.312346] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15292.320597] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15292.323713] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15292.326785] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15292.329409] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15292.331933] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15292.333981] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15293.988458] rcu: INFO: rcu_sched self-detected stall on CPU [15293.991586] rcu: 2-...!: (7737706 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [15293.996638] (t=13380299 jiffies g=258701 q=5120) [15293.999030] rcu: rcu_sched kthread starved for 13377094 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [15294.003580] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15294.007541] rcu: RCU grace-period kthread stack dump: [15294.009726] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15294.013812] Call Trace: [15294.015102] __schedule+0x351/0xcb0 [15294.016346] schedule+0xc0/0x180 [15294.017397] schedule_timeout+0xb4/0x190 [15294.019254] ? do_raw_spin_unlock+0x75/0x190 [15294.021200] ? __next_timer_interrupt+0x160/0x160 [15294.023227] ? prepare_to_swait_event+0xf3/0x140 [15294.025203] rcu_gp_kthread+0x8dd/0x15b0 [15294.026785] ? force_qs_rnp+0x2c0/0x2c0 [15294.028578] kthread+0x1d1/0x200 [15294.030177] ? set_kthread_struct+0x70/0x70 [15294.032083] ret_from_fork+0x1f/0x30 [15294.033559] rcu: Stack dump where RCU GP kthread last ran: [15294.035728] NMI backtrace for cpu 2 [15294.037442] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15294.041760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15294.046051] Call Trace: [15294.047064] [15294.047714] ? dump_stack+0xbb/0x10e [15294.049336] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15294.051403] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15294.053490] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15294.055770] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15294.058296] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15294.060362] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [15294.062654] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [15294.064849] ? update_process_times+0x6d/0xc0 [15294.067080] ? tick_sched_handle.isra.18+0x27/0x90 [15294.069271] ? tick_sched_timer+0x53/0xb0 [15294.071084] ? __hrtimer_run_queues+0x1c2/0x4d0 [15294.073191] ? tick_sched_handle.isra.18+0x90/0x90 [15294.076161] ? hrtimer_interrupt+0x128/0x2c0 [15294.078297] ? smp_apic_timer_interrupt+0xa6/0x270 [15294.082199] ? apic_timer_interrupt+0xf/0x20 [15294.083814] [15294.084712] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15294.086824] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15294.088641] ? do_raw_spin_lock+0x177/0x190 [15294.090039] ? _raw_spin_lock+0x21/0x30 [15294.091088] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15294.093177] ? obd_mod_free+0x46/0x140 [ptlrpc] [15294.094785] ? mdc_commit_open+0x91/0xb0 [mdc] [15294.096779] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15294.101467] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15294.103841] ? after_reply+0xe2c/0x1780 [ptlrpc] [15294.105612] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15294.107762] ? __next_timer_interrupt+0x160/0x160 [15294.109574] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15294.112102] ? do_wait_intr+0xf0/0xf0 [15294.113837] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15294.116134] ? kthread+0x1d1/0x200 [15294.117305] ? set_kthread_struct+0x70/0x70 [15294.119026] ? ret_from_fork+0x1f/0x30 [15294.120698] NMI backtrace for cpu 2 [15294.121869] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15294.126531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15294.130321] Call Trace: [15294.131639] [15294.132232] ? dump_stack+0xbb/0x10e [15294.133325] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15294.135401] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15294.137474] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15294.140360] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15294.143626] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15294.148089] ? rcu_dump_cpu_stacks+0x11b/0x171 [15294.150376] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15294.154118] ? update_process_times+0x6d/0xc0 [15294.157658] ? tick_sched_handle.isra.18+0x27/0x90 [15294.162372] ? tick_sched_timer+0x53/0xb0 [15294.164107] ? __hrtimer_run_queues+0x1c2/0x4d0 [15294.166883] ? tick_sched_handle.isra.18+0x90/0x90 [15294.173058] ? hrtimer_interrupt+0x128/0x2c0 [15294.174185] ? smp_apic_timer_interrupt+0xa6/0x270 [15294.178584] ? apic_timer_interrupt+0xf/0x20 [15294.183715] [15294.184404] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15294.188504] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15294.194149] ? do_raw_spin_lock+0x177/0x190 [15294.195465] ? _raw_spin_lock+0x21/0x30 [15294.198587] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15294.202213] ? obd_mod_free+0x46/0x140 [ptlrpc] [15294.209689] ? mdc_commit_open+0x91/0xb0 [mdc] [15294.211215] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15294.216101] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15294.223414] ? after_reply+0xe2c/0x1780 [ptlrpc] [15294.229173] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15294.235116] ? __next_timer_interrupt+0x160/0x160 [15294.238572] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15294.245489] ? do_wait_intr+0xf0/0xf0 [15294.249367] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15294.253539] ? kthread+0x1d1/0x200 [15294.254706] ? set_kthread_struct+0x70/0x70 [15294.257708] ? ret_from_fork+0x1f/0x30 [15295.972093] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13378s! [15295.975939] Showing busy workqueues and worker pools: [15295.977391] workqueue events: flags=0x0 [15295.979753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15295.983812] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15295.989800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15295.993441] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15295.998062] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.001337] pending: kfree_rcu_monitor [15296.004238] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15296.006922] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15296.010358] workqueue events_long: flags=0x0 [15296.012119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.015467] pending: lu_objects_flush [obdclass] [15296.017661] workqueue events_unbound: flags=0x2 [15296.019562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15296.021938] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15296.025050] workqueue events_power_efficient: flags=0x80 [15296.028145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.032275] pending: check_lifetime [15296.034458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.037263] pending: neigh_periodic_work [15296.039524] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.044024] pending: neigh_periodic_work [15296.046507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.048578] pending: do_cache_clean [15296.049930] workqueue mm_percpu_wq: flags=0x8 [15296.051582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.053771] pending: vmstat_update [15296.055396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.058173] pending: vmstat_update [15296.059587] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15296.063009] pending: vmstat_update [15296.064439] workqueue writeback: flags=0x4a [15296.065927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15296.068163] pending: wb_workfn, wb_workfn [15296.069600] workqueue xprtiod: flags=0xa [15296.071242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15296.074013] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15296.076944] workqueue ipv6_addrconf: flags=0x40008 [15296.078316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15296.080291] pending: addrconf_verify_work [15296.081555] workqueue knbd0-recv: flags=0x1a [15296.082713] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15296.084490] in-flight: 651:recv_work [15296.085530] workqueue ptlrpc_pinger: flags=0x2 [15296.087383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15296.088908] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15296.090605] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13378s workers=4 idle: 37 31846 28897 [15296.092672] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15285s workers=2 idle: 653 [15308.059082] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15308.065419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15308.091139] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15308.097545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15308.100248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15308.102523] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15308.111859] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15308.116338] RAX: 0000000000007f97 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15308.120679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15308.122927] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15308.129795] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15308.132978] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15308.138453] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15308.143458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15308.146650] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15308.155672] Call Trace: [15308.156752] [15308.157782] ? show_regs.cold.9+0x22/0x2f [15308.159351] ? watchdog_timer_fn.cold.6+0x62/0x10a [15308.162226] ? __hrtimer_run_queues+0x1c2/0x4d0 [15308.165077] ? watchdog+0x40/0x40 [15308.166485] ? hrtimer_interrupt+0x128/0x2c0 [15308.168345] ? smp_apic_timer_interrupt+0xa6/0x270 [15308.170817] ? apic_timer_interrupt+0xf/0x20 [15308.172232] [15308.172948] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15308.174914] do_raw_spin_lock+0x177/0x190 [15308.176314] _raw_spin_lock+0x21/0x30 [15308.177572] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15308.180142] ? do_raw_spin_unlock+0x75/0x190 [15308.182071] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15308.184737] ? do_wait_intr+0xf0/0xf0 [15308.186316] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15308.188723] mdc_close+0x53b/0xd70 [mdc] [15308.191128] lmv_close+0x18d/0x490 [lmv] [15308.192902] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15308.195353] ll_md_real_close+0xa5/0x2a0 [lustre] [15308.197285] ll_md_close+0x274/0x800 [lustre] [15308.199139] ll_file_release+0x128/0xc50 [lustre] [15308.201193] __fput+0x10d/0x440 [15308.202133] ____fput+0x12/0x20 [15308.203188] task_work_run+0xc6/0x110 [15308.204445] exit_to_usermode_loop+0x1dd/0x1f0 [15308.206764] do_syscall_64+0x3d6/0x440 [15308.208218] entry_SYSCALL_64_after_hwframe+0x49/0xae [15308.210242] RIP: 0033:0x7fa749552cd8 [15308.211662] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15308.222815] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15308.229019] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15308.235732] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15308.240905] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15308.250191] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15308.257710] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15316.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15316.060018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15316.083954] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15316.090081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15316.094210] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15316.097882] RIP: 0010:native_safe_halt+0x2e/0x50 [15316.101635] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15316.111591] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15316.116707] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15316.122698] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15316.126514] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15316.130024] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15316.132798] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15316.135908] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15316.139160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15316.141529] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15316.144306] Call Trace: [15316.145178] [15316.145859] ? show_regs.cold.9+0x22/0x2f [15316.147002] ? watchdog_timer_fn.cold.6+0x62/0x10a [15316.148613] ? __hrtimer_run_queues+0x1c2/0x4d0 [15316.150562] ? watchdog+0x40/0x40 [15316.151889] ? hrtimer_interrupt+0x128/0x2c0 [15316.154636] ? smp_apic_timer_interrupt+0xa6/0x270 [15316.155977] ? apic_timer_interrupt+0xf/0x20 [15316.158027] [15316.158868] ? native_safe_halt+0x2e/0x50 [15316.160565] kvm_wait+0xcd/0xe0 [15316.161654] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15316.163759] do_raw_spin_lock+0x177/0x190 [15316.165851] _raw_spin_lock+0x21/0x30 [15316.166999] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15316.169203] process_one_work+0x2c8/0x700 [15316.170767] worker_thread+0x296/0x6e0 [15316.172353] ? rescuer_thread+0x570/0x570 [15316.174081] kthread+0x1d1/0x200 [15316.175098] ? set_kthread_struct+0x70/0x70 [15316.176595] ret_from_fork+0x1f/0x30 [15320.058345] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15320.060877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15320.090900] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15320.095971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15320.106549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15320.111077] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15320.119752] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15320.122883] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15320.126444] RDX: 0000000000006ae8 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15320.130704] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15320.132937] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15320.138371] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15320.141167] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15320.144737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15320.148036] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15320.151796] Call Trace: [15320.153122] [15320.154040] ? show_regs.cold.9+0x22/0x2f [15320.155991] ? watchdog_timer_fn.cold.6+0x62/0x10a [15320.157689] ? __hrtimer_run_queues+0x1c2/0x4d0 [15320.159747] ? watchdog+0x40/0x40 [15320.161272] ? hrtimer_interrupt+0x128/0x2c0 [15320.163243] ? smp_apic_timer_interrupt+0xa6/0x270 [15320.165362] ? apic_timer_interrupt+0xf/0x20 [15320.167351] [15320.168164] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15320.170257] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15320.172394] do_raw_spin_lock+0x177/0x190 [15320.174261] _raw_spin_lock+0x21/0x30 [15320.176023] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15320.180045] obd_mod_free+0x46/0x140 [ptlrpc] [15320.182231] mdc_commit_open+0x91/0xb0 [mdc] [15320.184117] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15320.186412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15320.188471] after_reply+0xe2c/0x1780 [ptlrpc] [15320.190979] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15320.192651] ? __next_timer_interrupt+0x160/0x160 [15320.194204] ptlrpcd+0xa82/0xe40 [ptlrpc] [15320.195864] ? do_wait_intr+0xf0/0xf0 [15320.197155] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15320.198997] kthread+0x1d1/0x200 [15320.200013] ? set_kthread_struct+0x70/0x70 [15320.201240] ret_from_fork+0x1f/0x30 [15324.054191] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15324.057252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15324.090035] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15324.097069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15324.103399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15324.109469] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15324.126036] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15324.130075] RAX: 0000000000002bd3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15324.135147] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15324.143551] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15324.150494] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15324.172304] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15324.178310] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15324.182822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15324.185529] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15324.188924] Call Trace: [15324.189944] [15324.190584] ? show_regs.cold.9+0x22/0x2f [15324.192017] ? watchdog_timer_fn.cold.6+0x62/0x10a [15324.193859] ? __hrtimer_run_queues+0x1c2/0x4d0 [15324.195292] ? watchdog+0x40/0x40 [15324.196165] ? hrtimer_interrupt+0x128/0x2c0 [15324.199606] ? smp_apic_timer_interrupt+0xa6/0x270 [15324.202339] ? apic_timer_interrupt+0xf/0x20 [15324.204439] [15324.205438] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15324.208522] do_raw_spin_lock+0x177/0x190 [15324.210553] _raw_spin_lock+0x21/0x30 [15324.212493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15324.215017] ? do_raw_spin_unlock+0x75/0x190 [15324.217766] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15324.221561] ? do_wait_intr+0xf0/0xf0 [15324.222993] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15324.225461] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15324.229786] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15324.233133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15324.236207] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15324.238887] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15324.241022] mdc_intent_lock+0x353/0x7e0 [mdc] [15324.243010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15324.245142] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15324.248663] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15324.250760] lmv_intent_open+0x338/0x1150 [lmv] [15324.252714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15324.254688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15324.257082] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15324.258843] ll_intent_lock+0xc5/0xaa0 [lustre] [15324.260934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15324.263503] ll_lookup_it+0xe0e/0x2270 [lustre] [15324.265497] ? do_raw_spin_unlock+0x75/0x190 [15324.267091] ? _raw_spin_unlock+0x12/0x30 [15324.268858] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15324.271020] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15324.273129] ll_atomic_open+0x36a/0xd70 [lustre] [15324.275123] lookup_open+0x356/0xb40 [15324.276670] do_last+0x313/0xfc0 [15324.277859] ? nd_jump_root+0xe5/0x160 [15324.280004] ? path_init+0x437/0x520 [15324.281841] path_openat+0xf7/0x500 [15324.283709] ? do_raw_spin_unlock+0x75/0x190 [15324.285531] do_filp_open+0x99/0x140 [15324.287043] ? getname_flags+0x6e/0x330 [15324.288756] ? __check_object_size+0xff/0x256 [15324.290493] ? do_raw_spin_unlock+0x75/0x190 [15324.292026] ? _raw_spin_unlock+0x12/0x30 [15324.293782] do_sys_openat2+0x2b4/0x410 [15324.295348] do_sys_open+0x73/0xa0 [15324.297146] __x64_sys_openat+0x24/0x30 [15324.298872] do_syscall_64+0xc1/0x440 [15324.300119] entry_SYSCALL_64_after_hwframe+0x49/0xae [15324.302733] RIP: 0033:0x7fa749552332 [15324.304016] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15324.313064] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15324.316677] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15324.319675] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15324.322669] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15324.324914] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15324.328591] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15326.179032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13409s! [15326.181911] Showing busy workqueues and worker pools: [15326.189490] workqueue events: flags=0x0 [15326.192486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15326.199820] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15326.210012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15326.213485] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15326.222059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.229006] pending: kfree_rcu_monitor [15326.230556] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15326.239767] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15326.249788] workqueue events_long: flags=0x0 [15326.253853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.257565] pending: lu_objects_flush [obdclass] [15326.267053] workqueue events_unbound: flags=0x2 [15326.271861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15326.276888] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15326.289701] workqueue events_power_efficient: flags=0x80 [15326.293373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.296512] pending: check_lifetime [15326.298399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.301136] pending: neigh_periodic_work [15326.303336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.306652] pending: neigh_periodic_work [15326.308380] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.311614] pending: do_cache_clean [15326.313022] workqueue mm_percpu_wq: flags=0x8 [15326.321577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.330566] pending: vmstat_update [15326.336866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.343130] pending: vmstat_update [15326.345741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15326.348921] pending: vmstat_update [15326.353125] workqueue writeback: flags=0x4a [15326.354866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15326.358194] pending: wb_workfn, wb_workfn [15326.360659] workqueue xprtiod: flags=0xa [15326.362038] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15326.365453] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15326.369383] workqueue ipv6_addrconf: flags=0x40008 [15326.371661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15326.375496] pending: addrconf_verify_work [15326.380521] workqueue knbd0-recv: flags=0x1a [15326.382348] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15326.384195] in-flight: 651:recv_work [15326.386335] workqueue ptlrpc_pinger: flags=0x2 [15326.387877] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15326.390411] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15326.392159] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13409s workers=4 idle: 37 31846 28897 [15326.399467] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15316s workers=2 idle: 653 [15336.059229] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15336.061904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15336.079196] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15336.083760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15336.087375] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15336.089365] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15336.094981] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15336.097978] RAX: 0000000000006909 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15336.101192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15336.104022] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15336.106892] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15336.109780] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15336.112758] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15336.115897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15336.117928] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15336.120100] Call Trace: [15336.121041] [15336.121890] ? show_regs.cold.9+0x22/0x2f [15336.123612] ? watchdog_timer_fn.cold.6+0x62/0x10a [15336.125538] ? __hrtimer_run_queues+0x1c2/0x4d0 [15336.127406] ? watchdog+0x40/0x40 [15336.129196] ? hrtimer_interrupt+0x128/0x2c0 [15336.130912] ? smp_apic_timer_interrupt+0xa6/0x270 [15336.133019] ? apic_timer_interrupt+0xf/0x20 [15336.134762] [15336.135397] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15336.137299] do_raw_spin_lock+0x177/0x190 [15336.138876] _raw_spin_lock+0x21/0x30 [15336.140323] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15336.142697] ? do_raw_spin_unlock+0x75/0x190 [15336.144462] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15336.146655] ? do_wait_intr+0xf0/0xf0 [15336.148274] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15336.150569] mdc_close+0x53b/0xd70 [mdc] [15336.152248] lmv_close+0x18d/0x490 [lmv] [15336.154202] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15336.156509] ll_md_real_close+0xa5/0x2a0 [lustre] [15336.158184] ll_md_close+0x274/0x800 [lustre] [15336.159483] ll_file_release+0x128/0xc50 [lustre] [15336.161131] __fput+0x10d/0x440 [15336.162714] ____fput+0x12/0x20 [15336.163868] task_work_run+0xc6/0x110 [15336.165503] exit_to_usermode_loop+0x1dd/0x1f0 [15336.167364] do_syscall_64+0x3d6/0x440 [15336.168789] entry_SYSCALL_64_after_hwframe+0x49/0xae [15336.170860] RIP: 0033:0x7fa749552cd8 [15336.172429] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15336.179874] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15336.182997] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15336.185428] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15336.187818] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15336.190814] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15336.193694] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15344.057230] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15344.065107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15344.110970] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15344.122464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15344.131590] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15344.141024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15344.153790] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15344.181820] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15344.193238] RAX: 0000000000007f12 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15344.201287] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15344.209047] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15344.216965] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15344.231366] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15344.246304] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15344.254801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15344.260988] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15344.269354] Call Trace: [15344.272444] [15344.273416] ? show_regs.cold.9+0x22/0x2f [15344.277771] ? watchdog_timer_fn.cold.6+0x62/0x10a [15344.283281] ? __hrtimer_run_queues+0x1c2/0x4d0 [15344.287977] ? watchdog+0x40/0x40 [15344.290182] ? hrtimer_interrupt+0x128/0x2c0 [15344.294837] ? smp_apic_timer_interrupt+0xa6/0x270 [15344.305351] ? apic_timer_interrupt+0xf/0x20 [15344.310016] [15344.311071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15344.320773] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15344.327240] do_raw_spin_lock+0x177/0x190 [15344.331167] _raw_spin_lock+0x21/0x30 [15344.336483] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15344.347009] process_one_work+0x2c8/0x700 [15344.352995] worker_thread+0x296/0x6e0 [15344.363270] ? rescuer_thread+0x570/0x570 [15344.371421] kthread+0x1d1/0x200 [15344.375740] ? set_kthread_struct+0x70/0x70 [15344.383786] ret_from_fork+0x1f/0x30 [15348.058024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15348.061174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15348.094667] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15348.109692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15348.118188] RIP: 0010:native_safe_halt+0x2e/0x50 [15348.125220] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15348.147534] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15348.168548] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15348.175167] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15348.182348] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15348.192902] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15348.197856] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15348.207458] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15348.215925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15348.226331] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15348.235702] Call Trace: [15348.237684] [15348.238726] ? show_regs.cold.9+0x22/0x2f [15348.241836] ? watchdog_timer_fn.cold.6+0x62/0x10a [15348.246523] ? __hrtimer_run_queues+0x1c2/0x4d0 [15348.249487] ? watchdog+0x40/0x40 [15348.250847] ? hrtimer_interrupt+0x128/0x2c0 [15348.252170] ? smp_apic_timer_interrupt+0xa6/0x270 [15348.256194] ? apic_timer_interrupt+0xf/0x20 [15348.260411] [15348.263588] ? native_safe_halt+0x2e/0x50 [15348.268602] kvm_wait+0xcd/0xe0 [15348.271556] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15348.280667] do_raw_spin_lock+0x177/0x190 [15348.285297] _raw_spin_lock+0x21/0x30 [15348.291245] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15348.298227] obd_mod_free+0x46/0x140 [ptlrpc] [15348.304760] mdc_commit_open+0x91/0xb0 [mdc] [15348.307491] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15348.311161] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15348.315099] after_reply+0xe2c/0x1780 [ptlrpc] [15348.323328] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15348.329279] ? __next_timer_interrupt+0x160/0x160 [15348.332096] ptlrpcd+0xa82/0xe40 [ptlrpc] [15348.336168] ? do_wait_intr+0xf0/0xf0 [15348.338702] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15348.343297] kthread+0x1d1/0x200 [15348.345651] ? set_kthread_struct+0x70/0x70 [15348.348217] ret_from_fork+0x1f/0x30 [15352.057852] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15352.069225] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15352.142684] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15352.167742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15352.180898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15352.183583] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15352.220838] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15352.228128] RAX: 0000000000002f24 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15352.235578] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15352.241196] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15352.245746] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15352.263719] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15352.280206] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15352.296506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15352.307306] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15352.319686] Call Trace: [15352.320638] [15352.321559] ? show_regs.cold.9+0x22/0x2f [15352.332165] ? watchdog_timer_fn.cold.6+0x62/0x10a [15352.337322] ? __hrtimer_run_queues+0x1c2/0x4d0 [15352.339521] ? watchdog+0x40/0x40 [15352.346173] ? hrtimer_interrupt+0x128/0x2c0 [15352.349119] ? smp_apic_timer_interrupt+0xa6/0x270 [15352.356077] ? apic_timer_interrupt+0xf/0x20 [15352.359928] [15352.362238] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15352.366480] do_raw_spin_lock+0x177/0x190 [15352.368267] _raw_spin_lock+0x21/0x30 [15352.377585] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15352.389496] ? do_raw_spin_unlock+0x75/0x190 [15352.394291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15352.406173] ? do_wait_intr+0xf0/0xf0 [15352.415770] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15352.426909] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15352.435527] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15352.442229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15352.450286] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15352.457734] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15352.460713] mdc_intent_lock+0x353/0x7e0 [mdc] [15352.465027] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15352.471522] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15352.481519] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15352.485324] lmv_intent_open+0x338/0x1150 [lmv] [15352.489385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15352.494288] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15352.496852] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15352.500259] ll_intent_lock+0xc5/0xaa0 [lustre] [15352.504371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15352.512105] ll_lookup_it+0xe0e/0x2270 [lustre] [15352.514792] ? do_raw_spin_unlock+0x75/0x190 [15352.517294] ? _raw_spin_unlock+0x12/0x30 [15352.519299] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15352.529596] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15352.534070] ll_atomic_open+0x36a/0xd70 [lustre] [15352.545555] lookup_open+0x356/0xb40 [15352.556983] do_last+0x313/0xfc0 [15352.562300] ? nd_jump_root+0xe5/0x160 [15352.574467] ? path_init+0x437/0x520 [15352.575710] path_openat+0xf7/0x500 [15352.583279] ? do_raw_spin_unlock+0x75/0x190 [15352.588186] do_filp_open+0x99/0x140 [15352.594334] ? getname_flags+0x6e/0x330 [15352.598882] ? __check_object_size+0xff/0x256 [15352.601337] ? do_raw_spin_unlock+0x75/0x190 [15352.606108] ? _raw_spin_unlock+0x12/0x30 [15352.612207] do_sys_openat2+0x2b4/0x410 [15352.618590] do_sys_open+0x73/0xa0 [15352.621508] __x64_sys_openat+0x24/0x30 [15352.625970] do_syscall_64+0xc1/0x440 [15352.629973] entry_SYSCALL_64_after_hwframe+0x49/0xae [15352.634957] RIP: 0033:0x7fa749552332 [15352.636227] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15352.653988] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15352.658601] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15352.664174] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15352.668953] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15352.672040] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15352.675808] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15356.896896] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13439s! [15356.911343] Showing busy workqueues and worker pools: [15356.913175] workqueue events: flags=0x0 [15356.921066] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15356.923537] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15356.932828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15356.938050] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15356.943639] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15356.948532] pending: kfree_rcu_monitor [15356.951699] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15356.959629] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15356.964109] workqueue events_long: flags=0x0 [15356.970454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15356.978203] pending: lu_objects_flush [obdclass] [15356.982259] workqueue events_unbound: flags=0x2 [15356.986782] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15356.989440] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15356.994767] workqueue events_power_efficient: flags=0x80 [15357.005702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15357.009540] pending: check_lifetime [15357.011935] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15357.015019] pending: neigh_periodic_work [15357.017062] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15357.020129] pending: neigh_periodic_work [15357.021604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15357.026537] pending: do_cache_clean [15357.027863] workqueue mm_percpu_wq: flags=0x8 [15357.031157] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15357.039343] pending: vmstat_update [15357.041917] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15357.045193] pending: vmstat_update [15357.048153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15357.058083] pending: vmstat_update [15357.063236] workqueue writeback: flags=0x4a [15357.066148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15357.069299] pending: wb_workfn, wb_workfn [15357.073334] workqueue xprtiod: flags=0xa [15357.079030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15357.084978] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15357.089659] workqueue ipv6_addrconf: flags=0x40008 [15357.092479] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15357.095226] pending: addrconf_verify_work [15357.097667] workqueue knbd0-recv: flags=0x1a [15357.100424] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15357.104078] in-flight: 651:recv_work [15357.106815] workqueue ptlrpc_pinger: flags=0x2 [15357.109637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15357.117090] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15357.120173] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13439s workers=4 idle: 37 31846 28897 [15357.124086] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15347s workers=2 idle: 653 [15364.060641] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15364.073001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15364.121062] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15364.132811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15364.142957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15364.146141] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15364.158271] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15364.167601] RAX: 0000000000005881 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15364.178783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15364.187991] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15364.198045] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15364.208071] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15364.211165] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15364.222836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15364.227469] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15364.236093] Call Trace: [15364.239159] [15364.240334] ? show_regs.cold.9+0x22/0x2f [15364.250380] ? watchdog_timer_fn.cold.6+0x62/0x10a [15364.254566] ? __hrtimer_run_queues+0x1c2/0x4d0 [15364.256681] ? watchdog+0x40/0x40 [15364.257973] ? hrtimer_interrupt+0x128/0x2c0 [15364.260282] ? smp_apic_timer_interrupt+0xa6/0x270 [15364.262374] ? apic_timer_interrupt+0xf/0x20 [15364.264451] [15364.265424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15364.268223] do_raw_spin_lock+0x177/0x190 [15364.269996] _raw_spin_lock+0x21/0x30 [15364.271857] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15364.275514] ? do_raw_spin_unlock+0x75/0x190 [15364.277902] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15364.280334] ? do_wait_intr+0xf0/0xf0 [15364.281752] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15364.286689] mdc_close+0x53b/0xd70 [mdc] [15364.288230] lmv_close+0x18d/0x490 [lmv] [15364.290323] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15364.294787] ll_md_real_close+0xa5/0x2a0 [lustre] [15364.297513] ll_md_close+0x274/0x800 [lustre] [15364.301526] ll_file_release+0x128/0xc50 [lustre] [15364.305915] __fput+0x10d/0x440 [15364.307437] ____fput+0x12/0x20 [15364.308644] task_work_run+0xc6/0x110 [15364.312742] exit_to_usermode_loop+0x1dd/0x1f0 [15364.315206] do_syscall_64+0x3d6/0x440 [15364.317314] entry_SYSCALL_64_after_hwframe+0x49/0xae [15364.319981] RIP: 0033:0x7fa749552cd8 [15364.321787] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15364.331674] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15364.337558] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15364.340585] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15364.348083] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15364.352374] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15364.355487] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15372.057021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15372.060972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15372.113934] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15372.127614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15372.134330] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15372.150626] RIP: 0010:native_safe_halt+0x2e/0x50 [15372.157681] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15372.178191] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15372.187023] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15372.199036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15372.205546] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15372.211090] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15372.217984] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15372.226145] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15372.239694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15372.242739] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15372.246031] Call Trace: [15372.247967] [15372.249291] ? show_regs.cold.9+0x22/0x2f [15372.251782] ? watchdog_timer_fn.cold.6+0x62/0x10a [15372.254476] ? __hrtimer_run_queues+0x1c2/0x4d0 [15372.256883] ? watchdog+0x40/0x40 [15372.258679] ? hrtimer_interrupt+0x128/0x2c0 [15372.260882] ? smp_apic_timer_interrupt+0xa6/0x270 [15372.266229] ? apic_timer_interrupt+0xf/0x20 [15372.271368] [15372.272383] ? native_safe_halt+0x2e/0x50 [15372.275108] kvm_wait+0xcd/0xe0 [15372.277765] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15372.287884] do_raw_spin_lock+0x177/0x190 [15372.292863] _raw_spin_lock+0x21/0x30 [15372.298353] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15372.306930] process_one_work+0x2c8/0x700 [15372.310203] worker_thread+0x296/0x6e0 [15372.314668] ? rescuer_thread+0x570/0x570 [15372.326368] kthread+0x1d1/0x200 [15372.332856] ? set_kthread_struct+0x70/0x70 [15372.341862] ret_from_fork+0x1f/0x30 [15376.058274] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15376.061061] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15376.092455] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15376.097213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15376.101929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15376.104221] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15376.111103] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15376.114008] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15376.116793] RDX: 0000000000007763 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15376.119336] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15376.122087] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15376.125073] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15376.127250] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15376.130284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15376.132401] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15376.134616] Call Trace: [15376.135706] [15376.136489] ? show_regs.cold.9+0x22/0x2f [15376.138030] ? watchdog_timer_fn.cold.6+0x62/0x10a [15376.140035] ? __hrtimer_run_queues+0x1c2/0x4d0 [15376.142022] ? watchdog+0x40/0x40 [15376.143449] ? hrtimer_interrupt+0x128/0x2c0 [15376.145136] ? smp_apic_timer_interrupt+0xa6/0x270 [15376.147008] ? apic_timer_interrupt+0xf/0x20 [15376.148812] [15376.149835] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15376.151959] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15376.154182] do_raw_spin_lock+0x177/0x190 [15376.155794] _raw_spin_lock+0x21/0x30 [15376.157050] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15376.159180] obd_mod_free+0x46/0x140 [ptlrpc] [15376.161158] mdc_commit_open+0x91/0xb0 [mdc] [15376.162804] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15376.164925] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15376.167178] after_reply+0xe2c/0x1780 [ptlrpc] [15376.169402] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15376.171528] ? __next_timer_interrupt+0x160/0x160 [15376.172840] ptlrpcd+0xa82/0xe40 [ptlrpc] [15376.174456] ? do_wait_intr+0xf0/0xf0 [15376.175472] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15376.177073] kthread+0x1d1/0x200 [15376.178002] ? set_kthread_struct+0x70/0x70 [15376.179123] ret_from_fork+0x1f/0x30 [15384.055509] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15384.066018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15384.128045] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15384.140966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15384.157296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15384.163922] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15384.177761] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15384.185604] RAX: 00000000000047ab RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15384.189956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15384.201163] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15384.207013] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15384.210948] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15384.214567] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15384.219426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15384.223513] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15384.226976] Call Trace: [15384.228434] [15384.229970] ? show_regs.cold.9+0x22/0x2f [15384.235432] ? watchdog_timer_fn.cold.6+0x62/0x10a [15384.238010] ? __hrtimer_run_queues+0x1c2/0x4d0 [15384.240793] ? watchdog+0x40/0x40 [15384.243963] ? hrtimer_interrupt+0x128/0x2c0 [15384.247694] ? smp_apic_timer_interrupt+0xa6/0x270 [15384.249419] ? apic_timer_interrupt+0xf/0x20 [15384.256876] [15384.257857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15384.267998] do_raw_spin_lock+0x177/0x190 [15384.275292] _raw_spin_lock+0x21/0x30 [15384.279376] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15384.286707] ? do_raw_spin_unlock+0x75/0x190 [15384.290940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15384.293867] ? do_wait_intr+0xf0/0xf0 [15384.298320] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15384.302432] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15384.305336] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15384.309320] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15384.313060] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15384.321927] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15384.326499] mdc_intent_lock+0x353/0x7e0 [mdc] [15384.328053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15384.340037] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15384.347114] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15384.354978] lmv_intent_open+0x338/0x1150 [lmv] [15384.360450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15384.364302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15384.368800] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15384.372319] ll_intent_lock+0xc5/0xaa0 [lustre] [15384.380049] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15384.384029] ll_lookup_it+0xe0e/0x2270 [lustre] [15384.389172] ? do_raw_spin_unlock+0x75/0x190 [15384.399344] ? _raw_spin_unlock+0x12/0x30 [15384.403751] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15384.408528] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15384.412125] ll_atomic_open+0x36a/0xd70 [lustre] [15384.420740] lookup_open+0x356/0xb40 [15384.422184] do_last+0x313/0xfc0 [15384.425368] ? nd_jump_root+0xe5/0x160 [15384.428004] ? path_init+0x437/0x520 [15384.432518] path_openat+0xf7/0x500 [15384.434687] ? do_raw_spin_unlock+0x75/0x190 [15384.436855] do_filp_open+0x99/0x140 [15384.438533] ? getname_flags+0x6e/0x330 [15384.441825] ? __check_object_size+0xff/0x256 [15384.444716] ? do_raw_spin_unlock+0x75/0x190 [15384.452346] ? _raw_spin_unlock+0x12/0x30 [15384.455009] do_sys_openat2+0x2b4/0x410 [15384.456431] do_sys_open+0x73/0xa0 [15384.459071] __x64_sys_openat+0x24/0x30 [15384.460802] do_syscall_64+0xc1/0x440 [15384.463588] entry_SYSCALL_64_after_hwframe+0x49/0xae [15384.482100] RIP: 0033:0x7fa749552332 [15384.485951] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15384.519150] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15384.522358] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15384.531322] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15384.537045] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15384.543543] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15384.556917] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15387.616139] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13470s! [15387.632010] Showing busy workqueues and worker pools: [15387.636449] workqueue events: flags=0x0 [15387.646096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15387.656108] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15387.670558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15387.674053] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15387.680972] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.690642] pending: kfree_rcu_monitor [15387.697081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15387.709751] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15387.719312] workqueue events_long: flags=0x0 [15387.721331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.727974] pending: lu_objects_flush [obdclass] [15387.732775] workqueue events_unbound: flags=0x2 [15387.737637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15387.743936] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15387.756605] workqueue events_power_efficient: flags=0x80 [15387.759133] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.762374] pending: check_lifetime [15387.764075] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.767085] pending: neigh_periodic_work [15387.768192] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.772884] pending: neigh_periodic_work [15387.775792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.779328] pending: do_cache_clean [15387.782372] workqueue mm_percpu_wq: flags=0x8 [15387.784636] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.786866] pending: vmstat_update [15387.789266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.796980] pending: vmstat_update [15387.798649] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15387.802068] pending: vmstat_update [15387.804562] workqueue writeback: flags=0x4a [15387.811125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15387.813848] pending: wb_workfn, wb_workfn [15387.816749] workqueue xprtiod: flags=0xa [15387.821612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15387.824452] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15387.830671] workqueue ipv6_addrconf: flags=0x40008 [15387.833316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15387.837619] pending: addrconf_verify_work [15387.840608] workqueue knbd0-recv: flags=0x1a [15387.844851] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15387.849730] in-flight: 651:recv_work [15387.852510] workqueue ptlrpc_pinger: flags=0x2 [15387.854769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15387.861785] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15387.865256] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13470s workers=4 idle: 37 31846 28897 [15387.869566] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15377s workers=2 idle: 653 [15392.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15392.064900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15392.107429] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15392.122316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15392.129201] RIP: 0010:native_safe_halt+0x2e/0x50 [15392.132673] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15392.166507] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15392.174894] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15392.187376] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15392.194595] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15392.199448] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15392.205037] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15392.212454] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15392.220121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15392.226450] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15392.230027] Call Trace: [15392.231432] [15392.232778] ? show_regs.cold.9+0x22/0x2f [15392.235299] ? watchdog_timer_fn.cold.6+0x62/0x10a [15392.237556] ? __hrtimer_run_queues+0x1c2/0x4d0 [15392.240752] ? watchdog+0x40/0x40 [15392.243707] ? hrtimer_interrupt+0x128/0x2c0 [15392.250782] ? smp_apic_timer_interrupt+0xa6/0x270 [15392.262045] ? apic_timer_interrupt+0xf/0x20 [15392.264266] [15392.266636] ? native_safe_halt+0x2e/0x50 [15392.269181] kvm_wait+0xcd/0xe0 [15392.271545] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15392.274671] do_raw_spin_lock+0x177/0x190 [15392.282964] _raw_spin_lock+0x21/0x30 [15392.284296] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15392.287339] ? do_raw_spin_unlock+0x75/0x190 [15392.289039] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15392.291458] ? do_wait_intr+0xf0/0xf0 [15392.293377] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15392.295372] mdc_close+0x53b/0xd70 [mdc] [15392.296588] lmv_close+0x18d/0x490 [lmv] [15392.297759] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15392.299693] ll_md_real_close+0xa5/0x2a0 [lustre] [15392.302324] ll_md_close+0x274/0x800 [lustre] [15392.304464] ll_file_release+0x128/0xc50 [lustre] [15392.306501] __fput+0x10d/0x440 [15392.307784] ____fput+0x12/0x20 [15392.309032] task_work_run+0xc6/0x110 [15392.310608] exit_to_usermode_loop+0x1dd/0x1f0 [15392.312413] do_syscall_64+0x3d6/0x440 [15392.313892] entry_SYSCALL_64_after_hwframe+0x49/0xae [15392.316019] RIP: 0033:0x7fa749552cd8 [15392.317543] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15392.325231] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15392.328279] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15392.331268] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15392.334258] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15392.337019] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15392.339906] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15400.058188] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15400.065373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15400.106649] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15400.120292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15400.128644] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15400.135688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15400.138054] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15400.149143] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15400.153637] RAX: 0000000000007f89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15400.160014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15400.164230] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15400.168630] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15400.176275] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15400.182889] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15400.186668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15400.190615] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15400.194866] Call Trace: [15400.195834] [15400.197445] ? show_regs.cold.9+0x22/0x2f [15400.200737] ? watchdog_timer_fn.cold.6+0x62/0x10a [15400.202896] ? __hrtimer_run_queues+0x1c2/0x4d0 [15400.207573] ? watchdog+0x40/0x40 [15400.209177] ? hrtimer_interrupt+0x128/0x2c0 [15400.211041] ? smp_apic_timer_interrupt+0xa6/0x270 [15400.213361] ? apic_timer_interrupt+0xf/0x20 [15400.215687] [15400.216906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15400.220455] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15400.222895] do_raw_spin_lock+0x177/0x190 [15400.224466] _raw_spin_lock+0x21/0x30 [15400.226243] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15400.228309] process_one_work+0x2c8/0x700 [15400.232716] worker_thread+0x296/0x6e0 [15400.234062] ? rescuer_thread+0x570/0x570 [15400.236378] kthread+0x1d1/0x200 [15400.237603] ? set_kthread_struct+0x70/0x70 [15400.239367] ret_from_fork+0x1f/0x30 [15404.058025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15404.061781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15404.087565] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15404.092909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15404.097438] RIP: 0010:native_safe_halt+0x2e/0x50 [15404.099825] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15404.109781] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15404.112533] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15404.116688] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15404.120012] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15404.123511] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15404.126278] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15404.129981] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15404.133629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15404.136422] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15404.140513] Call Trace: [15404.142264] [15404.144051] ? show_regs.cold.9+0x22/0x2f [15404.147119] ? watchdog_timer_fn.cold.6+0x62/0x10a [15404.149313] ? __hrtimer_run_queues+0x1c2/0x4d0 [15404.152178] ? watchdog+0x40/0x40 [15404.153796] ? hrtimer_interrupt+0x128/0x2c0 [15404.156462] ? smp_apic_timer_interrupt+0xa6/0x270 [15404.158723] ? apic_timer_interrupt+0xf/0x20 [15404.160668] [15404.161683] ? native_safe_halt+0x2e/0x50 [15404.163450] kvm_wait+0xcd/0xe0 [15404.165396] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15404.168235] do_raw_spin_lock+0x177/0x190 [15404.169826] _raw_spin_lock+0x21/0x30 [15404.172182] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15404.175673] obd_mod_free+0x46/0x140 [ptlrpc] [15404.179693] mdc_commit_open+0x91/0xb0 [mdc] [15404.181262] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15404.186291] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15404.189563] after_reply+0xe2c/0x1780 [ptlrpc] [15404.191430] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15404.196562] ? __next_timer_interrupt+0x160/0x160 [15404.198681] ptlrpcd+0xa82/0xe40 [ptlrpc] [15404.206261] ? do_wait_intr+0xf0/0xf0 [15404.208300] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15404.213486] kthread+0x1d1/0x200 [15404.215210] ? set_kthread_struct+0x70/0x70 [15404.217626] ret_from_fork+0x1f/0x30 [15412.054034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15412.059221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15412.078187] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15412.083658] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15412.087244] RIP: 0010:native_safe_halt+0x2e/0x50 [15412.088888] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15412.096438] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15412.100058] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15412.103323] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15412.106465] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15412.109413] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15412.112639] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15412.115979] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15412.118801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15412.120363] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15412.123045] Call Trace: [15412.127198] [15412.127960] ? show_regs.cold.9+0x22/0x2f [15412.130031] ? watchdog_timer_fn.cold.6+0x62/0x10a [15412.132038] ? __hrtimer_run_queues+0x1c2/0x4d0 [15412.134136] ? watchdog+0x40/0x40 [15412.140304] ? hrtimer_interrupt+0x128/0x2c0 [15412.141473] ? smp_apic_timer_interrupt+0xa6/0x270 [15412.142661] ? apic_timer_interrupt+0xf/0x20 [15412.146315] [15412.148138] ? native_safe_halt+0x2e/0x50 [15412.155365] kvm_wait+0xcd/0xe0 [15412.156668] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15412.161833] do_raw_spin_lock+0x177/0x190 [15412.163420] _raw_spin_lock+0x21/0x30 [15412.165469] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15412.167901] ? do_raw_spin_unlock+0x75/0x190 [15412.169876] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15412.172087] ? do_wait_intr+0xf0/0xf0 [15412.173835] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15412.176151] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15412.178548] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15412.181501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15412.183892] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15412.187710] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15412.189940] mdc_intent_lock+0x353/0x7e0 [mdc] [15412.193748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15412.197379] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15412.199676] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15412.203531] lmv_intent_open+0x338/0x1150 [lmv] [15412.206941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15412.210088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15412.213162] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15412.214815] ll_intent_lock+0xc5/0xaa0 [lustre] [15412.216860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15412.220370] ll_lookup_it+0xe0e/0x2270 [lustre] [15412.223617] ? do_raw_spin_unlock+0x75/0x190 [15412.225563] ? _raw_spin_unlock+0x12/0x30 [15412.227377] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15412.229519] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15412.232014] ll_atomic_open+0x36a/0xd70 [lustre] [15412.234573] lookup_open+0x356/0xb40 [15412.236252] do_last+0x313/0xfc0 [15412.238123] ? nd_jump_root+0xe5/0x160 [15412.239628] ? path_init+0x437/0x520 [15412.241386] path_openat+0xf7/0x500 [15412.243064] ? do_raw_spin_unlock+0x75/0x190 [15412.245273] do_filp_open+0x99/0x140 [15412.247014] ? getname_flags+0x6e/0x330 [15412.251513] ? __check_object_size+0xff/0x256 [15412.252855] ? do_raw_spin_unlock+0x75/0x190 [15412.254659] ? _raw_spin_unlock+0x12/0x30 [15412.256769] do_sys_openat2+0x2b4/0x410 [15412.258256] do_sys_open+0x73/0xa0 [15412.259585] __x64_sys_openat+0x24/0x30 [15412.261074] do_syscall_64+0xc1/0x440 [15412.262447] entry_SYSCALL_64_after_hwframe+0x49/0xae [15412.265637] RIP: 0033:0x7fa749552332 [15412.266913] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15412.280028] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15412.285318] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15412.287768] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15412.291479] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15412.299198] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15412.306061] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15418.338014] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13501s! [15418.342088] Showing busy workqueues and worker pools: [15418.346175] workqueue events: flags=0x0 [15418.347192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15418.350525] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15418.356768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15418.366885] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15418.374492] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.379047] pending: kfree_rcu_monitor [15418.382105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15418.387907] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15418.394804] workqueue events_long: flags=0x0 [15418.401425] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.408402] pending: lu_objects_flush [obdclass] [15418.412288] workqueue events_unbound: flags=0x2 [15418.415650] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15418.419226] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15418.430942] workqueue events_power_efficient: flags=0x80 [15418.436744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.443056] pending: check_lifetime [15418.445462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.453525] pending: neigh_periodic_work [15418.457370] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.462187] pending: neigh_periodic_work [15418.466552] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.469428] pending: do_cache_clean [15418.470848] workqueue mm_percpu_wq: flags=0x8 [15418.480354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.488953] pending: vmstat_update [15418.491479] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.499106] pending: vmstat_update [15418.502050] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15418.504415] pending: vmstat_update [15418.505545] workqueue writeback: flags=0x4a [15418.510586] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15418.514005] pending: wb_workfn, wb_workfn [15418.516232] workqueue xprtiod: flags=0xa [15418.517248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15418.522341] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15418.526263] workqueue ipv6_addrconf: flags=0x40008 [15418.528876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15418.535493] pending: addrconf_verify_work [15418.538896] workqueue knbd0-recv: flags=0x1a [15418.543214] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15418.550496] in-flight: 651:recv_work [15418.556075] workqueue ptlrpc_pinger: flags=0x2 [15418.563102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15418.572004] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15418.579956] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13501s workers=4 idle: 37 31846 28897 [15418.586208] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15408s workers=2 idle: 653 [15420.059042] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15420.061452] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15420.080568] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15420.084352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15420.087362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15420.089515] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15420.096044] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15420.098182] RAX: 0000000000000b07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15420.100491] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15420.102689] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15420.104635] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15420.106977] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15420.109474] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15420.112093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15420.113731] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15420.115909] Call Trace: [15420.116795] [15420.117511] ? show_regs.cold.9+0x22/0x2f [15420.118962] ? watchdog_timer_fn.cold.6+0x62/0x10a [15420.120713] ? __hrtimer_run_queues+0x1c2/0x4d0 [15420.122341] ? watchdog+0x40/0x40 [15420.123769] ? hrtimer_interrupt+0x128/0x2c0 [15420.129105] ? smp_apic_timer_interrupt+0xa6/0x270 [15420.134672] ? apic_timer_interrupt+0xf/0x20 [15420.138047] [15420.139043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15420.141370] do_raw_spin_lock+0x177/0x190 [15420.143340] _raw_spin_lock+0x21/0x30 [15420.145124] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15420.151702] ? do_raw_spin_unlock+0x75/0x190 [15420.153470] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15420.156848] ? do_wait_intr+0xf0/0xf0 [15420.158958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15420.162053] mdc_close+0x53b/0xd70 [mdc] [15420.164450] lmv_close+0x18d/0x490 [lmv] [15420.167007] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15420.171760] ll_md_real_close+0xa5/0x2a0 [lustre] [15420.176400] ll_md_close+0x274/0x800 [lustre] [15420.178461] ll_file_release+0x128/0xc50 [lustre] [15420.181956] __fput+0x10d/0x440 [15420.183458] ____fput+0x12/0x20 [15420.184478] task_work_run+0xc6/0x110 [15420.185705] exit_to_usermode_loop+0x1dd/0x1f0 [15420.192917] do_syscall_64+0x3d6/0x440 [15420.196520] entry_SYSCALL_64_after_hwframe+0x49/0xae [15420.198869] RIP: 0033:0x7fa749552cd8 [15420.202430] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15420.219015] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15420.225982] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15420.231688] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15420.234934] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15420.241103] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15420.247916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15428.060406] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15428.074053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15428.126783] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15428.143368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15428.150694] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15428.157076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15428.160234] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15428.179671] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15428.188925] RAX: 0000000000005008 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15428.195753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15428.201823] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15428.208646] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15428.214593] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15428.218517] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15428.224615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15428.230128] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15428.234733] Call Trace: [15428.241166] [15428.242242] ? show_regs.cold.9+0x22/0x2f [15428.247683] ? watchdog_timer_fn.cold.6+0x62/0x10a [15428.251054] ? __hrtimer_run_queues+0x1c2/0x4d0 [15428.256395] ? watchdog+0x40/0x40 [15428.261058] ? hrtimer_interrupt+0x128/0x2c0 [15428.263821] ? smp_apic_timer_interrupt+0xa6/0x270 [15428.271443] ? apic_timer_interrupt+0xf/0x20 [15428.273906] [15428.277717] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15428.285003] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15428.290374] do_raw_spin_lock+0x177/0x190 [15428.295569] _raw_spin_lock+0x21/0x30 [15428.299976] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15428.305416] process_one_work+0x2c8/0x700 [15428.309463] worker_thread+0x296/0x6e0 [15428.311261] ? rescuer_thread+0x570/0x570 [15428.314478] kthread+0x1d1/0x200 [15428.315976] ? set_kthread_struct+0x70/0x70 [15428.320824] ret_from_fork+0x1f/0x30 [15432.058029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15432.060971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15432.079352] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15432.084104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15432.088203] RIP: 0010:native_safe_halt+0x2e/0x50 [15432.090110] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15432.098309] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15432.101988] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15432.105127] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15432.108031] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15432.110963] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15432.113114] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15432.115943] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15432.119111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15432.121770] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15432.124769] Call Trace: [15432.125700] [15432.126620] ? show_regs.cold.9+0x22/0x2f [15432.128087] ? watchdog_timer_fn.cold.6+0x62/0x10a [15432.130229] ? __hrtimer_run_queues+0x1c2/0x4d0 [15432.132164] ? watchdog+0x40/0x40 [15432.133770] ? hrtimer_interrupt+0x128/0x2c0 [15432.135762] ? smp_apic_timer_interrupt+0xa6/0x270 [15432.137742] ? apic_timer_interrupt+0xf/0x20 [15432.139706] [15432.140733] ? native_safe_halt+0x2e/0x50 [15432.142509] kvm_wait+0xcd/0xe0 [15432.143712] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15432.146045] do_raw_spin_lock+0x177/0x190 [15432.147581] _raw_spin_lock+0x21/0x30 [15432.149437] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15432.151886] obd_mod_free+0x46/0x140 [ptlrpc] [15432.154104] mdc_commit_open+0x91/0xb0 [mdc] [15432.156278] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15432.159088] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15432.162528] after_reply+0xe2c/0x1780 [ptlrpc] [15432.165899] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15432.168796] ? __next_timer_interrupt+0x160/0x160 [15432.170813] ptlrpcd+0xa82/0xe40 [ptlrpc] [15432.172898] ? do_wait_intr+0xf0/0xf0 [15432.174407] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15432.176555] kthread+0x1d1/0x200 [15432.177933] ? set_kthread_struct+0x70/0x70 [15432.179767] ret_from_fork+0x1f/0x30 [15444.054013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15444.058113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15444.139379] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15444.157569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15444.169468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15444.174438] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15444.190881] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15444.196727] RAX: 0000000000005224 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15444.201906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15444.212405] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15444.226773] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15444.234117] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15444.243031] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15444.250529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15444.257549] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15444.264719] Call Trace: [15444.265843] [15444.267427] ? show_regs.cold.9+0x22/0x2f [15444.270268] ? watchdog_timer_fn.cold.6+0x62/0x10a [15444.280508] ? __hrtimer_run_queues+0x1c2/0x4d0 [15444.289687] ? watchdog+0x40/0x40 [15444.295476] ? hrtimer_interrupt+0x128/0x2c0 [15444.309517] ? smp_apic_timer_interrupt+0xa6/0x270 [15444.325728] ? apic_timer_interrupt+0xf/0x20 [15444.327249] [15444.327999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15444.336412] do_raw_spin_lock+0x177/0x190 [15444.339102] _raw_spin_lock+0x21/0x30 [15444.342943] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15444.351017] ? do_raw_spin_unlock+0x75/0x190 [15444.356946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15444.358855] ? do_wait_intr+0xf0/0xf0 [15444.368536] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15444.386789] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15444.400246] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15444.406944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15444.416261] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15444.424370] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15444.433003] mdc_intent_lock+0x353/0x7e0 [mdc] [15444.442549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15444.452234] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15444.459525] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15444.466511] lmv_intent_open+0x338/0x1150 [lmv] [15444.480371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15444.489782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15444.496321] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15444.497927] ll_intent_lock+0xc5/0xaa0 [lustre] [15444.504770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15444.512040] ll_lookup_it+0xe0e/0x2270 [lustre] [15444.518098] ? do_raw_spin_unlock+0x75/0x190 [15444.522774] ? _raw_spin_unlock+0x12/0x30 [15444.531468] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15444.543180] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15444.552950] ll_atomic_open+0x36a/0xd70 [lustre] [15444.554717] lookup_open+0x356/0xb40 [15444.567092] do_last+0x313/0xfc0 [15444.568208] ? nd_jump_root+0xe5/0x160 [15444.583911] ? path_init+0x437/0x520 [15444.586346] path_openat+0xf7/0x500 [15444.587786] ? do_raw_spin_unlock+0x75/0x190 [15444.591737] do_filp_open+0x99/0x140 [15444.593915] ? getname_flags+0x6e/0x330 [15444.602375] ? __check_object_size+0xff/0x256 [15444.608825] ? do_raw_spin_unlock+0x75/0x190 [15444.611447] ? _raw_spin_unlock+0x12/0x30 [15444.615887] do_sys_openat2+0x2b4/0x410 [15444.617826] do_sys_open+0x73/0xa0 [15444.622636] __x64_sys_openat+0x24/0x30 [15444.624456] do_syscall_64+0xc1/0x440 [15444.633272] entry_SYSCALL_64_after_hwframe+0x49/0xae [15444.635720] RIP: 0033:0x7fa749552332 [15444.637016] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15444.663781] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15444.669936] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15444.678219] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15444.684068] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15444.687954] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15444.695812] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15448.061522] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15448.069807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15448.126923] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15448.136290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15448.143924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15448.150551] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15448.167556] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15448.176355] RAX: 0000000000000088 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15448.179893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15448.186926] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15448.189319] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15448.195853] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15448.200843] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15448.209451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15448.217358] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15448.223817] Call Trace: [15448.228789] [15448.229328] ? show_regs.cold.9+0x22/0x2f [15448.230605] ? watchdog_timer_fn.cold.6+0x62/0x10a [15448.232240] ? __hrtimer_run_queues+0x1c2/0x4d0 [15448.233802] ? watchdog+0x40/0x40 [15448.235057] ? hrtimer_interrupt+0x128/0x2c0 [15448.236471] ? smp_apic_timer_interrupt+0xa6/0x270 [15448.238028] ? apic_timer_interrupt+0xf/0x20 [15448.239592] [15448.240355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15448.242363] do_raw_spin_lock+0x177/0x190 [15448.243852] _raw_spin_lock+0x21/0x30 [15448.245286] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15448.247428] ? do_raw_spin_unlock+0x75/0x190 [15448.248957] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15448.250768] ? do_wait_intr+0xf0/0xf0 [15448.251986] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15448.254030] mdc_close+0x53b/0xd70 [mdc] [15448.255387] lmv_close+0x18d/0x490 [lmv] [15448.256681] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15448.258647] ll_md_real_close+0xa5/0x2a0 [lustre] [15448.260449] ll_md_close+0x274/0x800 [lustre] [15448.262143] ll_file_release+0x128/0xc50 [lustre] [15448.263857] __fput+0x10d/0x440 [15448.264960] ____fput+0x12/0x20 [15448.265982] task_work_run+0xc6/0x110 [15448.267277] exit_to_usermode_loop+0x1dd/0x1f0 [15448.268696] do_syscall_64+0x3d6/0x440 [15448.269853] entry_SYSCALL_64_after_hwframe+0x49/0xae [15448.271442] RIP: 0033:0x7fa749552cd8 [15448.272752] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15448.286481] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15448.291993] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15448.297183] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15448.300921] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15448.307635] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15448.315551] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15449.057027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13531s! [15449.073371] Showing busy workqueues and worker pools: [15449.076442] workqueue events: flags=0x0 [15449.081180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15449.084574] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15449.098452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15449.108643] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15449.119742] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.124823] pending: kfree_rcu_monitor [15449.130647] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15449.142679] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15449.154277] workqueue events_long: flags=0x0 [15449.156892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.161178] pending: lu_objects_flush [obdclass] [15449.164898] workqueue events_unbound: flags=0x2 [15449.167747] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15449.169932] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15449.177557] workqueue events_power_efficient: flags=0x80 [15449.182898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.190630] pending: check_lifetime [15449.192495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.200333] pending: neigh_periodic_work [15449.202754] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.207999] pending: neigh_periodic_work [15449.210928] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.220491] pending: do_cache_clean [15449.227167] workqueue mm_percpu_wq: flags=0x8 [15449.228884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.234713] pending: vmstat_update [15449.235940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.240805] pending: vmstat_update [15449.244721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15449.248567] pending: vmstat_update [15449.254182] workqueue writeback: flags=0x4a [15449.255691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15449.265897] pending: wb_workfn, wb_workfn [15449.270323] workqueue xprtiod: flags=0xa [15449.272413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15449.279218] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15449.284864] workqueue ipv6_addrconf: flags=0x40008 [15449.293816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15449.299466] pending: addrconf_verify_work [15449.301551] workqueue knbd0-recv: flags=0x1a [15449.306467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15449.314049] in-flight: 651:recv_work [15449.319522] workqueue ptlrpc_pinger: flags=0x2 [15449.321534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15449.324892] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15449.327947] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13532s workers=4 idle: 37 31846 28897 [15449.334450] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15439s workers=2 idle: 653 [15456.057054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15456.061335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15456.112596] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15456.119045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15456.124533] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15456.127997] RIP: 0010:native_safe_halt+0x2e/0x50 [15456.135207] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15456.148821] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15456.154598] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15456.161865] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15456.166684] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15456.173631] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15456.179893] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15456.184432] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15456.190224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15456.193797] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15456.199736] Call Trace: [15456.201905] [15456.203714] ? show_regs.cold.9+0x22/0x2f [15456.207245] ? watchdog_timer_fn.cold.6+0x62/0x10a [15456.215527] ? __hrtimer_run_queues+0x1c2/0x4d0 [15456.223852] ? watchdog+0x40/0x40 [15456.232442] ? hrtimer_interrupt+0x128/0x2c0 [15456.238777] ? smp_apic_timer_interrupt+0xa6/0x270 [15456.243501] ? apic_timer_interrupt+0xf/0x20 [15456.249538] [15456.251768] ? native_safe_halt+0x2e/0x50 [15456.253670] kvm_wait+0xcd/0xe0 [15456.259364] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15456.264271] do_raw_spin_lock+0x177/0x190 [15456.267520] _raw_spin_lock+0x21/0x30 [15456.270287] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15456.273146] process_one_work+0x2c8/0x700 [15456.275677] worker_thread+0x296/0x6e0 [15456.277609] ? rescuer_thread+0x570/0x570 [15456.279356] kthread+0x1d1/0x200 [15456.285973] ? set_kthread_struct+0x70/0x70 [15456.287774] ret_from_fork+0x1f/0x30 [15460.058017] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15460.062103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15460.095337] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15460.103952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15460.116209] RIP: 0010:native_safe_halt+0x2e/0x50 [15460.119674] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15460.141089] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15460.146102] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15460.152677] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15460.156597] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15460.162237] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15460.166020] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15460.172339] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15460.175616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15460.178841] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15460.184017] Call Trace: [15460.185357] [15460.186097] ? show_regs.cold.9+0x22/0x2f [15460.187471] ? watchdog_timer_fn.cold.6+0x62/0x10a [15460.193418] ? __hrtimer_run_queues+0x1c2/0x4d0 [15460.196055] ? watchdog+0x40/0x40 [15460.198498] ? hrtimer_interrupt+0x128/0x2c0 [15460.200433] ? smp_apic_timer_interrupt+0xa6/0x270 [15460.203217] ? apic_timer_interrupt+0xf/0x20 [15460.205940] [15460.206978] ? native_safe_halt+0x2e/0x50 [15460.208712] kvm_wait+0xcd/0xe0 [15460.210301] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15460.214574] do_raw_spin_lock+0x177/0x190 [15460.216688] _raw_spin_lock+0x21/0x30 [15460.218338] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15460.220683] obd_mod_free+0x46/0x140 [ptlrpc] [15460.222790] mdc_commit_open+0x91/0xb0 [mdc] [15460.224828] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15460.227433] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15460.229936] after_reply+0xe2c/0x1780 [ptlrpc] [15460.233825] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15460.236382] ? __next_timer_interrupt+0x160/0x160 [15460.238228] ptlrpcd+0xa82/0xe40 [ptlrpc] [15460.240103] ? do_wait_intr+0xf0/0xf0 [15460.246953] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15460.255613] kthread+0x1d1/0x200 [15460.258206] ? set_kthread_struct+0x70/0x70 [15460.263597] ret_from_fork+0x1f/0x30 [15473.994063] rcu: INFO: rcu_sched self-detected stall on CPU [15473.998154] rcu: 2-...!: (7851030 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [15474.006196] (t=13560305 jiffies g=258701 q=5178) [15474.012703] rcu: rcu_sched kthread starved for 13557100 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [15474.027138] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15474.034748] rcu: RCU grace-period kthread stack dump: [15474.053958] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15474.063206] Call Trace: [15474.072710] __schedule+0x351/0xcb0 [15474.073975] schedule+0xc0/0x180 [15474.075071] schedule_timeout+0xb4/0x190 [15474.086993] ? do_raw_spin_unlock+0x75/0x190 [15474.091172] ? __next_timer_interrupt+0x160/0x160 [15474.096334] ? prepare_to_swait_event+0xf3/0x140 [15474.097747] rcu_gp_kthread+0x8dd/0x15b0 [15474.111187] ? force_qs_rnp+0x2c0/0x2c0 [15474.118780] kthread+0x1d1/0x200 [15474.122004] ? set_kthread_struct+0x70/0x70 [15474.123109] ret_from_fork+0x1f/0x30 [15474.126384] rcu: Stack dump where RCU GP kthread last ran: [15474.128706] Sending NMI from CPU 2 to CPUs 0: [15474.146973] NMI backtrace for cpu 0 [15474.146981] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15474.146983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15474.146985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.146987] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15474.146989] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 [15474.146992] RAX: 000000000000291d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15474.146993] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15474.146995] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15474.146996] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15474.146998] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15474.146999] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15474.147000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15474.147002] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15474.147003] Call Trace: [15474.147004] [15474.147005] ? show_regs.cold.9+0x22/0x2f [15474.147006] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15474.147007] ? nmi_cpu_backtrace_handler+0x15/0x40 [15474.147008] ? nmi_handle+0xbd/0x210 [15474.147009] ? default_do_nmi+0x4b/0x1b0 [15474.147010] ? do_nmi+0x280/0x380 [15474.147011] ? end_repeat_nmi+0x16/0x1f [15474.147012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.147019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.147020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.147088] [15474.147091] do_raw_spin_lock+0x177/0x190 [15474.147096] _raw_spin_lock+0x21/0x30 [15474.147097] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15474.147098] ? do_raw_spin_unlock+0x75/0x190 [15474.147100] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15474.147101] ? do_wait_intr+0xf0/0xf0 [15474.147102] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15474.147103] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15474.147104] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15474.147105] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.147107] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15474.147108] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15474.147109] mdc_intent_lock+0x353/0x7e0 [mdc] [15474.147110] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.147112] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15474.147113] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15474.147114] lmv_intent_open+0x338/0x1150 [lmv] [15474.147115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.147116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.147117] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15474.147118] ll_intent_lock+0xc5/0xaa0 [lustre] [15474.147120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.147121] ll_lookup_it+0xe0e/0x2270 [lustre] [15474.147122] ? do_raw_spin_unlock+0x75/0x190 [15474.147123] ? _raw_spin_unlock+0x12/0x30 [15474.147124] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15474.147126] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15474.147127] ll_atomic_open+0x36a/0xd70 [lustre] [15474.147128] lookup_open+0x356/0xb40 [15474.147129] do_last+0x313/0xfc0 [15474.147130] ? nd_jump_root+0xe5/0x160 [15474.147131] ? path_init+0x437/0x520 [15474.147132] path_openat+0xf7/0x500 [15474.147133] ? do_raw_spin_unlock+0x75/0x190 [15474.147134] do_filp_open+0x99/0x140 [15474.147135] ? getname_flags+0x6e/0x330 [15474.147136] ? __check_object_size+0xff/0x256 [15474.147137] ? do_raw_spin_unlock+0x75/0x190 [15474.147138] ? _raw_spin_unlock+0x12/0x30 [15474.147139] do_sys_openat2+0x2b4/0x410 [15474.147140] do_sys_open+0x73/0xa0 [15474.147141] __x64_sys_openat+0x24/0x30 [15474.147142] do_syscall_64+0xc1/0x440 [15474.147143] entry_SYSCALL_64_after_hwframe+0x49/0xae [15474.147145] RIP: 0033:0x7fa749552332 [15474.147146] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15474.147154] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15474.147156] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15474.147158] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15474.147159] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15474.147161] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15474.147162] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15474.153401] NMI backtrace for cpu 2 [15474.741245] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15474.748365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15474.754389] Call Trace: [15474.755719] [15474.758703] ? dump_stack+0xbb/0x10e [15474.761230] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15474.764696] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15474.768460] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15474.773384] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15474.776476] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15474.782670] ? rcu_dump_cpu_stacks+0x11b/0x171 [15474.784032] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15474.787277] ? update_process_times+0x6d/0xc0 [15474.789440] ? tick_sched_handle.isra.18+0x27/0x90 [15474.793097] ? tick_sched_timer+0x53/0xb0 [15474.794294] ? __hrtimer_run_queues+0x1c2/0x4d0 [15474.795526] ? tick_sched_handle.isra.18+0x90/0x90 [15474.802968] ? hrtimer_interrupt+0x128/0x2c0 [15474.806111] ? smp_apic_timer_interrupt+0xa6/0x270 [15474.809014] ? apic_timer_interrupt+0xf/0x20 [15474.812439] [15474.813114] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15474.814422] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15474.815778] ? do_raw_spin_lock+0x177/0x190 [15474.818089] ? _raw_spin_lock+0x21/0x30 [15474.819541] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15474.823441] ? obd_mod_free+0x46/0x140 [ptlrpc] [15474.829765] ? mdc_commit_open+0x91/0xb0 [mdc] [15474.832470] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15474.835993] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15474.840453] ? after_reply+0xe2c/0x1780 [ptlrpc] [15474.846238] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15474.852583] ? __next_timer_interrupt+0x160/0x160 [15474.857735] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15474.859217] ? do_wait_intr+0xf0/0xf0 [15474.860434] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15474.871221] ? kthread+0x1d1/0x200 [15474.872440] ? set_kthread_struct+0x70/0x70 [15474.879815] ? ret_from_fork+0x1f/0x30 [15476.059991] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15476.068244] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15476.086376] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15476.091427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15476.095696] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15476.098315] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15476.106620] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15476.109977] RAX: 0000000000000117 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15476.112880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15476.116204] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15476.119562] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15476.122501] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15476.125772] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15476.129150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15476.132203] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15476.135631] Call Trace: [15476.136725] [15476.137630] ? show_regs.cold.9+0x22/0x2f [15476.139349] ? watchdog_timer_fn.cold.6+0x62/0x10a [15476.141521] ? __hrtimer_run_queues+0x1c2/0x4d0 [15476.143386] ? watchdog+0x40/0x40 [15476.144922] ? hrtimer_interrupt+0x128/0x2c0 [15476.146781] ? smp_apic_timer_interrupt+0xa6/0x270 [15476.149534] ? apic_timer_interrupt+0xf/0x20 [15476.151300] [15476.152638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15476.154901] do_raw_spin_lock+0x177/0x190 [15476.156726] _raw_spin_lock+0x21/0x30 [15476.158653] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15476.161294] ? do_raw_spin_unlock+0x75/0x190 [15476.163035] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15476.165492] ? do_wait_intr+0xf0/0xf0 [15476.167018] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15476.169428] mdc_close+0x53b/0xd70 [mdc] [15476.171045] lmv_close+0x18d/0x490 [lmv] [15476.172873] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15476.175933] ll_md_real_close+0xa5/0x2a0 [lustre] [15476.178183] ll_md_close+0x274/0x800 [lustre] [15476.180269] ll_file_release+0x128/0xc50 [lustre] [15476.182747] __fput+0x10d/0x440 [15476.184172] ____fput+0x12/0x20 [15476.185469] task_work_run+0xc6/0x110 [15476.187380] exit_to_usermode_loop+0x1dd/0x1f0 [15476.189271] do_syscall_64+0x3d6/0x440 [15476.191361] entry_SYSCALL_64_after_hwframe+0x49/0xae [15476.193753] RIP: 0033:0x7fa749552cd8 [15476.195705] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15476.204796] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15476.208120] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15476.211566] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15476.214357] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15476.217105] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15476.220280] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15479.781847] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13562s! [15479.786487] Showing busy workqueues and worker pools: [15479.789088] workqueue events: flags=0x0 [15479.791682] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15479.795411] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15479.801718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15479.805696] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15479.810472] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.812512] pending: kfree_rcu_monitor [15479.814478] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15479.817898] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15479.822385] workqueue events_long: flags=0x0 [15479.824335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.828134] pending: lu_objects_flush [obdclass] [15479.830576] workqueue events_unbound: flags=0x2 [15479.832226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15479.835325] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15479.840278] workqueue events_power_efficient: flags=0x80 [15479.843074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.846279] pending: check_lifetime [15479.850264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.855121] pending: neigh_periodic_work [15479.857187] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.862793] pending: neigh_periodic_work [15479.867273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.871827] pending: do_cache_clean [15479.878223] workqueue mm_percpu_wq: flags=0x8 [15479.880305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.890023] pending: vmstat_update [15479.891304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.895346] pending: vmstat_update [15479.898326] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15479.914494] pending: vmstat_update [15479.917806] workqueue writeback: flags=0x4a [15479.924329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15479.929167] pending: wb_workfn, wb_workfn [15479.932362] workqueue xprtiod: flags=0xa [15479.938481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15479.944888] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15479.953366] workqueue ipv6_addrconf: flags=0x40008 [15479.956401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15479.963460] pending: addrconf_verify_work [15479.965137] workqueue knbd0-recv: flags=0x1a [15479.979322] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15479.991651] in-flight: 651:recv_work [15479.993122] workqueue ptlrpc_pinger: flags=0x2 [15479.999933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15480.003369] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15480.007961] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13562s workers=4 idle: 37 31846 28897 [15480.013840] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15469s workers=2 idle: 653 [15484.057024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15484.062028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15484.094556] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15484.100638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15484.104848] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15484.108567] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15484.111174] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15484.121003] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15484.124667] RAX: 0000000000007d1b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15484.128822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15484.131932] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15484.135431] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15484.139672] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15484.143193] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15484.146521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15484.149274] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15484.153601] Call Trace: [15484.155326] [15484.156439] ? show_regs.cold.9+0x22/0x2f [15484.158767] ? watchdog_timer_fn.cold.6+0x62/0x10a [15484.161388] ? __hrtimer_run_queues+0x1c2/0x4d0 [15484.164357] ? watchdog+0x40/0x40 [15484.166098] ? hrtimer_interrupt+0x128/0x2c0 [15484.174472] ? smp_apic_timer_interrupt+0xa6/0x270 [15484.177000] ? apic_timer_interrupt+0xf/0x20 [15484.181082] [15484.181682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15484.183490] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15484.190055] do_raw_spin_lock+0x177/0x190 [15484.192526] _raw_spin_lock+0x21/0x30 [15484.194373] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15484.198246] process_one_work+0x2c8/0x700 [15484.202371] worker_thread+0x296/0x6e0 [15484.206926] ? rescuer_thread+0x570/0x570 [15484.210056] kthread+0x1d1/0x200 [15484.217729] ? set_kthread_struct+0x70/0x70 [15484.221250] ret_from_fork+0x1f/0x30 [15500.058021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15500.062018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15500.085920] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15500.090602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15500.095247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15500.098332] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15500.107667] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15500.111383] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15500.114694] RDX: 0000000000006385 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15500.117701] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15500.121445] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15500.124993] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15500.128471] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15500.132400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15500.134735] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15500.137595] Call Trace: [15500.138285] [15500.139261] ? show_regs.cold.9+0x22/0x2f [15500.140649] ? watchdog_timer_fn.cold.6+0x62/0x10a [15500.143403] ? __hrtimer_run_queues+0x1c2/0x4d0 [15500.145635] ? watchdog+0x40/0x40 [15500.147481] ? hrtimer_interrupt+0x128/0x2c0 [15500.149663] ? smp_apic_timer_interrupt+0xa6/0x270 [15500.151660] ? apic_timer_interrupt+0xf/0x20 [15500.154138] [15500.155041] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15500.157769] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15500.160609] do_raw_spin_lock+0x177/0x190 [15500.162238] _raw_spin_lock+0x21/0x30 [15500.164047] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15500.166558] obd_mod_free+0x46/0x140 [ptlrpc] [15500.168687] mdc_commit_open+0x91/0xb0 [mdc] [15500.170648] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15500.173292] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15500.176384] after_reply+0xe2c/0x1780 [ptlrpc] [15500.178751] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15500.181641] ? __next_timer_interrupt+0x160/0x160 [15500.184005] ptlrpcd+0xa82/0xe40 [ptlrpc] [15500.186828] ? do_wait_intr+0xf0/0xf0 [15500.190707] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15500.194593] kthread+0x1d1/0x200 [15500.196286] ? set_kthread_struct+0x70/0x70 [15500.198514] ret_from_fork+0x1f/0x30 [15504.054022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15504.056007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15504.059095] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15504.067329] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15504.073658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15504.073717] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15504.073723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15504.073725] RIP: 0010:native_safe_halt+0x2e/0x50 [15504.073736] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15504.073739] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15504.073742] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15504.073744] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15504.073745] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15504.073747] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15504.073748] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15504.073751] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15504.073753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15504.073754] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15504.073760] Call Trace: [15504.073765] [15504.073770] ? show_regs.cold.9+0x22/0x2f [15504.073777] ? watchdog_timer_fn.cold.6+0x62/0x10a [15504.073782] ? __hrtimer_run_queues+0x1c2/0x4d0 [15504.073784] ? watchdog+0x40/0x40 [15504.073787] ? hrtimer_interrupt+0x128/0x2c0 [15504.073790] ? smp_apic_timer_interrupt+0xa6/0x270 [15504.073795] ? apic_timer_interrupt+0xf/0x20 [15504.073798] [15504.073800] ? native_safe_halt+0x2e/0x50 [15504.073802] kvm_wait+0xcd/0xe0 [15504.073808] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15504.073814] do_raw_spin_lock+0x177/0x190 [15504.073817] _raw_spin_lock+0x21/0x30 [15504.073820] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15504.074109] ? do_raw_spin_unlock+0x75/0x190 [15504.074115] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15504.074313] ? do_wait_intr+0xf0/0xf0 [15504.074320] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15504.074645] mdc_close+0x53b/0xd70 [mdc] [15504.074681] lmv_close+0x18d/0x490 [lmv] [15504.074704] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15504.074787] ll_md_real_close+0xa5/0x2a0 [lustre] [15504.074838] ll_md_close+0x274/0x800 [lustre] [15504.074891] ll_file_release+0x128/0xc50 [lustre] [15504.074944] __fput+0x10d/0x440 [15504.074953] ____fput+0x12/0x20 [15504.074956] task_work_run+0xc6/0x110 [15504.074962] exit_to_usermode_loop+0x1dd/0x1f0 [15504.074968] do_syscall_64+0x3d6/0x440 [15504.074972] entry_SYSCALL_64_after_hwframe+0x49/0xae [15504.074978] RIP: 0033:0x7fa749552cd8 [15504.074984] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15504.074987] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15504.074991] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15504.074992] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15504.074994] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15504.074996] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15504.074997] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15504.179624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15504.182595] RIP: 0010:native_safe_halt+0x2e/0x50 [15504.183812] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15504.188936] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15504.191284] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15504.193592] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15504.195655] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15504.197534] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15504.199615] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15504.201499] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15504.204249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15504.205768] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15504.208145] Call Trace: [15504.208773] [15504.209322] ? show_regs.cold.9+0x22/0x2f [15504.210592] ? watchdog_timer_fn.cold.6+0x62/0x10a [15504.211919] ? __hrtimer_run_queues+0x1c2/0x4d0 [15504.213192] ? watchdog+0x40/0x40 [15504.214273] ? hrtimer_interrupt+0x128/0x2c0 [15504.215328] ? smp_apic_timer_interrupt+0xa6/0x270 [15504.216622] ? apic_timer_interrupt+0xf/0x20 [15504.217955] [15504.218840] ? native_safe_halt+0x2e/0x50 [15504.219830] kvm_wait+0xcd/0xe0 [15504.220743] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15504.221992] do_raw_spin_lock+0x177/0x190 [15504.223194] _raw_spin_lock+0x21/0x30 [15504.224218] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15504.225907] ? do_raw_spin_unlock+0x75/0x190 [15504.227193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15504.228777] ? do_wait_intr+0xf0/0xf0 [15504.229781] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15504.231383] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15504.233049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15504.234824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15504.236289] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15504.237870] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15504.239429] mdc_intent_lock+0x353/0x7e0 [mdc] [15504.240801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15504.242725] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15504.244448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15504.246234] lmv_intent_open+0x338/0x1150 [lmv] [15504.247482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15504.249096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15504.250737] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15504.251976] ll_intent_lock+0xc5/0xaa0 [lustre] [15504.253397] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15504.255323] ll_lookup_it+0xe0e/0x2270 [lustre] [15504.256730] ? do_raw_spin_unlock+0x75/0x190 [15504.257946] ? _raw_spin_unlock+0x12/0x30 [15504.259144] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15504.260677] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15504.261941] ll_atomic_open+0x36a/0xd70 [lustre] [15504.263446] lookup_open+0x356/0xb40 [15504.264386] do_last+0x313/0xfc0 [15504.265313] ? nd_jump_root+0xe5/0x160 [15504.266351] ? path_init+0x437/0x520 [15504.267294] path_openat+0xf7/0x500 [15504.268358] ? do_raw_spin_unlock+0x75/0x190 [15504.269640] do_filp_open+0x99/0x140 [15504.270463] ? getname_flags+0x6e/0x330 [15504.271613] ? __check_object_size+0xff/0x256 [15504.272798] ? do_raw_spin_unlock+0x75/0x190 [15504.274058] ? _raw_spin_unlock+0x12/0x30 [15504.275258] do_sys_openat2+0x2b4/0x410 [15504.276312] do_sys_open+0x73/0xa0 [15504.277340] __x64_sys_openat+0x24/0x30 [15504.278322] do_syscall_64+0xc1/0x440 [15504.279338] entry_SYSCALL_64_after_hwframe+0x49/0xae [15504.280715] RIP: 0033:0x7fa749552332 [15504.281672] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15504.286689] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15504.288720] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15504.290616] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15504.292715] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15504.294639] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15504.296693] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15510.498537] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13593s! [15510.504495] Showing busy workqueues and worker pools: [15510.509203] workqueue events: flags=0x0 [15510.511741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15510.518404] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15510.528407] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15510.533495] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15510.545662] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.552380] pending: kfree_rcu_monitor [15510.553758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15510.560194] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15510.570356] workqueue events_long: flags=0x0 [15510.575597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.583176] pending: lu_objects_flush [obdclass] [15510.584977] workqueue events_unbound: flags=0x2 [15510.587338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15510.590724] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15510.599447] workqueue events_power_efficient: flags=0x80 [15510.603787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.616967] pending: check_lifetime [15510.621375] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.625457] pending: neigh_periodic_work [15510.627902] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.631852] pending: neigh_periodic_work [15510.634919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.639286] pending: do_cache_clean [15510.641185] workqueue mm_percpu_wq: flags=0x8 [15510.643779] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.646868] pending: vmstat_update [15510.649239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.655238] pending: vmstat_update [15510.657107] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15510.663312] pending: vmstat_update [15510.665098] workqueue writeback: flags=0x4a [15510.670879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15510.673399] pending: wb_workfn, wb_workfn [15510.676467] workqueue xprtiod: flags=0xa [15510.681413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15510.685159] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15510.695942] workqueue ipv6_addrconf: flags=0x40008 [15510.698394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15510.702880] pending: addrconf_verify_work [15510.705598] workqueue knbd0-recv: flags=0x1a [15510.708120] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15510.711969] in-flight: 651:recv_work [15510.715212] workqueue ptlrpc_pinger: flags=0x2 [15510.722141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15510.729244] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15510.732851] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13593s workers=4 idle: 37 31846 28897 [15510.737480] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15500s workers=2 idle: 653 [15512.058148] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15512.066458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15512.109399] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15512.128730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15512.131656] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15512.138781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15512.144036] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15512.154677] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15512.159914] RAX: 0000000000001c09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15512.163822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15512.172365] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15512.177946] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15512.180738] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15512.183913] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15512.187745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15512.192105] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15512.195635] Call Trace: [15512.196670] [15512.197751] ? show_regs.cold.9+0x22/0x2f [15512.200836] ? watchdog_timer_fn.cold.6+0x62/0x10a [15512.202718] ? __hrtimer_run_queues+0x1c2/0x4d0 [15512.204638] ? watchdog+0x40/0x40 [15512.205954] ? hrtimer_interrupt+0x128/0x2c0 [15512.209312] ? smp_apic_timer_interrupt+0xa6/0x270 [15512.212118] ? apic_timer_interrupt+0xf/0x20 [15512.213881] [15512.214634] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15512.216765] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15512.219160] do_raw_spin_lock+0x177/0x190 [15512.220644] _raw_spin_lock+0x21/0x30 [15512.221693] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15512.223238] process_one_work+0x2c8/0x700 [15512.224298] worker_thread+0x296/0x6e0 [15512.225215] ? rescuer_thread+0x570/0x570 [15512.226356] kthread+0x1d1/0x200 [15512.227150] ? set_kthread_struct+0x70/0x70 [15512.228737] ret_from_fork+0x1f/0x30 [15528.058203] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15528.063409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15528.093983] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15528.099339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15528.103172] RIP: 0010:native_safe_halt+0x2e/0x50 [15528.104946] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15528.112664] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15528.115899] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15528.118965] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15528.122134] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15528.125406] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15528.128314] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15528.131494] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15528.135038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15528.137480] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15528.142853] Call Trace: [15528.144004] [15528.144965] ? show_regs.cold.9+0x22/0x2f [15528.147095] ? watchdog_timer_fn.cold.6+0x62/0x10a [15528.150304] ? __hrtimer_run_queues+0x1c2/0x4d0 [15528.152719] ? watchdog+0x40/0x40 [15528.154635] ? hrtimer_interrupt+0x128/0x2c0 [15528.156427] ? smp_apic_timer_interrupt+0xa6/0x270 [15528.158861] ? apic_timer_interrupt+0xf/0x20 [15528.160808] [15528.161612] ? native_safe_halt+0x2e/0x50 [15528.163452] kvm_wait+0xcd/0xe0 [15528.164762] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15528.166941] do_raw_spin_lock+0x177/0x190 [15528.168573] _raw_spin_lock+0x21/0x30 [15528.170136] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15528.173002] obd_mod_free+0x46/0x140 [ptlrpc] [15528.175245] mdc_commit_open+0x91/0xb0 [mdc] [15528.177168] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15528.181433] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15528.183056] after_reply+0xe2c/0x1780 [ptlrpc] [15528.186857] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15528.189691] ? __next_timer_interrupt+0x160/0x160 [15528.191713] ptlrpcd+0xa82/0xe40 [ptlrpc] [15528.193950] ? do_wait_intr+0xf0/0xf0 [15528.195591] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15528.199551] kthread+0x1d1/0x200 [15528.200801] ? set_kthread_struct+0x70/0x70 [15528.202535] ret_from_fork+0x1f/0x30 [15532.059412] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15532.067860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15532.100778] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15532.111360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15532.117371] RIP: 0010:native_safe_halt+0x2e/0x50 [15532.121544] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15532.132242] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15532.135338] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15532.138412] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15532.140406] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15532.144401] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15532.147534] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15532.150511] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15532.152786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15532.154328] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15532.156175] Call Trace: [15532.156796] [15532.157380] ? show_regs.cold.9+0x22/0x2f [15532.158914] ? watchdog_timer_fn.cold.6+0x62/0x10a [15532.164531] ? __hrtimer_run_queues+0x1c2/0x4d0 [15532.167000] ? watchdog+0x40/0x40 [15532.171863] ? hrtimer_interrupt+0x128/0x2c0 [15532.174416] ? smp_apic_timer_interrupt+0xa6/0x270 [15532.177379] ? apic_timer_interrupt+0xf/0x20 [15532.179209] [15532.183612] ? native_safe_halt+0x2e/0x50 [15532.186485] kvm_wait+0xcd/0xe0 [15532.188409] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15532.191735] do_raw_spin_lock+0x177/0x190 [15532.196055] _raw_spin_lock+0x21/0x30 [15532.198174] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15532.202281] ? do_raw_spin_unlock+0x75/0x190 [15532.204311] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15532.207365] ? do_wait_intr+0xf0/0xf0 [15532.208912] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15532.213113] mdc_close+0x53b/0xd70 [mdc] [15532.215122] lmv_close+0x18d/0x490 [lmv] [15532.216921] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15532.219330] ll_md_real_close+0xa5/0x2a0 [lustre] [15532.222982] ll_md_close+0x274/0x800 [lustre] [15532.225604] ll_file_release+0x128/0xc50 [lustre] [15532.229012] __fput+0x10d/0x440 [15532.230154] ____fput+0x12/0x20 [15532.232270] task_work_run+0xc6/0x110 [15532.233992] exit_to_usermode_loop+0x1dd/0x1f0 [15532.236963] do_syscall_64+0x3d6/0x440 [15532.239019] entry_SYSCALL_64_after_hwframe+0x49/0xae [15532.242028] RIP: 0033:0x7fa749552cd8 [15532.245442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15532.255214] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15532.265961] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15532.269888] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15532.274267] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15532.277836] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15532.281436] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15536.055682] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15536.061984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15536.081721] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15536.086158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15536.090244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15536.092435] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15536.098396] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15536.100702] RAX: 0000000000006270 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15536.103233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15536.105434] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15536.107759] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15536.110131] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15536.112910] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15536.115403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15536.117468] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15536.120063] Call Trace: [15536.120750] [15536.121424] ? show_regs.cold.9+0x22/0x2f [15536.122956] ? watchdog_timer_fn.cold.6+0x62/0x10a [15536.124745] ? __hrtimer_run_queues+0x1c2/0x4d0 [15536.126382] ? watchdog+0x40/0x40 [15536.127660] ? hrtimer_interrupt+0x128/0x2c0 [15536.129383] ? smp_apic_timer_interrupt+0xa6/0x270 [15536.131299] ? apic_timer_interrupt+0xf/0x20 [15536.132745] [15536.133645] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15536.135675] do_raw_spin_lock+0x177/0x190 [15536.136931] _raw_spin_lock+0x21/0x30 [15536.138313] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15536.140346] ? do_raw_spin_unlock+0x75/0x190 [15536.142143] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15536.143991] ? do_wait_intr+0xf0/0xf0 [15536.145165] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15536.146895] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15536.148428] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15536.150140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15536.152092] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15536.153763] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15536.155222] mdc_intent_lock+0x353/0x7e0 [mdc] [15536.156537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15536.158194] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15536.160105] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15536.161732] lmv_intent_open+0x338/0x1150 [lmv] [15536.163764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15536.165438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15536.167023] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15536.168263] ll_intent_lock+0xc5/0xaa0 [lustre] [15536.169682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15536.171164] ll_lookup_it+0xe0e/0x2270 [lustre] [15536.172434] ? do_raw_spin_unlock+0x75/0x190 [15536.173691] ? _raw_spin_unlock+0x12/0x30 [15536.175128] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15536.176861] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15536.178449] ll_atomic_open+0x36a/0xd70 [lustre] [15536.180005] lookup_open+0x356/0xb40 [15536.181126] do_last+0x313/0xfc0 [15536.182117] ? nd_jump_root+0xe5/0x160 [15536.183404] ? path_init+0x437/0x520 [15536.184443] path_openat+0xf7/0x500 [15536.185453] ? do_raw_spin_unlock+0x75/0x190 [15536.186536] do_filp_open+0x99/0x140 [15536.193738] ? getname_flags+0x6e/0x330 [15536.194953] ? __check_object_size+0xff/0x256 [15536.198175] ? do_raw_spin_unlock+0x75/0x190 [15536.200361] ? _raw_spin_unlock+0x12/0x30 [15536.202308] do_sys_openat2+0x2b4/0x410 [15536.203459] do_sys_open+0x73/0xa0 [15536.207824] __x64_sys_openat+0x24/0x30 [15536.209396] do_syscall_64+0xc1/0x440 [15536.210577] entry_SYSCALL_64_after_hwframe+0x49/0xae [15536.212800] RIP: 0033:0x7fa749552332 [15536.213998] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15536.220187] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15536.222937] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15536.225217] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15536.229706] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15536.236436] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15536.241077] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15540.057021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15540.059721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15540.080891] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15540.089744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15540.098605] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15540.104970] RIP: 0010:native_safe_halt+0x2e/0x50 [15540.108730] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15540.122892] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15540.127631] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15540.133834] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15540.140087] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15540.144511] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15540.147226] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15540.150355] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15540.153822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15540.156310] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15540.159201] Call Trace: [15540.160142] [15540.161541] ? show_regs.cold.9+0x22/0x2f [15540.163073] ? watchdog_timer_fn.cold.6+0x62/0x10a [15540.165777] ? __hrtimer_run_queues+0x1c2/0x4d0 [15540.167797] ? watchdog+0x40/0x40 [15540.168946] ? hrtimer_interrupt+0x128/0x2c0 [15540.170657] ? smp_apic_timer_interrupt+0xa6/0x270 [15540.172375] ? apic_timer_interrupt+0xf/0x20 [15540.173890] [15540.174588] ? native_safe_halt+0x2e/0x50 [15540.176036] kvm_wait+0xcd/0xe0 [15540.177438] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15540.181114] do_raw_spin_lock+0x177/0x190 [15540.183543] _raw_spin_lock+0x21/0x30 [15540.185885] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15540.190668] process_one_work+0x2c8/0x700 [15540.192698] worker_thread+0x296/0x6e0 [15540.196435] ? rescuer_thread+0x570/0x570 [15540.203407] kthread+0x1d1/0x200 [15540.204577] ? set_kthread_struct+0x70/0x70 [15540.206291] ret_from_fork+0x1f/0x30 [15541.217210] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13624s! [15541.220485] Showing busy workqueues and worker pools: [15541.223132] workqueue events: flags=0x0 [15541.225863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15541.229012] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15541.233265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15541.235643] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15541.239134] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.242424] pending: kfree_rcu_monitor [15541.243937] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15541.246241] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15541.249493] workqueue events_long: flags=0x0 [15541.251131] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.253233] pending: lu_objects_flush [obdclass] [15541.255094] workqueue events_unbound: flags=0x2 [15541.257035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15541.258991] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15541.262253] workqueue events_power_efficient: flags=0x80 [15541.264129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.266766] pending: check_lifetime [15541.268216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.271110] pending: neigh_periodic_work [15541.273859] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.276987] pending: neigh_periodic_work [15541.278218] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.281127] pending: do_cache_clean [15541.282539] workqueue mm_percpu_wq: flags=0x8 [15541.284054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.285966] pending: vmstat_update [15541.287191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.291399] pending: vmstat_update [15541.292538] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15541.294409] pending: vmstat_update [15541.295628] workqueue writeback: flags=0x4a [15541.296851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15541.298838] pending: wb_workfn, wb_workfn [15541.300184] workqueue xprtiod: flags=0xa [15541.301424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15541.303797] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15541.306352] workqueue ipv6_addrconf: flags=0x40008 [15541.307672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15541.309434] pending: addrconf_verify_work [15541.310715] workqueue knbd0-recv: flags=0x1a [15541.312023] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15541.313798] in-flight: 651:recv_work [15541.315032] workqueue ptlrpc_pinger: flags=0x2 [15541.316373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15541.318394] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15541.320290] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13624s workers=4 idle: 37 31846 28897 [15541.324030] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15531s workers=2 idle: 653 [15556.058039] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15556.060754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15556.084228] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15556.088056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15556.091010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15556.092660] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15556.101688] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15556.105818] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15556.112460] RDX: 00000000000008b1 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15556.119877] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15556.126155] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15556.131920] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15556.136251] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15556.138453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15556.148171] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15556.151826] Call Trace: [15556.152929] [15556.155927] ? show_regs.cold.9+0x22/0x2f [15556.161837] ? watchdog_timer_fn.cold.6+0x62/0x10a [15556.163670] ? __hrtimer_run_queues+0x1c2/0x4d0 [15556.165399] ? watchdog+0x40/0x40 [15556.168009] ? hrtimer_interrupt+0x128/0x2c0 [15556.172003] ? smp_apic_timer_interrupt+0xa6/0x270 [15556.177141] ? apic_timer_interrupt+0xf/0x20 [15556.178649] [15556.179313] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15556.183663] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15556.187150] do_raw_spin_lock+0x177/0x190 [15556.189568] _raw_spin_lock+0x21/0x30 [15556.191994] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15556.198111] obd_mod_free+0x46/0x140 [ptlrpc] [15556.203784] mdc_commit_open+0x91/0xb0 [mdc] [15556.205530] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15556.207840] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15556.213781] after_reply+0xe2c/0x1780 [ptlrpc] [15556.215257] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15556.221679] ? __next_timer_interrupt+0x160/0x160 [15556.223125] ptlrpcd+0xa82/0xe40 [ptlrpc] [15556.227431] ? do_wait_intr+0xf0/0xf0 [15556.228740] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15556.231910] kthread+0x1d1/0x200 [15556.235041] ? set_kthread_struct+0x70/0x70 [15556.240601] ret_from_fork+0x1f/0x30 [15560.061006] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15560.072051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15560.106308] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15560.113916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15560.118719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15560.122331] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15560.142285] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15560.151599] RAX: 0000000000007c21 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15560.154126] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15560.161513] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15560.166156] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15560.174622] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15560.179439] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15560.184720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15560.187286] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15560.191561] Call Trace: [15560.192970] [15560.193997] ? show_regs.cold.9+0x22/0x2f [15560.196464] ? watchdog_timer_fn.cold.6+0x62/0x10a [15560.199043] ? __hrtimer_run_queues+0x1c2/0x4d0 [15560.201976] ? watchdog+0x40/0x40 [15560.205097] ? hrtimer_interrupt+0x128/0x2c0 [15560.207457] ? smp_apic_timer_interrupt+0xa6/0x270 [15560.218915] ? apic_timer_interrupt+0xf/0x20 [15560.224921] [15560.227391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15560.232502] do_raw_spin_lock+0x177/0x190 [15560.234046] _raw_spin_lock+0x21/0x30 [15560.235312] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15560.241889] ? do_raw_spin_unlock+0x75/0x190 [15560.244367] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15560.250652] ? do_wait_intr+0xf0/0xf0 [15560.252174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15560.254645] mdc_close+0x53b/0xd70 [mdc] [15560.257308] lmv_close+0x18d/0x490 [lmv] [15560.259304] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15560.262520] ll_md_real_close+0xa5/0x2a0 [lustre] [15560.265705] ll_md_close+0x274/0x800 [lustre] [15560.269906] ll_file_release+0x128/0xc50 [lustre] [15560.275399] __fput+0x10d/0x440 [15560.276636] ____fput+0x12/0x20 [15560.277845] task_work_run+0xc6/0x110 [15560.287612] exit_to_usermode_loop+0x1dd/0x1f0 [15560.290765] do_syscall_64+0x3d6/0x440 [15560.292605] entry_SYSCALL_64_after_hwframe+0x49/0xae [15560.297713] RIP: 0033:0x7fa749552cd8 [15560.300790] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15560.316805] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15560.324400] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15560.331856] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15560.340239] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15560.346448] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15560.359940] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15568.055783] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15568.058163] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15568.066296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel [15568.071446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15568.071530] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15568.071537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15568.071543] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15568.071846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15568.071857] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15568.071861] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15568.071866] RAX: 000000000000558a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15568.071868] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15568.071871] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15568.071873] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15568.071876] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15568.071879] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15568.071882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15568.071884] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15568.071894] Call Trace: [15568.071899] [15568.071903] ? show_regs.cold.9+0x22/0x2f [15568.071912] ? watchdog_timer_fn.cold.6+0x62/0x10a [15568.071919] ? __hrtimer_run_queues+0x1c2/0x4d0 [15568.071923] ? watchdog+0x40/0x40 [15568.071927] ? hrtimer_interrupt+0x128/0x2c0 [15568.071931] ? smp_apic_timer_interrupt+0xa6/0x270 [15568.071940] ? apic_timer_interrupt+0xf/0x20 [15568.071944] [15568.071947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15568.071952] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15568.071957] do_raw_spin_lock+0x177/0x190 [15568.071961] _raw_spin_lock+0x21/0x30 [15568.071967] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15568.130855] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15568.130884] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15568.130893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15568.130896] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15568.130914] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15568.130919] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15568.130924] RAX: 0000000000003f0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15568.130926] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15568.130929] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15568.130931] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15568.130933] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15568.131041] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15568.131046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15568.131048] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15568.131056] Call Trace: [15568.131061] [15568.131067] ? show_regs.cold.9+0x22/0x2f [15568.131077] ? watchdog_timer_fn.cold.6+0x62/0x10a [15568.131083] ? __hrtimer_run_queues+0x1c2/0x4d0 [15568.131086] ? watchdog+0x40/0x40 [15568.131090] ? hrtimer_interrupt+0x128/0x2c0 [15568.131093] ? smp_apic_timer_interrupt+0xa6/0x270 [15568.131101] ? apic_timer_interrupt+0xf/0x20 [15568.131193] [15568.131196] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15568.131204] do_raw_spin_lock+0x177/0x190 [15568.131209] _raw_spin_lock+0x21/0x30 [15568.131214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15568.131525] ? do_raw_spin_unlock+0x75/0x190 [15568.131855] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15568.132236] ? do_wait_intr+0xf0/0xf0 [15568.132246] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15568.132516] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15568.134935] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15568.135682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15568.135789] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15568.135832] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15568.135858] mdc_intent_lock+0x353/0x7e0 [mdc] [15568.135885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15568.136326] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15568.136561] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15568.136592] lmv_intent_open+0x338/0x1150 [lmv] [15568.136624] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15568.136710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15568.136786] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15568.136805] ll_intent_lock+0xc5/0xaa0 [lustre] [15568.136883] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15568.136975] ll_lookup_it+0xe0e/0x2270 [lustre] [15568.137630] ? do_raw_spin_unlock+0x75/0x190 [15568.137639] ? _raw_spin_unlock+0x12/0x30 [15568.182354] process_one_work+0x2c8/0x700 [15568.182374] worker_thread+0x296/0x6e0 [15568.182379] ? rescuer_thread+0x570/0x570 [15568.182384] kthread+0x1d1/0x200 [15568.182388] ? set_kthread_struct+0x70/0x70 [15568.182393] ret_from_fork+0x1f/0x30 [15568.489611] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15568.495202] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15568.498812] ll_atomic_open+0x36a/0xd70 [lustre] [15568.500681] lookup_open+0x356/0xb40 [15568.502567] do_last+0x313/0xfc0 [15568.503960] ? nd_jump_root+0xe5/0x160 [15568.505633] ? path_init+0x437/0x520 [15568.511522] path_openat+0xf7/0x500 [15568.514324] ? do_raw_spin_unlock+0x75/0x190 [15568.517472] do_filp_open+0x99/0x140 [15568.521349] ? getname_flags+0x6e/0x330 [15568.524571] ? __check_object_size+0xff/0x256 [15568.527654] ? do_raw_spin_unlock+0x75/0x190 [15568.528932] ? _raw_spin_unlock+0x12/0x30 [15568.532815] do_sys_openat2+0x2b4/0x410 [15568.534907] do_sys_open+0x73/0xa0 [15568.539451] __x64_sys_openat+0x24/0x30 [15568.545492] do_syscall_64+0xc1/0x440 [15568.547141] entry_SYSCALL_64_after_hwframe+0x49/0xae [15568.551358] RIP: 0033:0x7fa749552332 [15568.552535] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15568.570892] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15568.574700] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15568.579232] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15568.585689] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15568.590857] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15568.594491] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15571.427290] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13654s! [15571.434727] Showing busy workqueues and worker pools: [15571.442416] workqueue events: flags=0x0 [15571.448693] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15571.456975] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15571.467066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15571.476612] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15571.490090] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.499392] pending: kfree_rcu_monitor [15571.503919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15571.512299] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15571.521910] workqueue events_long: flags=0x0 [15571.524405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.531762] pending: lu_objects_flush [obdclass] [15571.540881] workqueue events_unbound: flags=0x2 [15571.553311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15571.564500] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15571.586685] workqueue events_power_efficient: flags=0x80 [15571.589345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.605628] pending: check_lifetime [15571.609140] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.614989] pending: neigh_periodic_work [15571.616193] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.625016] pending: neigh_periodic_work [15571.627251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.632413] pending: do_cache_clean [15571.633832] workqueue mm_percpu_wq: flags=0x8 [15571.636476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.640353] pending: vmstat_update [15571.641698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.645277] pending: vmstat_update [15571.650259] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15571.656025] pending: vmstat_update [15571.660271] workqueue writeback: flags=0x4a [15571.667966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15571.678742] pending: wb_workfn, wb_workfn [15571.681417] workqueue xprtiod: flags=0xa [15571.684746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15571.694101] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15571.707495] workqueue ipv6_addrconf: flags=0x40008 [15571.716399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15571.723104] pending: addrconf_verify_work [15571.726069] workqueue knbd0-recv: flags=0x1a [15571.728371] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15571.735255] in-flight: 651:recv_work [15571.739040] workqueue ptlrpc_pinger: flags=0x2 [15571.743473] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15571.752059] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15571.762482] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13654s workers=4 idle: 37 31846 28897 [15571.770827] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15561s workers=2 idle: 653 [15584.058342] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15584.064359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15584.100985] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15584.106113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15584.113051] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15584.117116] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15584.126654] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15584.130320] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15584.133679] RDX: 0000000000007e0a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15584.136838] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15584.140066] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15584.143096] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15584.146459] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15584.150303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15584.153071] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15584.156618] Call Trace: [15584.157783] [15584.158432] ? show_regs.cold.9+0x22/0x2f [15584.160371] ? watchdog_timer_fn.cold.6+0x62/0x10a [15584.162301] ? __hrtimer_run_queues+0x1c2/0x4d0 [15584.164546] ? watchdog+0x40/0x40 [15584.165840] ? hrtimer_interrupt+0x128/0x2c0 [15584.167658] ? smp_apic_timer_interrupt+0xa6/0x270 [15584.169351] ? apic_timer_interrupt+0xf/0x20 [15584.171032] [15584.171828] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15584.174096] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15584.176414] do_raw_spin_lock+0x177/0x190 [15584.178822] _raw_spin_lock+0x21/0x30 [15584.182195] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15584.185325] obd_mod_free+0x46/0x140 [ptlrpc] [15584.187269] mdc_commit_open+0x91/0xb0 [mdc] [15584.189155] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15584.191429] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15584.193681] after_reply+0xe2c/0x1780 [ptlrpc] [15584.195923] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15584.199221] ? __next_timer_interrupt+0x160/0x160 [15584.201144] ptlrpcd+0xa82/0xe40 [ptlrpc] [15584.206885] ? do_wait_intr+0xf0/0xf0 [15584.208531] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15584.212422] kthread+0x1d1/0x200 [15584.217056] ? set_kthread_struct+0x70/0x70 [15584.218257] ret_from_fork+0x1f/0x30 [15588.059125] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15588.062320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15588.081132] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15588.085104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15588.088741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15588.090625] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15588.096382] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15588.099241] RAX: 00000000000016f6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15588.101830] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15588.104476] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15588.107763] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15588.111192] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15588.114577] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15588.117728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15588.120078] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15588.122611] Call Trace: [15588.123538] [15588.124225] ? show_regs.cold.9+0x22/0x2f [15588.125615] ? watchdog_timer_fn.cold.6+0x62/0x10a [15588.127323] ? __hrtimer_run_queues+0x1c2/0x4d0 [15588.128853] ? watchdog+0x40/0x40 [15588.129862] ? hrtimer_interrupt+0x128/0x2c0 [15588.131804] ? smp_apic_timer_interrupt+0xa6/0x270 [15588.133530] ? apic_timer_interrupt+0xf/0x20 [15588.135876] [15588.136572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15588.138186] do_raw_spin_lock+0x177/0x190 [15588.139682] _raw_spin_lock+0x21/0x30 [15588.140877] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15588.143097] ? do_raw_spin_unlock+0x75/0x190 [15588.144572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15588.146549] ? do_wait_intr+0xf0/0xf0 [15588.148125] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15588.151307] mdc_close+0x53b/0xd70 [mdc] [15588.156107] lmv_close+0x18d/0x490 [lmv] [15588.158371] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15588.161418] ll_md_real_close+0xa5/0x2a0 [lustre] [15588.163865] ll_md_close+0x274/0x800 [lustre] [15588.166321] ll_file_release+0x128/0xc50 [lustre] [15588.169466] __fput+0x10d/0x440 [15588.170746] ____fput+0x12/0x20 [15588.175549] task_work_run+0xc6/0x110 [15588.176885] exit_to_usermode_loop+0x1dd/0x1f0 [15588.181935] do_syscall_64+0x3d6/0x440 [15588.183606] entry_SYSCALL_64_after_hwframe+0x49/0xae [15588.185690] RIP: 0033:0x7fa749552cd8 [15588.187223] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15588.195091] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15588.197966] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15588.200797] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15588.203372] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15588.206627] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15588.210152] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15596.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15596.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15596.058826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15596.058904] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15596.058912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15596.058914] RIP: 0010:native_safe_halt+0x2e/0x50 [15596.058933] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15596.058936] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15596.058941] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15596.058944] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15596.058946] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15596.058948] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15596.058951] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15596.058954] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15596.058956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15596.058958] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15596.058967] Call Trace: [15596.058973] [15596.058978] ? show_regs.cold.9+0x22/0x2f [15596.058987] ? watchdog_timer_fn.cold.6+0x62/0x10a [15596.058993] ? __hrtimer_run_queues+0x1c2/0x4d0 [15596.058998] ? watchdog+0x40/0x40 [15596.059002] ? hrtimer_interrupt+0x128/0x2c0 [15596.059006] ? smp_apic_timer_interrupt+0xa6/0x270 [15596.059013] ? apic_timer_interrupt+0xf/0x20 [15596.059017] [15596.059020] ? native_safe_halt+0x2e/0x50 [15596.059023] kvm_wait+0xcd/0xe0 [15596.059031] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15596.059040] do_raw_spin_lock+0x177/0x190 [15596.059044] _raw_spin_lock+0x21/0x30 [15596.059049] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15596.059344] ? do_raw_spin_unlock+0x75/0x190 [15596.059351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15596.059561] ? do_wait_intr+0xf0/0xf0 [15596.059569] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15596.059769] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15596.065076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15596.065173] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15596.065182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15596.065189] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15596.095583] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15596.096180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15596.096319] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15596.096370] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15596.097418] mdc_intent_lock+0x353/0x7e0 [mdc] [15596.097450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15596.097628] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15596.098355] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15596.098394] lmv_intent_open+0x338/0x1150 [lmv] [15596.098433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15596.098557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15596.098658] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15596.098683] ll_intent_lock+0xc5/0xaa0 [lustre] [15596.098776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15596.099308] ll_lookup_it+0xe0e/0x2270 [lustre] [15596.099408] ? do_raw_spin_unlock+0x75/0x190 [15596.099415] ? _raw_spin_unlock+0x12/0x30 [15596.099422] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15596.099459] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15596.099480] ll_atomic_open+0x36a/0xd70 [lustre] [15596.099576] lookup_open+0x356/0xb40 [15596.099587] do_last+0x313/0xfc0 [15596.099593] ? nd_jump_root+0xe5/0x160 [15596.099597] ? path_init+0x437/0x520 [15596.099601] path_openat+0xf7/0x500 [15596.099606] ? do_raw_spin_unlock+0x75/0x190 [15596.099610] do_filp_open+0x99/0x140 [15596.099616] ? getname_flags+0x6e/0x330 [15596.099621] ? __check_object_size+0xff/0x256 [15596.099639] ? do_raw_spin_unlock+0x75/0x190 [15596.099643] ? _raw_spin_unlock+0x12/0x30 [15596.099648] do_sys_openat2+0x2b4/0x410 [15596.099653] do_sys_open+0x73/0xa0 [15596.099657] __x64_sys_openat+0x24/0x30 [15596.099661] do_syscall_64+0xc1/0x440 [15596.099669] entry_SYSCALL_64_after_hwframe+0x49/0xae [15596.099675] RIP: 0033:0x7fa749552332 [15596.099681] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15596.099685] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15596.099690] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15596.099693] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15596.099696] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15596.099698] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15596.099700] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15596.383439] [15596.387994] RIP: 0010:native_safe_halt+0x2e/0x50 [15596.391299] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15596.405093] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15596.409013] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15596.412863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15596.416128] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15596.420228] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15596.424241] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15596.428533] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15596.433986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15596.436441] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15596.440038] Call Trace: [15596.441262] [15596.442688] ? show_regs.cold.9+0x22/0x2f [15596.444382] ? watchdog_timer_fn.cold.6+0x62/0x10a [15596.446484] ? __hrtimer_run_queues+0x1c2/0x4d0 [15596.448534] ? watchdog+0x40/0x40 [15596.450564] ? hrtimer_interrupt+0x128/0x2c0 [15596.452988] ? smp_apic_timer_interrupt+0xa6/0x270 [15596.456122] ? apic_timer_interrupt+0xf/0x20 [15596.458312] [15596.459300] ? native_safe_halt+0x2e/0x50 [15596.461408] kvm_wait+0xcd/0xe0 [15596.462845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15596.468662] do_raw_spin_lock+0x177/0x190 [15596.470611] _raw_spin_lock+0x21/0x30 [15596.472236] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15596.474879] process_one_work+0x2c8/0x700 [15596.476530] worker_thread+0x296/0x6e0 [15596.478398] ? rescuer_thread+0x570/0x570 [15596.480329] kthread+0x1d1/0x200 [15596.481931] ? set_kthread_struct+0x70/0x70 [15596.483868] ret_from_fork+0x1f/0x30 [15602.144825] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13684s! [15602.147780] Showing busy workqueues and worker pools: [15602.149566] workqueue events: flags=0x0 [15602.151110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15602.154511] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15602.158840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15602.161426] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15602.165482] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.168058] pending: kfree_rcu_monitor [15602.169500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15602.172126] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15602.175070] workqueue events_long: flags=0x0 [15602.176334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.178716] pending: lu_objects_flush [obdclass] [15602.180512] workqueue events_unbound: flags=0x2 [15602.181877] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15602.183842] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15602.186499] workqueue events_power_efficient: flags=0x80 [15602.188171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.190584] pending: check_lifetime [15602.191704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.193687] pending: neigh_periodic_work [15602.195096] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.197243] pending: neigh_periodic_work [15602.198536] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.200717] pending: do_cache_clean [15602.201993] workqueue mm_percpu_wq: flags=0x8 [15602.203239] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.205230] pending: vmstat_update [15602.206292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.208275] pending: vmstat_update [15602.209561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15602.211722] pending: vmstat_update [15602.212931] workqueue writeback: flags=0x4a [15602.214128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15602.216037] pending: wb_workfn, wb_workfn [15602.222449] workqueue xprtiod: flags=0xa [15602.223794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15602.226035] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15602.228617] workqueue ipv6_addrconf: flags=0x40008 [15602.230706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15602.233072] pending: addrconf_verify_work [15602.234344] workqueue knbd0-recv: flags=0x1a [15602.235982] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15602.237988] in-flight: 651:recv_work [15602.239360] workqueue ptlrpc_pinger: flags=0x2 [15602.240871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15602.242605] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15602.244514] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13685s workers=4 idle: 37 31846 28897 [15602.247053] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15592s workers=2 idle: 653 [15612.058021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15612.061475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15612.081965] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15612.087533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15612.091941] RIP: 0010:native_safe_halt+0x2e/0x50 [15612.094687] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15612.102468] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15612.105672] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15612.108918] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15612.112081] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15612.114452] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15612.118081] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15612.121287] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15612.124699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15612.127463] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15612.130703] Call Trace: [15612.131874] [15612.132717] ? show_regs.cold.9+0x22/0x2f [15612.134382] ? watchdog_timer_fn.cold.6+0x62/0x10a [15612.136109] ? __hrtimer_run_queues+0x1c2/0x4d0 [15612.137444] ? watchdog+0x40/0x40 [15612.138419] ? hrtimer_interrupt+0x128/0x2c0 [15612.140346] ? smp_apic_timer_interrupt+0xa6/0x270 [15612.142381] ? apic_timer_interrupt+0xf/0x20 [15612.144706] [15612.145579] ? native_safe_halt+0x2e/0x50 [15612.146736] kvm_wait+0xcd/0xe0 [15612.147637] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15612.149252] do_raw_spin_lock+0x177/0x190 [15612.151107] _raw_spin_lock+0x21/0x30 [15612.152646] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15612.154979] obd_mod_free+0x46/0x140 [ptlrpc] [15612.156804] mdc_commit_open+0x91/0xb0 [mdc] [15612.158403] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15612.160573] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15612.163198] after_reply+0xe2c/0x1780 [ptlrpc] [15612.165866] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15612.167716] ? __next_timer_interrupt+0x160/0x160 [15612.177404] ptlrpcd+0xa82/0xe40 [ptlrpc] [15612.179983] ? do_wait_intr+0xf0/0xf0 [15612.182024] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15612.188033] kthread+0x1d1/0x200 [15612.189552] ? set_kthread_struct+0x70/0x70 [15612.191567] ret_from_fork+0x1f/0x30 [15616.060174] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15616.067543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15616.092528] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15616.104017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15616.109083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15616.115515] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15616.132361] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15616.137711] RAX: 000000000000260d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15616.149609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15616.153007] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15616.154784] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15616.158761] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15616.162042] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15616.164110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15616.168258] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15616.172085] Call Trace: [15616.176722] [15616.178109] ? show_regs.cold.9+0x22/0x2f [15616.182954] ? watchdog_timer_fn.cold.6+0x62/0x10a [15616.184697] ? __hrtimer_run_queues+0x1c2/0x4d0 [15616.186502] ? watchdog+0x40/0x40 [15616.187634] ? hrtimer_interrupt+0x128/0x2c0 [15616.189841] ? smp_apic_timer_interrupt+0xa6/0x270 [15616.191640] ? apic_timer_interrupt+0xf/0x20 [15616.196818] [15616.198918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15616.201072] do_raw_spin_lock+0x177/0x190 [15616.205764] _raw_spin_lock+0x21/0x30 [15616.207762] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15616.210671] ? do_raw_spin_unlock+0x75/0x190 [15616.212921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15616.216874] ? do_wait_intr+0xf0/0xf0 [15616.219048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15616.222601] mdc_close+0x53b/0xd70 [mdc] [15616.224587] lmv_close+0x18d/0x490 [lmv] [15616.226280] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15616.229444] ll_md_real_close+0xa5/0x2a0 [lustre] [15616.231074] ll_md_close+0x274/0x800 [lustre] [15616.236081] ll_file_release+0x128/0xc50 [lustre] [15616.239993] __fput+0x10d/0x440 [15616.241574] ____fput+0x12/0x20 [15616.244017] task_work_run+0xc6/0x110 [15616.249753] exit_to_usermode_loop+0x1dd/0x1f0 [15616.251006] do_syscall_64+0x3d6/0x440 [15616.251924] entry_SYSCALL_64_after_hwframe+0x49/0xae [15616.254911] RIP: 0033:0x7fa749552cd8 [15616.256416] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15616.267942] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15616.275123] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15616.279784] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15616.286418] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15616.288778] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15616.291855] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15624.057237] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15624.062312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15624.095660] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15624.102611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15624.108498] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15624.114357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15624.118512] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15624.127536] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15624.134838] RAX: 0000000000007e3a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15624.139558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15624.142664] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15624.147361] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15624.152179] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15624.156098] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15624.160695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15624.165631] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15624.170551] Call Trace: [15624.172715] [15624.173648] ? show_regs.cold.9+0x22/0x2f [15624.177153] ? watchdog_timer_fn.cold.6+0x62/0x10a [15624.179795] ? __hrtimer_run_queues+0x1c2/0x4d0 [15624.181317] ? watchdog+0x40/0x40 [15624.184518] ? hrtimer_interrupt+0x128/0x2c0 [15624.186415] ? smp_apic_timer_interrupt+0xa6/0x270 [15624.189671] ? apic_timer_interrupt+0xf/0x20 [15624.192822] [15624.195248] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15624.197661] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15624.200667] do_raw_spin_lock+0x177/0x190 [15624.202688] _raw_spin_lock+0x21/0x30 [15624.206865] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15624.208881] process_one_work+0x2c8/0x700 [15624.212407] worker_thread+0x296/0x6e0 [15624.215397] ? rescuer_thread+0x570/0x570 [15624.217177] kthread+0x1d1/0x200 [15624.219561] ? set_kthread_struct+0x70/0x70 [15624.221538] ret_from_fork+0x1f/0x30 [15628.057907] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15628.071869] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15628.133094] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15628.147941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15628.166522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15628.175316] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15628.197123] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15628.200359] RAX: 000000000000201e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15628.207251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15628.211594] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15628.215426] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15628.219283] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15628.223498] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15628.228072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15628.231475] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15628.235519] Call Trace: [15628.236712] [15628.237754] ? show_regs.cold.9+0x22/0x2f [15628.240017] ? watchdog_timer_fn.cold.6+0x62/0x10a [15628.242488] ? __hrtimer_run_queues+0x1c2/0x4d0 [15628.244675] ? watchdog+0x40/0x40 [15628.246749] ? hrtimer_interrupt+0x128/0x2c0 [15628.249456] ? smp_apic_timer_interrupt+0xa6/0x270 [15628.253242] ? apic_timer_interrupt+0xf/0x20 [15628.255437] [15628.256439] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15628.259300] do_raw_spin_lock+0x177/0x190 [15628.261337] _raw_spin_lock+0x21/0x30 [15628.266410] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15628.273101] ? do_raw_spin_unlock+0x75/0x190 [15628.275620] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15628.278673] ? do_wait_intr+0xf0/0xf0 [15628.281141] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15628.286561] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15628.289622] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15628.295431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15628.304426] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15628.309136] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15628.311750] mdc_intent_lock+0x353/0x7e0 [mdc] [15628.314163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15628.317757] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15628.321542] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15628.325496] lmv_intent_open+0x338/0x1150 [lmv] [15628.329239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15628.334480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15628.339240] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15628.343621] ll_intent_lock+0xc5/0xaa0 [lustre] [15628.346731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15628.350122] ll_lookup_it+0xe0e/0x2270 [lustre] [15628.353073] ? do_raw_spin_unlock+0x75/0x190 [15628.355098] ? _raw_spin_unlock+0x12/0x30 [15628.357175] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15628.360679] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15628.363299] ll_atomic_open+0x36a/0xd70 [lustre] [15628.365555] lookup_open+0x356/0xb40 [15628.367928] do_last+0x313/0xfc0 [15628.369785] ? nd_jump_root+0xe5/0x160 [15628.372161] ? path_init+0x437/0x520 [15628.374057] path_openat+0xf7/0x500 [15628.375845] ? do_raw_spin_unlock+0x75/0x190 [15628.378326] do_filp_open+0x99/0x140 [15628.380257] ? getname_flags+0x6e/0x330 [15628.383298] ? __check_object_size+0xff/0x256 [15628.385158] ? do_raw_spin_unlock+0x75/0x190 [15628.387455] ? _raw_spin_unlock+0x12/0x30 [15628.392148] do_sys_openat2+0x2b4/0x410 [15628.394922] do_sys_open+0x73/0xa0 [15628.397674] __x64_sys_openat+0x24/0x30 [15628.399557] do_syscall_64+0xc1/0x440 [15628.402144] entry_SYSCALL_64_after_hwframe+0x49/0xae [15628.404421] RIP: 0033:0x7fa749552332 [15628.406735] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15628.420881] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15628.430241] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15628.437587] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15628.447326] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15628.455390] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15628.460246] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15632.362826] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13715s! [15632.370612] Showing busy workqueues and worker pools: [15632.375913] workqueue events: flags=0x0 [15632.378600] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15632.386605] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15632.398376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15632.407328] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15632.412503] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.415968] pending: kfree_rcu_monitor [15632.417856] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15632.424178] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15632.430124] workqueue events_long: flags=0x0 [15632.434692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.437716] pending: lu_objects_flush [obdclass] [15632.441801] workqueue events_unbound: flags=0x2 [15632.446001] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15632.451352] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15632.466681] workqueue events_power_efficient: flags=0x80 [15632.470782] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.475654] pending: check_lifetime [15632.481280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.497789] pending: neigh_periodic_work [15632.505040] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.515015] pending: neigh_periodic_work [15632.522242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.531438] pending: do_cache_clean [15632.533161] workqueue mm_percpu_wq: flags=0x8 [15632.535804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.539477] pending: vmstat_update [15632.541444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.545006] pending: vmstat_update [15632.547216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15632.551103] pending: vmstat_update [15632.553351] workqueue writeback: flags=0x4a [15632.555519] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15632.558838] pending: wb_workfn, wb_workfn [15632.561188] workqueue xprtiod: flags=0xa [15632.563424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15632.568621] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15632.573246] workqueue ipv6_addrconf: flags=0x40008 [15632.581801] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15632.589301] pending: addrconf_verify_work [15632.600972] workqueue knbd0-recv: flags=0x1a [15632.609733] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15632.614566] in-flight: 651:recv_work [15632.618478] workqueue ptlrpc_pinger: flags=0x2 [15632.624377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15632.629273] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15632.631812] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13715s workers=4 idle: 37 31846 28897 [15632.639666] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15622s workers=2 idle: 653 [15640.058025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15640.063316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15640.096273] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15640.101309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15640.104422] RIP: 0010:native_safe_halt+0x2e/0x50 [15640.105998] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15640.118671] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15640.122988] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15640.127409] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15640.133008] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15640.137037] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15640.140785] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15640.146206] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15640.150155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15640.159462] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15640.165680] Call Trace: [15640.168598] [15640.172326] ? show_regs.cold.9+0x22/0x2f [15640.175221] ? watchdog_timer_fn.cold.6+0x62/0x10a [15640.180613] ? __hrtimer_run_queues+0x1c2/0x4d0 [15640.183650] ? watchdog+0x40/0x40 [15640.186553] ? hrtimer_interrupt+0x128/0x2c0 [15640.189532] ? smp_apic_timer_interrupt+0xa6/0x270 [15640.192713] ? apic_timer_interrupt+0xf/0x20 [15640.197760] [15640.198907] ? native_safe_halt+0x2e/0x50 [15640.201828] kvm_wait+0xcd/0xe0 [15640.206946] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15640.208825] do_raw_spin_lock+0x177/0x190 [15640.210602] _raw_spin_lock+0x21/0x30 [15640.213786] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15640.216977] obd_mod_free+0x46/0x140 [ptlrpc] [15640.219671] mdc_commit_open+0x91/0xb0 [mdc] [15640.225703] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15640.233167] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15640.235674] after_reply+0xe2c/0x1780 [ptlrpc] [15640.242627] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15640.246514] ? __next_timer_interrupt+0x160/0x160 [15640.249006] ptlrpcd+0xa82/0xe40 [ptlrpc] [15640.255849] ? do_wait_intr+0xf0/0xf0 [15640.257002] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15640.258494] kthread+0x1d1/0x200 [15640.260536] ? set_kthread_struct+0x70/0x70 [15640.265021] ret_from_fork+0x1f/0x30 [15644.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15644.063547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15644.111115] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15644.123257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15644.128483] RIP: 0010:native_safe_halt+0x2e/0x50 [15644.145299] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15644.159722] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15644.172910] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15644.183374] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15644.191587] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15644.196276] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15644.198392] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15644.203754] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15644.207228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15644.209523] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15644.212500] Call Trace: [15644.213402] [15644.214166] ? show_regs.cold.9+0x22/0x2f [15644.215947] ? watchdog_timer_fn.cold.6+0x62/0x10a [15644.217319] ? __hrtimer_run_queues+0x1c2/0x4d0 [15644.218926] ? watchdog+0x40/0x40 [15644.220188] ? hrtimer_interrupt+0x128/0x2c0 [15644.221475] ? smp_apic_timer_interrupt+0xa6/0x270 [15644.223083] ? apic_timer_interrupt+0xf/0x20 [15644.224438] [15644.225175] ? native_safe_halt+0x2e/0x50 [15644.226307] kvm_wait+0xcd/0xe0 [15644.227311] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15644.229158] do_raw_spin_lock+0x177/0x190 [15644.230609] _raw_spin_lock+0x21/0x30 [15644.232241] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15644.238246] ? do_raw_spin_unlock+0x75/0x190 [15644.239847] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15644.241609] ? do_wait_intr+0xf0/0xf0 [15644.248561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15644.254421] mdc_close+0x53b/0xd70 [mdc] [15644.259483] lmv_close+0x18d/0x490 [lmv] [15644.261932] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15644.267239] ll_md_real_close+0xa5/0x2a0 [lustre] [15644.273334] ll_md_close+0x274/0x800 [lustre] [15644.278945] ll_file_release+0x128/0xc50 [lustre] [15644.285322] __fput+0x10d/0x440 [15644.286240] ____fput+0x12/0x20 [15644.289089] task_work_run+0xc6/0x110 [15644.291193] exit_to_usermode_loop+0x1dd/0x1f0 [15644.294291] do_syscall_64+0x3d6/0x440 [15644.296009] entry_SYSCALL_64_after_hwframe+0x49/0xae [15644.300191] RIP: 0033:0x7fa749552cd8 [15644.302329] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15644.329079] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15644.333470] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15644.336762] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15644.344728] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15644.351229] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15644.354393] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15652.059991] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15652.062770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15652.087882] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15652.096824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15652.101840] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15652.105924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15652.109463] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15652.120061] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15652.123946] RAX: 0000000000007f92 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15652.128397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15652.131894] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15652.135312] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15652.138371] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15652.141765] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15652.145333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15652.148293] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15652.154740] Call Trace: [15652.155648] [15652.158357] ? show_regs.cold.9+0x22/0x2f [15652.162204] ? watchdog_timer_fn.cold.6+0x62/0x10a [15652.164978] ? __hrtimer_run_queues+0x1c2/0x4d0 [15652.167189] ? watchdog+0x40/0x40 [15652.168750] ? hrtimer_interrupt+0x128/0x2c0 [15652.170786] ? smp_apic_timer_interrupt+0xa6/0x270 [15652.173237] ? apic_timer_interrupt+0xf/0x20 [15652.174935] [15652.175708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15652.177401] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15652.179276] do_raw_spin_lock+0x177/0x190 [15652.180815] _raw_spin_lock+0x21/0x30 [15652.182216] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15652.184266] process_one_work+0x2c8/0x700 [15652.185652] worker_thread+0x296/0x6e0 [15652.187359] ? rescuer_thread+0x570/0x570 [15652.189313] kthread+0x1d1/0x200 [15652.190465] ? set_kthread_struct+0x70/0x70 [15652.192060] ret_from_fork+0x1f/0x30 [15653.997318] rcu: INFO: rcu_sched self-detected stall on CPU [15654.001074] rcu: 2-...!: (7983028 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [15654.006480] (t=13740308 jiffies g=258701 q=5222) [15654.009297] rcu: rcu_sched kthread starved for 13737103 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [15654.016287] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15654.021598] rcu: RCU grace-period kthread stack dump: [15654.025208] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15654.033692] Call Trace: [15654.034777] __schedule+0x351/0xcb0 [15654.039297] schedule+0xc0/0x180 [15654.044060] schedule_timeout+0xb4/0x190 [15654.046241] ? do_raw_spin_unlock+0x75/0x190 [15654.048272] ? __next_timer_interrupt+0x160/0x160 [15654.050742] ? prepare_to_swait_event+0xf3/0x140 [15654.053537] rcu_gp_kthread+0x8dd/0x15b0 [15654.056593] ? force_qs_rnp+0x2c0/0x2c0 [15654.059394] kthread+0x1d1/0x200 [15654.061334] ? set_kthread_struct+0x70/0x70 [15654.063838] ret_from_fork+0x1f/0x30 [15654.066700] rcu: Stack dump where RCU GP kthread last ran: [15654.068604] NMI backtrace for cpu 2 [15654.071423] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15654.078915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15654.081670] Call Trace: [15654.083432] [15654.083866] ? dump_stack+0xbb/0x10e [15654.088166] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15654.090093] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15654.093807] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15654.096188] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15654.100313] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15654.102644] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [15654.107667] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [15654.115036] ? update_process_times+0x6d/0xc0 [15654.120390] ? tick_sched_handle.isra.18+0x27/0x90 [15654.122451] ? tick_sched_timer+0x53/0xb0 [15654.124544] ? __hrtimer_run_queues+0x1c2/0x4d0 [15654.131707] ? tick_sched_handle.isra.18+0x90/0x90 [15654.134363] ? hrtimer_interrupt+0x128/0x2c0 [15654.139341] ? smp_apic_timer_interrupt+0xa6/0x270 [15654.141550] ? apic_timer_interrupt+0xf/0x20 [15654.143387] [15654.144435] ? native_safe_halt+0x2e/0x50 [15654.147368] ? kvm_wait+0xcd/0xe0 [15654.149177] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15654.152063] ? do_raw_spin_lock+0x177/0x190 [15654.153567] ? _raw_spin_lock+0x21/0x30 [15654.158284] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15654.162170] ? obd_mod_free+0x46/0x140 [ptlrpc] [15654.168736] ? mdc_commit_open+0x91/0xb0 [mdc] [15654.174154] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15654.179033] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15654.185055] ? after_reply+0xe2c/0x1780 [ptlrpc] [15654.187947] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15654.190706] ? __next_timer_interrupt+0x160/0x160 [15654.193160] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15654.195705] ? do_wait_intr+0xf0/0xf0 [15654.197360] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15654.200232] ? kthread+0x1d1/0x200 [15654.201742] ? set_kthread_struct+0x70/0x70 [15654.203775] ? ret_from_fork+0x1f/0x30 [15654.205401] NMI backtrace for cpu 2 [15654.207191] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15654.211723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15654.215273] Call Trace: [15654.216160] [15654.216981] ? dump_stack+0xbb/0x10e [15654.218071] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15654.220707] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15654.223875] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15654.225664] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15654.230540] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15654.233119] ? rcu_dump_cpu_stacks+0x11b/0x171 [15654.235316] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15654.236708] ? update_process_times+0x6d/0xc0 [15654.237742] ? tick_sched_handle.isra.18+0x27/0x90 [15654.246565] ? tick_sched_timer+0x53/0xb0 [15654.248302] ? __hrtimer_run_queues+0x1c2/0x4d0 [15654.250683] ? tick_sched_handle.isra.18+0x90/0x90 [15654.253688] ? hrtimer_interrupt+0x128/0x2c0 [15654.255453] ? smp_apic_timer_interrupt+0xa6/0x270 [15654.258336] ? apic_timer_interrupt+0xf/0x20 [15654.259803] [15654.260526] ? native_safe_halt+0x2e/0x50 [15654.270152] ? kvm_wait+0xcd/0xe0 [15654.273243] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15654.279186] ? do_raw_spin_lock+0x177/0x190 [15654.281232] ? _raw_spin_lock+0x21/0x30 [15654.283774] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15654.288201] ? obd_mod_free+0x46/0x140 [ptlrpc] [15654.291931] ? mdc_commit_open+0x91/0xb0 [mdc] [15654.296039] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15654.302648] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15654.304899] ? after_reply+0xe2c/0x1780 [ptlrpc] [15654.309347] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15654.320185] ? __next_timer_interrupt+0x160/0x160 [15654.324267] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15654.327050] ? do_wait_intr+0xf0/0xf0 [15654.336079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15654.340417] ? kthread+0x1d1/0x200 [15654.344327] ? set_kthread_struct+0x70/0x70 [15654.348627] ? ret_from_fork+0x1f/0x30 [15660.054037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15660.056557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15660.089211] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15660.098938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15660.104888] RIP: 0010:native_safe_halt+0x2e/0x50 [15660.108173] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15660.124105] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15660.131076] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15660.136966] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15660.143359] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15660.147462] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15660.153090] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15660.158277] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15660.164940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15660.171327] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15660.182630] Call Trace: [15660.187984] [15660.189612] ? show_regs.cold.9+0x22/0x2f [15660.195742] ? watchdog_timer_fn.cold.6+0x62/0x10a [15660.200833] ? __hrtimer_run_queues+0x1c2/0x4d0 [15660.206296] ? watchdog+0x40/0x40 [15660.208979] ? hrtimer_interrupt+0x128/0x2c0 [15660.211463] ? smp_apic_timer_interrupt+0xa6/0x270 [15660.215437] ? apic_timer_interrupt+0xf/0x20 [15660.217411] [15660.219241] ? native_safe_halt+0x2e/0x50 [15660.223817] kvm_wait+0xcd/0xe0 [15660.225112] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15660.234474] do_raw_spin_lock+0x177/0x190 [15660.241314] _raw_spin_lock+0x21/0x30 [15660.243021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15660.245006] ? do_raw_spin_unlock+0x75/0x190 [15660.249239] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15660.252566] ? do_wait_intr+0xf0/0xf0 [15660.254495] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15660.257017] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15660.259303] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15660.261940] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15660.264312] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15660.267526] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15660.272680] mdc_intent_lock+0x353/0x7e0 [mdc] [15660.274510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15660.277924] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15660.286072] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15660.291576] lmv_intent_open+0x338/0x1150 [lmv] [15660.294105] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15660.304638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15660.309255] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15660.310861] ll_intent_lock+0xc5/0xaa0 [lustre] [15660.313521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15660.316082] ll_lookup_it+0xe0e/0x2270 [lustre] [15660.318263] ? do_raw_spin_unlock+0x75/0x190 [15660.320282] ? _raw_spin_unlock+0x12/0x30 [15660.322005] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15660.325944] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15660.328256] ll_atomic_open+0x36a/0xd70 [lustre] [15660.332466] lookup_open+0x356/0xb40 [15660.334804] do_last+0x313/0xfc0 [15660.336231] ? nd_jump_root+0xe5/0x160 [15660.338015] ? path_init+0x437/0x520 [15660.341024] path_openat+0xf7/0x500 [15660.342731] ? do_raw_spin_unlock+0x75/0x190 [15660.343939] do_filp_open+0x99/0x140 [15660.346260] ? getname_flags+0x6e/0x330 [15660.347443] ? __check_object_size+0xff/0x256 [15660.349472] ? do_raw_spin_unlock+0x75/0x190 [15660.350388] ? _raw_spin_unlock+0x12/0x30 [15660.351293] do_sys_openat2+0x2b4/0x410 [15660.352654] do_sys_open+0x73/0xa0 [15660.353319] __x64_sys_openat+0x24/0x30 [15660.354170] do_syscall_64+0xc1/0x440 [15660.355089] entry_SYSCALL_64_after_hwframe+0x49/0xae [15660.359113] RIP: 0033:0x7fa749552332 [15660.360165] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15660.367991] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15660.371866] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15660.381693] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15660.383757] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15660.400908] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15660.408054] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15663.073833] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13745s! [15663.078528] Showing busy workqueues and worker pools: [15663.081545] workqueue events: flags=0x0 [15663.083090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15663.086454] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15663.090439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15663.092429] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15663.095686] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.098260] pending: kfree_rcu_monitor [15663.100268] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15663.103257] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15663.107513] workqueue events_long: flags=0x0 [15663.109245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.115288] pending: lu_objects_flush [obdclass] [15663.117827] workqueue events_unbound: flags=0x2 [15663.123199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15663.134557] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15663.141459] workqueue events_power_efficient: flags=0x80 [15663.145229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.150963] pending: check_lifetime [15663.152412] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.155907] pending: neigh_periodic_work [15663.157917] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.160985] pending: neigh_periodic_work [15663.162839] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.165885] pending: do_cache_clean [15663.167699] workqueue mm_percpu_wq: flags=0x8 [15663.169596] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.172249] pending: vmstat_update [15663.174144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.176829] pending: vmstat_update [15663.178258] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15663.181818] pending: vmstat_update [15663.183636] workqueue writeback: flags=0x4a [15663.185625] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15663.188762] pending: wb_workfn, wb_workfn [15663.190989] workqueue xprtiod: flags=0xa [15663.193114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15663.195989] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15663.199056] workqueue ipv6_addrconf: flags=0x40008 [15663.200824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15663.203221] pending: addrconf_verify_work [15663.205721] workqueue knbd0-recv: flags=0x1a [15663.210100] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15663.216117] in-flight: 651:recv_work [15663.224825] workqueue ptlrpc_pinger: flags=0x2 [15663.229717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15663.234787] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15663.242146] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13746s workers=4 idle: 37 31846 28897 [15663.246737] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15653s workers=2 idle: 653 [15672.059104] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15672.061295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15672.090113] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15672.096565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15672.099818] RIP: 0010:native_safe_halt+0x2e/0x50 [15672.101849] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15672.109741] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15672.113004] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15672.116520] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15672.122075] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15672.124817] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15672.128610] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15672.134809] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15672.137722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15672.140033] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15672.151968] Call Trace: [15672.152917] [15672.153634] ? show_regs.cold.9+0x22/0x2f [15672.157731] ? watchdog_timer_fn.cold.6+0x62/0x10a [15672.159830] ? __hrtimer_run_queues+0x1c2/0x4d0 [15672.163745] ? watchdog+0x40/0x40 [15672.166139] ? hrtimer_interrupt+0x128/0x2c0 [15672.171046] ? smp_apic_timer_interrupt+0xa6/0x270 [15672.175443] ? apic_timer_interrupt+0xf/0x20 [15672.177796] [15672.181696] ? native_safe_halt+0x2e/0x50 [15672.183137] kvm_wait+0xcd/0xe0 [15672.185975] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15672.189124] do_raw_spin_lock+0x177/0x190 [15672.191977] _raw_spin_lock+0x21/0x30 [15672.195220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15672.200924] ? do_raw_spin_unlock+0x75/0x190 [15672.202964] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15672.208168] ? do_wait_intr+0xf0/0xf0 [15672.210130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15672.219127] mdc_close+0x53b/0xd70 [mdc] [15672.224742] lmv_close+0x18d/0x490 [lmv] [15672.242110] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15672.244002] ll_md_real_close+0xa5/0x2a0 [lustre] [15672.256011] ll_md_close+0x274/0x800 [lustre] [15672.262941] ll_file_release+0x128/0xc50 [lustre] [15672.268154] __fput+0x10d/0x440 [15672.273951] ____fput+0x12/0x20 [15672.277104] task_work_run+0xc6/0x110 [15672.279007] exit_to_usermode_loop+0x1dd/0x1f0 [15672.281331] do_syscall_64+0x3d6/0x440 [15672.290274] entry_SYSCALL_64_after_hwframe+0x49/0xae [15672.295160] RIP: 0033:0x7fa749552cd8 [15672.296348] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15672.314611] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15672.317829] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15672.321850] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15672.328579] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15672.333353] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15672.335921] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15680.057025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15680.058674] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15680.060819] Modules linked in: lustre(O) [15680.063554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15680.063628] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15680.063642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15680.063645] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15680.063659] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15680.063664] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15680.063668] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15680.063671] RDX: 00000000000064c9 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15680.063673] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15680.063675] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15680.063678] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15680.063732] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15680.063737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15680.063739] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15680.063748] Call Trace: [15680.063753] [15680.063758] ? show_regs.cold.9+0x22/0x2f [15680.063769] ? watchdog_timer_fn.cold.6+0x62/0x10a [15680.063775] ? __hrtimer_run_queues+0x1c2/0x4d0 [15680.063780] ? watchdog+0x40/0x40 [15680.063784] ? hrtimer_interrupt+0x128/0x2c0 [15680.063788] ? smp_apic_timer_interrupt+0xa6/0x270 [15680.063795] ? apic_timer_interrupt+0xf/0x20 [15680.063800] [15680.063803] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15680.063808] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15680.063813] do_raw_spin_lock+0x177/0x190 [15680.063818] _raw_spin_lock+0x21/0x30 [15680.063829] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15680.066096] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [15680.085149] obd_mod_free+0x46/0x140 [ptlrpc] [15680.085452] mdc_commit_open+0x91/0xb0 [mdc] [15680.085486] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15680.085699] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15680.085912] after_reply+0xe2c/0x1780 [ptlrpc] [15680.086135] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15680.086359] ? __next_timer_interrupt+0x160/0x160 [15680.086369] ptlrpcd+0xa82/0xe40 [ptlrpc] [15680.086586] ? do_wait_intr+0xf0/0xf0 [15680.086595] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15680.086810] kthread+0x1d1/0x200 [15680.086816] ? set_kthread_struct+0x70/0x70 [15680.086821] ret_from_fork+0x1f/0x30 [15680.232136] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15680.245726] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15680.249860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15680.253140] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15680.255752] RIP: 0010:native_safe_halt+0x2e/0x50 [15680.257152] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15680.263738] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15680.266635] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15680.269165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15680.273494] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15680.280897] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15680.287159] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15680.290662] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15680.293678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15680.298419] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15680.304656] Call Trace: [15680.306132] [15680.308594] ? show_regs.cold.9+0x22/0x2f [15680.312401] ? watchdog_timer_fn.cold.6+0x62/0x10a [15680.314284] ? __hrtimer_run_queues+0x1c2/0x4d0 [15680.315943] ? watchdog+0x40/0x40 [15680.317305] ? hrtimer_interrupt+0x128/0x2c0 [15680.318952] ? smp_apic_timer_interrupt+0xa6/0x270 [15680.321179] ? apic_timer_interrupt+0xf/0x20 [15680.322975] [15680.323861] ? native_safe_halt+0x2e/0x50 [15680.328261] kvm_wait+0xcd/0xe0 [15680.330520] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15680.334192] do_raw_spin_lock+0x177/0x190 [15680.336113] _raw_spin_lock+0x21/0x30 [15680.337862] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15680.343390] process_one_work+0x2c8/0x700 [15680.344968] worker_thread+0x296/0x6e0 [15680.349537] ? rescuer_thread+0x570/0x570 [15680.351387] kthread+0x1d1/0x200 [15680.352999] ? set_kthread_struct+0x70/0x70 [15680.355382] ret_from_fork+0x1f/0x30 [15688.054027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15688.060484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15688.099814] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15688.106736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15688.109478] RIP: 0010:native_safe_halt+0x2e/0x50 [15688.112098] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15688.120441] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15688.125059] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15688.128661] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15688.135075] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15688.139546] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15688.143557] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15688.155939] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15688.158046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15688.163753] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15688.166964] Call Trace: [15688.167634] [15688.168146] ? show_regs.cold.9+0x22/0x2f [15688.172387] ? watchdog_timer_fn.cold.6+0x62/0x10a [15688.174082] ? __hrtimer_run_queues+0x1c2/0x4d0 [15688.186742] ? watchdog+0x40/0x40 [15688.189559] ? hrtimer_interrupt+0x128/0x2c0 [15688.191117] ? smp_apic_timer_interrupt+0xa6/0x270 [15688.192451] ? apic_timer_interrupt+0xf/0x20 [15688.194494] [15688.195874] ? native_safe_halt+0x2e/0x50 [15688.198832] kvm_wait+0xcd/0xe0 [15688.201010] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15688.204146] do_raw_spin_lock+0x177/0x190 [15688.207426] _raw_spin_lock+0x21/0x30 [15688.209072] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15688.211344] ? do_raw_spin_unlock+0x75/0x190 [15688.213399] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15688.216015] ? do_wait_intr+0xf0/0xf0 [15688.218298] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15688.221773] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15688.224081] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15688.227780] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15688.230583] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15688.232924] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15688.235048] mdc_intent_lock+0x353/0x7e0 [mdc] [15688.237126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15688.239556] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15688.242299] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15688.244495] lmv_intent_open+0x338/0x1150 [lmv] [15688.246813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15688.249219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15688.252568] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15688.254534] ll_intent_lock+0xc5/0xaa0 [lustre] [15688.256686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15688.259656] ll_lookup_it+0xe0e/0x2270 [lustre] [15688.261797] ? do_raw_spin_unlock+0x75/0x190 [15688.263825] ? _raw_spin_unlock+0x12/0x30 [15688.265672] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15688.268007] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15688.270289] ll_atomic_open+0x36a/0xd70 [lustre] [15688.271921] lookup_open+0x356/0xb40 [15688.273231] do_last+0x313/0xfc0 [15688.274609] ? nd_jump_root+0xe5/0x160 [15688.276248] ? path_init+0x437/0x520 [15688.277942] path_openat+0xf7/0x500 [15688.279348] ? do_raw_spin_unlock+0x75/0x190 [15688.281406] do_filp_open+0x99/0x140 [15688.282940] ? getname_flags+0x6e/0x330 [15688.284553] ? __check_object_size+0xff/0x256 [15688.286824] ? do_raw_spin_unlock+0x75/0x190 [15688.289683] ? _raw_spin_unlock+0x12/0x30 [15688.290999] do_sys_openat2+0x2b4/0x410 [15688.292255] do_sys_open+0x73/0xa0 [15688.294083] __x64_sys_openat+0x24/0x30 [15688.296802] do_syscall_64+0xc1/0x440 [15688.298665] entry_SYSCALL_64_after_hwframe+0x49/0xae [15688.301025] RIP: 0033:0x7fa749552332 [15688.303352] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15688.312866] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15688.316968] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15688.321376] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15688.325114] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15688.331199] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15688.336947] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15693.281182] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13776s! [15693.288193] Showing busy workqueues and worker pools: [15693.289778] workqueue events: flags=0x0 [15693.291284] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15693.294172] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15693.299074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15693.301975] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15693.305577] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.308776] pending: kfree_rcu_monitor [15693.310599] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15693.313663] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15693.317731] workqueue events_long: flags=0x0 [15693.319477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.322377] pending: lu_objects_flush [obdclass] [15693.324802] workqueue events_unbound: flags=0x2 [15693.326585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15693.328447] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15693.332170] workqueue events_power_efficient: flags=0x80 [15693.334417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.336744] pending: check_lifetime [15693.338378] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.341135] pending: neigh_periodic_work [15693.342749] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.345548] pending: neigh_periodic_work [15693.347372] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.350119] pending: do_cache_clean [15693.351680] workqueue mm_percpu_wq: flags=0x8 [15693.353517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.356267] pending: vmstat_update [15693.357722] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.360525] pending: vmstat_update [15693.362127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15693.365028] pending: vmstat_update [15693.366543] workqueue writeback: flags=0x4a [15693.368402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15693.371040] pending: wb_workfn, wb_workfn [15693.372861] workqueue xprtiod: flags=0xa [15693.374477] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15693.377322] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15693.380535] workqueue ipv6_addrconf: flags=0x40008 [15693.382613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15693.385401] pending: addrconf_verify_work [15693.387310] workqueue knbd0-recv: flags=0x1a [15693.389051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15693.392273] in-flight: 651:recv_work [15693.393867] workqueue ptlrpc_pinger: flags=0x2 [15693.395556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15693.398251] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15693.401408] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13776s workers=4 idle: 37 31846 28897 [15693.405963] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15683s workers=2 idle: 653 [15700.059062] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15700.062309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15700.087176] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15700.093465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15700.098048] RIP: 0010:native_safe_halt+0x2e/0x50 [15700.099756] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15700.107783] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15700.113319] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15700.121455] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15700.124987] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15700.130919] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15700.134337] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15700.139648] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15700.141558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15700.147944] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15700.154786] Call Trace: [15700.155914] [15700.159089] ? show_regs.cold.9+0x22/0x2f [15700.162630] ? watchdog_timer_fn.cold.6+0x62/0x10a [15700.166251] ? __hrtimer_run_queues+0x1c2/0x4d0 [15700.168576] ? watchdog+0x40/0x40 [15700.170744] ? hrtimer_interrupt+0x128/0x2c0 [15700.172862] ? smp_apic_timer_interrupt+0xa6/0x270 [15700.176082] ? apic_timer_interrupt+0xf/0x20 [15700.178575] [15700.179516] ? native_safe_halt+0x2e/0x50 [15700.181399] kvm_wait+0xcd/0xe0 [15700.182804] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15700.186112] do_raw_spin_lock+0x177/0x190 [15700.188507] _raw_spin_lock+0x21/0x30 [15700.189746] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15700.192254] ? do_raw_spin_unlock+0x75/0x190 [15700.193649] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15700.196900] ? do_wait_intr+0xf0/0xf0 [15700.199536] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15700.204070] mdc_close+0x53b/0xd70 [mdc] [15700.205812] lmv_close+0x18d/0x490 [lmv] [15700.208615] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15700.212048] ll_md_real_close+0xa5/0x2a0 [lustre] [15700.214227] ll_md_close+0x274/0x800 [lustre] [15700.217533] ll_file_release+0x128/0xc50 [lustre] [15700.220237] __fput+0x10d/0x440 [15700.221611] ____fput+0x12/0x20 [15700.223118] task_work_run+0xc6/0x110 [15700.224590] exit_to_usermode_loop+0x1dd/0x1f0 [15700.226990] do_syscall_64+0x3d6/0x440 [15700.228612] entry_SYSCALL_64_after_hwframe+0x49/0xae [15700.231769] RIP: 0033:0x7fa749552cd8 [15700.234093] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15700.244548] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15700.249102] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15700.253021] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15700.257457] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15700.261467] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15700.265365] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15708.057130] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15708.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15708.059973] Modules linked in: lustre(O) [15708.065144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15708.065230] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15708.065238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15708.065241] RIP: 0010:native_safe_halt+0x2e/0x50 [15708.065255] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15708.065260] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15708.065265] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15708.065267] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15708.065269] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15708.065271] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15708.065274] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15708.065277] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15708.067240] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15708.067317] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15708.067323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15708.067329] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15708.067905] RIP: 0010:native_safe_halt+0x2e/0x50 [15708.067916] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15708.067920] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15708.067924] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15708.067926] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15708.067929] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15708.067931] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15708.067933] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15708.067936] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15708.067939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15708.067942] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15708.067951] Call Trace: [15708.067956] [15708.067962] ? show_regs.cold.9+0x22/0x2f [15708.067973] ? watchdog_timer_fn.cold.6+0x62/0x10a [15708.067979] ? __hrtimer_run_queues+0x1c2/0x4d0 [15708.067983] ? watchdog+0x40/0x40 [15708.067988] ? hrtimer_interrupt+0x128/0x2c0 [15708.067993] ? smp_apic_timer_interrupt+0xa6/0x270 [15708.067999] ? apic_timer_interrupt+0xf/0x20 [15708.068004] [15708.068007] ? native_safe_halt+0x2e/0x50 [15708.068012] kvm_wait+0xcd/0xe0 [15708.068019] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15708.068028] do_raw_spin_lock+0x177/0x190 [15708.068033] _raw_spin_lock+0x21/0x30 [15708.068038] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15708.068279] process_one_work+0x2c8/0x700 [15708.068287] worker_thread+0x296/0x6e0 [15708.068293] ? rescuer_thread+0x570/0x570 [15708.068298] kthread+0x1d1/0x200 [15708.068302] ? set_kthread_struct+0x70/0x70 [15708.068307] ret_from_fork+0x1f/0x30 [15708.316462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15708.319215] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15708.322376] Call Trace: [15708.323602] [15708.324590] ? show_regs.cold.9+0x22/0x2f [15708.326395] ? watchdog_timer_fn.cold.6+0x62/0x10a [15708.328667] ? __hrtimer_run_queues+0x1c2/0x4d0 [15708.330776] ? watchdog+0x40/0x40 [15708.332408] ? hrtimer_interrupt+0x128/0x2c0 [15708.334326] ? smp_apic_timer_interrupt+0xa6/0x270 [15708.336626] ? apic_timer_interrupt+0xf/0x20 [15708.338379] [15708.339418] ? native_safe_halt+0x2e/0x50 [15708.341267] kvm_wait+0xcd/0xe0 [15708.342635] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15708.345014] do_raw_spin_lock+0x177/0x190 [15708.346755] _raw_spin_lock+0x21/0x30 [15708.348787] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15708.351105] obd_mod_free+0x46/0x140 [ptlrpc] [15708.353253] mdc_commit_open+0x91/0xb0 [mdc] [15708.355215] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15708.358476] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15708.361650] after_reply+0xe2c/0x1780 [ptlrpc] [15708.364225] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15708.367146] ? __next_timer_interrupt+0x160/0x160 [15708.369581] ptlrpcd+0xa82/0xe40 [ptlrpc] [15708.371730] ? do_wait_intr+0xf0/0xf0 [15708.373689] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15708.376206] kthread+0x1d1/0x200 [15708.377785] ? set_kthread_struct+0x70/0x70 [15708.379832] ret_from_fork+0x1f/0x30 [15720.057735] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15720.071287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15720.097975] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15720.102936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15720.105923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15720.108635] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15720.117247] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15720.120483] RAX: 0000000000003688 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15720.126071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15720.130165] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15720.132277] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15720.136830] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15720.140021] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15720.143094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15720.145773] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15720.148731] Call Trace: [15720.150384] [15720.151131] ? show_regs.cold.9+0x22/0x2f [15720.153501] ? watchdog_timer_fn.cold.6+0x62/0x10a [15720.156714] ? __hrtimer_run_queues+0x1c2/0x4d0 [15720.159229] ? watchdog+0x40/0x40 [15720.160750] ? hrtimer_interrupt+0x128/0x2c0 [15720.161962] ? smp_apic_timer_interrupt+0xa6/0x270 [15720.163734] ? apic_timer_interrupt+0xf/0x20 [15720.167040] [15720.168075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15720.170730] do_raw_spin_lock+0x177/0x190 [15720.172189] _raw_spin_lock+0x21/0x30 [15720.173249] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15720.176390] ? do_raw_spin_unlock+0x75/0x190 [15720.178774] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15720.180874] ? do_wait_intr+0xf0/0xf0 [15720.181967] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15720.183997] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15720.186111] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15720.188478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15720.190586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15720.192647] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15720.194434] mdc_intent_lock+0x353/0x7e0 [mdc] [15720.196114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15720.198225] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15720.200604] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15720.202683] lmv_intent_open+0x338/0x1150 [lmv] [15720.204413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15720.206472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15720.208536] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15720.210214] ll_intent_lock+0xc5/0xaa0 [lustre] [15720.212050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15720.215599] ll_lookup_it+0xe0e/0x2270 [lustre] [15720.218759] ? do_raw_spin_unlock+0x75/0x190 [15720.221377] ? _raw_spin_unlock+0x12/0x30 [15720.223920] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15720.225236] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15720.227844] ll_atomic_open+0x36a/0xd70 [lustre] [15720.230433] lookup_open+0x356/0xb40 [15720.233773] do_last+0x313/0xfc0 [15720.235395] ? nd_jump_root+0xe5/0x160 [15720.236269] ? path_init+0x437/0x520 [15720.237162] path_openat+0xf7/0x500 [15720.238116] ? do_raw_spin_unlock+0x75/0x190 [15720.240148] do_filp_open+0x99/0x140 [15720.242198] ? getname_flags+0x6e/0x330 [15720.244144] ? __check_object_size+0xff/0x256 [15720.245965] ? do_raw_spin_unlock+0x75/0x190 [15720.247756] ? _raw_spin_unlock+0x12/0x30 [15720.249683] do_sys_openat2+0x2b4/0x410 [15720.251238] do_sys_open+0x73/0xa0 [15720.252156] __x64_sys_openat+0x24/0x30 [15720.253924] do_syscall_64+0xc1/0x440 [15720.255965] entry_SYSCALL_64_after_hwframe+0x49/0xae [15720.258797] RIP: 0033:0x7fa749552332 [15720.261142] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15720.271404] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15720.275494] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15720.277765] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15720.281080] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15720.284922] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15720.288485] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15723.489359] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13806s! [15723.492810] Showing busy workqueues and worker pools: [15723.494956] workqueue events: flags=0x0 [15723.497172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15723.499407] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15723.505076] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15723.509027] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15723.514838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.518485] pending: kfree_rcu_monitor [15723.519850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15723.524275] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15723.527150] workqueue events_long: flags=0x0 [15723.529773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.539996] pending: lu_objects_flush [obdclass] [15723.542713] workqueue events_unbound: flags=0x2 [15723.548362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15723.550514] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15723.561184] workqueue events_power_efficient: flags=0x80 [15723.568435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.572395] pending: check_lifetime [15723.574255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.577959] pending: neigh_periodic_work [15723.580008] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.583638] pending: neigh_periodic_work [15723.586219] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.590215] pending: do_cache_clean [15723.591959] workqueue mm_percpu_wq: flags=0x8 [15723.593844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.599593] pending: vmstat_update [15723.601743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.604910] pending: vmstat_update [15723.607054] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15723.612967] pending: vmstat_update [15723.615119] workqueue writeback: flags=0x4a [15723.616537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15723.619501] pending: wb_workfn, wb_workfn [15723.623771] workqueue xprtiod: flags=0xa [15723.626826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15723.630934] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15723.634433] workqueue ipv6_addrconf: flags=0x40008 [15723.636153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15723.641791] pending: addrconf_verify_work [15723.643525] workqueue knbd0-recv: flags=0x1a [15723.647758] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15723.649867] in-flight: 651:recv_work [15723.651995] workqueue ptlrpc_pinger: flags=0x2 [15723.653936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15723.657439] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15723.662861] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13806s workers=4 idle: 37 31846 28897 [15723.667643] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15713s workers=2 idle: 653 [15728.059096] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15728.062881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15728.083013] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15728.092140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15728.099615] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15728.104738] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15728.120984] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15728.127719] RAX: 00000000000016a7 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15728.131620] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15728.138379] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15728.144333] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15728.148995] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15728.155132] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15728.163032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15728.165762] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15728.172608] Call Trace: [15728.173589] [15728.174936] ? show_regs.cold.9+0x22/0x2f [15728.178322] ? watchdog_timer_fn.cold.6+0x62/0x10a [15728.182606] ? __hrtimer_run_queues+0x1c2/0x4d0 [15728.184938] ? watchdog+0x40/0x40 [15728.186079] ? hrtimer_interrupt+0x128/0x2c0 [15728.187117] ? smp_apic_timer_interrupt+0xa6/0x270 [15728.189344] ? apic_timer_interrupt+0xf/0x20 [15728.192823] [15728.193971] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15728.197816] do_raw_spin_lock+0x177/0x190 [15728.201237] _raw_spin_lock+0x21/0x30 [15728.202505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15728.204163] ? do_raw_spin_unlock+0x75/0x190 [15728.208553] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15728.211165] ? do_wait_intr+0xf0/0xf0 [15728.213068] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15728.216575] mdc_close+0x53b/0xd70 [mdc] [15728.219491] lmv_close+0x18d/0x490 [lmv] [15728.221744] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15728.226734] ll_md_real_close+0xa5/0x2a0 [lustre] [15728.228874] ll_md_close+0x274/0x800 [lustre] [15728.231377] ll_file_release+0x128/0xc50 [lustre] [15728.233940] __fput+0x10d/0x440 [15728.236577] ____fput+0x12/0x20 [15728.239561] task_work_run+0xc6/0x110 [15728.241781] exit_to_usermode_loop+0x1dd/0x1f0 [15728.245040] do_syscall_64+0x3d6/0x440 [15728.247710] entry_SYSCALL_64_after_hwframe+0x49/0xae [15728.252259] RIP: 0033:0x7fa749552cd8 [15728.253945] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15728.265767] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15728.270487] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15728.276410] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15728.285915] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15728.288283] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15728.297966] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15736.057102] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15736.058094] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15736.061253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [15736.065983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15736.066219] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15736.066255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15736.066262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15736.066374] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15736.066407] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15736.066415] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15736.066448] RDX: 00000000000055ff RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15736.066478] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15736.066480] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15736.066659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15736.066662] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15736.066729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15736.066733] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15736.066741] Call Trace: [15736.066747] [15736.066753] ? show_regs.cold.9+0x22/0x2f [15736.066763] ? watchdog_timer_fn.cold.6+0x62/0x10a [15736.066770] ? __hrtimer_run_queues+0x1c2/0x4d0 [15736.066774] ? watchdog+0x40/0x40 [15736.066778] ? hrtimer_interrupt+0x128/0x2c0 [15736.066783] ? smp_apic_timer_interrupt+0xa6/0x270 [15736.066791] ? apic_timer_interrupt+0xf/0x20 [15736.066796] [15736.066799] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15736.066804] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15736.066809] do_raw_spin_lock+0x177/0x190 [15736.066813] _raw_spin_lock+0x21/0x30 [15736.066820] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15736.067177] obd_mod_free+0x46/0x140 [ptlrpc] [15736.111489] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15736.111534] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15736.111541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15736.111548] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15736.111828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15736.111839] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15736.111844] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15736.111848] RAX: 0000000000001aab RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15736.111850] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15736.111853] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15736.111855] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15736.111857] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15736.111860] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15736.111863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15736.111866] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15736.111876] Call Trace: [15736.111882] [15736.111887] ? show_regs.cold.9+0x22/0x2f [15736.111897] ? watchdog_timer_fn.cold.6+0x62/0x10a [15736.111902] ? __hrtimer_run_queues+0x1c2/0x4d0 [15736.111907] ? watchdog+0x40/0x40 [15736.111912] ? hrtimer_interrupt+0x128/0x2c0 [15736.111915] ? smp_apic_timer_interrupt+0xa6/0x270 [15736.111924] ? apic_timer_interrupt+0xf/0x20 [15736.111928] [15736.111931] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15736.111936] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15736.111942] do_raw_spin_lock+0x177/0x190 [15736.111946] _raw_spin_lock+0x21/0x30 [15736.111952] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15736.112160] process_one_work+0x2c8/0x700 [15736.112167] worker_thread+0x296/0x6e0 [15736.112171] ? rescuer_thread+0x570/0x570 [15736.112174] kthread+0x1d1/0x200 [15736.112179] ? set_kthread_struct+0x70/0x70 [15736.112183] ret_from_fork+0x1f/0x30 [15736.483719] mdc_commit_open+0x91/0xb0 [mdc] [15736.485160] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15736.492009] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15736.500969] after_reply+0xe2c/0x1780 [ptlrpc] [15736.504508] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15736.508212] ? __next_timer_interrupt+0x160/0x160 [15736.513811] ptlrpcd+0xa82/0xe40 [ptlrpc] [15736.517090] ? do_wait_intr+0xf0/0xf0 [15736.520529] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15736.522624] kthread+0x1d1/0x200 [15736.529374] ? set_kthread_struct+0x70/0x70 [15736.534955] ret_from_fork+0x1f/0x30 [15748.054336] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15748.058295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15748.079402] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15748.084200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15748.088166] RIP: 0010:native_safe_halt+0x2e/0x50 [15748.090256] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15748.101089] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15748.106251] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15748.110295] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15748.113170] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15748.116434] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15748.122958] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15748.128183] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15748.135729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15748.138820] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15748.143513] Call Trace: [15748.144534] [15748.145979] ? show_regs.cold.9+0x22/0x2f [15748.149629] ? watchdog_timer_fn.cold.6+0x62/0x10a [15748.151635] ? __hrtimer_run_queues+0x1c2/0x4d0 [15748.153722] ? watchdog+0x40/0x40 [15748.155089] ? hrtimer_interrupt+0x128/0x2c0 [15748.156907] ? smp_apic_timer_interrupt+0xa6/0x270 [15748.158572] ? apic_timer_interrupt+0xf/0x20 [15748.159978] [15748.160533] ? native_safe_halt+0x2e/0x50 [15748.161525] kvm_wait+0xcd/0xe0 [15748.162616] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15748.164335] do_raw_spin_lock+0x177/0x190 [15748.166247] _raw_spin_lock+0x21/0x30 [15748.167970] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15748.169499] ? do_raw_spin_unlock+0x75/0x190 [15748.171686] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15748.173497] ? do_wait_intr+0xf0/0xf0 [15748.174458] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15748.175929] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15748.177344] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15748.179320] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15748.182721] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15748.185646] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15748.187029] mdc_intent_lock+0x353/0x7e0 [mdc] [15748.188570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15748.190578] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15748.192438] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15748.194032] lmv_intent_open+0x338/0x1150 [lmv] [15748.195194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15748.196404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15748.197765] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15748.199027] ll_intent_lock+0xc5/0xaa0 [lustre] [15748.200617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15748.202661] ll_lookup_it+0xe0e/0x2270 [lustre] [15748.204149] ? do_raw_spin_unlock+0x75/0x190 [15748.205412] ? _raw_spin_unlock+0x12/0x30 [15748.206672] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15748.208116] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15748.209570] ll_atomic_open+0x36a/0xd70 [lustre] [15748.211337] lookup_open+0x356/0xb40 [15748.212519] do_last+0x313/0xfc0 [15748.213708] ? nd_jump_root+0xe5/0x160 [15748.214904] ? path_init+0x437/0x520 [15748.216027] path_openat+0xf7/0x500 [15748.217011] ? do_raw_spin_unlock+0x75/0x190 [15748.218441] do_filp_open+0x99/0x140 [15748.219682] ? getname_flags+0x6e/0x330 [15748.221034] ? __check_object_size+0xff/0x256 [15748.222350] ? do_raw_spin_unlock+0x75/0x190 [15748.223643] ? _raw_spin_unlock+0x12/0x30 [15748.224984] do_sys_openat2+0x2b4/0x410 [15748.226226] do_sys_open+0x73/0xa0 [15748.227218] __x64_sys_openat+0x24/0x30 [15748.228945] do_syscall_64+0xc1/0x440 [15748.229935] entry_SYSCALL_64_after_hwframe+0x49/0xae [15748.231135] RIP: 0033:0x7fa749552332 [15748.232001] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15748.237464] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15748.240224] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15748.242693] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15748.245185] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15748.247588] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15748.249381] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15753.696657] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13836s! [15753.700455] Showing busy workqueues and worker pools: [15753.706640] workqueue events: flags=0x0 [15753.708482] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15753.716168] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15753.723953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15753.728409] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15753.732804] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.735906] pending: kfree_rcu_monitor [15753.737708] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15753.740558] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15753.744155] workqueue events_long: flags=0x0 [15753.745892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.748577] pending: lu_objects_flush [obdclass] [15753.750491] workqueue events_unbound: flags=0x2 [15753.752149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15753.754575] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15753.757913] workqueue events_power_efficient: flags=0x80 [15753.759571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.761431] pending: check_lifetime [15753.762470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.765715] pending: neigh_periodic_work [15753.767519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.770473] pending: neigh_periodic_work [15753.771969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.774864] pending: do_cache_clean [15753.777048] workqueue mm_percpu_wq: flags=0x8 [15753.778250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.780925] pending: vmstat_update [15753.785162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.791014] pending: vmstat_update [15753.792346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15753.799503] pending: vmstat_update [15753.802251] workqueue writeback: flags=0x4a [15753.807018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15753.809438] pending: wb_workfn, wb_workfn [15753.814664] workqueue xprtiod: flags=0xa [15753.817144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15753.828292] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15753.836255] workqueue ipv6_addrconf: flags=0x40008 [15753.841525] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15753.844242] pending: addrconf_verify_work [15753.848739] workqueue knbd0-recv: flags=0x1a [15753.850628] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15753.856145] in-flight: 651:recv_work [15753.859973] workqueue ptlrpc_pinger: flags=0x2 [15753.864091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15753.868593] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15753.875128] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13836s workers=4 idle: 37 31846 28897 [15753.881232] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15743s workers=2 idle: 653 [15756.062433] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15756.068342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15756.099861] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15756.104591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15756.113279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15756.120598] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15756.143124] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15756.150485] RAX: 0000000000000c30 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15756.153313] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15756.160429] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15756.168271] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15756.170322] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15756.174440] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15756.180621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15756.184590] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15756.191800] Call Trace: [15756.192940] [15756.195015] ? show_regs.cold.9+0x22/0x2f [15756.198975] ? watchdog_timer_fn.cold.6+0x62/0x10a [15756.200700] ? __hrtimer_run_queues+0x1c2/0x4d0 [15756.208810] ? watchdog+0x40/0x40 [15756.210371] ? hrtimer_interrupt+0x128/0x2c0 [15756.214472] ? smp_apic_timer_interrupt+0xa6/0x270 [15756.217674] ? apic_timer_interrupt+0xf/0x20 [15756.221176] [15756.222045] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15756.228293] do_raw_spin_lock+0x177/0x190 [15756.229780] _raw_spin_lock+0x21/0x30 [15756.232909] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15756.236791] ? do_raw_spin_unlock+0x75/0x190 [15756.239335] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15756.243592] ? do_wait_intr+0xf0/0xf0 [15756.245724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15756.247713] mdc_close+0x53b/0xd70 [mdc] [15756.252811] lmv_close+0x18d/0x490 [lmv] [15756.254257] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15756.257525] ll_md_real_close+0xa5/0x2a0 [lustre] [15756.261733] ll_md_close+0x274/0x800 [lustre] [15756.263447] ll_file_release+0x128/0xc50 [lustre] [15756.267274] __fput+0x10d/0x440 [15756.268669] ____fput+0x12/0x20 [15756.270302] task_work_run+0xc6/0x110 [15756.276921] exit_to_usermode_loop+0x1dd/0x1f0 [15756.283529] do_syscall_64+0x3d6/0x440 [15756.289991] entry_SYSCALL_64_after_hwframe+0x49/0xae [15756.291836] RIP: 0033:0x7fa749552cd8 [15756.298788] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15756.316954] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15756.323468] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15756.333310] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15756.338286] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15756.344683] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15756.348338] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15764.057085] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15764.058025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15764.062045] Modules linked in: [15764.065833] Modules linked in: [15764.065836] lustre(O) [15764.065852] lustre(O) [15764.067879] osp(O) [15764.069095] osp(O) [15764.070018] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15764.070086] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15764.070093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15764.070099] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15764.070407] RIP: 0010:native_safe_halt+0x2e/0x50 [15764.070416] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15764.070420] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15764.070425] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15764.070428] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15764.070430] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15764.070432] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15764.070434] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15764.070438] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15764.070441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15764.070443] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15764.070451] Call Trace: [15764.070457] [15764.070461] ? show_regs.cold.9+0x22/0x2f [15764.070471] ? watchdog_timer_fn.cold.6+0x62/0x10a [15764.070477] ? __hrtimer_run_queues+0x1c2/0x4d0 [15764.070481] ? watchdog+0x40/0x40 [15764.070486] ? hrtimer_interrupt+0x128/0x2c0 [15764.070490] ? smp_apic_timer_interrupt+0xa6/0x270 [15764.070497] ? apic_timer_interrupt+0xf/0x20 [15764.070502] [15764.070505] ? native_safe_halt+0x2e/0x50 [15764.070508] kvm_wait+0xcd/0xe0 [15764.070516] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15764.070523] do_raw_spin_lock+0x177/0x190 [15764.070528] _raw_spin_lock+0x21/0x30 [15764.070532] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15764.070866] process_one_work+0x2c8/0x700 [15764.070874] worker_thread+0x296/0x6e0 [15764.070878] ? rescuer_thread+0x570/0x570 [15764.070882] kthread+0x1d1/0x200 [15764.070887] ? set_kthread_struct+0x70/0x70 [15764.070890] ret_from_fork+0x1f/0x30 [15764.214916] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15764.236832] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15764.242519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15764.247143] RIP: 0010:native_safe_halt+0x2e/0x50 [15764.249522] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15764.258454] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15764.261821] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15764.265460] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15764.268889] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15764.272727] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15764.275911] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15764.278982] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15764.281924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15764.284664] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15764.287360] Call Trace: [15764.288465] [15764.289808] ? show_regs.cold.9+0x22/0x2f [15764.292149] ? watchdog_timer_fn.cold.6+0x62/0x10a [15764.294460] ? __hrtimer_run_queues+0x1c2/0x4d0 [15764.296384] ? watchdog+0x40/0x40 [15764.297408] ? hrtimer_interrupt+0x128/0x2c0 [15764.299827] ? smp_apic_timer_interrupt+0xa6/0x270 [15764.302409] ? apic_timer_interrupt+0xf/0x20 [15764.304298] [15764.305292] ? native_safe_halt+0x2e/0x50 [15764.307181] kvm_wait+0xcd/0xe0 [15764.308520] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15764.310850] do_raw_spin_lock+0x177/0x190 [15764.312388] _raw_spin_lock+0x21/0x30 [15764.313986] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15764.316710] obd_mod_free+0x46/0x140 [ptlrpc] [15764.318963] mdc_commit_open+0x91/0xb0 [mdc] [15764.320877] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15764.324153] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15764.326934] after_reply+0xe2c/0x1780 [ptlrpc] [15764.329429] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15764.332419] ? __next_timer_interrupt+0x160/0x160 [15764.335200] ptlrpcd+0xa82/0xe40 [ptlrpc] [15764.336764] ? do_wait_intr+0xf0/0xf0 [15764.338077] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15764.340287] kthread+0x1d1/0x200 [15764.341398] ? set_kthread_struct+0x70/0x70 [15764.343127] ret_from_fork+0x1f/0x30 [15780.054035] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15780.061776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15780.094315] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15780.103059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15780.107982] RIP: 0010:native_safe_halt+0x2e/0x50 [15780.110410] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15780.129204] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15780.136000] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15780.138886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15780.143029] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15780.147099] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15780.151941] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15780.156014] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15780.161339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15780.165008] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15780.169357] Call Trace: [15780.170929] [15780.171974] ? show_regs.cold.9+0x22/0x2f [15780.173971] ? watchdog_timer_fn.cold.6+0x62/0x10a [15780.177331] ? __hrtimer_run_queues+0x1c2/0x4d0 [15780.179938] ? watchdog+0x40/0x40 [15780.182640] ? hrtimer_interrupt+0x128/0x2c0 [15780.184484] ? smp_apic_timer_interrupt+0xa6/0x270 [15780.186665] ? apic_timer_interrupt+0xf/0x20 [15780.190637] [15780.191863] ? native_safe_halt+0x2e/0x50 [15780.194068] kvm_wait+0xcd/0xe0 [15780.195743] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15780.198752] do_raw_spin_lock+0x177/0x190 [15780.201122] _raw_spin_lock+0x21/0x30 [15780.203340] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15780.206471] ? do_raw_spin_unlock+0x75/0x190 [15780.208006] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15780.211359] ? do_wait_intr+0xf0/0xf0 [15780.213419] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15780.216969] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15780.220389] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15780.223770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15780.226948] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15780.229766] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15780.231963] mdc_intent_lock+0x353/0x7e0 [mdc] [15780.235493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15780.238418] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15780.242567] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15780.245398] lmv_intent_open+0x338/0x1150 [lmv] [15780.247368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15780.251476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15780.254845] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15780.257554] ll_intent_lock+0xc5/0xaa0 [lustre] [15780.259169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15780.264082] ll_lookup_it+0xe0e/0x2270 [lustre] [15780.266926] ? do_raw_spin_unlock+0x75/0x190 [15780.268472] ? _raw_spin_unlock+0x12/0x30 [15780.269895] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15780.274101] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15780.279453] ll_atomic_open+0x36a/0xd70 [lustre] [15780.284724] lookup_open+0x356/0xb40 [15780.287490] do_last+0x313/0xfc0 [15780.289152] ? nd_jump_root+0xe5/0x160 [15780.291844] ? path_init+0x437/0x520 [15780.297865] path_openat+0xf7/0x500 [15780.299872] ? do_raw_spin_unlock+0x75/0x190 [15780.302128] do_filp_open+0x99/0x140 [15780.309374] ? getname_flags+0x6e/0x330 [15780.318013] ? __check_object_size+0xff/0x256 [15780.319248] ? do_raw_spin_unlock+0x75/0x190 [15780.322905] ? _raw_spin_unlock+0x12/0x30 [15780.324235] do_sys_openat2+0x2b4/0x410 [15780.328044] do_sys_open+0x73/0xa0 [15780.330212] __x64_sys_openat+0x24/0x30 [15780.331778] do_syscall_64+0xc1/0x440 [15780.332858] entry_SYSCALL_64_after_hwframe+0x49/0xae [15780.334974] RIP: 0033:0x7fa749552332 [15780.336334] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15780.347052] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15780.352219] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15780.358622] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15780.361005] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15780.364196] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15780.367971] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15783.904045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13866s! [15783.908788] Showing busy workqueues and worker pools: [15783.912028] workqueue events: flags=0x0 [15783.913989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15783.917018] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15783.922118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15783.925150] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15783.929088] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15783.932198] pending: kfree_rcu_monitor [15783.934111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15783.937174] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15783.941303] workqueue events_long: flags=0x0 [15783.943197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15783.946058] pending: lu_objects_flush [obdclass] [15783.948553] workqueue events_unbound: flags=0x2 [15783.950907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15783.953604] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15783.957412] workqueue events_power_efficient: flags=0x80 [15783.959797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15783.962543] pending: check_lifetime [15783.964167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15783.967010] pending: neigh_periodic_work [15783.968950] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15783.971818] pending: neigh_periodic_work [15783.973642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15783.980557] pending: do_cache_clean [15783.982943] workqueue mm_percpu_wq: flags=0x8 [15783.986450] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15783.990161] pending: vmstat_update [15783.992033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15784.003990] pending: vmstat_update [15784.006468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15784.010340] pending: vmstat_update [15784.016555] workqueue writeback: flags=0x4a [15784.022895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15784.029687] pending: wb_workfn, wb_workfn [15784.034824] workqueue xprtiod: flags=0xa [15784.037796] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15784.042620] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15784.046949] workqueue ipv6_addrconf: flags=0x40008 [15784.049479] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15784.053065] pending: addrconf_verify_work [15784.055201] workqueue knbd0-recv: flags=0x1a [15784.057279] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15784.059034] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15784.061771] in-flight: 651:recv_work [15784.061821] workqueue ptlrpc_pinger: flags=0x2 [15784.064886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15784.064972] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15784.064980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15784.064983] RIP: 0010:native_safe_halt+0x2e/0x50 [15784.064997] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15784.065001] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15784.065005] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15784.065007] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15784.065010] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15784.065012] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15784.065014] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15784.065017] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15784.065020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15784.065023] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15784.065032] Call Trace: [15784.065039] [15784.065046] ? show_regs.cold.9+0x22/0x2f [15784.065055] ? watchdog_timer_fn.cold.6+0x62/0x10a [15784.065062] ? __hrtimer_run_queues+0x1c2/0x4d0 [15784.065066] ? watchdog+0x40/0x40 [15784.065070] ? hrtimer_interrupt+0x128/0x2c0 [15784.065074] ? smp_apic_timer_interrupt+0xa6/0x270 [15784.065081] ? apic_timer_interrupt+0xf/0x20 [15784.068045] pwq 8: [15784.071616] [15784.071629] ? native_safe_halt+0x2e/0x50 [15784.071640] kvm_wait+0xcd/0xe0 [15784.071654] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15784.071736] do_raw_spin_lock+0x177/0x190 [15784.071744] _raw_spin_lock+0x21/0x30 [15784.071749] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15784.095886] cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15784.095905] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15784.096378] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13866s workers=4 idle: 37 31846 28897 [15784.096395] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15773s workers=2 idle: 653 [15784.202522] ? do_raw_spin_unlock+0x75/0x190 [15784.204424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15784.206946] ? do_wait_intr+0xf0/0xf0 [15784.208898] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15784.213003] mdc_close+0x53b/0xd70 [mdc] [15784.215406] lmv_close+0x18d/0x490 [lmv] [15784.217547] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15784.220878] ll_md_real_close+0xa5/0x2a0 [lustre] [15784.223871] ll_md_close+0x274/0x800 [lustre] [15784.226680] ll_file_release+0x128/0xc50 [lustre] [15784.232888] __fput+0x10d/0x440 [15784.234649] ____fput+0x12/0x20 [15784.235667] task_work_run+0xc6/0x110 [15784.237271] exit_to_usermode_loop+0x1dd/0x1f0 [15784.239108] do_syscall_64+0x3d6/0x440 [15784.240572] entry_SYSCALL_64_after_hwframe+0x49/0xae [15784.242567] RIP: 0033:0x7fa749552cd8 [15784.244418] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15784.258723] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15784.265202] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15784.269039] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15784.273016] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15784.278496] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15784.285672] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15792.057528] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15792.058270] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15792.062859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15792.063139] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15792.063147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15792.063154] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15792.068141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15792.068214] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15792.068221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15792.068223] RIP: 0010:native_safe_halt+0x2e/0x50 [15792.068236] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15792.068241] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15792.068245] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15792.068247] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15792.068249] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15792.068250] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15792.068253] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15792.068256] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15792.068258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15792.068260] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15792.068270] Call Trace: [15792.068275] [15792.068281] ? show_regs.cold.9+0x22/0x2f [15792.068291] ? watchdog_timer_fn.cold.6+0x62/0x10a [15792.068297] ? __hrtimer_run_queues+0x1c2/0x4d0 [15792.068301] ? watchdog+0x40/0x40 [15792.068305] ? hrtimer_interrupt+0x128/0x2c0 [15792.068308] ? smp_apic_timer_interrupt+0xa6/0x270 [15792.068315] ? apic_timer_interrupt+0xf/0x20 [15792.068319] [15792.068322] ? native_safe_halt+0x2e/0x50 [15792.068326] kvm_wait+0xcd/0xe0 [15792.068333] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15792.068339] do_raw_spin_lock+0x177/0x190 [15792.068343] _raw_spin_lock+0x21/0x30 [15792.068360] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15792.068685] obd_mod_free+0x46/0x140 [ptlrpc] [15792.104679] [15792.113371] mdc_commit_open+0x91/0xb0 [mdc] [15792.116681] RIP: 0010:native_safe_halt+0x2e/0x50 [15792.118924] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15792.157215] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15792.160758] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15792.161157] after_reply+0xe2c/0x1780 [ptlrpc] [15792.161346] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15792.161652] ? __next_timer_interrupt+0x160/0x160 [15792.161662] ptlrpcd+0xa82/0xe40 [ptlrpc] [15792.161928] ? do_wait_intr+0xf0/0xf0 [15792.161936] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15792.162144] kthread+0x1d1/0x200 [15792.162151] ? set_kthread_struct+0x70/0x70 [15792.162155] ret_from_fork+0x1f/0x30 [15792.335886] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15792.341733] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15792.344658] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15792.347769] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15792.351999] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15792.355702] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15792.360758] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15792.364564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15792.367570] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15792.369626] Call Trace: [15792.370228] [15792.375507] ? show_regs.cold.9+0x22/0x2f [15792.376550] ? watchdog_timer_fn.cold.6+0x62/0x10a [15792.377819] ? __hrtimer_run_queues+0x1c2/0x4d0 [15792.384456] ? watchdog+0x40/0x40 [15792.385336] ? hrtimer_interrupt+0x128/0x2c0 [15792.396965] ? smp_apic_timer_interrupt+0xa6/0x270 [15792.398294] ? apic_timer_interrupt+0xf/0x20 [15792.399460] [15792.400064] ? native_safe_halt+0x2e/0x50 [15792.405414] kvm_wait+0xcd/0xe0 [15792.407213] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15792.412582] do_raw_spin_lock+0x177/0x190 [15792.414860] _raw_spin_lock+0x21/0x30 [15792.416805] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15792.419715] process_one_work+0x2c8/0x700 [15792.421709] worker_thread+0x296/0x6e0 [15792.423407] ? rescuer_thread+0x570/0x570 [15792.425682] kthread+0x1d1/0x200 [15792.427270] ? set_kthread_struct+0x70/0x70 [15792.429390] ret_from_fork+0x1f/0x30 [15812.054016] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15812.056453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15812.062515] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15812.089825] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15812.089844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15812.089846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15812.089864] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15812.089868] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15812.089877] RAX: 000000000000283a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15812.089879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15812.089881] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15812.089883] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15812.089886] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15812.089889] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15812.089892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15812.089894] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15812.089902] Call Trace: [15812.089907] [15812.089913] ? show_regs.cold.9+0x22/0x2f [15812.089921] ? watchdog_timer_fn.cold.6+0x62/0x10a [15812.089928] ? __hrtimer_run_queues+0x1c2/0x4d0 [15812.089933] ? watchdog+0x40/0x40 [15812.089938] ? hrtimer_interrupt+0x128/0x2c0 [15812.089942] ? smp_apic_timer_interrupt+0xa6/0x270 [15812.089950] ? apic_timer_interrupt+0xf/0x20 [15812.089961] [15812.089964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15812.089973] do_raw_spin_lock+0x177/0x190 [15812.089978] _raw_spin_lock+0x21/0x30 [15812.089982] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15812.095068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15812.095215] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15812.095223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15812.095226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15812.095241] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15812.095246] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15812.095251] RAX: 0000000000005a1a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15812.095254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15812.095256] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15812.095258] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15812.095261] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15812.095265] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15812.095268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15812.095270] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15812.095279] Call Trace: [15812.095284] [15812.095291] ? show_regs.cold.9+0x22/0x2f [15812.095300] ? watchdog_timer_fn.cold.6+0x62/0x10a [15812.095307] ? __hrtimer_run_queues+0x1c2/0x4d0 [15812.095311] ? watchdog+0x40/0x40 [15812.095316] ? hrtimer_interrupt+0x128/0x2c0 [15812.095321] ? smp_apic_timer_interrupt+0xa6/0x270 [15812.095328] ? apic_timer_interrupt+0xf/0x20 [15812.095333] [15812.095336] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15812.095343] do_raw_spin_lock+0x177/0x190 [15812.095349] _raw_spin_lock+0x21/0x30 [15812.095355] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15812.095710] ? do_raw_spin_unlock+0x75/0x190 [15812.095774] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15812.095973] ? do_wait_intr+0xf0/0xf0 [15812.095983] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15812.096256] mdc_close+0x53b/0xd70 [mdc] [15812.096301] lmv_close+0x18d/0x490 [lmv] [15812.096333] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15812.096445] ll_md_real_close+0xa5/0x2a0 [lustre] [15812.096528] ll_md_close+0x274/0x800 [lustre] [15812.096702] ll_file_release+0x128/0xc50 [lustre] [15812.096850] __fput+0x10d/0x440 [15812.096860] ____fput+0x12/0x20 [15812.096865] task_work_run+0xc6/0x110 [15812.096872] exit_to_usermode_loop+0x1dd/0x1f0 [15812.096879] do_syscall_64+0x3d6/0x440 [15812.096884] entry_SYSCALL_64_after_hwframe+0x49/0xae [15812.096891] RIP: 0033:0x7fa749552cd8 [15812.096898] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15812.096902] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15812.096906] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15812.096909] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15812.096911] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15812.096913] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15812.096916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15812.335308] ? do_raw_spin_unlock+0x75/0x190 [15812.337464] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15812.340318] ? do_wait_intr+0xf0/0xf0 [15812.342022] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15812.344366] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15812.346841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15812.349717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15812.352685] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15812.355014] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15812.357258] mdc_intent_lock+0x353/0x7e0 [mdc] [15812.358971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15812.361492] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15812.364353] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15812.366698] lmv_intent_open+0x338/0x1150 [lmv] [15812.368612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15812.371280] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15812.374323] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15812.376277] ll_intent_lock+0xc5/0xaa0 [lustre] [15812.378821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15812.381969] ll_lookup_it+0xe0e/0x2270 [lustre] [15812.384329] ? do_raw_spin_unlock+0x75/0x190 [15812.386339] ? _raw_spin_unlock+0x12/0x30 [15812.388278] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15812.390973] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15812.393329] ll_atomic_open+0x36a/0xd70 [lustre] [15812.395499] lookup_open+0x356/0xb40 [15812.397279] do_last+0x313/0xfc0 [15812.398666] ? nd_jump_root+0xe5/0x160 [15812.400585] ? path_init+0x437/0x520 [15812.402374] path_openat+0xf7/0x500 [15812.404254] ? do_raw_spin_unlock+0x75/0x190 [15812.405859] do_filp_open+0x99/0x140 [15812.407687] ? getname_flags+0x6e/0x330 [15812.409426] ? __check_object_size+0xff/0x256 [15812.411474] ? do_raw_spin_unlock+0x75/0x190 [15812.413437] ? _raw_spin_unlock+0x12/0x30 [15812.415466] do_sys_openat2+0x2b4/0x410 [15812.416981] do_sys_open+0x73/0xa0 [15812.418513] __x64_sys_openat+0x24/0x30 [15812.420378] do_syscall_64+0xc1/0x440 [15812.422019] entry_SYSCALL_64_after_hwframe+0x49/0xae [15812.423858] RIP: 0033:0x7fa749552332 [15812.425638] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15812.434785] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15812.437507] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15812.440446] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15812.443373] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15812.446470] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15812.449527] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15814.113044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13896s! [15814.115886] Showing busy workqueues and worker pools: [15814.117019] workqueue events: flags=0x0 [15814.117784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15814.119159] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15814.122376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15814.124130] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15814.126256] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.127677] pending: kfree_rcu_monitor [15814.128521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15814.129941] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15814.132029] workqueue events_long: flags=0x0 [15814.132952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.134436] pending: lu_objects_flush [obdclass] [15814.135752] workqueue events_unbound: flags=0x2 [15814.137010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15814.138326] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15814.140144] workqueue events_power_efficient: flags=0x80 [15814.141150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.142261] pending: check_lifetime [15814.142938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.144044] pending: neigh_periodic_work [15814.144744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.146290] pending: neigh_periodic_work [15814.147035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.148152] pending: do_cache_clean [15814.148847] workqueue mm_percpu_wq: flags=0x8 [15814.149550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.150683] pending: vmstat_update [15814.151335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.152460] pending: vmstat_update [15814.153132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15814.154248] pending: vmstat_update [15814.154943] workqueue writeback: flags=0x4a [15814.155798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15814.157094] pending: wb_workfn, wb_workfn [15814.157926] workqueue xprtiod: flags=0xa [15814.158588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15814.159933] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15814.161522] workqueue ipv6_addrconf: flags=0x40008 [15814.162422] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15814.163930] pending: addrconf_verify_work [15814.164988] workqueue knbd0-recv: flags=0x1a [15814.166602] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15814.168669] in-flight: 651:recv_work [15814.169593] workqueue ptlrpc_pinger: flags=0x2 [15814.170858] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15814.172097] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15814.173478] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13897s workers=4 idle: 37 31846 28897 [15814.175008] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15804s workers=2 idle: 653 [15820.057018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15820.058073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15820.059282] Modules linked in: [15820.063368] Modules linked in: [15820.063370] lustre(O) osp(O) [15820.065422] lustre(O) [15820.066717] ofd(O) [15820.068250] osp(O) [15820.069037] lod(O) [15820.069846] ofd(O) [15820.070761] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs [15820.071927] lod(O) [15820.072657] ata_generic [15820.084271] mdt(O) [15820.085194] crct10dif_pclmul [15820.086373] mdd(O) [15820.086888] crc32_pclmul [15820.088264] mgs(O) [15820.089016] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15820.089038] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15820.089044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15820.089049] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15820.090344] lquota(O) [15820.091325] [15820.091330] RIP: 0010:native_safe_halt+0x2e/0x50 [15820.096023] lfsck(O) [15820.101043] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15820.101057] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000216 [15820.105540] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [15820.108486] ORIG_RAX: ffffffffffffff13 [15820.109611] ptlrpc_gss(O) [15820.110275] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15820.111629] ptlrpc(O) [15820.112788] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15820.112795] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15820.112797] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15820.112800] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15820.112803] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15820.112805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15820.112808] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15820.112817] Call Trace: [15820.112823] [15820.112828] ? show_regs.cold.9+0x22/0x2f [15820.112841] ? watchdog_timer_fn.cold.6+0x62/0x10a [15820.112847] ? __hrtimer_run_queues+0x1c2/0x4d0 [15820.112851] ? watchdog+0x40/0x40 [15820.112855] ? hrtimer_interrupt+0x128/0x2c0 [15820.112869] ? smp_apic_timer_interrupt+0xa6/0x270 [15820.112877] ? apic_timer_interrupt+0xf/0x20 [15820.112882] [15820.112884] ? native_safe_halt+0x2e/0x50 [15820.112888] kvm_wait+0xcd/0xe0 [15820.112895] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15820.112905] do_raw_spin_lock+0x177/0x190 [15820.112910] _raw_spin_lock+0x21/0x30 [15820.112915] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15820.120686] obdclass(O) [15820.124263] process_one_work+0x2c8/0x700 [15820.127408] ksocklnd(O) [15820.129280] worker_thread+0x296/0x6e0 [15820.130633] lnet(O) [15820.134058] ? rescuer_thread+0x570/0x570 [15820.135161] libcfs(O) [15820.138429] kthread+0x1d1/0x200 [15820.141721] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15820.141767] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15820.141774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15820.141777] RIP: 0010:native_safe_halt+0x2e/0x50 [15820.141790] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15820.141794] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15820.141799] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15820.141801] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15820.141803] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15820.141805] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15820.141807] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15820.141810] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15820.141813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15820.141815] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15820.141823] Call Trace: [15820.141829] [15820.141834] ? show_regs.cold.9+0x22/0x2f [15820.141845] ? watchdog_timer_fn.cold.6+0x62/0x10a [15820.141852] ? __hrtimer_run_queues+0x1c2/0x4d0 [15820.141856] ? watchdog+0x40/0x40 [15820.141869] ? hrtimer_interrupt+0x128/0x2c0 [15820.141874] ? smp_apic_timer_interrupt+0xa6/0x270 [15820.145391] ? set_kthread_struct+0x70/0x70 [15820.148470] ? apic_timer_interrupt+0xf/0x20 [15820.152216] ret_from_fork+0x1f/0x30 [15820.154411] [15820.291899] ? native_safe_halt+0x2e/0x50 [15820.294385] kvm_wait+0xcd/0xe0 [15820.295766] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15820.299883] do_raw_spin_lock+0x177/0x190 [15820.303219] _raw_spin_lock+0x21/0x30 [15820.304693] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15820.307355] obd_mod_free+0x46/0x140 [ptlrpc] [15820.309321] mdc_commit_open+0x91/0xb0 [mdc] [15820.311315] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15820.313564] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15820.316174] after_reply+0xe2c/0x1780 [ptlrpc] [15820.318220] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15820.320550] ? __next_timer_interrupt+0x160/0x160 [15820.322442] ptlrpcd+0xa82/0xe40 [ptlrpc] [15820.324534] ? do_wait_intr+0xf0/0xf0 [15820.326243] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15820.328849] kthread+0x1d1/0x200 [15820.330231] ? set_kthread_struct+0x70/0x70 [15820.331844] ret_from_fork+0x1f/0x30 [15834.000448] rcu: INFO: rcu_sched self-detected stall on CPU [15834.003776] rcu: 2-...!: (8122145 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [15834.010297] (t=13920311 jiffies g=258701 q=5239) [15834.012308] rcu: rcu_sched kthread starved for 13917106 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [15834.015987] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15834.019573] rcu: RCU grace-period kthread stack dump: [15834.021829] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15834.025988] Call Trace: [15834.027021] __schedule+0x351/0xcb0 [15834.029434] schedule+0xc0/0x180 [15834.031038] schedule_timeout+0xb4/0x190 [15834.033028] ? do_raw_spin_unlock+0x75/0x190 [15834.035040] ? __next_timer_interrupt+0x160/0x160 [15834.036881] ? prepare_to_swait_event+0xf3/0x140 [15834.038522] rcu_gp_kthread+0x8dd/0x15b0 [15834.040512] ? force_qs_rnp+0x2c0/0x2c0 [15834.042217] kthread+0x1d1/0x200 [15834.044224] ? set_kthread_struct+0x70/0x70 [15834.047019] ret_from_fork+0x1f/0x30 [15834.048424] rcu: Stack dump where RCU GP kthread last ran: [15834.050315] Sending NMI from CPU 2 to CPUs 1: [15834.052084] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [15834.053254] NMI backtrace for cpu 2 [15834.059152] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15834.064736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15834.069876] Call Trace: [15834.071841] [15834.072974] ? dump_stack+0xbb/0x10e [15834.074494] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15834.077180] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15834.079518] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15834.081584] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15834.083191] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15834.085468] ? rcu_dump_cpu_stacks+0x11b/0x171 [15834.087240] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15834.089450] ? update_process_times+0x6d/0xc0 [15834.090865] ? tick_sched_handle.isra.18+0x27/0x90 [15834.092489] ? tick_sched_timer+0x53/0xb0 [15834.094191] ? __hrtimer_run_queues+0x1c2/0x4d0 [15834.096014] ? tick_sched_handle.isra.18+0x90/0x90 [15834.097914] ? hrtimer_interrupt+0x128/0x2c0 [15834.099800] ? smp_apic_timer_interrupt+0xa6/0x270 [15834.101920] ? apic_timer_interrupt+0xf/0x20 [15834.103816] [15834.104787] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15834.106902] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15834.109495] ? do_raw_spin_lock+0x177/0x190 [15834.113161] ? _raw_spin_lock+0x21/0x30 [15834.115409] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15834.118223] ? obd_mod_free+0x46/0x140 [ptlrpc] [15834.120987] ? mdc_commit_open+0x91/0xb0 [mdc] [15834.122609] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15834.125540] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15834.130076] ? after_reply+0xe2c/0x1780 [ptlrpc] [15834.132628] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15834.136075] ? __next_timer_interrupt+0x160/0x160 [15834.137968] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15834.140119] ? do_wait_intr+0xf0/0xf0 [15834.142742] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15834.144480] ? kthread+0x1d1/0x200 [15834.146000] ? set_kthread_struct+0x70/0x70 [15834.147471] ? ret_from_fork+0x1f/0x30 [15840.054109] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15840.056790] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15840.059021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15840.074693] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15840.082782] Modules linked in: [15840.087580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15840.087583] lustre(O) osp(O) ofd(O) [15840.088858] RIP: 0010:native_safe_halt+0x2e/0x50 [15840.088873] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15840.088878] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15840.088883] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15840.088885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15840.088887] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15840.088889] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15840.088892] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15840.088895] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15840.088897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15840.088900] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15840.088907] Call Trace: [15840.088911] [15840.088916] ? show_regs.cold.9+0x22/0x2f [15840.088925] ? watchdog_timer_fn.cold.6+0x62/0x10a [15840.088930] ? __hrtimer_run_queues+0x1c2/0x4d0 [15840.088935] ? watchdog+0x40/0x40 [15840.088939] ? hrtimer_interrupt+0x128/0x2c0 [15840.088943] ? smp_apic_timer_interrupt+0xa6/0x270 [15840.088950] ? apic_timer_interrupt+0xf/0x20 [15840.088955] [15840.088957] ? native_safe_halt+0x2e/0x50 [15840.088961] kvm_wait+0xcd/0xe0 [15840.088969] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15840.088979] do_raw_spin_lock+0x177/0x190 [15840.088983] _raw_spin_lock+0x21/0x30 [15840.088988] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15840.089543] ? do_raw_spin_unlock+0x75/0x190 [15840.089550] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15840.089742] ? do_wait_intr+0xf0/0xf0 [15840.089751] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15840.089954] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15840.090453] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15840.090674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15840.090795] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15840.090841] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15840.090928] mdc_intent_lock+0x353/0x7e0 [mdc] [15840.090957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15840.091202] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15840.102161] lod(O) [15840.103535] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15840.103600] lmv_intent_open+0x338/0x1150 [lmv] [15840.103632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15840.103753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15840.103834] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15840.103962] ll_intent_lock+0xc5/0xaa0 [lustre] [15840.104164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15840.104366] ll_lookup_it+0xe0e/0x2270 [lustre] [15840.104450] ? do_raw_spin_unlock+0x75/0x190 [15840.104458] ? _raw_spin_unlock+0x12/0x30 [15840.104466] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15840.104493] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15840.104521] ll_atomic_open+0x36a/0xd70 [lustre] [15840.104704] lookup_open+0x356/0xb40 [15840.104715] do_last+0x313/0xfc0 [15840.110520] mdt(O) [15840.117815] ? nd_jump_root+0xe5/0x160 [15840.117837] ? path_init+0x437/0x520 [15840.117842] path_openat+0xf7/0x500 [15840.117860] ? do_raw_spin_unlock+0x75/0x190 [15840.117868] do_filp_open+0x99/0x140 [15840.117874] ? getname_flags+0x6e/0x330 [15840.117879] ? __check_object_size+0xff/0x256 [15840.117886] ? do_raw_spin_unlock+0x75/0x190 [15840.117889] ? _raw_spin_unlock+0x12/0x30 [15840.117896] do_sys_openat2+0x2b4/0x410 [15840.117900] do_sys_open+0x73/0xa0 [15840.117904] __x64_sys_openat+0x24/0x30 [15840.117908] do_syscall_64+0xc1/0x440 [15840.117914] entry_SYSCALL_64_after_hwframe+0x49/0xae [15840.117920] RIP: 0033:0x7fa749552332 [15840.117974] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15840.117978] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15840.117983] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15840.117986] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15840.117988] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15840.117990] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15840.117992] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15840.360255] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15840.378326] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15840.383806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15840.387996] RIP: 0010:native_safe_halt+0x2e/0x50 [15840.389988] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15840.406072] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15840.409682] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15840.413128] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15840.416860] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15840.420647] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15840.426938] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15840.431125] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15840.434828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15840.437351] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15840.440663] Call Trace: [15840.441751] [15840.442670] ? show_regs.cold.9+0x22/0x2f [15840.444381] ? watchdog_timer_fn.cold.6+0x62/0x10a [15840.446685] ? __hrtimer_run_queues+0x1c2/0x4d0 [15840.448731] ? watchdog+0x40/0x40 [15840.450113] ? hrtimer_interrupt+0x128/0x2c0 [15840.452064] ? smp_apic_timer_interrupt+0xa6/0x270 [15840.453931] ? apic_timer_interrupt+0xf/0x20 [15840.456045] [15840.456789] ? native_safe_halt+0x2e/0x50 [15840.458129] kvm_wait+0xcd/0xe0 [15840.459999] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15840.462365] do_raw_spin_lock+0x177/0x190 [15840.464670] _raw_spin_lock+0x21/0x30 [15840.465740] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15840.468214] ? do_raw_spin_unlock+0x75/0x190 [15840.471128] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15840.473871] ? do_wait_intr+0xf0/0xf0 [15840.474814] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15840.477019] mdc_close+0x53b/0xd70 [mdc] [15840.479100] lmv_close+0x18d/0x490 [lmv] [15840.480863] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15840.483406] ll_md_real_close+0xa5/0x2a0 [lustre] [15840.485817] ll_md_close+0x274/0x800 [lustre] [15840.487862] ll_file_release+0x128/0xc50 [lustre] [15840.490323] __fput+0x10d/0x440 [15840.491535] ____fput+0x12/0x20 [15840.492817] task_work_run+0xc6/0x110 [15840.495444] exit_to_usermode_loop+0x1dd/0x1f0 [15840.498109] do_syscall_64+0x3d6/0x440 [15840.500106] entry_SYSCALL_64_after_hwframe+0x49/0xae [15840.502997] RIP: 0033:0x7fa749552cd8 [15840.504869] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15840.514638] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15840.518097] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15840.521285] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15840.525040] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15840.528125] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15840.534991] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15844.324062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13927s! [15844.327905] Showing busy workqueues and worker pools: [15844.331985] workqueue events: flags=0x0 [15844.333719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15844.336851] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15844.345505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15844.348129] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15844.355351] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.358367] pending: kfree_rcu_monitor [15844.360695] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15844.363109] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15844.368963] workqueue events_long: flags=0x0 [15844.373764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.377520] pending: lu_objects_flush [obdclass] [15844.380575] workqueue events_unbound: flags=0x2 [15844.381981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15844.384778] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15844.388784] workqueue events_power_efficient: flags=0x80 [15844.390713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.393175] pending: check_lifetime [15844.394246] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.396770] pending: neigh_periodic_work [15844.400472] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.406138] pending: neigh_periodic_work [15844.410013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.413029] pending: do_cache_clean [15844.416990] workqueue mm_percpu_wq: flags=0x8 [15844.419670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.424991] pending: vmstat_update [15844.427499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.431111] pending: vmstat_update [15844.434839] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15844.438121] pending: vmstat_update [15844.440682] workqueue writeback: flags=0x4a [15844.442181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15844.447120] pending: wb_workfn, wb_workfn [15844.448914] workqueue xprtiod: flags=0xa [15844.453501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15844.458447] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15844.463560] workqueue ipv6_addrconf: flags=0x40008 [15844.465524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15844.472141] pending: addrconf_verify_work [15844.476128] workqueue knbd0-recv: flags=0x1a [15844.478808] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15844.482214] in-flight: 651:recv_work [15844.490089] workqueue ptlrpc_pinger: flags=0x2 [15844.492011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15844.497368] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15844.500463] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13927s workers=4 idle: 37 31846 28897 [15844.503676] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15834s workers=2 idle: 653 [15860.057432] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15860.058031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15860.067291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [15860.070650] Modules linked in: lustre(O) [15860.083976] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15860.084021] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15860.084026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15860.084031] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15860.085652] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15860.085822] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15860.085828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15860.085831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15860.085843] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15860.085847] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15860.085851] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15860.085853] RDX: 0000000000002dc9 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15860.085855] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15860.085857] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15860.085860] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15860.085862] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15860.085865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15860.085867] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15860.085881] Call Trace: [15860.085886] [15860.085890] ? show_regs.cold.9+0x22/0x2f [15860.085898] ? watchdog_timer_fn.cold.6+0x62/0x10a [15860.085903] ? __hrtimer_run_queues+0x1c2/0x4d0 [15860.085907] ? watchdog+0x40/0x40 [15860.085911] ? hrtimer_interrupt+0x128/0x2c0 [15860.085915] ? smp_apic_timer_interrupt+0xa6/0x270 [15860.085923] ? apic_timer_interrupt+0xf/0x20 [15860.085928] [15860.085930] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15860.085935] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15860.085939] do_raw_spin_lock+0x177/0x190 [15860.085943] _raw_spin_lock+0x21/0x30 [15860.085948] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15860.108359] [15860.108368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15860.108386] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15860.108482] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15860.108487] RAX: 0000000000005384 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15860.108489] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15860.108491] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15860.108493] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15860.108496] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15860.108499] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15860.108502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15860.108504] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15860.108512] Call Trace: [15860.108517] [15860.108522] ? show_regs.cold.9+0x22/0x2f [15860.108530] ? watchdog_timer_fn.cold.6+0x62/0x10a [15860.108536] ? __hrtimer_run_queues+0x1c2/0x4d0 [15860.108541] ? watchdog+0x40/0x40 [15860.108545] ? hrtimer_interrupt+0x128/0x2c0 [15860.108550] ? smp_apic_timer_interrupt+0xa6/0x270 [15860.108559] ? apic_timer_interrupt+0xf/0x20 [15860.108564] [15860.108567] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15860.108572] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15860.108579] do_raw_spin_lock+0x177/0x190 [15860.108584] _raw_spin_lock+0x21/0x30 [15860.108589] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15860.118017] obd_mod_free+0x46/0x140 [ptlrpc] [15860.118478] mdc_commit_open+0x91/0xb0 [mdc] [15860.118528] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15860.125467] process_one_work+0x2c8/0x700 [15860.125494] worker_thread+0x296/0x6e0 [15860.125503] ? rescuer_thread+0x570/0x570 [15860.125509] kthread+0x1d1/0x200 [15860.125514] ? set_kthread_struct+0x70/0x70 [15860.125519] ret_from_fork+0x1f/0x30 [15860.337060] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15860.339696] after_reply+0xe2c/0x1780 [ptlrpc] [15860.342511] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15860.344802] ? __next_timer_interrupt+0x160/0x160 [15860.346977] ptlrpcd+0xa82/0xe40 [ptlrpc] [15860.349466] ? do_wait_intr+0xf0/0xf0 [15860.351136] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15860.354122] kthread+0x1d1/0x200 [15860.355700] ? set_kthread_struct+0x70/0x70 [15860.357950] ret_from_fork+0x1f/0x30 [15868.064239] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [15868.072296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15868.109157] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15868.115057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15868.119786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15868.123432] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15868.133933] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15868.138433] RAX: 000000000000778a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15868.143919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15868.150048] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15868.154355] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15868.158438] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15868.163149] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15868.170566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15868.174370] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15868.181368] Call Trace: [15868.183296] [15868.186442] ? show_regs.cold.9+0x22/0x2f [15868.189405] ? watchdog_timer_fn.cold.6+0x62/0x10a [15868.192090] ? __hrtimer_run_queues+0x1c2/0x4d0 [15868.194130] ? watchdog+0x40/0x40 [15868.195431] ? hrtimer_interrupt+0x128/0x2c0 [15868.197974] ? smp_apic_timer_interrupt+0xa6/0x270 [15868.200141] ? apic_timer_interrupt+0xf/0x20 [15868.202300] [15868.203264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15868.205647] do_raw_spin_lock+0x177/0x190 [15868.208758] _raw_spin_lock+0x21/0x30 [15868.210083] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15868.211958] ? do_raw_spin_unlock+0x75/0x190 [15868.215190] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15868.219121] ? do_wait_intr+0xf0/0xf0 [15868.221419] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15868.224978] mdc_close+0x53b/0xd70 [mdc] [15868.228421] lmv_close+0x18d/0x490 [lmv] [15868.231050] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15868.234616] ll_md_real_close+0xa5/0x2a0 [lustre] [15868.238313] ll_md_close+0x274/0x800 [lustre] [15868.241345] ll_file_release+0x128/0xc50 [lustre] [15868.244415] __fput+0x10d/0x440 [15868.246391] ____fput+0x12/0x20 [15868.248053] task_work_run+0xc6/0x110 [15868.249909] exit_to_usermode_loop+0x1dd/0x1f0 [15868.255274] do_syscall_64+0x3d6/0x440 [15868.261320] entry_SYSCALL_64_after_hwframe+0x49/0xae [15868.263182] RIP: 0033:0x7fa749552cd8 [15868.264908] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15868.274411] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15868.278611] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15868.282414] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15868.286335] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15868.289336] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15868.292923] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15872.054031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15872.059915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15872.082792] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15872.088121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15872.092652] RIP: 0010:native_safe_halt+0x2e/0x50 [15872.095111] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15872.103470] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15872.106077] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15872.109159] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15872.111896] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15872.114745] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15872.117846] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15872.121417] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15872.125453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15872.127944] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15872.131184] Call Trace: [15872.132222] [15872.133139] ? show_regs.cold.9+0x22/0x2f [15872.135184] ? watchdog_timer_fn.cold.6+0x62/0x10a [15872.136597] ? __hrtimer_run_queues+0x1c2/0x4d0 [15872.138012] ? watchdog+0x40/0x40 [15872.138774] ? hrtimer_interrupt+0x128/0x2c0 [15872.139856] ? smp_apic_timer_interrupt+0xa6/0x270 [15872.142234] ? apic_timer_interrupt+0xf/0x20 [15872.143963] [15872.146159] ? native_safe_halt+0x2e/0x50 [15872.148175] kvm_wait+0xcd/0xe0 [15872.149621] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15872.152146] do_raw_spin_lock+0x177/0x190 [15872.153700] _raw_spin_lock+0x21/0x30 [15872.154985] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15872.157378] ? do_raw_spin_unlock+0x75/0x190 [15872.159384] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15872.162155] ? do_wait_intr+0xf0/0xf0 [15872.163628] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15872.166130] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15872.168536] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15872.170897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15872.173367] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15872.175378] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15872.177072] mdc_intent_lock+0x353/0x7e0 [mdc] [15872.178935] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15872.181448] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15872.184180] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15872.186368] lmv_intent_open+0x338/0x1150 [lmv] [15872.188291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15872.190569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15872.192726] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15872.194600] ll_intent_lock+0xc5/0xaa0 [lustre] [15872.196635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15872.199024] ll_lookup_it+0xe0e/0x2270 [lustre] [15872.201006] ? do_raw_spin_unlock+0x75/0x190 [15872.202976] ? _raw_spin_unlock+0x12/0x30 [15872.204624] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15872.207147] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15872.209207] ll_atomic_open+0x36a/0xd70 [lustre] [15872.211251] lookup_open+0x356/0xb40 [15872.212986] do_last+0x313/0xfc0 [15872.214407] ? nd_jump_root+0xe5/0x160 [15872.215560] ? path_init+0x437/0x520 [15872.217039] path_openat+0xf7/0x500 [15872.218531] ? do_raw_spin_unlock+0x75/0x190 [15872.220314] do_filp_open+0x99/0x140 [15872.221495] ? getname_flags+0x6e/0x330 [15872.223126] ? __check_object_size+0xff/0x256 [15872.224860] ? do_raw_spin_unlock+0x75/0x190 [15872.226658] ? _raw_spin_unlock+0x12/0x30 [15872.228333] do_sys_openat2+0x2b4/0x410 [15872.230397] do_sys_open+0x73/0xa0 [15872.231729] __x64_sys_openat+0x24/0x30 [15872.233568] do_syscall_64+0xc1/0x440 [15872.234748] entry_SYSCALL_64_after_hwframe+0x49/0xae [15872.235942] RIP: 0033:0x7fa749552332 [15872.237166] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15872.243641] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15872.246469] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15872.249618] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15872.252482] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15872.254990] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15872.258263] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15874.529054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13957s! [15874.533058] Showing busy workqueues and worker pools: [15874.535128] workqueue events: flags=0x0 [15874.537940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15874.546668] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15874.567871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15874.575480] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15874.580408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.584218] pending: kfree_rcu_monitor [15874.585987] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15874.592275] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15874.603438] workqueue events_long: flags=0x0 [15874.608446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.618458] pending: lu_objects_flush [obdclass] [15874.623668] workqueue events_unbound: flags=0x2 [15874.626274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15874.629430] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15874.638025] workqueue events_power_efficient: flags=0x80 [15874.639628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.645304] pending: check_lifetime [15874.648345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.651867] pending: neigh_periodic_work [15874.655410] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.665195] pending: neigh_periodic_work [15874.671916] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.676463] pending: do_cache_clean [15874.680815] workqueue mm_percpu_wq: flags=0x8 [15874.682860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.686449] pending: vmstat_update [15874.689702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.693252] pending: vmstat_update [15874.697964] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15874.702382] pending: vmstat_update [15874.704596] workqueue writeback: flags=0x4a [15874.706378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15874.713257] pending: wb_workfn, wb_workfn [15874.722304] workqueue xprtiod: flags=0xa [15874.723558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15874.730644] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15874.737363] workqueue ipv6_addrconf: flags=0x40008 [15874.742436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15874.747161] pending: addrconf_verify_work [15874.751573] workqueue knbd0-recv: flags=0x1a [15874.754514] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15874.760165] in-flight: 651:recv_work [15874.764221] workqueue ptlrpc_pinger: flags=0x2 [15874.765858] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15874.773144] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15874.784197] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13957s workers=4 idle: 37 31846 28897 [15874.792463] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15864s workers=2 idle: 653 [15888.057037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15888.058237] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15888.059659] Modules linked in: lustre(O) osp(O) [15888.064863] Modules linked in: [15888.064865] ofd(O) lod(O) [15888.067590] lustre(O) [15888.069863] mdt(O) [15888.071018] osp(O) [15888.075019] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15888.075272] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15888.075281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15888.075287] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15888.076898] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15888.076962] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15888.076969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15888.076972] RIP: 0010:native_safe_halt+0x2e/0x50 [15888.076984] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15888.076988] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15888.076992] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15888.076994] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15888.076996] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15888.076998] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15888.077000] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15888.077003] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15888.077006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15888.077009] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15888.077018] Call Trace: [15888.077024] [15888.077032] ? show_regs.cold.9+0x22/0x2f [15888.077042] ? watchdog_timer_fn.cold.6+0x62/0x10a [15888.077051] ? __hrtimer_run_queues+0x1c2/0x4d0 [15888.077056] ? watchdog+0x40/0x40 [15888.077060] ? hrtimer_interrupt+0x128/0x2c0 [15888.077065] ? smp_apic_timer_interrupt+0xa6/0x270 [15888.077071] ? apic_timer_interrupt+0xf/0x20 [15888.077076] [15888.077079] ? native_safe_halt+0x2e/0x50 [15888.077084] kvm_wait+0xcd/0xe0 [15888.077091] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15888.077098] do_raw_spin_lock+0x177/0x190 [15888.077102] _raw_spin_lock+0x21/0x30 [15888.077106] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15888.078838] [15888.078845] RIP: 0010:native_safe_halt+0x2e/0x50 [15888.078856] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15888.078861] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15888.078865] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15888.078868] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15888.078870] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15888.078872] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15888.078875] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15888.078878] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15888.078881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15888.078883] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15888.078890] Call Trace: [15888.078896] [15888.078901] ? show_regs.cold.9+0x22/0x2f [15888.078910] ? watchdog_timer_fn.cold.6+0x62/0x10a [15888.078918] ? __hrtimer_run_queues+0x1c2/0x4d0 [15888.078922] ? watchdog+0x40/0x40 [15888.078927] ? hrtimer_interrupt+0x128/0x2c0 [15888.078931] ? smp_apic_timer_interrupt+0xa6/0x270 [15888.078938] ? apic_timer_interrupt+0xf/0x20 [15888.078943] [15888.078945] ? native_safe_halt+0x2e/0x50 [15888.078950] kvm_wait+0xcd/0xe0 [15888.078957] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15888.078965] do_raw_spin_lock+0x177/0x190 [15888.078970] _raw_spin_lock+0x21/0x30 [15888.078974] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15888.107620] obd_mod_free+0x46/0x140 [ptlrpc] [15888.119267] process_one_work+0x2c8/0x700 [15888.119294] worker_thread+0x296/0x6e0 [15888.119299] ? rescuer_thread+0x570/0x570 [15888.119302] kthread+0x1d1/0x200 [15888.119306] ? set_kthread_struct+0x70/0x70 [15888.119310] ret_from_fork+0x1f/0x30 [15888.370212] mdc_commit_open+0x91/0xb0 [mdc] [15888.371533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15888.372913] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15888.374337] after_reply+0xe2c/0x1780 [ptlrpc] [15888.375695] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15888.377353] ? __next_timer_interrupt+0x160/0x160 [15888.378480] ptlrpcd+0xa82/0xe40 [ptlrpc] [15888.379613] ? do_wait_intr+0xf0/0xf0 [15888.380682] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15888.382017] kthread+0x1d1/0x200 [15888.382793] ? set_kthread_struct+0x70/0x70 [15888.383712] ret_from_fork+0x1f/0x30 [15896.065294] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15896.075997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15896.098520] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15896.106063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15896.119397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15896.125017] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15896.136655] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15896.145393] RAX: 0000000000007e07 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15896.154861] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15896.159006] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15896.163759] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15896.172869] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15896.183376] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15896.192825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15896.196945] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15896.205019] Call Trace: [15896.205997] [15896.209760] ? show_regs.cold.9+0x22/0x2f [15896.211235] ? watchdog_timer_fn.cold.6+0x62/0x10a [15896.216682] ? __hrtimer_run_queues+0x1c2/0x4d0 [15896.222138] ? watchdog+0x40/0x40 [15896.226961] ? hrtimer_interrupt+0x128/0x2c0 [15896.230016] ? smp_apic_timer_interrupt+0xa6/0x270 [15896.232555] ? apic_timer_interrupt+0xf/0x20 [15896.234100] [15896.241586] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15896.244989] do_raw_spin_lock+0x177/0x190 [15896.247858] _raw_spin_lock+0x21/0x30 [15896.251775] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15896.259684] ? do_raw_spin_unlock+0x75/0x190 [15896.261839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15896.265402] ? do_wait_intr+0xf0/0xf0 [15896.269365] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15896.271926] mdc_close+0x53b/0xd70 [mdc] [15896.273494] lmv_close+0x18d/0x490 [lmv] [15896.276711] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15896.280018] ll_md_real_close+0xa5/0x2a0 [lustre] [15896.282310] ll_md_close+0x274/0x800 [lustre] [15896.286477] ll_file_release+0x128/0xc50 [lustre] [15896.289851] __fput+0x10d/0x440 [15896.292780] ____fput+0x12/0x20 [15896.294588] task_work_run+0xc6/0x110 [15896.297844] exit_to_usermode_loop+0x1dd/0x1f0 [15896.299309] do_syscall_64+0x3d6/0x440 [15896.300447] entry_SYSCALL_64_after_hwframe+0x49/0xae [15896.304851] RIP: 0033:0x7fa749552cd8 [15896.306123] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15896.333822] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15896.344736] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15896.350868] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15896.354198] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15896.358377] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15896.361620] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15900.054092] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [15900.057576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15900.081347] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15900.087169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15900.091724] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15900.095042] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15900.106407] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15900.110886] RAX: 0000000000005a84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15900.113677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15900.117431] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15900.120993] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15900.124688] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15900.127405] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15900.131750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15900.135021] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15900.139717] Call Trace: [15900.141059] [15900.142269] ? show_regs.cold.9+0x22/0x2f [15900.144407] ? watchdog_timer_fn.cold.6+0x62/0x10a [15900.147354] ? __hrtimer_run_queues+0x1c2/0x4d0 [15900.150367] ? watchdog+0x40/0x40 [15900.152342] ? hrtimer_interrupt+0x128/0x2c0 [15900.155042] ? smp_apic_timer_interrupt+0xa6/0x270 [15900.157529] ? apic_timer_interrupt+0xf/0x20 [15900.160066] [15900.160907] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15900.163534] do_raw_spin_lock+0x177/0x190 [15900.165695] _raw_spin_lock+0x21/0x30 [15900.166917] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15900.169665] ? do_raw_spin_unlock+0x75/0x190 [15900.171940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15900.174345] ? do_wait_intr+0xf0/0xf0 [15900.176258] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15900.179015] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15900.182068] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15900.184820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15900.187855] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15900.190971] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15900.193599] mdc_intent_lock+0x353/0x7e0 [mdc] [15900.195930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15900.199060] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15900.201732] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15900.203573] lmv_intent_open+0x338/0x1150 [lmv] [15900.205069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15900.207424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15900.209930] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15900.212144] ll_intent_lock+0xc5/0xaa0 [lustre] [15900.214644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15900.217674] ll_lookup_it+0xe0e/0x2270 [lustre] [15900.219554] ? do_raw_spin_unlock+0x75/0x190 [15900.220986] ? _raw_spin_unlock+0x12/0x30 [15900.222420] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15900.224618] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15900.227083] ll_atomic_open+0x36a/0xd70 [lustre] [15900.230255] lookup_open+0x356/0xb40 [15900.231444] do_last+0x313/0xfc0 [15900.232728] ? nd_jump_root+0xe5/0x160 [15900.234654] ? path_init+0x437/0x520 [15900.236672] path_openat+0xf7/0x500 [15900.238379] ? do_raw_spin_unlock+0x75/0x190 [15900.240133] do_filp_open+0x99/0x140 [15900.241727] ? getname_flags+0x6e/0x330 [15900.243513] ? __check_object_size+0xff/0x256 [15900.244907] ? do_raw_spin_unlock+0x75/0x190 [15900.246631] ? _raw_spin_unlock+0x12/0x30 [15900.248750] do_sys_openat2+0x2b4/0x410 [15900.250857] do_sys_open+0x73/0xa0 [15900.251930] __x64_sys_openat+0x24/0x30 [15900.253943] do_syscall_64+0xc1/0x440 [15900.255705] entry_SYSCALL_64_after_hwframe+0x49/0xae [15900.258267] RIP: 0033:0x7fa749552332 [15900.260023] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15900.268521] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15900.270675] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15900.273450] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15900.276320] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15900.279625] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15900.282840] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15905.248141] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13988s! [15905.254691] Showing busy workqueues and worker pools: [15905.259544] workqueue events: flags=0x0 [15905.262431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15905.266829] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15905.273163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15905.276338] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15905.280551] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.286948] pending: kfree_rcu_monitor [15905.289814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15905.292816] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15905.295749] workqueue events_long: flags=0x0 [15905.297906] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.301019] pending: lu_objects_flush [obdclass] [15905.303557] workqueue events_unbound: flags=0x2 [15905.306209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15905.309595] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15905.316285] workqueue events_power_efficient: flags=0x80 [15905.319536] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.325911] pending: check_lifetime [15905.328186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.331220] pending: neigh_periodic_work [15905.333317] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.337292] pending: neigh_periodic_work [15905.340655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.347666] pending: do_cache_clean [15905.351528] workqueue mm_percpu_wq: flags=0x8 [15905.356477] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.361479] pending: vmstat_update [15905.362379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.365748] pending: vmstat_update [15905.367501] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15905.371869] pending: vmstat_update [15905.374152] workqueue writeback: flags=0x4a [15905.376379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15905.379670] pending: wb_workfn, wb_workfn [15905.382213] workqueue xprtiod: flags=0xa [15905.384289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15905.387682] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15905.392334] workqueue ipv6_addrconf: flags=0x40008 [15905.395615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15905.401329] pending: addrconf_verify_work [15905.405214] workqueue knbd0-recv: flags=0x1a [15905.408225] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15905.411878] in-flight: 651:recv_work [15905.414873] workqueue ptlrpc_pinger: flags=0x2 [15905.417869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15905.422608] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15905.427584] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13988s workers=4 idle: 37 31846 28897 [15905.433181] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15895s workers=2 idle: 653 [15916.057021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [15916.058027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [15916.061511] Modules linked in: [15916.068298] Modules linked in: [15916.068301] lustre(O) osp(O) ofd(O) [15916.070747] lustre(O) osp(O) ofd(O) lod(O) mdt(O) [15916.072264] lod(O) [15916.074054] mdd(O) [15916.076799] mdt(O) [15916.077823] mgs(O) [15916.079365] mdd(O) [15916.080335] lquota(O) [15916.081514] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [15916.082977] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [15916.084227] intel_rapl_msr [15916.094585] ptlrpc_gss(O) [15916.098037] intel_rapl_common [15916.099332] ptlrpc(O) [15916.100737] sb_edac [15916.103147] obdclass(O) [15916.104169] rapl [15916.106315] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [15916.107591] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15916.107618] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15916.107631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15916.107637] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15916.107956] RIP: 0010:native_safe_halt+0x2e/0x50 [15916.107969] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15916.107974] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15916.107978] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15916.107981] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15916.107983] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15916.107985] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15916.107988] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15916.107991] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15916.107994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15916.107996] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15916.108004] Call Trace: [15916.108214] [15916.108272] ? show_regs.cold.9+0x22/0x2f [15916.109104] sha512_ssse3 [15916.119117] ? watchdog_timer_fn.cold.6+0x62/0x10a [15916.132850] sha512_generic [15916.140813] ? __hrtimer_run_queues+0x1c2/0x4d0 [15916.144021] [15916.146040] ? watchdog+0x40/0x40 [15916.147390] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15916.154148] ? hrtimer_interrupt+0x128/0x2c0 [15916.157460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15916.159691] ? smp_apic_timer_interrupt+0xa6/0x270 [15916.162692] RIP: 0010:native_safe_halt+0x2e/0x50 [15916.164604] ? apic_timer_interrupt+0xf/0x20 [15916.167054] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15916.169156] [15916.171513] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 [15916.173087] ? native_safe_halt+0x2e/0x50 [15916.174971] ORIG_RAX: ffffffffffffff13 [15916.175740] kvm_wait+0xcd/0xe0 [15916.176265] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15916.177392] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15916.177408] do_raw_spin_lock+0x177/0x190 [15916.177412] _raw_spin_lock+0x21/0x30 [15916.177419] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15916.178101] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15916.179408] process_one_work+0x2c8/0x700 [15916.179423] worker_thread+0x296/0x6e0 [15916.179428] ? rescuer_thread+0x570/0x570 [15916.179432] kthread+0x1d1/0x200 [15916.179437] ? set_kthread_struct+0x70/0x70 [15916.180132] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15916.181539] ret_from_fork+0x1f/0x30 [15916.181983] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15916.254158] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15916.257703] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15916.264267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15916.267447] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15916.272522] Call Trace: [15916.274249] [15916.275223] ? show_regs.cold.9+0x22/0x2f [15916.277821] ? watchdog_timer_fn.cold.6+0x62/0x10a [15916.280154] ? __hrtimer_run_queues+0x1c2/0x4d0 [15916.283206] ? watchdog+0x40/0x40 [15916.287636] ? hrtimer_interrupt+0x128/0x2c0 [15916.290442] ? smp_apic_timer_interrupt+0xa6/0x270 [15916.294777] ? apic_timer_interrupt+0xf/0x20 [15916.297224] [15916.298440] ? native_safe_halt+0x2e/0x50 [15916.300438] kvm_wait+0xcd/0xe0 [15916.301788] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15916.307612] do_raw_spin_lock+0x177/0x190 [15916.310668] _raw_spin_lock+0x21/0x30 [15916.312325] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15916.315201] obd_mod_free+0x46/0x140 [ptlrpc] [15916.323839] mdc_commit_open+0x91/0xb0 [mdc] [15916.327688] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15916.331135] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15916.336811] after_reply+0xe2c/0x1780 [ptlrpc] [15916.341960] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15916.344345] ? __next_timer_interrupt+0x160/0x160 [15916.346454] ptlrpcd+0xa82/0xe40 [ptlrpc] [15916.348464] ? do_wait_intr+0xf0/0xf0 [15916.350046] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15916.352259] kthread+0x1d1/0x200 [15916.353621] ? set_kthread_struct+0x70/0x70 [15916.355248] ret_from_fork+0x1f/0x30 [15924.059009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15924.064104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15924.108111] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15924.120555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15924.130478] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15924.138986] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15924.156550] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15924.165158] RAX: 0000000000003e34 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15924.173404] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15924.180866] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15924.188224] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15924.195540] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15924.203471] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15924.217562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15924.220569] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15924.228263] Call Trace: [15924.229363] [15924.232522] ? show_regs.cold.9+0x22/0x2f [15924.235327] ? watchdog_timer_fn.cold.6+0x62/0x10a [15924.239379] ? __hrtimer_run_queues+0x1c2/0x4d0 [15924.244105] ? watchdog+0x40/0x40 [15924.245499] ? hrtimer_interrupt+0x128/0x2c0 [15924.250930] ? smp_apic_timer_interrupt+0xa6/0x270 [15924.253092] ? apic_timer_interrupt+0xf/0x20 [15924.256847] [15924.258000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15924.262310] do_raw_spin_lock+0x177/0x190 [15924.264412] _raw_spin_lock+0x21/0x30 [15924.269052] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15924.274133] ? do_raw_spin_unlock+0x75/0x190 [15924.278943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15924.282961] ? do_wait_intr+0xf0/0xf0 [15924.285126] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15924.290275] mdc_close+0x53b/0xd70 [mdc] [15924.295597] lmv_close+0x18d/0x490 [lmv] [15924.300021] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15924.305947] ll_md_real_close+0xa5/0x2a0 [lustre] [15924.310914] ll_md_close+0x274/0x800 [lustre] [15924.316199] ll_file_release+0x128/0xc50 [lustre] [15924.319965] __fput+0x10d/0x440 [15924.322196] ____fput+0x12/0x20 [15924.323715] task_work_run+0xc6/0x110 [15924.327245] exit_to_usermode_loop+0x1dd/0x1f0 [15924.334017] do_syscall_64+0x3d6/0x440 [15924.339008] entry_SYSCALL_64_after_hwframe+0x49/0xae [15924.345970] RIP: 0033:0x7fa749552cd8 [15924.351414] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15924.365366] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15924.373774] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15924.382754] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15924.387232] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15924.392874] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15924.399282] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15932.054062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15932.058912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15932.083816] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15932.088972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15932.092680] RIP: 0010:native_safe_halt+0x2e/0x50 [15932.094750] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15932.103121] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15932.106327] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15932.109422] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [15932.112418] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15932.115776] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15932.118831] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15932.122050] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15932.125737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15932.128157] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15932.131179] Call Trace: [15932.132323] [15932.133153] ? show_regs.cold.9+0x22/0x2f [15932.134845] ? watchdog_timer_fn.cold.6+0x62/0x10a [15932.136921] ? __hrtimer_run_queues+0x1c2/0x4d0 [15932.139128] ? watchdog+0x40/0x40 [15932.140531] ? hrtimer_interrupt+0x128/0x2c0 [15932.142371] ? smp_apic_timer_interrupt+0xa6/0x270 [15932.144434] ? apic_timer_interrupt+0xf/0x20 [15932.146211] [15932.147141] ? native_safe_halt+0x2e/0x50 [15932.148820] kvm_wait+0xcd/0xe0 [15932.150228] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15932.152432] do_raw_spin_lock+0x177/0x190 [15932.154087] _raw_spin_lock+0x21/0x30 [15932.155562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15932.158105] ? do_raw_spin_unlock+0x75/0x190 [15932.159857] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15932.162108] ? do_wait_intr+0xf0/0xf0 [15932.163591] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15932.166152] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15932.168504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15932.171179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15932.173483] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15932.175844] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15932.177888] mdc_intent_lock+0x353/0x7e0 [mdc] [15932.179738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15932.182085] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15932.184791] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15932.187726] lmv_intent_open+0x338/0x1150 [lmv] [15932.189973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15932.192380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15932.194798] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15932.196750] ll_intent_lock+0xc5/0xaa0 [lustre] [15932.198760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15932.201334] ll_lookup_it+0xe0e/0x2270 [lustre] [15932.203406] ? do_raw_spin_unlock+0x75/0x190 [15932.205354] ? _raw_spin_unlock+0x12/0x30 [15932.207069] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15932.209364] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15932.211571] ll_atomic_open+0x36a/0xd70 [lustre] [15932.213689] lookup_open+0x356/0xb40 [15932.215131] do_last+0x313/0xfc0 [15932.216783] ? nd_jump_root+0xe5/0x160 [15932.218393] ? path_init+0x437/0x520 [15932.219878] path_openat+0xf7/0x500 [15932.221460] ? do_raw_spin_unlock+0x75/0x190 [15932.223176] do_filp_open+0x99/0x140 [15932.224710] ? getname_flags+0x6e/0x330 [15932.226413] ? __check_object_size+0xff/0x256 [15932.228227] ? do_raw_spin_unlock+0x75/0x190 [15932.230054] ? _raw_spin_unlock+0x12/0x30 [15932.231685] do_sys_openat2+0x2b4/0x410 [15932.233339] do_sys_open+0x73/0xa0 [15932.234797] __x64_sys_openat+0x24/0x30 [15932.236499] do_syscall_64+0xc1/0x440 [15932.238108] entry_SYSCALL_64_after_hwframe+0x49/0xae [15932.240508] RIP: 0033:0x7fa749552332 [15932.242089] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15932.249816] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15932.252855] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15932.255857] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15932.258941] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15932.262100] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15932.265349] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15935.457055] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14018s! [15935.476232] Showing busy workqueues and worker pools: [15935.482952] workqueue events: flags=0x0 [15935.488146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15935.493999] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15935.500809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15935.504202] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15935.509920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.513171] pending: kfree_rcu_monitor [15935.515348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15935.519180] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15935.523766] workqueue events_long: flags=0x0 [15935.526131] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.529994] pending: lu_objects_flush [obdclass] [15935.532993] workqueue events_unbound: flags=0x2 [15935.535875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15935.539743] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15935.544705] workqueue events_power_efficient: flags=0x80 [15935.548015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.552091] pending: check_lifetime [15935.553951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.557856] pending: neigh_periodic_work [15935.560932] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.564001] pending: neigh_periodic_work [15935.567493] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.571526] pending: do_cache_clean [15935.574124] workqueue mm_percpu_wq: flags=0x8 [15935.576717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.580235] pending: vmstat_update [15935.582238] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.586197] pending: vmstat_update [15935.589112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15935.593041] pending: vmstat_update [15935.595082] workqueue writeback: flags=0x4a [15935.597805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15935.601949] pending: wb_workfn, wb_workfn [15935.604941] workqueue xprtiod: flags=0xa [15935.606868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15935.610822] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15935.614610] workqueue ipv6_addrconf: flags=0x40008 [15935.617695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15935.621526] pending: addrconf_verify_work [15935.624021] workqueue knbd0-recv: flags=0x1a [15935.625976] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15935.629664] in-flight: 651:recv_work [15935.631651] workqueue ptlrpc_pinger: flags=0x2 [15935.633703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15935.636859] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15935.639805] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14018s workers=4 idle: 37 31846 28897 [15935.643962] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15925s workers=2 idle: 653 [15944.058048] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15944.058048] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15944.058065] Modules linked in: lustre(O) [15944.065806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15944.065885] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15944.065893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15944.065899] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15944.069265] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15944.069356] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15944.069367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15944.069369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15944.069391] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15944.069395] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15944.069400] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15944.069403] RDX: 0000000000001337 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15944.069405] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15944.069407] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15944.069410] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15944.069418] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15944.069424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15944.069426] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15944.069434] Call Trace: [15944.069445] [15944.069451] ? show_regs.cold.9+0x22/0x2f [15944.069460] ? watchdog_timer_fn.cold.6+0x62/0x10a [15944.069471] ? __hrtimer_run_queues+0x1c2/0x4d0 [15944.069478] ? watchdog+0x40/0x40 [15944.069482] ? hrtimer_interrupt+0x128/0x2c0 [15944.069502] ? smp_apic_timer_interrupt+0xa6/0x270 [15944.069514] ? apic_timer_interrupt+0xf/0x20 [15944.069518] [15944.069521] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15944.069529] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15944.069534] do_raw_spin_lock+0x177/0x190 [15944.069539] _raw_spin_lock+0x21/0x30 [15944.069545] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15944.071157] [15944.071161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15944.071173] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15944.071177] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15944.071183] RAX: 0000000000007f89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15944.096070] obd_mod_free+0x46/0x140 [ptlrpc] [15944.096436] mdc_commit_open+0x91/0xb0 [mdc] [15944.096481] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15944.096840] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15944.105079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15944.109108] after_reply+0xe2c/0x1780 [ptlrpc] [15944.109728] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15944.110041] ? __next_timer_interrupt+0x160/0x160 [15944.110049] ptlrpcd+0xa82/0xe40 [ptlrpc] [15944.110203] ? do_wait_intr+0xf0/0xf0 [15944.110211] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15944.115975] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15944.115985] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15944.115988] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15944.115992] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15944.115996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15944.133111] kthread+0x1d1/0x200 [15944.133131] ? set_kthread_struct+0x70/0x70 [15944.133136] ret_from_fork+0x1f/0x30 [15944.259201] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15944.262522] Call Trace: [15944.263523] [15944.264224] ? show_regs.cold.9+0x22/0x2f [15944.265861] ? watchdog_timer_fn.cold.6+0x62/0x10a [15944.268046] ? __hrtimer_run_queues+0x1c2/0x4d0 [15944.269642] ? watchdog+0x40/0x40 [15944.270985] ? hrtimer_interrupt+0x128/0x2c0 [15944.272526] ? smp_apic_timer_interrupt+0xa6/0x270 [15944.275048] ? apic_timer_interrupt+0xf/0x20 [15944.276534] [15944.277242] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15944.279177] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15944.281168] do_raw_spin_lock+0x177/0x190 [15944.282616] _raw_spin_lock+0x21/0x30 [15944.284082] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15944.286228] process_one_work+0x2c8/0x700 [15944.288191] worker_thread+0x296/0x6e0 [15944.289621] ? rescuer_thread+0x570/0x570 [15944.291226] kthread+0x1d1/0x200 [15944.292525] ? set_kthread_struct+0x70/0x70 [15944.294057] ret_from_fork+0x1f/0x30 [15952.062237] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15952.065814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15952.110221] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15952.123584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15952.128087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15952.131178] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15952.150002] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15952.154497] RAX: 0000000000007715 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15952.158087] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15952.166876] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15952.176432] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15952.180407] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15952.184889] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15952.191743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15952.196871] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15952.205208] Call Trace: [15952.207142] [15952.209696] ? show_regs.cold.9+0x22/0x2f [15952.214835] ? watchdog_timer_fn.cold.6+0x62/0x10a [15952.217501] ? __hrtimer_run_queues+0x1c2/0x4d0 [15952.219220] ? watchdog+0x40/0x40 [15952.223166] ? hrtimer_interrupt+0x128/0x2c0 [15952.226757] ? smp_apic_timer_interrupt+0xa6/0x270 [15952.229498] ? apic_timer_interrupt+0xf/0x20 [15952.232619] [15952.233625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15952.236130] do_raw_spin_lock+0x177/0x190 [15952.237952] _raw_spin_lock+0x21/0x30 [15952.239605] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15952.242531] ? do_raw_spin_unlock+0x75/0x190 [15952.247395] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15952.251862] ? do_wait_intr+0xf0/0xf0 [15952.253931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15952.258882] mdc_close+0x53b/0xd70 [mdc] [15952.261330] lmv_close+0x18d/0x490 [lmv] [15952.265391] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15952.269905] ll_md_real_close+0xa5/0x2a0 [lustre] [15952.276074] ll_md_close+0x274/0x800 [lustre] [15952.279219] ll_file_release+0x128/0xc50 [lustre] [15952.281559] __fput+0x10d/0x440 [15952.282749] ____fput+0x12/0x20 [15952.283771] task_work_run+0xc6/0x110 [15952.285232] exit_to_usermode_loop+0x1dd/0x1f0 [15952.287187] do_syscall_64+0x3d6/0x440 [15952.288890] entry_SYSCALL_64_after_hwframe+0x49/0xae [15952.291246] RIP: 0033:0x7fa749552cd8 [15952.292833] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15952.306511] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15952.310251] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15952.314753] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15952.318422] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15952.325116] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15952.330846] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15960.054072] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15960.058002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15960.087553] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15960.092822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15960.097250] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15960.100748] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15960.109999] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15960.113310] RAX: 0000000000000f0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15960.117022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15960.119896] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15960.123441] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15960.125709] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15960.128577] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15960.134114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15960.137579] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15960.140788] Call Trace: [15960.142501] [15960.143291] ? show_regs.cold.9+0x22/0x2f [15960.144662] ? watchdog_timer_fn.cold.6+0x62/0x10a [15960.146176] ? __hrtimer_run_queues+0x1c2/0x4d0 [15960.148000] ? watchdog+0x40/0x40 [15960.149724] ? hrtimer_interrupt+0x128/0x2c0 [15960.151504] ? smp_apic_timer_interrupt+0xa6/0x270 [15960.153728] ? apic_timer_interrupt+0xf/0x20 [15960.155341] [15960.156529] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15960.159070] do_raw_spin_lock+0x177/0x190 [15960.160988] _raw_spin_lock+0x21/0x30 [15960.162397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15960.165132] ? do_raw_spin_unlock+0x75/0x190 [15960.167567] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15960.170305] ? do_wait_intr+0xf0/0xf0 [15960.172660] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15960.174522] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15960.179064] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15960.182024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15960.185012] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15960.187589] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15960.189565] mdc_intent_lock+0x353/0x7e0 [mdc] [15960.191850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15960.194197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15960.197506] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15960.199921] lmv_intent_open+0x338/0x1150 [lmv] [15960.201760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15960.203682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15960.206393] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15960.208563] ll_intent_lock+0xc5/0xaa0 [lustre] [15960.211570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15960.214890] ll_lookup_it+0xe0e/0x2270 [lustre] [15960.218026] ? do_raw_spin_unlock+0x75/0x190 [15960.219978] ? _raw_spin_unlock+0x12/0x30 [15960.221665] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15960.224178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15960.226588] ll_atomic_open+0x36a/0xd70 [lustre] [15960.229576] lookup_open+0x356/0xb40 [15960.231074] do_last+0x313/0xfc0 [15960.232705] ? nd_jump_root+0xe5/0x160 [15960.234535] ? path_init+0x437/0x520 [15960.236103] path_openat+0xf7/0x500 [15960.237900] ? do_raw_spin_unlock+0x75/0x190 [15960.239840] do_filp_open+0x99/0x140 [15960.241527] ? getname_flags+0x6e/0x330 [15960.243447] ? __check_object_size+0xff/0x256 [15960.245576] ? do_raw_spin_unlock+0x75/0x190 [15960.247510] ? _raw_spin_unlock+0x12/0x30 [15960.250167] do_sys_openat2+0x2b4/0x410 [15960.252061] do_sys_open+0x73/0xa0 [15960.253898] __x64_sys_openat+0x24/0x30 [15960.255932] do_syscall_64+0xc1/0x440 [15960.257954] entry_SYSCALL_64_after_hwframe+0x49/0xae [15960.261748] RIP: 0033:0x7fa749552332 [15960.263159] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15960.269805] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15960.274908] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15960.279999] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15960.282571] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15960.285920] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15960.289451] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15965.665040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14048s! [15965.667593] Showing busy workqueues and worker pools: [15965.669720] workqueue events: flags=0x0 [15965.671120] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15965.673567] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15965.677449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15965.679799] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15965.683197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.685427] pending: kfree_rcu_monitor [15965.686681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15965.688728] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15965.691765] workqueue events_long: flags=0x0 [15965.692955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.694978] pending: lu_objects_flush [obdclass] [15965.696561] workqueue events_unbound: flags=0x2 [15965.697866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15965.699848] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15965.703014] workqueue events_power_efficient: flags=0x80 [15965.704603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.707374] pending: check_lifetime [15965.708698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.711114] pending: neigh_periodic_work [15965.712518] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.716584] pending: neigh_periodic_work [15965.718194] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.720476] pending: do_cache_clean [15965.721871] workqueue mm_percpu_wq: flags=0x8 [15965.723731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.726213] pending: vmstat_update [15965.727805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.730708] pending: vmstat_update [15965.732212] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15965.735364] pending: vmstat_update [15965.736856] workqueue writeback: flags=0x4a [15965.738574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15965.740771] pending: wb_workfn, wb_workfn [15965.742587] workqueue xprtiod: flags=0xa [15965.744261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15965.746470] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15965.748965] workqueue ipv6_addrconf: flags=0x40008 [15965.751019] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15965.753152] pending: addrconf_verify_work [15965.754864] workqueue knbd0-recv: flags=0x1a [15965.757226] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15965.759825] in-flight: 651:recv_work [15965.761552] workqueue ptlrpc_pinger: flags=0x2 [15965.763197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15965.765811] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15965.768013] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14048s workers=4 idle: 37 31846 28897 [15965.771420] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15955s workers=2 idle: 653 [15972.058025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [15972.058561] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [15972.062525] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [15972.065831] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [15972.074672] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac [15972.090748] nfsv4 [15972.090758] rapl i2c_piix4 pcspkr [15972.101238] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15972.101277] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15972.102089] squashfs [15972.103711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15972.112705] ata_generic [15972.116687] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15972.117786] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15972.117887] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15972.117894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15972.117897] RIP: 0010:native_safe_halt+0x2e/0x50 [15972.117908] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15972.117972] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15972.117977] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [15972.117979] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [15972.117982] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [15972.117984] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [15972.117986] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15972.117989] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [15972.117991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15972.117994] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [15972.118054] Call Trace: [15972.118061] [15972.118066] ? show_regs.cold.9+0x22/0x2f [15972.118076] ? watchdog_timer_fn.cold.6+0x62/0x10a [15972.118082] ? __hrtimer_run_queues+0x1c2/0x4d0 [15972.118086] ? watchdog+0x40/0x40 [15972.118091] ? hrtimer_interrupt+0x128/0x2c0 [15972.118095] ? smp_apic_timer_interrupt+0xa6/0x270 [15972.118103] ? apic_timer_interrupt+0xf/0x20 [15972.118108] [15972.118111] ? native_safe_halt+0x2e/0x50 [15972.118116] kvm_wait+0xcd/0xe0 [15972.118123] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15972.118130] do_raw_spin_lock+0x177/0x190 [15972.118134] _raw_spin_lock+0x21/0x30 [15972.118139] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15972.118478] obd_mod_free+0x46/0x140 [ptlrpc] [15972.118742] mdc_commit_open+0x91/0xb0 [mdc] [15972.118783] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15972.119169] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15972.123411] [15972.123422] RIP: 0010:native_safe_halt+0x2e/0x50 [15972.123439] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15972.123443] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15972.123448] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15972.123450] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [15972.123452] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [15972.123454] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [15972.123456] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [15972.123459] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [15972.123462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15972.123464] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [15972.123472] Call Trace: [15972.123478] [15972.123484] ? show_regs.cold.9+0x22/0x2f [15972.123495] ? watchdog_timer_fn.cold.6+0x62/0x10a [15972.123500] ? __hrtimer_run_queues+0x1c2/0x4d0 [15972.123504] ? watchdog+0x40/0x40 [15972.123508] ? hrtimer_interrupt+0x128/0x2c0 [15972.123512] ? smp_apic_timer_interrupt+0xa6/0x270 [15972.123519] ? apic_timer_interrupt+0xf/0x20 [15972.125647] after_reply+0xe2c/0x1780 [ptlrpc] [15972.131249] [15972.140808] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15972.148454] ? native_safe_halt+0x2e/0x50 [15972.148473] kvm_wait+0xcd/0xe0 [15972.148480] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15972.148492] do_raw_spin_lock+0x177/0x190 [15972.148496] _raw_spin_lock+0x21/0x30 [15972.148500] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15972.148795] process_one_work+0x2c8/0x700 [15972.148803] worker_thread+0x296/0x6e0 [15972.148808] ? rescuer_thread+0x570/0x570 [15972.148812] kthread+0x1d1/0x200 [15972.148816] ? set_kthread_struct+0x70/0x70 [15972.148820] ret_from_fork+0x1f/0x30 [15972.270403] ? __next_timer_interrupt+0x160/0x160 [15972.271991] ptlrpcd+0xa82/0xe40 [ptlrpc] [15972.273804] ? do_wait_intr+0xf0/0xf0 [15972.275343] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15972.277374] kthread+0x1d1/0x200 [15972.278630] ? set_kthread_struct+0x70/0x70 [15972.280111] ret_from_fork+0x1f/0x30 [15980.070068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [15980.077178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15980.105336] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15980.111493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15980.116735] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15980.120362] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15980.131183] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15980.135490] RAX: 0000000000007d70 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15980.139601] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [15980.143631] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [15980.148165] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [15980.152534] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [15980.156710] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [15980.161480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15980.164662] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [15980.167837] Call Trace: [15980.169728] [15980.173327] ? show_regs.cold.9+0x22/0x2f [15980.178547] ? watchdog_timer_fn.cold.6+0x62/0x10a [15980.182473] ? __hrtimer_run_queues+0x1c2/0x4d0 [15980.186535] ? watchdog+0x40/0x40 [15980.191264] ? hrtimer_interrupt+0x128/0x2c0 [15980.192956] ? smp_apic_timer_interrupt+0xa6/0x270 [15980.195699] ? apic_timer_interrupt+0xf/0x20 [15980.197806] [15980.198605] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15980.203273] do_raw_spin_lock+0x177/0x190 [15980.205004] _raw_spin_lock+0x21/0x30 [15980.209330] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15980.217491] ? do_raw_spin_unlock+0x75/0x190 [15980.219741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15980.225777] ? do_wait_intr+0xf0/0xf0 [15980.230343] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15980.235233] mdc_close+0x53b/0xd70 [mdc] [15980.237439] lmv_close+0x18d/0x490 [lmv] [15980.239927] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15980.243014] ll_md_real_close+0xa5/0x2a0 [lustre] [15980.245224] ll_md_close+0x274/0x800 [lustre] [15980.247602] ll_file_release+0x128/0xc50 [lustre] [15980.249879] __fput+0x10d/0x440 [15980.251149] ____fput+0x12/0x20 [15980.252337] task_work_run+0xc6/0x110 [15980.254103] exit_to_usermode_loop+0x1dd/0x1f0 [15980.256084] do_syscall_64+0x3d6/0x440 [15980.257700] entry_SYSCALL_64_after_hwframe+0x49/0xae [15980.260016] RIP: 0033:0x7fa749552cd8 [15980.261408] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15980.270032] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15980.273534] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [15980.276714] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [15980.279959] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [15980.282916] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [15980.286051] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15992.056780] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [15992.059434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15992.122428] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15992.130690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15992.142281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15992.150537] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15992.172012] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15992.183746] RAX: 0000000000005442 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [15992.203217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [15992.213006] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [15992.222216] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [15992.229881] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [15992.237834] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [15992.249414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15992.254623] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [15992.266220] Call Trace: [15992.267632] [15992.268638] ? show_regs.cold.9+0x22/0x2f [15992.274905] ? watchdog_timer_fn.cold.6+0x62/0x10a [15992.281107] ? __hrtimer_run_queues+0x1c2/0x4d0 [15992.287171] ? watchdog+0x40/0x40 [15992.290531] ? hrtimer_interrupt+0x128/0x2c0 [15992.292831] ? smp_apic_timer_interrupt+0xa6/0x270 [15992.296392] ? apic_timer_interrupt+0xf/0x20 [15992.299101] [15992.300043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15992.306109] do_raw_spin_lock+0x177/0x190 [15992.308238] _raw_spin_lock+0x21/0x30 [15992.310373] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15992.313367] ? do_raw_spin_unlock+0x75/0x190 [15992.315969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15992.319076] ? do_wait_intr+0xf0/0xf0 [15992.321008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15992.324364] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15992.327819] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15992.331377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15992.334865] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15992.338228] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15992.341767] mdc_intent_lock+0x353/0x7e0 [mdc] [15992.344542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15992.347985] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15992.355866] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15992.360747] lmv_intent_open+0x338/0x1150 [lmv] [15992.363543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15992.374034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15992.383840] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15992.390940] ll_intent_lock+0xc5/0xaa0 [lustre] [15992.398051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15992.400939] ll_lookup_it+0xe0e/0x2270 [lustre] [15992.404926] ? do_raw_spin_unlock+0x75/0x190 [15992.407634] ? _raw_spin_unlock+0x12/0x30 [15992.409225] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15992.412448] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15992.414965] ll_atomic_open+0x36a/0xd70 [lustre] [15992.417851] lookup_open+0x356/0xb40 [15992.421305] do_last+0x313/0xfc0 [15992.424131] ? nd_jump_root+0xe5/0x160 [15992.426344] ? path_init+0x437/0x520 [15992.429363] path_openat+0xf7/0x500 [15992.430743] ? do_raw_spin_unlock+0x75/0x190 [15992.432330] do_filp_open+0x99/0x140 [15992.433646] ? getname_flags+0x6e/0x330 [15992.435040] ? __check_object_size+0xff/0x256 [15992.437098] ? do_raw_spin_unlock+0x75/0x190 [15992.440787] ? _raw_spin_unlock+0x12/0x30 [15992.447737] do_sys_openat2+0x2b4/0x410 [15992.452975] do_sys_open+0x73/0xa0 [15992.459363] __x64_sys_openat+0x24/0x30 [15992.460499] do_syscall_64+0xc1/0x440 [15992.463691] entry_SYSCALL_64_after_hwframe+0x49/0xae [15992.466384] RIP: 0033:0x7fa749552332 [15992.469208] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15992.482588] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15992.489339] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [15992.496881] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [15992.506446] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [15992.514607] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [15992.516459] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [15995.882686] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14078s! [15995.888769] Showing busy workqueues and worker pools: [15995.895407] workqueue events: flags=0x0 [15995.897273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15995.902465] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [15995.908780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15995.913871] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15995.920299] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15995.927267] pending: kfree_rcu_monitor [15995.931035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15995.939605] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [15995.954486] workqueue events_long: flags=0x0 [15995.963542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15995.973708] pending: lu_objects_flush [obdclass] [15995.975458] workqueue events_unbound: flags=0x2 [15995.985547] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [15995.994137] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15996.007054] workqueue events_power_efficient: flags=0x80 [15996.017992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15996.025148] pending: check_lifetime [15996.028268] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15996.032848] pending: neigh_periodic_work [15996.038348] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15996.045663] pending: neigh_periodic_work [15996.050465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15996.058622] pending: do_cache_clean [15996.064450] workqueue mm_percpu_wq: flags=0x8 [15996.068828] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15996.078449] pending: vmstat_update [15996.080471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15996.086457] pending: vmstat_update [15996.088124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15996.091518] pending: vmstat_update [15996.095419] workqueue writeback: flags=0x4a [15996.104964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15996.112856] pending: wb_workfn, wb_workfn [15996.115197] workqueue xprtiod: flags=0xa [15996.123413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15996.130517] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [15996.136570] workqueue ipv6_addrconf: flags=0x40008 [15996.138482] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15996.149064] pending: addrconf_verify_work [15996.154531] workqueue knbd0-recv: flags=0x1a [15996.157746] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15996.161925] in-flight: 651:recv_work [15996.164525] workqueue ptlrpc_pinger: flags=0x2 [15996.169644] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15996.177752] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [15996.185430] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14079s workers=4 idle: 37 31846 28897 [15996.195762] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15986s workers=2 idle: 653 [16000.057070] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16000.058075] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16000.060903] Modules linked in: lustre(O) osp(O) [16000.065216] Modules linked in: [16000.065219] ofd(O) lod(O) [16000.068299] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [16000.070585] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 [16000.072274] i2c_piix4 [16000.086552] pcspkr [16000.103197] pcspkr [16000.104378] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16000.104404] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16000.104410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16000.104417] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16000.104789] RIP: 0010:native_safe_halt+0x2e/0x50 [16000.104799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16000.104802] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16000.104807] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16000.104809] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16000.104811] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16000.104814] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16000.104816] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16000.104819] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16000.104822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16000.104824] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16000.104833] Call Trace: [16000.104838] [16000.104843] ? show_regs.cold.9+0x22/0x2f [16000.104853] ? watchdog_timer_fn.cold.6+0x62/0x10a [16000.104859] ? __hrtimer_run_queues+0x1c2/0x4d0 [16000.104959] ? watchdog+0x40/0x40 [16000.104967] ? hrtimer_interrupt+0x128/0x2c0 [16000.104972] ? smp_apic_timer_interrupt+0xa6/0x270 [16000.104978] ? apic_timer_interrupt+0xf/0x20 [16000.104983] [16000.104986] ? native_safe_halt+0x2e/0x50 [16000.104990] kvm_wait+0xcd/0xe0 [16000.104998] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16000.105006] do_raw_spin_lock+0x177/0x190 [16000.105010] _raw_spin_lock+0x21/0x30 [16000.105015] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16000.105495] process_one_work+0x2c8/0x700 [16000.105502] worker_thread+0x296/0x6e0 [16000.105507] ? rescuer_thread+0x570/0x570 [16000.105510] kthread+0x1d1/0x200 [16000.105515] ? set_kthread_struct+0x70/0x70 [16000.105519] ret_from_fork+0x1f/0x30 [16000.244552] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16000.256154] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16000.263949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16000.270519] RIP: 0010:native_safe_halt+0x2e/0x50 [16000.273503] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16000.287833] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16000.293282] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16000.298159] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16000.303151] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16000.312277] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16000.317289] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16000.321736] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16000.326359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16000.330200] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16000.334152] Call Trace: [16000.335524] [16000.336957] ? show_regs.cold.9+0x22/0x2f [16000.339828] ? watchdog_timer_fn.cold.6+0x62/0x10a [16000.342990] ? __hrtimer_run_queues+0x1c2/0x4d0 [16000.346838] ? watchdog+0x40/0x40 [16000.349074] ? hrtimer_interrupt+0x128/0x2c0 [16000.351714] ? smp_apic_timer_interrupt+0xa6/0x270 [16000.354943] ? apic_timer_interrupt+0xf/0x20 [16000.357227] [16000.358366] ? native_safe_halt+0x2e/0x50 [16000.361441] kvm_wait+0xcd/0xe0 [16000.363192] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16000.365804] do_raw_spin_lock+0x177/0x190 [16000.367863] _raw_spin_lock+0x21/0x30 [16000.370229] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16000.373167] obd_mod_free+0x46/0x140 [ptlrpc] [16000.375109] mdc_commit_open+0x91/0xb0 [mdc] [16000.377469] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16000.381087] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16000.384495] after_reply+0xe2c/0x1780 [ptlrpc] [16000.387813] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16000.395620] ? __next_timer_interrupt+0x160/0x160 [16000.397084] ptlrpcd+0xa82/0xe40 [ptlrpc] [16000.408281] ? do_wait_intr+0xf0/0xf0 [16000.412501] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16000.422148] kthread+0x1d1/0x200 [16000.424175] ? set_kthread_struct+0x70/0x70 [16000.427803] ret_from_fork+0x1f/0x30 [16008.065344] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16008.074589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16008.129220] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16008.140925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16008.160344] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16008.163103] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16008.181718] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16008.184998] RAX: 0000000000001710 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16008.190690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16008.194359] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16008.196899] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16008.199946] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16008.204911] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16008.214882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16008.218077] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16008.228581] Call Trace: [16008.230068] [16008.231055] ? show_regs.cold.9+0x22/0x2f [16008.234152] ? watchdog_timer_fn.cold.6+0x62/0x10a [16008.237514] ? __hrtimer_run_queues+0x1c2/0x4d0 [16008.240420] ? watchdog+0x40/0x40 [16008.242743] ? hrtimer_interrupt+0x128/0x2c0 [16008.245021] ? smp_apic_timer_interrupt+0xa6/0x270 [16008.251185] ? apic_timer_interrupt+0xf/0x20 [16008.256283] [16008.257097] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16008.259538] do_raw_spin_lock+0x177/0x190 [16008.263353] _raw_spin_lock+0x21/0x30 [16008.266347] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16008.268810] ? do_raw_spin_unlock+0x75/0x190 [16008.271494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16008.276553] ? do_wait_intr+0xf0/0xf0 [16008.278838] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16008.281540] mdc_close+0x53b/0xd70 [mdc] [16008.286400] lmv_close+0x18d/0x490 [lmv] [16008.289962] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16008.295425] ll_md_real_close+0xa5/0x2a0 [lustre] [16008.299760] ll_md_close+0x274/0x800 [lustre] [16008.303553] ll_file_release+0x128/0xc50 [lustre] [16008.306266] __fput+0x10d/0x440 [16008.307394] ____fput+0x12/0x20 [16008.310152] task_work_run+0xc6/0x110 [16008.314483] exit_to_usermode_loop+0x1dd/0x1f0 [16008.320768] do_syscall_64+0x3d6/0x440 [16008.322817] entry_SYSCALL_64_after_hwframe+0x49/0xae [16008.327956] RIP: 0033:0x7fa749552cd8 [16008.329307] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16008.348105] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16008.356237] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16008.360766] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16008.364794] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16008.372989] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16008.381117] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16014.003032] rcu: INFO: rcu_sched self-detected stall on CPU [16014.005097] rcu: 2-...!: (8253355 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [16014.009757] (t=14100314 jiffies g=258701 q=5275) [16014.011154] rcu: rcu_sched kthread starved for 14097109 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [16014.015886] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16014.023747] rcu: RCU grace-period kthread stack dump: [16014.027819] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16014.034368] Call Trace: [16014.036544] __schedule+0x351/0xcb0 [16014.043450] schedule+0xc0/0x180 [16014.045445] schedule_timeout+0xb4/0x190 [16014.047801] ? do_raw_spin_unlock+0x75/0x190 [16014.056984] ? __next_timer_interrupt+0x160/0x160 [16014.060847] ? prepare_to_swait_event+0xf3/0x140 [16014.065787] rcu_gp_kthread+0x8dd/0x15b0 [16014.068691] ? force_qs_rnp+0x2c0/0x2c0 [16014.072889] kthread+0x1d1/0x200 [16014.074301] ? set_kthread_struct+0x70/0x70 [16014.077460] ret_from_fork+0x1f/0x30 [16014.084839] rcu: Stack dump where RCU GP kthread last ran: [16014.091442] NMI backtrace for cpu 2 [16014.097777] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16014.105050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16014.115271] Call Trace: [16014.116536] [16014.118415] ? dump_stack+0xbb/0x10e [16014.120768] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16014.125054] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16014.128885] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16014.132198] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16014.142501] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16014.144026] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [16014.147916] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [16014.154498] ? update_process_times+0x6d/0xc0 [16014.158662] ? tick_sched_handle.isra.18+0x27/0x90 [16014.165761] ? tick_sched_timer+0x53/0xb0 [16014.170604] ? __hrtimer_run_queues+0x1c2/0x4d0 [16014.173523] ? tick_sched_handle.isra.18+0x90/0x90 [16014.178960] ? hrtimer_interrupt+0x128/0x2c0 [16014.185021] ? smp_apic_timer_interrupt+0xa6/0x270 [16014.189454] ? apic_timer_interrupt+0xf/0x20 [16014.192882] [16014.198654] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16014.204729] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16014.211148] ? do_raw_spin_lock+0x177/0x190 [16014.214519] ? _raw_spin_lock+0x21/0x30 [16014.216019] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16014.222402] ? obd_mod_free+0x46/0x140 [ptlrpc] [16014.228711] ? mdc_commit_open+0x91/0xb0 [mdc] [16014.231362] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16014.237625] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16014.243778] ? after_reply+0xe2c/0x1780 [ptlrpc] [16014.249847] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16014.255791] ? __next_timer_interrupt+0x160/0x160 [16014.259799] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16014.265346] ? do_wait_intr+0xf0/0xf0 [16014.266784] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16014.271670] ? kthread+0x1d1/0x200 [16014.275448] ? set_kthread_struct+0x70/0x70 [16014.284311] ? ret_from_fork+0x1f/0x30 [16014.289862] NMI backtrace for cpu 2 [16014.292772] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16014.304724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16014.313059] Call Trace: [16014.316892] [16014.319541] ? dump_stack+0xbb/0x10e [16014.323564] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16014.327200] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16014.328838] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16014.338794] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16014.342932] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16014.344673] ? rcu_dump_cpu_stacks+0x11b/0x171 [16014.346972] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16014.352767] ? update_process_times+0x6d/0xc0 [16014.356917] ? tick_sched_handle.isra.18+0x27/0x90 [16014.360921] ? tick_sched_timer+0x53/0xb0 [16014.364262] ? __hrtimer_run_queues+0x1c2/0x4d0 [16014.366853] ? tick_sched_handle.isra.18+0x90/0x90 [16014.372066] ? hrtimer_interrupt+0x128/0x2c0 [16014.377621] ? smp_apic_timer_interrupt+0xa6/0x270 [16014.384894] ? apic_timer_interrupt+0xf/0x20 [16014.386818] [16014.387776] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16014.393367] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16014.398948] ? do_raw_spin_lock+0x177/0x190 [16014.400581] ? _raw_spin_lock+0x21/0x30 [16014.401786] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16014.407382] ? obd_mod_free+0x46/0x140 [ptlrpc] [16014.410656] ? mdc_commit_open+0x91/0xb0 [mdc] [16014.413002] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16014.415966] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16014.419164] ? after_reply+0xe2c/0x1780 [ptlrpc] [16014.422004] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16014.424428] ? __next_timer_interrupt+0x160/0x160 [16014.426958] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16014.428957] ? do_wait_intr+0xf0/0xf0 [16014.433057] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16014.437279] ? kthread+0x1d1/0x200 [16014.439413] ? set_kthread_struct+0x70/0x70 [16014.441930] ? ret_from_fork+0x1f/0x30 [16024.054313] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16024.059141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16024.115494] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16024.128172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16024.141682] RIP: 0010:native_safe_halt+0x2e/0x50 [16024.143041] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16024.171582] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [16024.188613] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16024.196216] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16024.202315] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16024.216372] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16024.222491] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16024.232590] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16024.237476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16024.243480] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16024.245673] Call Trace: [16024.246500] [16024.247116] ? show_regs.cold.9+0x22/0x2f [16024.255138] ? watchdog_timer_fn.cold.6+0x62/0x10a [16024.258872] ? __hrtimer_run_queues+0x1c2/0x4d0 [16024.262406] ? watchdog+0x40/0x40 [16024.264753] ? hrtimer_interrupt+0x128/0x2c0 [16024.268752] ? smp_apic_timer_interrupt+0xa6/0x270 [16024.274898] ? apic_timer_interrupt+0xf/0x20 [16024.281119] [16024.281921] ? native_safe_halt+0x2e/0x50 [16024.287463] kvm_wait+0xcd/0xe0 [16024.289914] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16024.294211] do_raw_spin_lock+0x177/0x190 [16024.297666] _raw_spin_lock+0x21/0x30 [16024.300323] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16024.302438] ? do_raw_spin_unlock+0x75/0x190 [16024.304541] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16024.306765] ? do_wait_intr+0xf0/0xf0 [16024.308552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16024.312384] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16024.319549] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16024.322896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16024.326302] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16024.327906] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16024.329841] mdc_intent_lock+0x353/0x7e0 [mdc] [16024.332359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16024.334802] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16024.337873] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16024.340609] lmv_intent_open+0x338/0x1150 [lmv] [16024.343451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16024.346655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16024.350324] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16024.352983] ll_intent_lock+0xc5/0xaa0 [lustre] [16024.355738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16024.360666] ll_lookup_it+0xe0e/0x2270 [lustre] [16024.363055] ? do_raw_spin_unlock+0x75/0x190 [16024.365187] ? _raw_spin_unlock+0x12/0x30 [16024.367655] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16024.370697] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16024.383516] ll_atomic_open+0x36a/0xd70 [lustre] [16024.385784] lookup_open+0x356/0xb40 [16024.387658] do_last+0x313/0xfc0 [16024.389633] ? nd_jump_root+0xe5/0x160 [16024.391837] ? path_init+0x437/0x520 [16024.393886] path_openat+0xf7/0x500 [16024.395833] ? do_raw_spin_unlock+0x75/0x190 [16024.397840] do_filp_open+0x99/0x140 [16024.399793] ? getname_flags+0x6e/0x330 [16024.401603] ? __check_object_size+0xff/0x256 [16024.403600] ? do_raw_spin_unlock+0x75/0x190 [16024.405465] ? _raw_spin_unlock+0x12/0x30 [16024.407434] do_sys_openat2+0x2b4/0x410 [16024.413374] do_sys_open+0x73/0xa0 [16024.414917] __x64_sys_openat+0x24/0x30 [16024.419717] do_syscall_64+0xc1/0x440 [16024.423676] entry_SYSCALL_64_after_hwframe+0x49/0xae [16024.426611] RIP: 0033:0x7fa749552332 [16024.428690] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16024.445590] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16024.455564] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16024.458587] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16024.464246] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16024.470620] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16024.479311] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16026.592078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14109s! [16026.594934] Showing busy workqueues and worker pools: [16026.606800] workqueue events: flags=0x0 [16026.613374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16026.616710] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16026.626281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16026.632049] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16026.638833] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.641751] pending: kfree_rcu_monitor [16026.646708] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16026.649863] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16026.657331] workqueue events_long: flags=0x0 [16026.659211] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.663354] pending: lu_objects_flush [obdclass] [16026.665875] workqueue events_unbound: flags=0x2 [16026.668351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16026.671405] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16026.675759] workqueue events_power_efficient: flags=0x80 [16026.678447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.681692] pending: check_lifetime [16026.683471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.687790] pending: neigh_periodic_work [16026.690646] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.694763] pending: neigh_periodic_work [16026.697516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.700988] pending: do_cache_clean [16026.703088] workqueue mm_percpu_wq: flags=0x8 [16026.705276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.711649] pending: vmstat_update [16026.714461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.721311] pending: vmstat_update [16026.726646] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16026.734367] pending: vmstat_update [16026.737163] workqueue writeback: flags=0x4a [16026.740767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16026.747616] pending: wb_workfn, wb_workfn [16026.750389] workqueue xprtiod: flags=0xa [16026.751729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16026.754382] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16026.757378] workqueue ipv6_addrconf: flags=0x40008 [16026.760369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16026.764070] pending: addrconf_verify_work [16026.766594] workqueue knbd0-recv: flags=0x1a [16026.776201] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16026.781689] in-flight: 651:recv_work [16026.783806] workqueue ptlrpc_pinger: flags=0x2 [16026.786846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16026.790495] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16026.793458] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14109s workers=4 idle: 37 31846 28897 [16026.797203] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16016s workers=2 idle: 653 [16028.057385] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16028.061582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16028.098388] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16028.115712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16028.126976] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16028.135672] RIP: 0010:native_safe_halt+0x2e/0x50 [16028.141765] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16028.158515] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16028.164387] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16028.167534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16028.175492] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16028.178241] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16028.181141] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16028.184002] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16028.187094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16028.189288] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16028.191395] Call Trace: [16028.191987] [16028.192788] ? show_regs.cold.9+0x22/0x2f [16028.194446] ? watchdog_timer_fn.cold.6+0x62/0x10a [16028.196412] ? __hrtimer_run_queues+0x1c2/0x4d0 [16028.198156] ? watchdog+0x40/0x40 [16028.199551] ? hrtimer_interrupt+0x128/0x2c0 [16028.201335] ? smp_apic_timer_interrupt+0xa6/0x270 [16028.203259] ? apic_timer_interrupt+0xf/0x20 [16028.204893] [16028.205814] ? native_safe_halt+0x2e/0x50 [16028.207546] kvm_wait+0xcd/0xe0 [16028.208792] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16028.210903] do_raw_spin_lock+0x177/0x190 [16028.212582] _raw_spin_lock+0x21/0x30 [16028.213861] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16028.216121] process_one_work+0x2c8/0x700 [16028.217686] worker_thread+0x296/0x6e0 [16028.219096] ? rescuer_thread+0x570/0x570 [16028.220081] kthread+0x1d1/0x200 [16028.220865] ? set_kthread_struct+0x70/0x70 [16028.221917] ret_from_fork+0x1f/0x30 [16036.059253] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16036.065780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16036.110788] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16036.132063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16036.142312] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16036.146314] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16036.163310] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16036.170211] RAX: 000000000000010a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16036.174216] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16036.179256] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16036.186215] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16036.199016] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16036.210639] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16036.213566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16036.220759] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16036.224174] Call Trace: [16036.227249] [16036.229193] ? show_regs.cold.9+0x22/0x2f [16036.234928] ? watchdog_timer_fn.cold.6+0x62/0x10a [16036.237703] ? __hrtimer_run_queues+0x1c2/0x4d0 [16036.240624] ? watchdog+0x40/0x40 [16036.243731] ? hrtimer_interrupt+0x128/0x2c0 [16036.245285] ? smp_apic_timer_interrupt+0xa6/0x270 [16036.249463] ? apic_timer_interrupt+0xf/0x20 [16036.257663] [16036.262394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16036.266163] do_raw_spin_lock+0x177/0x190 [16036.272163] _raw_spin_lock+0x21/0x30 [16036.278218] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16036.281566] ? do_raw_spin_unlock+0x75/0x190 [16036.284315] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16036.286548] ? do_wait_intr+0xf0/0xf0 [16036.292574] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16036.298351] mdc_close+0x53b/0xd70 [mdc] [16036.301819] lmv_close+0x18d/0x490 [lmv] [16036.314027] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16036.324183] ll_md_real_close+0xa5/0x2a0 [lustre] [16036.325954] ll_md_close+0x274/0x800 [lustre] [16036.331733] ll_file_release+0x128/0xc50 [lustre] [16036.335657] __fput+0x10d/0x440 [16036.337363] ____fput+0x12/0x20 [16036.341575] task_work_run+0xc6/0x110 [16036.343637] exit_to_usermode_loop+0x1dd/0x1f0 [16036.349102] do_syscall_64+0x3d6/0x440 [16036.351535] entry_SYSCALL_64_after_hwframe+0x49/0xae [16036.354234] RIP: 0033:0x7fa749552cd8 [16036.357273] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16036.380359] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16036.388918] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16036.397149] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16036.404662] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16036.411235] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16036.414459] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16040.058098] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16040.067490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16040.119722] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16040.130517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16040.140624] RIP: 0010:native_safe_halt+0x2e/0x50 [16040.148279] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16040.168487] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16040.175518] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16040.180485] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16040.189268] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16040.197136] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16040.202547] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16040.212515] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16040.218965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16040.225707] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16040.232227] Call Trace: [16040.233918] [16040.236251] ? show_regs.cold.9+0x22/0x2f [16040.239264] ? watchdog_timer_fn.cold.6+0x62/0x10a [16040.242895] ? __hrtimer_run_queues+0x1c2/0x4d0 [16040.246300] ? watchdog+0x40/0x40 [16040.249489] ? hrtimer_interrupt+0x128/0x2c0 [16040.251151] ? smp_apic_timer_interrupt+0xa6/0x270 [16040.257427] ? apic_timer_interrupt+0xf/0x20 [16040.263255] [16040.265942] ? native_safe_halt+0x2e/0x50 [16040.269522] kvm_wait+0xcd/0xe0 [16040.274958] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16040.277033] do_raw_spin_lock+0x177/0x190 [16040.283361] _raw_spin_lock+0x21/0x30 [16040.291596] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16040.301252] obd_mod_free+0x46/0x140 [ptlrpc] [16040.306368] mdc_commit_open+0x91/0xb0 [mdc] [16040.317284] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16040.324830] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16040.335490] after_reply+0xe2c/0x1780 [ptlrpc] [16040.337841] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16040.342630] ? __next_timer_interrupt+0x160/0x160 [16040.353638] ptlrpcd+0xa82/0xe40 [ptlrpc] [16040.355437] ? do_wait_intr+0xf0/0xf0 [16040.367790] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16040.381427] kthread+0x1d1/0x200 [16040.385952] ? set_kthread_struct+0x70/0x70 [16040.397700] ret_from_fork+0x1f/0x30 [16052.057810] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16052.065467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16052.109765] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16052.124159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16052.131433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16052.137861] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16052.161683] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16052.168677] RAX: 0000000000004b15 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16052.178076] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16052.184983] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16052.193148] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16052.198155] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16052.218251] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16052.231165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16052.241021] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16052.252588] Call Trace: [16052.258083] [16052.260173] ? show_regs.cold.9+0x22/0x2f [16052.264319] ? watchdog_timer_fn.cold.6+0x62/0x10a [16052.269367] ? __hrtimer_run_queues+0x1c2/0x4d0 [16052.270914] ? watchdog+0x40/0x40 [16052.272050] ? hrtimer_interrupt+0x128/0x2c0 [16052.276061] ? smp_apic_timer_interrupt+0xa6/0x270 [16052.288094] ? apic_timer_interrupt+0xf/0x20 [16052.290184] [16052.291279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16052.300836] do_raw_spin_lock+0x177/0x190 [16052.302212] _raw_spin_lock+0x21/0x30 [16052.308130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16052.312022] ? do_raw_spin_unlock+0x75/0x190 [16052.318705] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16052.327457] ? do_wait_intr+0xf0/0xf0 [16052.329942] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16052.332833] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16052.338719] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16052.341628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16052.350717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16052.357970] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16052.361044] mdc_intent_lock+0x353/0x7e0 [mdc] [16052.366380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16052.369257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16052.373802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16052.377188] lmv_intent_open+0x338/0x1150 [lmv] [16052.379771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16052.386465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16052.394150] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16052.403948] ll_intent_lock+0xc5/0xaa0 [lustre] [16052.405660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16052.413024] ll_lookup_it+0xe0e/0x2270 [lustre] [16052.427369] ? do_raw_spin_unlock+0x75/0x190 [16052.432041] ? _raw_spin_unlock+0x12/0x30 [16052.441158] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16052.448380] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16052.458114] ll_atomic_open+0x36a/0xd70 [lustre] [16052.465771] lookup_open+0x356/0xb40 [16052.467051] do_last+0x313/0xfc0 [16052.469073] ? nd_jump_root+0xe5/0x160 [16052.471379] ? path_init+0x437/0x520 [16052.473369] path_openat+0xf7/0x500 [16052.478464] ? do_raw_spin_unlock+0x75/0x190 [16052.485298] do_filp_open+0x99/0x140 [16052.493184] ? getname_flags+0x6e/0x330 [16052.497990] ? __check_object_size+0xff/0x256 [16052.508477] ? do_raw_spin_unlock+0x75/0x190 [16052.511118] ? _raw_spin_unlock+0x12/0x30 [16052.513287] do_sys_openat2+0x2b4/0x410 [16052.515052] do_sys_open+0x73/0xa0 [16052.516798] __x64_sys_openat+0x24/0x30 [16052.518664] do_syscall_64+0xc1/0x440 [16052.523353] entry_SYSCALL_64_after_hwframe+0x49/0xae [16052.528630] RIP: 0033:0x7fa749552332 [16052.532290] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16052.543053] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16052.546646] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16052.551288] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16052.560830] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16052.568810] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16052.573097] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16056.057081] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16056.060758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16056.076790] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16056.080725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16056.083739] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16056.090984] RIP: 0010:native_safe_halt+0x2e/0x50 [16056.094454] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16056.104200] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16056.108905] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16056.117215] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16056.120885] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16056.126596] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16056.131016] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16056.134930] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16056.142251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16056.146090] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16056.150249] Call Trace: [16056.151243] [16056.152097] ? show_regs.cold.9+0x22/0x2f [16056.167039] ? watchdog_timer_fn.cold.6+0x62/0x10a [16056.176080] ? __hrtimer_run_queues+0x1c2/0x4d0 [16056.187756] ? watchdog+0x40/0x40 [16056.192750] ? hrtimer_interrupt+0x128/0x2c0 [16056.201006] ? smp_apic_timer_interrupt+0xa6/0x270 [16056.207806] ? apic_timer_interrupt+0xf/0x20 [16056.211129] [16056.213705] ? native_safe_halt+0x2e/0x50 [16056.218201] kvm_wait+0xcd/0xe0 [16056.219560] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16056.221896] do_raw_spin_lock+0x177/0x190 [16056.224088] _raw_spin_lock+0x21/0x30 [16056.227949] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16056.229983] process_one_work+0x2c8/0x700 [16056.231785] worker_thread+0x296/0x6e0 [16056.233214] ? rescuer_thread+0x570/0x570 [16056.234872] kthread+0x1d1/0x200 [16056.236106] ? set_kthread_struct+0x70/0x70 [16056.237218] ret_from_fork+0x1f/0x30 [16056.801043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14139s! [16056.809895] Showing busy workqueues and worker pools: [16056.813065] workqueue events: flags=0x0 [16056.815090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16056.818905] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16056.824949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16056.829627] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16056.837197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.843843] pending: kfree_rcu_monitor [16056.845734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16056.849972] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16056.855049] workqueue events_long: flags=0x0 [16056.856959] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.861574] pending: lu_objects_flush [obdclass] [16056.863454] workqueue events_unbound: flags=0x2 [16056.866175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16056.868325] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16056.875089] workqueue events_power_efficient: flags=0x80 [16056.878755] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.882564] pending: check_lifetime [16056.884008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.886237] pending: neigh_periodic_work [16056.887635] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.890264] pending: neigh_periodic_work [16056.891862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.894293] pending: do_cache_clean [16056.895875] workqueue mm_percpu_wq: flags=0x8 [16056.897690] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.900238] pending: vmstat_update [16056.901683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.904522] pending: vmstat_update [16056.906067] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16056.908657] pending: vmstat_update [16056.910250] workqueue writeback: flags=0x4a [16056.911839] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16056.914229] pending: wb_workfn, wb_workfn [16056.915887] workqueue xprtiod: flags=0xa [16056.917414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16056.920032] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16056.923373] workqueue ipv6_addrconf: flags=0x40008 [16056.925255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16056.929408] pending: addrconf_verify_work [16056.931727] workqueue knbd0-recv: flags=0x1a [16056.933655] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16056.936235] in-flight: 651:recv_work [16056.938261] workqueue ptlrpc_pinger: flags=0x2 [16056.940155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16056.942770] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16056.947509] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14139s workers=4 idle: 37 31846 28897 [16056.953049] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16046s workers=2 idle: 653 [16064.061935] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16064.069781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16064.110137] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16064.114863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16064.125119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16064.126740] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16064.135267] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16064.139197] RAX: 0000000000006cab RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16064.141891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16064.145650] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16064.156965] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16064.159911] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16064.163641] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16064.170883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16064.177767] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16064.182568] Call Trace: [16064.183437] [16064.184050] ? show_regs.cold.9+0x22/0x2f [16064.188054] ? watchdog_timer_fn.cold.6+0x62/0x10a [16064.193905] ? __hrtimer_run_queues+0x1c2/0x4d0 [16064.196676] ? watchdog+0x40/0x40 [16064.198689] ? hrtimer_interrupt+0x128/0x2c0 [16064.200972] ? smp_apic_timer_interrupt+0xa6/0x270 [16064.204569] ? apic_timer_interrupt+0xf/0x20 [16064.205980] [16064.206546] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16064.211111] do_raw_spin_lock+0x177/0x190 [16064.217925] _raw_spin_lock+0x21/0x30 [16064.219302] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16064.222241] ? do_raw_spin_unlock+0x75/0x190 [16064.223326] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16064.225570] ? do_wait_intr+0xf0/0xf0 [16064.227287] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16064.230915] mdc_close+0x53b/0xd70 [mdc] [16064.233622] lmv_close+0x18d/0x490 [lmv] [16064.235987] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16064.239262] ll_md_real_close+0xa5/0x2a0 [lustre] [16064.241015] ll_md_close+0x274/0x800 [lustre] [16064.243096] ll_file_release+0x128/0xc50 [lustre] [16064.245855] __fput+0x10d/0x440 [16064.247790] ____fput+0x12/0x20 [16064.248963] task_work_run+0xc6/0x110 [16064.251698] exit_to_usermode_loop+0x1dd/0x1f0 [16064.253851] do_syscall_64+0x3d6/0x440 [16064.257874] entry_SYSCALL_64_after_hwframe+0x49/0xae [16064.259839] RIP: 0033:0x7fa749552cd8 [16064.262055] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16064.270923] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16064.274413] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16064.277698] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16064.280952] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16064.284926] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16064.290073] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16068.058029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16068.062994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16068.092006] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16068.103088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16068.107343] RIP: 0010:native_safe_halt+0x2e/0x50 [16068.111788] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16068.126544] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16068.131126] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16068.134035] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16068.138788] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16068.142161] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16068.148646] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16068.151144] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16068.155729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16068.158026] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16068.161226] Call Trace: [16068.162133] [16068.162952] ? show_regs.cold.9+0x22/0x2f [16068.165179] ? watchdog_timer_fn.cold.6+0x62/0x10a [16068.168761] ? __hrtimer_run_queues+0x1c2/0x4d0 [16068.171830] ? watchdog+0x40/0x40 [16068.173763] ? hrtimer_interrupt+0x128/0x2c0 [16068.176496] ? smp_apic_timer_interrupt+0xa6/0x270 [16068.179318] ? apic_timer_interrupt+0xf/0x20 [16068.181793] [16068.183027] ? native_safe_halt+0x2e/0x50 [16068.185515] kvm_wait+0xcd/0xe0 [16068.187788] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16068.190752] do_raw_spin_lock+0x177/0x190 [16068.193033] _raw_spin_lock+0x21/0x30 [16068.195027] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16068.199362] obd_mod_free+0x46/0x140 [ptlrpc] [16068.202305] mdc_commit_open+0x91/0xb0 [mdc] [16068.205318] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16068.209146] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16068.212708] after_reply+0xe2c/0x1780 [ptlrpc] [16068.215829] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16068.219191] ? __next_timer_interrupt+0x160/0x160 [16068.222314] ptlrpcd+0xa82/0xe40 [ptlrpc] [16068.227045] ? do_wait_intr+0xf0/0xf0 [16068.229510] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16068.232556] kthread+0x1d1/0x200 [16068.234343] ? set_kthread_struct+0x70/0x70 [16068.236976] ret_from_fork+0x1f/0x30 [16084.054035] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16084.055744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16084.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16084.078536] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16084.084298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16084.084377] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16084.084384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16084.084390] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16084.090526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16084.090538] RIP: 0010:native_safe_halt+0x2e/0x50 [16084.090553] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16084.090558] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16084.090564] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16084.090566] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16084.090568] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16084.090569] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16084.090572] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16084.090575] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16084.090578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16084.090580] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16084.090588] Call Trace: [16084.090594] [16084.090600] ? show_regs.cold.9+0x22/0x2f [16084.090610] ? watchdog_timer_fn.cold.6+0x62/0x10a [16084.090616] ? __hrtimer_run_queues+0x1c2/0x4d0 [16084.090620] ? watchdog+0x40/0x40 [16084.090623] ? hrtimer_interrupt+0x128/0x2c0 [16084.090627] ? smp_apic_timer_interrupt+0xa6/0x270 [16084.090633] ? apic_timer_interrupt+0xf/0x20 [16084.090637] [16084.090640] ? native_safe_halt+0x2e/0x50 [16084.090643] kvm_wait+0xcd/0xe0 [16084.090650] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16084.090660] do_raw_spin_lock+0x177/0x190 [16084.090664] _raw_spin_lock+0x21/0x30 [16084.090668] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16084.091020] ? do_raw_spin_unlock+0x75/0x190 [16084.091028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16084.091456] ? do_wait_intr+0xf0/0xf0 [16084.091464] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16084.091658] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16084.091869] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16084.092419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16084.092535] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16084.092576] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16084.092607] mdc_intent_lock+0x353/0x7e0 [mdc] [16084.092634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16084.092716] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16084.092931] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16084.093232] lmv_intent_open+0x338/0x1150 [lmv] [16084.093277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16084.119172] [16084.119181] RIP: 0010:native_safe_halt+0x2e/0x50 [16084.119197] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16084.119202] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16084.119208] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16084.119211] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16084.119214] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16084.119216] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16084.119219] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16084.119222] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16084.119225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16084.119228] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16084.119236] Call Trace: [16084.119251] [16084.119257] ? show_regs.cold.9+0x22/0x2f [16084.119266] ? watchdog_timer_fn.cold.6+0x62/0x10a [16084.119273] ? __hrtimer_run_queues+0x1c2/0x4d0 [16084.119277] ? watchdog+0x40/0x40 [16084.119281] ? hrtimer_interrupt+0x128/0x2c0 [16084.119286] ? smp_apic_timer_interrupt+0xa6/0x270 [16084.119291] ? apic_timer_interrupt+0xf/0x20 [16084.119296] [16084.119298] ? native_safe_halt+0x2e/0x50 [16084.119302] kvm_wait+0xcd/0xe0 [16084.119310] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16084.119318] do_raw_spin_lock+0x177/0x190 [16084.119323] _raw_spin_lock+0x21/0x30 [16084.119327] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16084.120068] process_one_work+0x2c8/0x700 [16084.120076] worker_thread+0x296/0x6e0 [16084.120081] ? rescuer_thread+0x570/0x570 [16084.120084] kthread+0x1d1/0x200 [16084.120089] ? set_kthread_struct+0x70/0x70 [16084.120093] ret_from_fork+0x1f/0x30 [16084.370556] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16084.373161] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16084.374859] ll_intent_lock+0xc5/0xaa0 [lustre] [16084.377810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16084.380381] ll_lookup_it+0xe0e/0x2270 [lustre] [16084.381760] ? do_raw_spin_unlock+0x75/0x190 [16084.384874] ? _raw_spin_unlock+0x12/0x30 [16084.386627] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16084.390170] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16084.392757] ll_atomic_open+0x36a/0xd70 [lustre] [16084.394192] lookup_open+0x356/0xb40 [16084.395179] do_last+0x313/0xfc0 [16084.398695] ? nd_jump_root+0xe5/0x160 [16084.402206] ? path_init+0x437/0x520 [16084.403963] path_openat+0xf7/0x500 [16084.405466] ? do_raw_spin_unlock+0x75/0x190 [16084.408156] do_filp_open+0x99/0x140 [16084.410547] ? getname_flags+0x6e/0x330 [16084.412112] ? __check_object_size+0xff/0x256 [16084.414580] ? do_raw_spin_unlock+0x75/0x190 [16084.417254] ? _raw_spin_unlock+0x12/0x30 [16084.419251] do_sys_openat2+0x2b4/0x410 [16084.422001] do_sys_open+0x73/0xa0 [16084.423570] __x64_sys_openat+0x24/0x30 [16084.425256] do_syscall_64+0xc1/0x440 [16084.427006] entry_SYSCALL_64_after_hwframe+0x49/0xae [16084.430008] RIP: 0033:0x7fa749552332 [16084.431542] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16084.440196] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16084.443612] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16084.448885] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16084.452508] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16084.455889] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16084.460623] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16087.008685] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14169s! [16087.015226] Showing busy workqueues and worker pools: [16087.018552] workqueue events: flags=0x0 [16087.019964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16087.024885] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16087.031699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16087.035283] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16087.045650] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.050589] pending: kfree_rcu_monitor [16087.052114] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16087.056348] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16087.070395] workqueue events_long: flags=0x0 [16087.077233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.083913] pending: lu_objects_flush [obdclass] [16087.091989] workqueue events_unbound: flags=0x2 [16087.098674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16087.105333] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16087.111111] workqueue events_power_efficient: flags=0x80 [16087.113956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.120426] pending: check_lifetime [16087.122716] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.126208] pending: neigh_periodic_work [16087.128168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.131650] pending: neigh_periodic_work [16087.133542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.136830] pending: do_cache_clean [16087.139054] workqueue mm_percpu_wq: flags=0x8 [16087.141440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.144522] pending: vmstat_update [16087.145588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.148528] pending: vmstat_update [16087.151160] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16087.154884] pending: vmstat_update [16087.156142] workqueue writeback: flags=0x4a [16087.159707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16087.165332] pending: wb_workfn, wb_workfn [16087.167674] workqueue xprtiod: flags=0xa [16087.170082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16087.173956] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16087.181600] workqueue ipv6_addrconf: flags=0x40008 [16087.184129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16087.188604] pending: addrconf_verify_work [16087.190245] workqueue knbd0-recv: flags=0x1a [16087.192050] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16087.193998] in-flight: 651:recv_work [16087.195094] workqueue ptlrpc_pinger: flags=0x2 [16087.197020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16087.199176] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16087.201426] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14170s workers=4 idle: 37 31846 28897 [16087.204885] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16077s workers=2 idle: 653 [16092.059021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16092.062381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16092.096764] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16092.104545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16092.108335] RIP: 0010:native_safe_halt+0x2e/0x50 [16092.112292] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16092.128136] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16092.133803] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16092.138110] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16092.142803] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16092.150256] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16092.160122] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16092.170369] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16092.177728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16092.185781] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16092.191512] Call Trace: [16092.192760] [16092.195605] ? show_regs.cold.9+0x22/0x2f [16092.201804] ? watchdog_timer_fn.cold.6+0x62/0x10a [16092.205754] ? __hrtimer_run_queues+0x1c2/0x4d0 [16092.208824] ? watchdog+0x40/0x40 [16092.214471] ? hrtimer_interrupt+0x128/0x2c0 [16092.219920] ? smp_apic_timer_interrupt+0xa6/0x270 [16092.221766] ? apic_timer_interrupt+0xf/0x20 [16092.231206] [16092.233529] ? native_safe_halt+0x2e/0x50 [16092.236637] kvm_wait+0xcd/0xe0 [16092.239066] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16092.241828] do_raw_spin_lock+0x177/0x190 [16092.247180] _raw_spin_lock+0x21/0x30 [16092.252126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16092.259180] ? do_raw_spin_unlock+0x75/0x190 [16092.261846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16092.264584] ? do_wait_intr+0xf0/0xf0 [16092.266290] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16092.272532] mdc_close+0x53b/0xd70 [mdc] [16092.275656] lmv_close+0x18d/0x490 [lmv] [16092.277526] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16092.280484] ll_md_real_close+0xa5/0x2a0 [lustre] [16092.285750] ll_md_close+0x274/0x800 [lustre] [16092.289807] ll_file_release+0x128/0xc50 [lustre] [16092.293696] __fput+0x10d/0x440 [16092.296639] ____fput+0x12/0x20 [16092.297516] task_work_run+0xc6/0x110 [16092.298751] exit_to_usermode_loop+0x1dd/0x1f0 [16092.303741] do_syscall_64+0x3d6/0x440 [16092.307586] entry_SYSCALL_64_after_hwframe+0x49/0xae [16092.309574] RIP: 0033:0x7fa749552cd8 [16092.310428] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16092.322778] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16092.328896] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16092.336747] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16092.343528] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16092.349977] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16092.354062] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16096.059989] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16096.062658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16096.095927] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16096.106350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16096.112272] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16096.114702] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16096.124505] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16096.128969] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16096.132405] RDX: 0000000000005436 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16096.135150] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16096.138012] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16096.141145] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16096.144545] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16096.147533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16096.151553] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16096.154507] Call Trace: [16096.155544] [16096.156489] ? show_regs.cold.9+0x22/0x2f [16096.157982] ? watchdog_timer_fn.cold.6+0x62/0x10a [16096.159598] ? __hrtimer_run_queues+0x1c2/0x4d0 [16096.161342] ? watchdog+0x40/0x40 [16096.162761] ? hrtimer_interrupt+0x128/0x2c0 [16096.164272] ? smp_apic_timer_interrupt+0xa6/0x270 [16096.166418] ? apic_timer_interrupt+0xf/0x20 [16096.172491] [16096.173960] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16096.177821] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16096.184669] do_raw_spin_lock+0x177/0x190 [16096.188168] _raw_spin_lock+0x21/0x30 [16096.195690] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16096.198983] obd_mod_free+0x46/0x140 [ptlrpc] [16096.203311] mdc_commit_open+0x91/0xb0 [mdc] [16096.207765] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16096.211074] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16096.213406] after_reply+0xe2c/0x1780 [ptlrpc] [16096.218851] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16096.223677] ? __next_timer_interrupt+0x160/0x160 [16096.229184] ptlrpcd+0xa82/0xe40 [ptlrpc] [16096.230952] ? do_wait_intr+0xf0/0xf0 [16096.235174] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16096.240010] kthread+0x1d1/0x200 [16096.241857] ? set_kthread_struct+0x70/0x70 [16096.245271] ret_from_fork+0x1f/0x30 [16112.057462] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16112.060158] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16112.060683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16112.060757] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16112.060763] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16112.060766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16112.060778] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16112.060782] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16112.060786] RAX: 0000000000003c84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16112.060788] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16112.060790] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16112.060792] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16112.060794] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16112.060797] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16112.060800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16112.060802] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16112.060810] Call Trace: [16112.060815] [16112.060820] ? show_regs.cold.9+0x22/0x2f [16112.060829] ? watchdog_timer_fn.cold.6+0x62/0x10a [16112.060834] ? __hrtimer_run_queues+0x1c2/0x4d0 [16112.060838] ? watchdog+0x40/0x40 [16112.060842] ? hrtimer_interrupt+0x128/0x2c0 [16112.060845] ? smp_apic_timer_interrupt+0xa6/0x270 [16112.060852] ? apic_timer_interrupt+0xf/0x20 [16112.060856] [16112.060858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16112.060864] do_raw_spin_lock+0x177/0x190 [16112.060868] _raw_spin_lock+0x21/0x30 [16112.060872] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16112.061464] ? do_raw_spin_unlock+0x75/0x190 [16112.061470] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16112.061714] ? do_wait_intr+0xf0/0xf0 [16112.061721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16112.061924] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16112.062301] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16112.062643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16112.062767] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16112.062812] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16112.062836] mdc_intent_lock+0x353/0x7e0 [mdc] [16112.062861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16112.062943] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16112.063684] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16112.063718] lmv_intent_open+0x338/0x1150 [lmv] [16112.067308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [16112.086235] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16112.086369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16112.086561] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16112.086595] ll_intent_lock+0xc5/0xaa0 [lustre] [16112.086680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16112.086776] ll_lookup_it+0xe0e/0x2270 [lustre] [16112.086858] ? do_raw_spin_unlock+0x75/0x190 [16112.086865] ? _raw_spin_unlock+0x12/0x30 [16112.086871] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16112.086902] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16112.086921] ll_atomic_open+0x36a/0xd70 [lustre] [16112.087004] lookup_open+0x356/0xb40 [16112.087014] do_last+0x313/0xfc0 [16112.087019] ? nd_jump_root+0xe5/0x160 [16112.087024] ? path_init+0x437/0x520 [16112.087028] path_openat+0xf7/0x500 [16112.087032] ? do_raw_spin_unlock+0x75/0x190 [16112.087036] do_filp_open+0x99/0x140 [16112.087041] ? getname_flags+0x6e/0x330 [16112.087045] ? __check_object_size+0xff/0x256 [16112.087051] ? do_raw_spin_unlock+0x75/0x190 [16112.087055] ? _raw_spin_unlock+0x12/0x30 [16112.087058] do_sys_openat2+0x2b4/0x410 [16112.087062] do_sys_open+0x73/0xa0 [16112.087167] __x64_sys_openat+0x24/0x30 [16112.087172] do_syscall_64+0xc1/0x440 [16112.087178] entry_SYSCALL_64_after_hwframe+0x49/0xae [16112.087185] RIP: 0033:0x7fa749552332 [16112.087192] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16112.087196] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16112.087201] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16112.087203] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16112.087205] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16112.087207] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16112.087209] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16112.525989] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16112.542136] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16112.553935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16112.562629] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16112.568904] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16112.577586] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16112.599251] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16112.609387] RAX: 000000000000441b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16112.615251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16112.627101] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16112.631487] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16112.637764] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16112.645537] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16112.651142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16112.652774] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16112.658022] Call Trace: [16112.660759] [16112.662497] ? show_regs.cold.9+0x22/0x2f [16112.664751] ? watchdog_timer_fn.cold.6+0x62/0x10a [16112.670771] ? __hrtimer_run_queues+0x1c2/0x4d0 [16112.672799] ? watchdog+0x40/0x40 [16112.676474] ? hrtimer_interrupt+0x128/0x2c0 [16112.681906] ? smp_apic_timer_interrupt+0xa6/0x270 [16112.687442] ? apic_timer_interrupt+0xf/0x20 [16112.690721] [16112.692412] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16112.699906] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16112.703717] do_raw_spin_lock+0x177/0x190 [16112.706732] _raw_spin_lock+0x21/0x30 [16112.712233] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16112.721983] process_one_work+0x2c8/0x700 [16112.725882] worker_thread+0x296/0x6e0 [16112.733953] ? rescuer_thread+0x570/0x570 [16112.737922] kthread+0x1d1/0x200 [16112.744282] ? set_kthread_struct+0x70/0x70 [16112.746343] ret_from_fork+0x1f/0x30 [16117.217087] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14200s! [16117.221335] Showing busy workqueues and worker pools: [16117.223577] workqueue events: flags=0x0 [16117.225519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16117.228817] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16117.234359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16117.237642] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16117.242650] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.246708] pending: kfree_rcu_monitor [16117.249021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16117.252747] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16117.264082] workqueue events_long: flags=0x0 [16117.278223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.285346] pending: lu_objects_flush [obdclass] [16117.290146] workqueue events_unbound: flags=0x2 [16117.293245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16117.301039] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16117.309120] workqueue events_power_efficient: flags=0x80 [16117.316018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.318569] pending: check_lifetime [16117.324594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.328193] pending: neigh_periodic_work [16117.330297] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.337075] pending: neigh_periodic_work [16117.339774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.343289] pending: do_cache_clean [16117.345230] workqueue mm_percpu_wq: flags=0x8 [16117.347279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.354080] pending: vmstat_update [16117.355696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.359476] pending: vmstat_update [16117.361781] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16117.365566] pending: vmstat_update [16117.370333] workqueue writeback: flags=0x4a [16117.376561] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16117.378671] pending: wb_workfn, wb_workfn [16117.381345] workqueue xprtiod: flags=0xa [16117.384293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16117.386784] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16117.391105] workqueue ipv6_addrconf: flags=0x40008 [16117.393464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16117.397158] pending: addrconf_verify_work [16117.399264] workqueue knbd0-recv: flags=0x1a [16117.405156] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16117.409480] in-flight: 651:recv_work [16117.413667] workqueue ptlrpc_pinger: flags=0x2 [16117.416512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16117.421403] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16117.425222] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14200s workers=4 idle: 37 31846 28897 [16117.436446] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16107s workers=2 idle: 653 [16120.059021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16120.061445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16120.075539] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16120.078316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16120.080799] RIP: 0010:native_safe_halt+0x2e/0x50 [16120.082104] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16120.087090] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16120.089507] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16120.091483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16120.093824] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16120.095694] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16120.097793] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16120.099793] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16120.101924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16120.103932] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16120.107533] Call Trace: [16120.108571] [16120.109293] ? show_regs.cold.9+0x22/0x2f [16120.110974] ? watchdog_timer_fn.cold.6+0x62/0x10a [16120.113141] ? __hrtimer_run_queues+0x1c2/0x4d0 [16120.115245] ? watchdog+0x40/0x40 [16120.118245] ? hrtimer_interrupt+0x128/0x2c0 [16120.126737] ? smp_apic_timer_interrupt+0xa6/0x270 [16120.129246] ? apic_timer_interrupt+0xf/0x20 [16120.130954] [16120.132116] ? native_safe_halt+0x2e/0x50 [16120.133135] kvm_wait+0xcd/0xe0 [16120.133940] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16120.135006] do_raw_spin_lock+0x177/0x190 [16120.135817] _raw_spin_lock+0x21/0x30 [16120.136540] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16120.138767] ? do_raw_spin_unlock+0x75/0x190 [16120.139805] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16120.141285] ? do_wait_intr+0xf0/0xf0 [16120.142235] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16120.171445] mdc_close+0x53b/0xd70 [mdc] [16120.172986] lmv_close+0x18d/0x490 [lmv] [16120.174907] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16120.178016] ll_md_real_close+0xa5/0x2a0 [lustre] [16120.183275] ll_md_close+0x274/0x800 [lustre] [16120.185893] ll_file_release+0x128/0xc50 [lustre] [16120.189057] __fput+0x10d/0x440 [16120.190147] ____fput+0x12/0x20 [16120.191268] task_work_run+0xc6/0x110 [16120.193324] exit_to_usermode_loop+0x1dd/0x1f0 [16120.196014] do_syscall_64+0x3d6/0x440 [16120.197926] entry_SYSCALL_64_after_hwframe+0x49/0xae [16120.200091] RIP: 0033:0x7fa749552cd8 [16120.201595] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16120.208121] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16120.210379] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16120.212311] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16120.216782] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16120.219534] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16120.222572] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16124.058029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16124.062157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16124.092643] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16124.098082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16124.102389] RIP: 0010:native_safe_halt+0x2e/0x50 [16124.105000] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16124.114985] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16124.120910] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16124.127088] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16124.131900] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16124.139016] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16124.146242] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16124.150495] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16124.154257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16124.160233] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16124.170186] Call Trace: [16124.175810] [16124.177246] ? show_regs.cold.9+0x22/0x2f [16124.184904] ? watchdog_timer_fn.cold.6+0x62/0x10a [16124.192572] ? __hrtimer_run_queues+0x1c2/0x4d0 [16124.198105] ? watchdog+0x40/0x40 [16124.201461] ? hrtimer_interrupt+0x128/0x2c0 [16124.208932] ? smp_apic_timer_interrupt+0xa6/0x270 [16124.214502] ? apic_timer_interrupt+0xf/0x20 [16124.216676] [16124.222701] ? native_safe_halt+0x2e/0x50 [16124.225030] kvm_wait+0xcd/0xe0 [16124.227644] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16124.230580] do_raw_spin_lock+0x177/0x190 [16124.232379] _raw_spin_lock+0x21/0x30 [16124.235384] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16124.237858] obd_mod_free+0x46/0x140 [ptlrpc] [16124.242246] mdc_commit_open+0x91/0xb0 [mdc] [16124.243883] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16124.247021] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16124.250692] after_reply+0xe2c/0x1780 [ptlrpc] [16124.254410] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16124.259029] ? __next_timer_interrupt+0x160/0x160 [16124.260742] ptlrpcd+0xa82/0xe40 [ptlrpc] [16124.264551] ? do_wait_intr+0xf0/0xf0 [16124.266934] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16124.270909] kthread+0x1d1/0x200 [16124.274383] ? set_kthread_struct+0x70/0x70 [16124.275965] ret_from_fork+0x1f/0x30 [16140.057244] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16140.065669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16140.102904] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16140.117831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16140.124813] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16140.131587] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16140.135392] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16140.148180] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16140.152082] RAX: 0000000000007ecf RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16140.154157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16140.156336] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16140.159085] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16140.163246] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16140.166404] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16140.169765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16140.172155] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16140.176416] Call Trace: [16140.177595] [16140.178567] ? show_regs.cold.9+0x22/0x2f [16140.180163] ? watchdog_timer_fn.cold.6+0x62/0x10a [16140.182361] ? __hrtimer_run_queues+0x1c2/0x4d0 [16140.188078] ? watchdog+0x40/0x40 [16140.189170] ? hrtimer_interrupt+0x128/0x2c0 [16140.191401] ? smp_apic_timer_interrupt+0xa6/0x270 [16140.194216] ? apic_timer_interrupt+0xf/0x20 [16140.196290] [16140.197322] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16140.200107] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16140.203503] do_raw_spin_lock+0x177/0x190 [16140.205427] _raw_spin_lock+0x21/0x30 [16140.206898] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16140.212489] process_one_work+0x2c8/0x700 [16140.213642] worker_thread+0x296/0x6e0 [16140.215347] ? rescuer_thread+0x570/0x570 [16140.216796] kthread+0x1d1/0x200 [16140.220584] ? set_kthread_struct+0x70/0x70 [16140.222105] ret_from_fork+0x1f/0x30 [16144.055292] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16144.063167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16144.108024] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16144.127217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16144.135213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16144.139830] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16144.158234] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [16144.164839] RAX: 0000000000004494 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16144.169303] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16144.173980] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16144.180630] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16144.187734] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16144.191862] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16144.201688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16144.206732] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16144.211315] Call Trace: [16144.212882] [16144.214684] ? show_regs.cold.9+0x22/0x2f [16144.217300] ? watchdog_timer_fn.cold.6+0x62/0x10a [16144.220310] ? __hrtimer_run_queues+0x1c2/0x4d0 [16144.227635] ? watchdog+0x40/0x40 [16144.231668] ? hrtimer_interrupt+0x128/0x2c0 [16144.235231] ? smp_apic_timer_interrupt+0xa6/0x270 [16144.239695] ? apic_timer_interrupt+0xf/0x20 [16144.245568] [16144.246171] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16144.247891] do_raw_spin_lock+0x177/0x190 [16144.256019] _raw_spin_lock+0x21/0x30 [16144.260026] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16144.265006] ? do_raw_spin_unlock+0x75/0x190 [16144.267075] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16144.269490] ? do_wait_intr+0xf0/0xf0 [16144.273890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16144.279913] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16144.281914] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16144.291807] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16144.296360] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16144.300415] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16144.305717] mdc_intent_lock+0x353/0x7e0 [mdc] [16144.314008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16144.318312] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16144.320351] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16144.327661] lmv_intent_open+0x338/0x1150 [lmv] [16144.333715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16144.338727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16144.342757] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16144.346718] ll_intent_lock+0xc5/0xaa0 [lustre] [16144.352964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16144.358435] ll_lookup_it+0xe0e/0x2270 [lustre] [16144.361054] ? do_raw_spin_unlock+0x75/0x190 [16144.363845] ? _raw_spin_unlock+0x12/0x30 [16144.365331] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16144.368150] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16144.370502] ll_atomic_open+0x36a/0xd70 [lustre] [16144.371989] lookup_open+0x356/0xb40 [16144.373060] do_last+0x313/0xfc0 [16144.373877] ? nd_jump_root+0xe5/0x160 [16144.375032] ? path_init+0x437/0x520 [16144.376497] path_openat+0xf7/0x500 [16144.377627] ? do_raw_spin_unlock+0x75/0x190 [16144.378799] do_filp_open+0x99/0x140 [16144.379755] ? getname_flags+0x6e/0x330 [16144.380964] ? __check_object_size+0xff/0x256 [16144.382051] ? do_raw_spin_unlock+0x75/0x190 [16144.383137] ? _raw_spin_unlock+0x12/0x30 [16144.384060] do_sys_openat2+0x2b4/0x410 [16144.384992] do_sys_open+0x73/0xa0 [16144.385768] __x64_sys_openat+0x24/0x30 [16144.386788] do_syscall_64+0xc1/0x440 [16144.387809] entry_SYSCALL_64_after_hwframe+0x49/0xae [16144.389080] RIP: 0033:0x7fa749552332 [16144.390054] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16144.394283] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16144.396404] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16144.401356] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16144.405772] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16144.410370] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16144.417387] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16147.937229] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14230s! [16147.942155] Showing busy workqueues and worker pools: [16147.944464] workqueue events: flags=0x0 [16147.945770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16147.948741] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16147.955596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16147.959183] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16147.962195] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16147.965117] pending: kfree_rcu_monitor [16147.966696] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16147.969774] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16147.975955] workqueue events_long: flags=0x0 [16147.978188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16147.982623] pending: lu_objects_flush [obdclass] [16147.985221] workqueue events_unbound: flags=0x2 [16147.987700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16147.990779] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16147.995061] workqueue events_power_efficient: flags=0x80 [16147.997220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16148.002108] pending: check_lifetime [16148.004018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16148.007276] pending: neigh_periodic_work [16148.009201] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16148.013570] pending: neigh_periodic_work [16148.015498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16148.018048] pending: do_cache_clean [16148.019788] workqueue mm_percpu_wq: flags=0x8 [16148.022685] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16148.025111] pending: vmstat_update [16148.030599] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16148.034135] pending: vmstat_update [16148.036632] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16148.039036] pending: vmstat_update [16148.043797] workqueue writeback: flags=0x4a [16148.045644] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16148.048250] pending: wb_workfn, wb_workfn [16148.050807] workqueue xprtiod: flags=0xa [16148.052706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16148.056373] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16148.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16148.062327] workqueue ipv6_addrconf: flags=0x40008 [16148.065619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [16148.073443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16148.073468] pending: addrconf_verify_work [16148.074688] workqueue knbd0-recv: flags=0x1a [16148.078437] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16148.078511] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16148.078518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16148.078521] RIP: 0010:native_safe_halt+0x2e/0x50 [16148.078534] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16148.078541] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16148.078545] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16148.078548] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16148.078550] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16148.078552] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16148.078554] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16148.078558] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16148.078561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16148.078563] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16148.078572] Call Trace: [16148.078577] [16148.078583] ? show_regs.cold.9+0x22/0x2f [16148.078593] ? watchdog_timer_fn.cold.6+0x62/0x10a [16148.078599] ? __hrtimer_run_queues+0x1c2/0x4d0 [16148.078603] ? watchdog+0x40/0x40 [16148.078607] ? hrtimer_interrupt+0x128/0x2c0 [16148.078610] ? smp_apic_timer_interrupt+0xa6/0x270 [16148.078617] ? apic_timer_interrupt+0xf/0x20 [16148.078622] [16148.078624] ? native_safe_halt+0x2e/0x50 [16148.078628] kvm_wait+0xcd/0xe0 [16148.078635] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16148.078642] do_raw_spin_lock+0x177/0x190 [16148.078646] _raw_spin_lock+0x21/0x30 [16148.078655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16148.078935] ? do_raw_spin_unlock+0x75/0x190 [16148.078940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16148.079141] ? do_wait_intr+0xf0/0xf0 [16148.079149] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16148.079343] mdc_close+0x53b/0xd70 [mdc] [16148.079401] lmv_close+0x18d/0x490 [lmv] [16148.079432] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16148.079845] ll_md_real_close+0xa5/0x2a0 [lustre] [16148.079925] ll_md_close+0x274/0x800 [lustre] [16148.080007] ll_file_release+0x128/0xc50 [lustre] [16148.080086] __fput+0x10d/0x440 [16148.080095] ____fput+0x12/0x20 [16148.080100] task_work_run+0xc6/0x110 [16148.080107] exit_to_usermode_loop+0x1dd/0x1f0 [16148.080114] do_syscall_64+0x3d6/0x440 [16148.080130] entry_SYSCALL_64_after_hwframe+0x49/0xae [16148.080137] RIP: 0033:0x7fa749552cd8 [16148.080145] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16148.080149] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16148.080153] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16148.080155] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16148.080157] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16148.080160] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16148.080162] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16148.358957] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16148.363582] in-flight: 651:recv_work [16148.364710] workqueue ptlrpc_pinger: flags=0x2 [16148.373955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16148.375978] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16148.383367] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14231s workers=4 idle: 37 31846 28897 [16148.391274] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16138s workers=2 idle: 653 [16152.058007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16152.062740] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16152.094821] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16152.104234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16152.118542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16152.123343] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16152.139199] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16152.142733] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16152.146837] RDX: 0000000000007c1e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16152.149435] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16152.153189] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16152.156597] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16152.159671] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16152.167517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16152.175201] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16152.179548] Call Trace: [16152.180655] [16152.181733] ? show_regs.cold.9+0x22/0x2f [16152.184480] ? watchdog_timer_fn.cold.6+0x62/0x10a [16152.186596] ? __hrtimer_run_queues+0x1c2/0x4d0 [16152.189535] ? watchdog+0x40/0x40 [16152.191080] ? hrtimer_interrupt+0x128/0x2c0 [16152.197407] ? smp_apic_timer_interrupt+0xa6/0x270 [16152.200743] ? apic_timer_interrupt+0xf/0x20 [16152.204142] [16152.205068] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16152.209280] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16152.213667] do_raw_spin_lock+0x177/0x190 [16152.216553] _raw_spin_lock+0x21/0x30 [16152.219354] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16152.222063] obd_mod_free+0x46/0x140 [ptlrpc] [16152.224468] mdc_commit_open+0x91/0xb0 [mdc] [16152.226632] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16152.229611] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16152.235630] after_reply+0xe2c/0x1780 [ptlrpc] [16152.238247] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16152.244318] ? __next_timer_interrupt+0x160/0x160 [16152.246425] ptlrpcd+0xa82/0xe40 [ptlrpc] [16152.249099] ? do_wait_intr+0xf0/0xf0 [16152.250748] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16152.253501] kthread+0x1d1/0x200 [16152.256196] ? set_kthread_struct+0x70/0x70 [16152.258379] ret_from_fork+0x1f/0x30 [16168.057674] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16168.065612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16168.094605] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16168.100718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16168.107156] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16168.113653] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16168.115730] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16168.123527] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16168.128886] RAX: 0000000000006f84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16168.132950] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16168.139163] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16168.147614] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16168.155907] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16168.164203] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16168.168219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16168.171717] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16168.175828] Call Trace: [16168.178650] [16168.186913] ? show_regs.cold.9+0x22/0x2f [16168.188117] ? watchdog_timer_fn.cold.6+0x62/0x10a [16168.192109] ? __hrtimer_run_queues+0x1c2/0x4d0 [16168.193985] ? watchdog+0x40/0x40 [16168.195585] ? hrtimer_interrupt+0x128/0x2c0 [16168.198926] ? smp_apic_timer_interrupt+0xa6/0x270 [16168.200329] ? apic_timer_interrupt+0xf/0x20 [16168.201471] [16168.207299] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16168.211913] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16168.215636] do_raw_spin_lock+0x177/0x190 [16168.218733] _raw_spin_lock+0x21/0x30 [16168.219740] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16168.223684] process_one_work+0x2c8/0x700 [16168.227614] worker_thread+0x296/0x6e0 [16168.231057] ? rescuer_thread+0x570/0x570 [16168.234140] kthread+0x1d1/0x200 [16168.235691] ? set_kthread_struct+0x70/0x70 [16168.238131] ret_from_fork+0x1f/0x30 [16176.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16176.056464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16176.059092] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16176.088008] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16176.088027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16176.088031] RIP: 0010:native_safe_halt+0x2e/0x50 [16176.088048] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16176.088053] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16176.088059] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16176.088062] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16176.088064] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16176.088066] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16176.088068] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16176.088072] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16176.088075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16176.088077] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16176.088086] Call Trace: [16176.088093] [16176.090281] ? show_regs.cold.9+0x22/0x2f [16176.090312] ? watchdog_timer_fn.cold.6+0x62/0x10a [16176.090320] ? __hrtimer_run_queues+0x1c2/0x4d0 [16176.090325] ? watchdog+0x40/0x40 [16176.090329] ? hrtimer_interrupt+0x128/0x2c0 [16176.090334] ? smp_apic_timer_interrupt+0xa6/0x270 [16176.090416] ? apic_timer_interrupt+0xf/0x20 [16176.090421] [16176.090495] ? native_safe_halt+0x2e/0x50 [16176.090502] kvm_wait+0xcd/0xe0 [16176.090514] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16176.090527] do_raw_spin_lock+0x177/0x190 [16176.090533] _raw_spin_lock+0x21/0x30 [16176.090538] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16176.090864] ? do_raw_spin_unlock+0x75/0x190 [16176.090871] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16176.091065] ? do_wait_intr+0xf0/0xf0 [16176.091075] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16176.107564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [16176.118964] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16176.141676] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16176.141793] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16176.141802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16176.141805] RIP: 0010:native_safe_halt+0x2e/0x50 [16176.141823] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16176.141828] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16176.141833] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16176.141835] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16176.141838] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16176.141840] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16176.141843] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16176.141847] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16176.141850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16176.141853] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16176.141861] Call Trace: [16176.141868] [16176.141876] ? show_regs.cold.9+0x22/0x2f [16176.141888] ? watchdog_timer_fn.cold.6+0x62/0x10a [16176.141895] ? __hrtimer_run_queues+0x1c2/0x4d0 [16176.141900] ? watchdog+0x40/0x40 [16176.141905] ? hrtimer_interrupt+0x128/0x2c0 [16176.141909] ? smp_apic_timer_interrupt+0xa6/0x270 [16176.141917] ? apic_timer_interrupt+0xf/0x20 [16176.141921] [16176.141924] ? native_safe_halt+0x2e/0x50 [16176.141928] kvm_wait+0xcd/0xe0 [16176.141937] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16176.141951] do_raw_spin_lock+0x177/0x190 [16176.141957] _raw_spin_lock+0x21/0x30 [16176.141961] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16176.154403] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16176.154755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16176.154880] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16176.154920] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16176.154952] mdc_intent_lock+0x353/0x7e0 [mdc] [16176.154981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16176.155070] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16176.155621] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16176.155664] lmv_intent_open+0x338/0x1150 [lmv] [16176.155713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16176.155806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16176.155889] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16176.155912] ll_intent_lock+0xc5/0xaa0 [lustre] [16176.155995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16176.156078] ll_lookup_it+0xe0e/0x2270 [lustre] [16176.156606] ? do_raw_spin_unlock+0x75/0x190 [16176.156618] ? _raw_spin_unlock+0x12/0x30 [16176.156626] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16176.156664] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16176.156686] ll_atomic_open+0x36a/0xd70 [lustre] [16176.156771] lookup_open+0x356/0xb40 [16176.156781] do_last+0x313/0xfc0 [16176.156787] ? nd_jump_root+0xe5/0x160 [16176.156792] ? path_init+0x437/0x520 [16176.156798] path_openat+0xf7/0x500 [16176.156803] ? do_raw_spin_unlock+0x75/0x190 [16176.156808] do_filp_open+0x99/0x140 [16176.156815] ? getname_flags+0x6e/0x330 [16176.156820] ? __check_object_size+0xff/0x256 [16176.156830] ? do_raw_spin_unlock+0x75/0x190 [16176.156834] ? _raw_spin_unlock+0x12/0x30 [16176.156839] do_sys_openat2+0x2b4/0x410 [16176.156844] do_sys_open+0x73/0xa0 [16176.156849] __x64_sys_openat+0x24/0x30 [16176.156853] do_syscall_64+0xc1/0x440 [16176.156861] entry_SYSCALL_64_after_hwframe+0x49/0xae [16176.156867] RIP: 0033:0x7fa749552332 [16176.156877] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16176.156881] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16176.156886] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16176.156889] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16176.156891] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16176.156894] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16176.156896] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16176.722598] ? do_raw_spin_unlock+0x75/0x190 [16176.724675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16176.727932] ? do_wait_intr+0xf0/0xf0 [16176.728917] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16176.730411] mdc_close+0x53b/0xd70 [mdc] [16176.736042] lmv_close+0x18d/0x490 [lmv] [16176.737671] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16176.740150] ll_md_real_close+0xa5/0x2a0 [lustre] [16176.742842] ll_md_close+0x274/0x800 [lustre] [16176.745612] ll_file_release+0x128/0xc50 [lustre] [16176.747545] __fput+0x10d/0x440 [16176.748376] ____fput+0x12/0x20 [16176.749191] task_work_run+0xc6/0x110 [16176.752033] exit_to_usermode_loop+0x1dd/0x1f0 [16176.753968] do_syscall_64+0x3d6/0x440 [16176.756374] entry_SYSCALL_64_after_hwframe+0x49/0xae [16176.758987] RIP: 0033:0x7fa749552cd8 [16176.761322] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16176.770791] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16176.778406] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16176.781623] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16176.787848] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16176.793927] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16176.797865] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16178.657938] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14261s! [16178.665387] Showing busy workqueues and worker pools: [16178.673898] workqueue events: flags=0x0 [16178.677197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16178.683104] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16178.689693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16178.698264] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16178.709575] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.713385] pending: kfree_rcu_monitor [16178.717231] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16178.729717] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16178.739963] workqueue events_long: flags=0x0 [16178.747179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.754132] pending: lu_objects_flush [obdclass] [16178.760051] workqueue events_unbound: flags=0x2 [16178.767565] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16178.776902] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16178.785127] workqueue events_power_efficient: flags=0x80 [16178.787098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.791098] pending: check_lifetime [16178.792219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.797357] pending: neigh_periodic_work [16178.802660] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.807929] pending: neigh_periodic_work [16178.813633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.823491] pending: do_cache_clean [16178.833223] workqueue mm_percpu_wq: flags=0x8 [16178.836241] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.845970] pending: vmstat_update [16178.850014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.857016] pending: vmstat_update [16178.859038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16178.862190] pending: vmstat_update [16178.864085] workqueue writeback: flags=0x4a [16178.865960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16178.870533] pending: wb_workfn, wb_workfn [16178.873092] workqueue xprtiod: flags=0xa [16178.875232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16178.878700] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16178.887600] workqueue ipv6_addrconf: flags=0x40008 [16178.893908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16178.899651] pending: addrconf_verify_work [16178.901233] workqueue knbd0-recv: flags=0x1a [16178.914084] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16178.919713] in-flight: 651:recv_work [16178.925669] workqueue ptlrpc_pinger: flags=0x2 [16178.931274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16178.938184] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16178.941091] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14261s workers=4 idle: 37 31846 28897 [16178.949181] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16168s workers=2 idle: 653 [16180.058614] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16180.064207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16180.096152] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16180.104004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16180.111486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16180.116374] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16180.132877] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16180.137816] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16180.142374] RDX: 0000000000001bf2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16180.145946] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16180.150283] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16180.154150] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16180.158888] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16180.163246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16180.166951] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16180.170321] Call Trace: [16180.171413] [16180.172395] ? show_regs.cold.9+0x22/0x2f [16180.174396] ? watchdog_timer_fn.cold.6+0x62/0x10a [16180.176741] ? __hrtimer_run_queues+0x1c2/0x4d0 [16180.178479] ? watchdog+0x40/0x40 [16180.180473] ? hrtimer_interrupt+0x128/0x2c0 [16180.182239] ? smp_apic_timer_interrupt+0xa6/0x270 [16180.184883] ? apic_timer_interrupt+0xf/0x20 [16180.186331] [16180.187123] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16180.189417] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16180.191410] do_raw_spin_lock+0x177/0x190 [16180.193417] _raw_spin_lock+0x21/0x30 [16180.195126] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16180.197378] obd_mod_free+0x46/0x140 [ptlrpc] [16180.199465] mdc_commit_open+0x91/0xb0 [mdc] [16180.201044] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16180.203260] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16180.205471] after_reply+0xe2c/0x1780 [ptlrpc] [16180.211015] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16180.215433] ? __next_timer_interrupt+0x160/0x160 [16180.216809] ptlrpcd+0xa82/0xe40 [ptlrpc] [16180.218538] ? do_wait_intr+0xf0/0xf0 [16180.219778] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16180.221980] kthread+0x1d1/0x200 [16180.223103] ? set_kthread_struct+0x70/0x70 [16180.224497] ret_from_fork+0x1f/0x30 [16194.006225] rcu: INFO: rcu_sched self-detected stall on CPU [16194.008624] rcu: 2-...!: (8381359 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [16194.019900] (t=14280317 jiffies g=258701 q=5333) [16194.022245] rcu: rcu_sched kthread starved for 14277112 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [16194.035514] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16194.043373] rcu: RCU grace-period kthread stack dump: [16194.045601] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16194.050146] Call Trace: [16194.051330] __schedule+0x351/0xcb0 [16194.052867] schedule+0xc0/0x180 [16194.054428] schedule_timeout+0xb4/0x190 [16194.056257] ? do_raw_spin_unlock+0x75/0x190 [16194.058232] ? __next_timer_interrupt+0x160/0x160 [16194.060427] ? prepare_to_swait_event+0xf3/0x140 [16194.063274] rcu_gp_kthread+0x8dd/0x15b0 [16194.064288] ? force_qs_rnp+0x2c0/0x2c0 [16194.066133] kthread+0x1d1/0x200 [16194.071692] ? set_kthread_struct+0x70/0x70 [16194.074327] ret_from_fork+0x1f/0x30 [16194.079361] rcu: Stack dump where RCU GP kthread last ran: [16194.082285] Sending NMI from CPU 2 to CPUs 0: [16194.088429] NMI backtrace for cpu 0 [16194.088436] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16194.088438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16194.088440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16194.088442] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16194.088445] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 [16194.088448] RAX: 00000000000060df RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16194.088449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16194.088451] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16194.088452] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16194.088454] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16194.088455] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16194.088457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16194.088458] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16194.088459] Call Trace: [16194.088460] [16194.088461] ? show_regs.cold.9+0x22/0x2f [16194.088463] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [16194.088464] ? nmi_cpu_backtrace_handler+0x15/0x40 [16194.088465] ? nmi_handle+0xbd/0x210 [16194.088466] ? default_do_nmi+0x4b/0x1b0 [16194.088467] ? do_nmi+0x280/0x380 [16194.088468] ? end_repeat_nmi+0x16/0x1f [16194.088469] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16194.088471] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16194.088472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16194.088473] [16194.088473] do_raw_spin_lock+0x177/0x190 [16194.088475] _raw_spin_lock+0x21/0x30 [16194.088476] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16194.088477] ? do_raw_spin_unlock+0x75/0x190 [16194.088478] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16194.088479] ? do_wait_intr+0xf0/0xf0 [16194.088480] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16194.088482] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16194.088483] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16194.088484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.088485] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16194.088486] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16194.088487] mdc_intent_lock+0x353/0x7e0 [mdc] [16194.088488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.088489] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16194.088490] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16194.088492] lmv_intent_open+0x338/0x1150 [lmv] [16194.088493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.088494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.088496] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16194.088497] ll_intent_lock+0xc5/0xaa0 [lustre] [16194.088498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.088499] ll_lookup_it+0xe0e/0x2270 [lustre] [16194.088500] ? do_raw_spin_unlock+0x75/0x190 [16194.088501] ? _raw_spin_unlock+0x12/0x30 [16194.088503] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16194.088504] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16194.088505] ll_atomic_open+0x36a/0xd70 [lustre] [16194.088507] lookup_open+0x356/0xb40 [16194.088508] do_last+0x313/0xfc0 [16194.088508] ? nd_jump_root+0xe5/0x160 [16194.088510] ? path_init+0x437/0x520 [16194.088510] path_openat+0xf7/0x500 [16194.088511] ? do_raw_spin_unlock+0x75/0x190 [16194.088512] do_filp_open+0x99/0x140 [16194.088513] ? getname_flags+0x6e/0x330 [16194.088514] ? __check_object_size+0xff/0x256 [16194.088515] ? do_raw_spin_unlock+0x75/0x190 [16194.088516] ? _raw_spin_unlock+0x12/0x30 [16194.088517] do_sys_openat2+0x2b4/0x410 [16194.088518] do_sys_open+0x73/0xa0 [16194.088519] __x64_sys_openat+0x24/0x30 [16194.088520] do_syscall_64+0xc1/0x440 [16194.088522] entry_SYSCALL_64_after_hwframe+0x49/0xae [16194.088523] RIP: 0033:0x7fa749552332 [16194.088524] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16194.088527] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16194.088530] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16194.088531] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16194.088532] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16194.088534] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16194.088535] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16194.089331] NMI backtrace for cpu 2 [16194.310479] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16194.314352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16194.317287] Call Trace: [16194.318282] [16194.319383] ? dump_stack+0xbb/0x10e [16194.320988] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16194.323604] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16194.325444] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16194.327575] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16194.329608] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16194.331628] ? rcu_dump_cpu_stacks+0x11b/0x171 [16194.333377] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16194.335642] ? update_process_times+0x6d/0xc0 [16194.337559] ? tick_sched_handle.isra.18+0x27/0x90 [16194.339297] ? tick_sched_timer+0x53/0xb0 [16194.340788] ? __hrtimer_run_queues+0x1c2/0x4d0 [16194.342422] ? tick_sched_handle.isra.18+0x90/0x90 [16194.344075] ? hrtimer_interrupt+0x128/0x2c0 [16194.345553] ? smp_apic_timer_interrupt+0xa6/0x270 [16194.347485] ? apic_timer_interrupt+0xf/0x20 [16194.349105] [16194.350140] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16194.351967] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16194.353910] ? do_raw_spin_lock+0x177/0x190 [16194.355575] ? _raw_spin_lock+0x21/0x30 [16194.356570] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16194.358378] ? obd_mod_free+0x46/0x140 [ptlrpc] [16194.360117] ? mdc_commit_open+0x91/0xb0 [mdc] [16194.361756] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16194.363883] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16194.365952] ? after_reply+0xe2c/0x1780 [ptlrpc] [16194.367592] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16194.369802] ? __next_timer_interrupt+0x160/0x160 [16194.371965] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16194.373559] ? do_wait_intr+0xf0/0xf0 [16194.374852] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16194.377160] ? kthread+0x1d1/0x200 [16194.379659] ? set_kthread_struct+0x70/0x70 [16194.381075] ? ret_from_fork+0x1f/0x30 [16196.057082] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16196.061682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16196.091805] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16196.096661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16196.101554] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16196.104892] RIP: 0010:native_safe_halt+0x2e/0x50 [16196.107815] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16196.119812] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16196.125775] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16196.133505] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16196.136349] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16196.145095] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16196.151597] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16196.158023] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16196.168165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16196.174825] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16196.186707] Call Trace: [16196.187629] [16196.188272] ? show_regs.cold.9+0x22/0x2f [16196.192580] ? watchdog_timer_fn.cold.6+0x62/0x10a [16196.195136] ? __hrtimer_run_queues+0x1c2/0x4d0 [16196.197293] ? watchdog+0x40/0x40 [16196.198626] ? hrtimer_interrupt+0x128/0x2c0 [16196.200607] ? smp_apic_timer_interrupt+0xa6/0x270 [16196.203947] ? apic_timer_interrupt+0xf/0x20 [16196.206794] [16196.208791] ? native_safe_halt+0x2e/0x50 [16196.212569] kvm_wait+0xcd/0xe0 [16196.215190] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16196.219378] do_raw_spin_lock+0x177/0x190 [16196.221620] _raw_spin_lock+0x21/0x30 [16196.227201] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16196.230323] process_one_work+0x2c8/0x700 [16196.232585] worker_thread+0x296/0x6e0 [16196.235504] ? rescuer_thread+0x570/0x570 [16196.237546] kthread+0x1d1/0x200 [16196.239477] ? set_kthread_struct+0x70/0x70 [16196.241436] ret_from_fork+0x1f/0x30 [16204.059364] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16204.064145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16204.081413] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16204.085243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16204.088733] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16204.090835] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16204.098101] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16204.101228] RAX: 0000000000005893 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16204.104735] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16204.107304] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16204.110753] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16204.114561] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16204.117131] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16204.119774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16204.121772] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16204.124354] Call Trace: [16204.125304] [16204.126233] ? show_regs.cold.9+0x22/0x2f [16204.127721] ? watchdog_timer_fn.cold.6+0x62/0x10a [16204.129450] ? __hrtimer_run_queues+0x1c2/0x4d0 [16204.131000] ? watchdog+0x40/0x40 [16204.132231] ? hrtimer_interrupt+0x128/0x2c0 [16204.133727] ? smp_apic_timer_interrupt+0xa6/0x270 [16204.135748] ? apic_timer_interrupt+0xf/0x20 [16204.137222] [16204.137980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16204.140132] do_raw_spin_lock+0x177/0x190 [16204.141476] _raw_spin_lock+0x21/0x30 [16204.142738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16204.145089] ? do_raw_spin_unlock+0x75/0x190 [16204.146586] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16204.148681] ? do_wait_intr+0xf0/0xf0 [16204.150237] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16204.152578] mdc_close+0x53b/0xd70 [mdc] [16204.154236] lmv_close+0x18d/0x490 [lmv] [16204.155636] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16204.157838] ll_md_real_close+0xa5/0x2a0 [lustre] [16204.159997] ll_md_close+0x274/0x800 [lustre] [16204.162286] ll_file_release+0x128/0xc50 [lustre] [16204.169165] __fput+0x10d/0x440 [16204.170776] ____fput+0x12/0x20 [16204.174001] task_work_run+0xc6/0x110 [16204.175654] exit_to_usermode_loop+0x1dd/0x1f0 [16204.177452] do_syscall_64+0x3d6/0x440 [16204.179074] entry_SYSCALL_64_after_hwframe+0x49/0xae [16204.182229] RIP: 0033:0x7fa749552cd8 [16204.183785] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16204.195048] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16204.200658] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16204.204303] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16204.210556] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16204.218247] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16204.224293] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16209.378591] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14292s! [16209.382545] Showing busy workqueues and worker pools: [16209.393303] workqueue events: flags=0x0 [16209.396256] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16209.400150] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16209.406283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16209.410165] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16209.418502] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.423430] pending: kfree_rcu_monitor [16209.427333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16209.431264] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16209.436296] workqueue events_long: flags=0x0 [16209.438838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.442497] pending: lu_objects_flush [obdclass] [16209.444341] workqueue events_unbound: flags=0x2 [16209.450339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16209.453586] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16209.459584] workqueue events_power_efficient: flags=0x80 [16209.462696] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.465642] pending: check_lifetime [16209.467419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.470663] pending: neigh_periodic_work [16209.472913] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.475811] pending: neigh_periodic_work [16209.477362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.482327] pending: do_cache_clean [16209.484302] workqueue mm_percpu_wq: flags=0x8 [16209.485805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.491860] pending: vmstat_update [16209.493542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.503907] pending: vmstat_update [16209.506491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16209.511036] pending: vmstat_update [16209.513209] workqueue writeback: flags=0x4a [16209.515198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16209.518317] pending: wb_workfn, wb_workfn [16209.524684] workqueue xprtiod: flags=0xa [16209.526299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16209.528808] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16209.533566] workqueue ipv6_addrconf: flags=0x40008 [16209.538229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16209.541538] pending: addrconf_verify_work [16209.549156] workqueue knbd0-recv: flags=0x1a [16209.554393] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16209.566170] in-flight: 651:recv_work [16209.568806] workqueue ptlrpc_pinger: flags=0x2 [16209.572741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16209.582544] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16209.587642] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14292s workers=4 idle: 37 31846 28897 [16209.593123] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16199s workers=2 idle: 653 [16220.058511] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16220.067489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16220.114543] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16220.119164] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16220.123448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16220.126272] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16220.146276] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16220.150261] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16220.154295] RDX: 0000000000003d17 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16220.158759] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16220.163511] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16220.167427] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16220.174423] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16220.180794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16220.182756] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16220.187438] Call Trace: [16220.188372] [16220.189568] ? show_regs.cold.9+0x22/0x2f [16220.192808] ? watchdog_timer_fn.cold.6+0x62/0x10a [16220.195406] ? __hrtimer_run_queues+0x1c2/0x4d0 [16220.197272] ? watchdog+0x40/0x40 [16220.198343] ? hrtimer_interrupt+0x128/0x2c0 [16220.205099] ? smp_apic_timer_interrupt+0xa6/0x270 [16220.207100] ? apic_timer_interrupt+0xf/0x20 [16220.209426] [16220.210908] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16220.213195] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16220.215768] do_raw_spin_lock+0x177/0x190 [16220.217895] _raw_spin_lock+0x21/0x30 [16220.221665] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16220.225737] obd_mod_free+0x46/0x140 [ptlrpc] [16220.228701] mdc_commit_open+0x91/0xb0 [mdc] [16220.231028] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16220.234276] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16220.237547] after_reply+0xe2c/0x1780 [ptlrpc] [16220.242369] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16220.248873] ? __next_timer_interrupt+0x160/0x160 [16220.253016] ptlrpcd+0xa82/0xe40 [ptlrpc] [16220.256548] ? do_wait_intr+0xf0/0xf0 [16220.258424] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16220.261189] kthread+0x1d1/0x200 [16220.262338] ? set_kthread_struct+0x70/0x70 [16220.263784] ret_from_fork+0x1f/0x30 [16224.057012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16224.061182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16224.083661] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16224.089655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16224.093419] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16224.096051] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16224.098031] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16224.106402] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16224.109990] RAX: 0000000000007910 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16224.113127] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16224.116113] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16224.119316] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16224.127346] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16224.132870] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16224.139136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16224.141138] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16224.143659] Call Trace: [16224.144711] [16224.148196] ? show_regs.cold.9+0x22/0x2f [16224.151420] ? watchdog_timer_fn.cold.6+0x62/0x10a [16224.153180] ? __hrtimer_run_queues+0x1c2/0x4d0 [16224.157293] ? watchdog+0x40/0x40 [16224.160613] ? hrtimer_interrupt+0x128/0x2c0 [16224.163816] ? smp_apic_timer_interrupt+0xa6/0x270 [16224.166200] ? apic_timer_interrupt+0xf/0x20 [16224.167880] [16224.168716] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16224.170731] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16224.172744] do_raw_spin_lock+0x177/0x190 [16224.174081] _raw_spin_lock+0x21/0x30 [16224.175156] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16224.176909] process_one_work+0x2c8/0x700 [16224.178055] worker_thread+0x296/0x6e0 [16224.179144] ? rescuer_thread+0x570/0x570 [16224.180253] kthread+0x1d1/0x200 [16224.181296] ? set_kthread_struct+0x70/0x70 [16224.182941] ret_from_fork+0x1f/0x30 [16232.059310] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16232.065259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16232.093011] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16232.098399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16232.102736] RIP: 0010:native_safe_halt+0x2e/0x50 [16232.105016] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16232.114714] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [16232.118244] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16232.122173] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16232.125794] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16232.130112] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16232.134988] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16232.138933] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16232.143236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16232.145880] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16232.149293] Call Trace: [16232.150657] [16232.151612] ? show_regs.cold.9+0x22/0x2f [16232.153730] ? watchdog_timer_fn.cold.6+0x62/0x10a [16232.156116] ? __hrtimer_run_queues+0x1c2/0x4d0 [16232.158336] ? watchdog+0x40/0x40 [16232.159794] ? hrtimer_interrupt+0x128/0x2c0 [16232.161989] ? smp_apic_timer_interrupt+0xa6/0x270 [16232.165626] ? apic_timer_interrupt+0xf/0x20 [16232.167736] [16232.168819] ? native_safe_halt+0x2e/0x50 [16232.171221] kvm_wait+0xcd/0xe0 [16232.172808] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16232.175433] do_raw_spin_lock+0x177/0x190 [16232.177470] _raw_spin_lock+0x21/0x30 [16232.179480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16232.183710] ? do_raw_spin_unlock+0x75/0x190 [16232.185919] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16232.188725] ? do_wait_intr+0xf0/0xf0 [16232.190479] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16232.193330] mdc_close+0x53b/0xd70 [mdc] [16232.195349] lmv_close+0x18d/0x490 [lmv] [16232.197321] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16232.200184] ll_md_real_close+0xa5/0x2a0 [lustre] [16232.202763] ll_md_close+0x274/0x800 [lustre] [16232.205641] ll_file_release+0x128/0xc50 [lustre] [16232.208180] __fput+0x10d/0x440 [16232.209902] ____fput+0x12/0x20 [16232.211679] task_work_run+0xc6/0x110 [16232.213675] exit_to_usermode_loop+0x1dd/0x1f0 [16232.215977] do_syscall_64+0x3d6/0x440 [16232.217809] entry_SYSCALL_64_after_hwframe+0x49/0xae [16232.220016] RIP: 0033:0x7fa749552cd8 [16232.221573] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16232.232628] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16232.237297] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16232.241436] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16232.244704] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16232.248512] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16232.252321] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16236.054025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16236.062108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16236.102977] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16236.107420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16236.112615] RIP: 0010:native_safe_halt+0x2e/0x50 [16236.116317] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16236.128849] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16236.132172] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16236.144142] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16236.153949] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16236.158218] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16236.164612] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16236.169469] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16236.173863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16236.176608] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16236.180179] Call Trace: [16236.181851] [16236.182923] ? show_regs.cold.9+0x22/0x2f [16236.185891] ? watchdog_timer_fn.cold.6+0x62/0x10a [16236.188606] ? __hrtimer_run_queues+0x1c2/0x4d0 [16236.190934] ? watchdog+0x40/0x40 [16236.193107] ? hrtimer_interrupt+0x128/0x2c0 [16236.195514] ? smp_apic_timer_interrupt+0xa6/0x270 [16236.198162] ? apic_timer_interrupt+0xf/0x20 [16236.200724] [16236.201853] ? native_safe_halt+0x2e/0x50 [16236.204504] kvm_wait+0xcd/0xe0 [16236.206444] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16236.209490] do_raw_spin_lock+0x177/0x190 [16236.210884] _raw_spin_lock+0x21/0x30 [16236.213131] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16236.215961] ? do_raw_spin_unlock+0x75/0x190 [16236.218548] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16236.221964] ? do_wait_intr+0xf0/0xf0 [16236.224252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16236.227606] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16236.231049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16236.235110] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16236.238555] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16236.241846] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16236.244896] mdc_intent_lock+0x353/0x7e0 [mdc] [16236.247028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16236.249045] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16236.251968] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16236.254524] lmv_intent_open+0x338/0x1150 [lmv] [16236.256841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16236.259357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16236.261892] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16236.263730] ll_intent_lock+0xc5/0xaa0 [lustre] [16236.265879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16236.268366] ll_lookup_it+0xe0e/0x2270 [lustre] [16236.270491] ? do_raw_spin_unlock+0x75/0x190 [16236.272517] ? _raw_spin_unlock+0x12/0x30 [16236.274369] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16236.276563] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16236.279105] ll_atomic_open+0x36a/0xd70 [lustre] [16236.281462] lookup_open+0x356/0xb40 [16236.283240] do_last+0x313/0xfc0 [16236.285258] ? nd_jump_root+0xe5/0x160 [16236.287152] ? path_init+0x437/0x520 [16236.288842] path_openat+0xf7/0x500 [16236.290218] ? do_raw_spin_unlock+0x75/0x190 [16236.292106] do_filp_open+0x99/0x140 [16236.293714] ? getname_flags+0x6e/0x330 [16236.295345] ? __check_object_size+0xff/0x256 [16236.297221] ? do_raw_spin_unlock+0x75/0x190 [16236.299199] ? _raw_spin_unlock+0x12/0x30 [16236.300847] do_sys_openat2+0x2b4/0x410 [16236.302063] do_sys_open+0x73/0xa0 [16236.303537] __x64_sys_openat+0x24/0x30 [16236.305245] do_syscall_64+0xc1/0x440 [16236.306909] entry_SYSCALL_64_after_hwframe+0x49/0xae [16236.309348] RIP: 0033:0x7fa749552332 [16236.311515] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16236.319524] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16236.322834] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16236.325545] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16236.329417] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16236.332484] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16236.337305] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16240.098406] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14322s! [16240.105873] Showing busy workqueues and worker pools: [16240.108407] workqueue events: flags=0x0 [16240.110011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16240.115604] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16240.120875] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16240.124063] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16240.129770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.132857] pending: kfree_rcu_monitor [16240.134203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16240.139835] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16240.144795] workqueue events_long: flags=0x0 [16240.149765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.153379] pending: lu_objects_flush [obdclass] [16240.155304] workqueue events_unbound: flags=0x2 [16240.161050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16240.167689] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16240.178928] workqueue events_power_efficient: flags=0x80 [16240.183119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.192076] pending: check_lifetime [16240.200196] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.211239] pending: neigh_periodic_work [16240.216374] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.222749] pending: neigh_periodic_work [16240.227126] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.230690] pending: do_cache_clean [16240.236147] workqueue mm_percpu_wq: flags=0x8 [16240.240033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.247536] pending: vmstat_update [16240.251018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.262576] pending: vmstat_update [16240.268115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16240.278551] pending: vmstat_update [16240.283101] workqueue writeback: flags=0x4a [16240.288925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16240.293212] pending: wb_workfn, wb_workfn [16240.295380] workqueue xprtiod: flags=0xa [16240.297125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16240.299973] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16240.303647] workqueue ipv6_addrconf: flags=0x40008 [16240.305731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16240.308820] pending: addrconf_verify_work [16240.310932] workqueue knbd0-recv: flags=0x1a [16240.312437] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16240.314865] in-flight: 651:recv_work [16240.316501] workqueue ptlrpc_pinger: flags=0x2 [16240.318086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16240.321066] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16240.323243] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14323s workers=4 idle: 37 31846 28897 [16240.325803] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16230s workers=2 idle: 653 [16248.058889] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16248.073005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16248.097318] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16248.102631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16248.106733] RIP: 0010:native_safe_halt+0x2e/0x50 [16248.111186] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16248.129431] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16248.139869] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16248.144249] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16248.148457] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16248.151434] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16248.154709] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16248.159516] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16248.162957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16248.165504] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16248.168802] Call Trace: [16248.169865] [16248.170754] ? show_regs.cold.9+0x22/0x2f [16248.172523] ? watchdog_timer_fn.cold.6+0x62/0x10a [16248.174967] ? __hrtimer_run_queues+0x1c2/0x4d0 [16248.177672] ? watchdog+0x40/0x40 [16248.179024] ? hrtimer_interrupt+0x128/0x2c0 [16248.181265] ? smp_apic_timer_interrupt+0xa6/0x270 [16248.183339] ? apic_timer_interrupt+0xf/0x20 [16248.185784] [16248.186730] ? native_safe_halt+0x2e/0x50 [16248.188482] kvm_wait+0xcd/0xe0 [16248.189931] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16248.192621] do_raw_spin_lock+0x177/0x190 [16248.194398] _raw_spin_lock+0x21/0x30 [16248.195917] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16248.198309] obd_mod_free+0x46/0x140 [ptlrpc] [16248.200674] mdc_commit_open+0x91/0xb0 [mdc] [16248.202542] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16248.204828] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16248.207361] after_reply+0xe2c/0x1780 [ptlrpc] [16248.209762] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16248.212319] ? __next_timer_interrupt+0x160/0x160 [16248.214955] ptlrpcd+0xa82/0xe40 [ptlrpc] [16248.217787] ? do_wait_intr+0xf0/0xf0 [16248.219187] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16248.221601] kthread+0x1d1/0x200 [16248.223141] ? set_kthread_struct+0x70/0x70 [16248.225100] ret_from_fork+0x1f/0x30 [16252.058663] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16252.072313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16252.124510] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16252.143232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16252.154078] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16252.159282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16252.162880] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16252.177451] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16252.182911] RAX: 0000000000005614 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16252.189274] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16252.195356] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16252.205371] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16252.209252] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16252.223836] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16252.236090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16252.245208] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16252.249391] Call Trace: [16252.252803] [16252.255065] ? show_regs.cold.9+0x22/0x2f [16252.259403] ? watchdog_timer_fn.cold.6+0x62/0x10a [16252.272899] ? __hrtimer_run_queues+0x1c2/0x4d0 [16252.277801] ? watchdog+0x40/0x40 [16252.282976] ? hrtimer_interrupt+0x128/0x2c0 [16252.285750] ? smp_apic_timer_interrupt+0xa6/0x270 [16252.295461] ? apic_timer_interrupt+0xf/0x20 [16252.300675] [16252.302849] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16252.309129] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16252.320650] do_raw_spin_lock+0x177/0x190 [16252.327821] _raw_spin_lock+0x21/0x30 [16252.331373] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16252.338382] process_one_work+0x2c8/0x700 [16252.342467] worker_thread+0x296/0x6e0 [16252.345358] ? rescuer_thread+0x570/0x570 [16252.347759] kthread+0x1d1/0x200 [16252.349587] ? set_kthread_struct+0x70/0x70 [16252.351751] ret_from_fork+0x1f/0x30 [16260.060580] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16260.065040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16260.111215] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16260.115512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16260.121401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16260.124045] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16260.133561] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16260.135821] RAX: 0000000000007f0d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16260.144466] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16260.147563] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16260.151829] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16260.158451] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16260.163192] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16260.172744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16260.175711] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16260.181124] Call Trace: [16260.182221] [16260.182911] ? show_regs.cold.9+0x22/0x2f [16260.184453] ? watchdog_timer_fn.cold.6+0x62/0x10a [16260.185916] ? __hrtimer_run_queues+0x1c2/0x4d0 [16260.188142] ? watchdog+0x40/0x40 [16260.189424] ? hrtimer_interrupt+0x128/0x2c0 [16260.191361] ? smp_apic_timer_interrupt+0xa6/0x270 [16260.193355] ? apic_timer_interrupt+0xf/0x20 [16260.195224] [16260.196002] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16260.198220] do_raw_spin_lock+0x177/0x190 [16260.199749] _raw_spin_lock+0x21/0x30 [16260.201317] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16260.203566] ? do_raw_spin_unlock+0x75/0x190 [16260.205652] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16260.207980] ? do_wait_intr+0xf0/0xf0 [16260.210675] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16260.213021] mdc_close+0x53b/0xd70 [mdc] [16260.214458] lmv_close+0x18d/0x490 [lmv] [16260.215824] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16260.228790] ll_md_real_close+0xa5/0x2a0 [lustre] [16260.237682] ll_md_close+0x274/0x800 [lustre] [16260.244451] ll_file_release+0x128/0xc50 [lustre] [16260.249687] __fput+0x10d/0x440 [16260.255317] ____fput+0x12/0x20 [16260.256574] task_work_run+0xc6/0x110 [16260.258501] exit_to_usermode_loop+0x1dd/0x1f0 [16260.260360] do_syscall_64+0x3d6/0x440 [16260.262297] entry_SYSCALL_64_after_hwframe+0x49/0xae [16260.268449] RIP: 0033:0x7fa749552cd8 [16260.270369] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16260.282648] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16260.291022] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16260.296850] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16260.305771] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16260.313706] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16260.320818] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16268.054042] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16268.058445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16268.089504] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16268.105096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16268.111384] RIP: 0010:native_safe_halt+0x2e/0x50 [16268.112704] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16268.126152] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16268.131983] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16268.134827] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16268.139586] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16268.145107] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16268.149033] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16268.153156] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16268.158052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16268.164934] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16268.168577] Call Trace: [16268.169516] [16268.170753] ? show_regs.cold.9+0x22/0x2f [16268.173924] ? watchdog_timer_fn.cold.6+0x62/0x10a [16268.180193] ? __hrtimer_run_queues+0x1c2/0x4d0 [16268.184274] ? watchdog+0x40/0x40 [16268.185428] ? hrtimer_interrupt+0x128/0x2c0 [16268.190276] ? smp_apic_timer_interrupt+0xa6/0x270 [16268.192213] ? apic_timer_interrupt+0xf/0x20 [16268.194309] [16268.197541] ? native_safe_halt+0x2e/0x50 [16268.200081] kvm_wait+0xcd/0xe0 [16268.202588] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16268.209257] do_raw_spin_lock+0x177/0x190 [16268.214601] _raw_spin_lock+0x21/0x30 [16268.216634] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16268.224413] ? do_raw_spin_unlock+0x75/0x190 [16268.228512] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16268.231062] ? do_wait_intr+0xf0/0xf0 [16268.235459] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16268.244046] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16268.250294] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16268.253686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16268.258245] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16268.261192] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16268.271687] mdc_intent_lock+0x353/0x7e0 [mdc] [16268.272937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16268.274669] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16268.278445] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16268.281947] lmv_intent_open+0x338/0x1150 [lmv] [16268.283478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16268.289596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16268.294714] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16268.298570] ll_intent_lock+0xc5/0xaa0 [lustre] [16268.300908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16268.304122] ll_lookup_it+0xe0e/0x2270 [lustre] [16268.306418] ? do_raw_spin_unlock+0x75/0x190 [16268.308373] ? _raw_spin_unlock+0x12/0x30 [16268.310417] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16268.312426] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16268.314737] ll_atomic_open+0x36a/0xd70 [lustre] [16268.318911] lookup_open+0x356/0xb40 [16268.320909] do_last+0x313/0xfc0 [16268.323401] ? nd_jump_root+0xe5/0x160 [16268.332927] ? path_init+0x437/0x520 [16268.336048] path_openat+0xf7/0x500 [16268.338210] ? do_raw_spin_unlock+0x75/0x190 [16268.340603] do_filp_open+0x99/0x140 [16268.342970] ? getname_flags+0x6e/0x330 [16268.347257] ? __check_object_size+0xff/0x256 [16268.352029] ? do_raw_spin_unlock+0x75/0x190 [16268.356638] ? _raw_spin_unlock+0x12/0x30 [16268.359007] do_sys_openat2+0x2b4/0x410 [16268.361888] do_sys_open+0x73/0xa0 [16268.367214] __x64_sys_openat+0x24/0x30 [16268.368378] do_syscall_64+0xc1/0x440 [16268.374030] entry_SYSCALL_64_after_hwframe+0x49/0xae [16268.377366] RIP: 0033:0x7fa749552332 [16268.381286] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16268.390030] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16268.395099] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16268.398420] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16268.405208] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16268.411364] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16268.418566] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16270.818375] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14353s! [16270.822428] Showing busy workqueues and worker pools: [16270.826334] workqueue events: flags=0x0 [16270.828455] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16270.835150] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16270.844029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16270.850328] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16270.855918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.861344] pending: kfree_rcu_monitor [16270.863961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16270.871321] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16270.876170] workqueue events_long: flags=0x0 [16270.878022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.884913] pending: lu_objects_flush [obdclass] [16270.886870] workqueue events_unbound: flags=0x2 [16270.893463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16270.896628] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16270.901196] workqueue events_power_efficient: flags=0x80 [16270.903280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.908319] pending: check_lifetime [16270.910133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.916105] pending: neigh_periodic_work [16270.924138] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.933265] pending: neigh_periodic_work [16270.934786] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.944383] pending: do_cache_clean [16270.945841] workqueue mm_percpu_wq: flags=0x8 [16270.954706] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.957061] pending: vmstat_update [16270.961782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.965338] pending: vmstat_update [16270.967347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16270.970666] pending: vmstat_update [16270.972446] workqueue writeback: flags=0x4a [16270.974288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16270.977072] pending: wb_workfn, wb_workfn [16270.982066] workqueue xprtiod: flags=0xa [16270.986895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16270.990881] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16270.996168] workqueue ipv6_addrconf: flags=0x40008 [16270.999107] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16271.003154] pending: addrconf_verify_work [16271.004716] workqueue knbd0-recv: flags=0x1a [16271.007656] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16271.010849] in-flight: 651:recv_work [16271.012072] workqueue ptlrpc_pinger: flags=0x2 [16271.014672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16271.018298] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16271.022114] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14353s workers=4 idle: 37 31846 28897 [16271.025841] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16260s workers=2 idle: 653 [16276.058016] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16276.062450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16276.087057] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16276.092888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16276.098062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16276.100971] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16276.113782] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16276.118293] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16276.122481] RDX: 00000000000039d5 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16276.125939] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16276.129353] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16276.136410] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16276.146921] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16276.160304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16276.166830] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16276.172232] Call Trace: [16276.173032] [16276.173679] ? show_regs.cold.9+0x22/0x2f [16276.175032] ? watchdog_timer_fn.cold.6+0x62/0x10a [16276.176573] ? __hrtimer_run_queues+0x1c2/0x4d0 [16276.178739] ? watchdog+0x40/0x40 [16276.180208] ? hrtimer_interrupt+0x128/0x2c0 [16276.183335] ? smp_apic_timer_interrupt+0xa6/0x270 [16276.185852] ? apic_timer_interrupt+0xf/0x20 [16276.187788] [16276.188763] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16276.190411] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16276.192455] do_raw_spin_lock+0x177/0x190 [16276.193954] _raw_spin_lock+0x21/0x30 [16276.195033] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16276.197342] obd_mod_free+0x46/0x140 [ptlrpc] [16276.199227] mdc_commit_open+0x91/0xb0 [mdc] [16276.200579] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16276.202538] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16276.204665] after_reply+0xe2c/0x1780 [ptlrpc] [16276.206417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16276.207961] ? __next_timer_interrupt+0x160/0x160 [16276.209680] ptlrpcd+0xa82/0xe40 [ptlrpc] [16276.211167] ? do_wait_intr+0xf0/0xf0 [16276.212314] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16276.214150] kthread+0x1d1/0x200 [16276.215203] ? set_kthread_struct+0x70/0x70 [16276.216393] ret_from_fork+0x1f/0x30 [16280.057816] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16280.064133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16280.086321] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16280.090443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16280.095190] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16280.098674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16280.101521] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16280.109207] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16280.112816] RAX: 0000000000003727 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16280.115469] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16280.119168] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16280.122291] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16280.125066] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16280.128263] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16280.132139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16280.134897] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16280.138062] Call Trace: [16280.139217] [16280.140583] ? show_regs.cold.9+0x22/0x2f [16280.142181] ? watchdog_timer_fn.cold.6+0x62/0x10a [16280.144468] ? __hrtimer_run_queues+0x1c2/0x4d0 [16280.146092] ? watchdog+0x40/0x40 [16280.147608] ? hrtimer_interrupt+0x128/0x2c0 [16280.149855] ? smp_apic_timer_interrupt+0xa6/0x270 [16280.151965] ? apic_timer_interrupt+0xf/0x20 [16280.153882] [16280.154670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16280.156954] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16280.159237] do_raw_spin_lock+0x177/0x190 [16280.160716] _raw_spin_lock+0x21/0x30 [16280.162097] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16280.165060] process_one_work+0x2c8/0x700 [16280.166967] worker_thread+0x296/0x6e0 [16280.168800] ? rescuer_thread+0x570/0x570 [16280.170848] kthread+0x1d1/0x200 [16280.172018] ? set_kthread_struct+0x70/0x70 [16280.173563] ret_from_fork+0x1f/0x30 [16288.059033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16288.066204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16288.112848] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16288.132751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16288.138755] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16288.141909] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16288.153842] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16288.157721] RAX: 000000000000583b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16288.161204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16288.164663] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16288.169812] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16288.177239] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16288.181363] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16288.186117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16288.188735] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16288.196688] Call Trace: [16288.198158] [16288.198947] ? show_regs.cold.9+0x22/0x2f [16288.200734] ? watchdog_timer_fn.cold.6+0x62/0x10a [16288.204891] ? __hrtimer_run_queues+0x1c2/0x4d0 [16288.207866] ? watchdog+0x40/0x40 [16288.210168] ? hrtimer_interrupt+0x128/0x2c0 [16288.213841] ? smp_apic_timer_interrupt+0xa6/0x270 [16288.217624] ? apic_timer_interrupt+0xf/0x20 [16288.218945] [16288.219517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16288.227218] do_raw_spin_lock+0x177/0x190 [16288.228349] _raw_spin_lock+0x21/0x30 [16288.230591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16288.236367] ? do_raw_spin_unlock+0x75/0x190 [16288.238905] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16288.241634] ? do_wait_intr+0xf0/0xf0 [16288.243368] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16288.246144] mdc_close+0x53b/0xd70 [mdc] [16288.248180] lmv_close+0x18d/0x490 [lmv] [16288.250972] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16288.257594] ll_md_real_close+0xa5/0x2a0 [lustre] [16288.263203] ll_md_close+0x274/0x800 [lustre] [16288.267947] ll_file_release+0x128/0xc50 [lustre] [16288.269680] __fput+0x10d/0x440 [16288.273206] ____fput+0x12/0x20 [16288.276257] task_work_run+0xc6/0x110 [16288.280093] exit_to_usermode_loop+0x1dd/0x1f0 [16288.284457] do_syscall_64+0x3d6/0x440 [16288.286158] entry_SYSCALL_64_after_hwframe+0x49/0xae [16288.289423] RIP: 0033:0x7fa749552cd8 [16288.290792] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16288.309850] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16288.315119] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16288.322611] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16288.330965] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16288.333513] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16288.348002] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16296.054813] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16296.065612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16296.118072] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16296.133323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16296.148916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16296.156216] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16296.177477] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16296.184768] RAX: 0000000000000482 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16296.195794] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16296.203960] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16296.207522] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16296.210785] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16296.215754] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16296.220681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16296.224693] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16296.227814] Call Trace: [16296.231373] [16296.238392] ? show_regs.cold.9+0x22/0x2f [16296.243051] ? watchdog_timer_fn.cold.6+0x62/0x10a [16296.250616] ? __hrtimer_run_queues+0x1c2/0x4d0 [16296.260103] ? watchdog+0x40/0x40 [16296.264167] ? hrtimer_interrupt+0x128/0x2c0 [16296.268888] ? smp_apic_timer_interrupt+0xa6/0x270 [16296.270849] ? apic_timer_interrupt+0xf/0x20 [16296.273066] [16296.273674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16296.277157] do_raw_spin_lock+0x177/0x190 [16296.280627] _raw_spin_lock+0x21/0x30 [16296.282656] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16296.287853] ? do_raw_spin_unlock+0x75/0x190 [16296.289753] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16296.292053] ? do_wait_intr+0xf0/0xf0 [16296.293679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16296.299495] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16296.306056] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16296.316499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16296.320702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16296.323313] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16296.330154] mdc_intent_lock+0x353/0x7e0 [mdc] [16296.336232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16296.347623] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16296.362837] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16296.364948] lmv_intent_open+0x338/0x1150 [lmv] [16296.377382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16296.379391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16296.384107] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16296.385800] ll_intent_lock+0xc5/0xaa0 [lustre] [16296.387632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16296.390544] ll_lookup_it+0xe0e/0x2270 [lustre] [16296.394610] ? do_raw_spin_unlock+0x75/0x190 [16296.399209] ? _raw_spin_unlock+0x12/0x30 [16296.407590] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16296.409707] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16296.412565] ll_atomic_open+0x36a/0xd70 [lustre] [16296.417123] lookup_open+0x356/0xb40 [16296.418397] do_last+0x313/0xfc0 [16296.427888] ? nd_jump_root+0xe5/0x160 [16296.432530] ? path_init+0x437/0x520 [16296.436888] path_openat+0xf7/0x500 [16296.442075] ? do_raw_spin_unlock+0x75/0x190 [16296.446174] do_filp_open+0x99/0x140 [16296.450158] ? getname_flags+0x6e/0x330 [16296.454143] ? __check_object_size+0xff/0x256 [16296.459515] ? do_raw_spin_unlock+0x75/0x190 [16296.464801] ? _raw_spin_unlock+0x12/0x30 [16296.468905] do_sys_openat2+0x2b4/0x410 [16296.473817] do_sys_open+0x73/0xa0 [16296.479335] __x64_sys_openat+0x24/0x30 [16296.480649] do_syscall_64+0xc1/0x440 [16296.482767] entry_SYSCALL_64_after_hwframe+0x49/0xae [16296.491222] RIP: 0033:0x7fa749552332 [16296.498646] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16296.521441] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16296.529914] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16296.532861] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16296.541174] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16296.547021] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16296.556234] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16301.537046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14384s! [16301.541545] Showing busy workqueues and worker pools: [16301.543523] workqueue events: flags=0x0 [16301.544801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16301.548084] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16301.553539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16301.556807] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16301.561627] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.565046] pending: kfree_rcu_monitor [16301.566924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16301.571272] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16301.576279] workqueue events_long: flags=0x0 [16301.578443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.581363] pending: lu_objects_flush [obdclass] [16301.583707] workqueue events_unbound: flags=0x2 [16301.585393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16301.588365] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16301.592269] workqueue events_power_efficient: flags=0x80 [16301.594573] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.597332] pending: check_lifetime [16301.599122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.601794] pending: neigh_periodic_work [16301.603483] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.606213] pending: neigh_periodic_work [16301.608048] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.610858] pending: do_cache_clean [16301.614133] workqueue mm_percpu_wq: flags=0x8 [16301.616095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.619440] pending: vmstat_update [16301.621486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.624276] pending: vmstat_update [16301.625697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16301.628445] pending: vmstat_update [16301.629859] workqueue writeback: flags=0x4a [16301.631693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16301.636741] pending: wb_workfn, wb_workfn [16301.638137] workqueue xprtiod: flags=0xa [16301.640262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16301.643204] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16301.646698] workqueue ipv6_addrconf: flags=0x40008 [16301.649060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16301.652254] pending: addrconf_verify_work [16301.654388] workqueue knbd0-recv: flags=0x1a [16301.656391] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16301.659388] in-flight: 651:recv_work [16301.660869] workqueue ptlrpc_pinger: flags=0x2 [16301.662815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16301.665815] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16301.669388] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14384s workers=4 idle: 37 31846 28897 [16301.673626] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16291s workers=2 idle: 653 [16304.058472] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16304.065191] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16304.086184] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16304.094129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16304.098883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16304.102728] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16304.115507] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16304.119287] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16304.124551] RDX: 0000000000002c4f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16304.129816] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16304.134694] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16304.137849] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16304.143061] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16304.147712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16304.150435] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16304.153579] Call Trace: [16304.154687] [16304.155724] ? show_regs.cold.9+0x22/0x2f [16304.157671] ? watchdog_timer_fn.cold.6+0x62/0x10a [16304.159882] ? __hrtimer_run_queues+0x1c2/0x4d0 [16304.162020] ? watchdog+0x40/0x40 [16304.163936] ? hrtimer_interrupt+0x128/0x2c0 [16304.165962] ? smp_apic_timer_interrupt+0xa6/0x270 [16304.168088] ? apic_timer_interrupt+0xf/0x20 [16304.170478] [16304.171481] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16304.174026] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16304.176878] do_raw_spin_lock+0x177/0x190 [16304.178899] _raw_spin_lock+0x21/0x30 [16304.180714] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16304.183267] obd_mod_free+0x46/0x140 [ptlrpc] [16304.185620] mdc_commit_open+0x91/0xb0 [mdc] [16304.187591] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16304.189842] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16304.192258] after_reply+0xe2c/0x1780 [ptlrpc] [16304.194665] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16304.197294] ? __next_timer_interrupt+0x160/0x160 [16304.199483] ptlrpcd+0xa82/0xe40 [ptlrpc] [16304.201642] ? do_wait_intr+0xf0/0xf0 [16304.203274] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16304.205927] kthread+0x1d1/0x200 [16304.207526] ? set_kthread_struct+0x70/0x70 [16304.209599] ret_from_fork+0x1f/0x30 [16308.059195] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16308.064599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16308.105596] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16308.119926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16308.125230] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16308.133676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16308.138605] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16308.155374] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16308.157829] RAX: 0000000000007a8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16308.165365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16308.176730] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16308.188422] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16308.199840] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16308.202177] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16308.225705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16308.235084] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16308.247435] Call Trace: [16308.250408] [16308.258583] ? show_regs.cold.9+0x22/0x2f [16308.259935] ? watchdog_timer_fn.cold.6+0x62/0x10a [16308.265877] ? __hrtimer_run_queues+0x1c2/0x4d0 [16308.267404] ? watchdog+0x40/0x40 [16308.268588] ? hrtimer_interrupt+0x128/0x2c0 [16308.275358] ? smp_apic_timer_interrupt+0xa6/0x270 [16308.283458] ? apic_timer_interrupt+0xf/0x20 [16308.286937] [16308.287732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16308.294792] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16308.300415] do_raw_spin_lock+0x177/0x190 [16308.301833] _raw_spin_lock+0x21/0x30 [16308.303148] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16308.316020] process_one_work+0x2c8/0x700 [16308.321201] worker_thread+0x296/0x6e0 [16308.324330] ? rescuer_thread+0x570/0x570 [16308.329005] kthread+0x1d1/0x200 [16308.332378] ? set_kthread_struct+0x70/0x70 [16308.341288] ret_from_fork+0x1f/0x30 [16316.059027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16316.064119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16316.102154] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16316.111733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16316.118461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16316.124092] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16316.136547] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16316.145312] RAX: 00000000000060d1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16316.151696] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16316.157270] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16316.161418] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16316.168544] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16316.172109] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16316.177162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16316.181048] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16316.190114] Call Trace: [16316.197021] [16316.198703] ? show_regs.cold.9+0x22/0x2f [16316.203864] ? watchdog_timer_fn.cold.6+0x62/0x10a [16316.207339] ? __hrtimer_run_queues+0x1c2/0x4d0 [16316.210097] ? watchdog+0x40/0x40 [16316.212484] ? hrtimer_interrupt+0x128/0x2c0 [16316.215127] ? smp_apic_timer_interrupt+0xa6/0x270 [16316.221138] ? apic_timer_interrupt+0xf/0x20 [16316.223655] [16316.224720] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16316.233099] do_raw_spin_lock+0x177/0x190 [16316.234294] _raw_spin_lock+0x21/0x30 [16316.235276] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16316.242714] ? do_raw_spin_unlock+0x75/0x190 [16316.244119] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16316.248809] ? do_wait_intr+0xf0/0xf0 [16316.250276] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16316.260996] mdc_close+0x53b/0xd70 [mdc] [16316.262364] lmv_close+0x18d/0x490 [lmv] [16316.264482] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16316.268729] ll_md_real_close+0xa5/0x2a0 [lustre] [16316.270846] ll_md_close+0x274/0x800 [lustre] [16316.274867] ll_file_release+0x128/0xc50 [lustre] [16316.287577] __fput+0x10d/0x440 [16316.294808] ____fput+0x12/0x20 [16316.299723] task_work_run+0xc6/0x110 [16316.302188] exit_to_usermode_loop+0x1dd/0x1f0 [16316.305048] do_syscall_64+0x3d6/0x440 [16316.307048] entry_SYSCALL_64_after_hwframe+0x49/0xae [16316.308898] RIP: 0033:0x7fa749552cd8 [16316.311832] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16316.335183] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16316.342447] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16316.346525] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16316.351791] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16316.355702] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16316.372691] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16328.066440] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16328.076033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16328.145359] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16328.160312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16328.176459] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16328.183857] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16328.207855] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16328.219949] RAX: 0000000000002f89 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16328.231928] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16328.245559] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16328.256726] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16328.267131] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16328.283903] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16328.296178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16328.305197] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16328.316167] Call Trace: [16328.322116] [16328.322852] ? show_regs.cold.9+0x22/0x2f [16328.324275] ? watchdog_timer_fn.cold.6+0x62/0x10a [16328.332103] ? __hrtimer_run_queues+0x1c2/0x4d0 [16328.338694] ? watchdog+0x40/0x40 [16328.340265] ? hrtimer_interrupt+0x128/0x2c0 [16328.351150] ? smp_apic_timer_interrupt+0xa6/0x270 [16328.358169] ? apic_timer_interrupt+0xf/0x20 [16328.367157] [16328.367946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16328.379577] do_raw_spin_lock+0x177/0x190 [16328.382290] _raw_spin_lock+0x21/0x30 [16328.388840] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16328.398314] ? do_raw_spin_unlock+0x75/0x190 [16328.408093] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16328.415291] ? do_wait_intr+0xf0/0xf0 [16328.420359] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16328.428251] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16328.437762] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16328.444334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16328.452546] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16328.459359] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16328.464087] mdc_intent_lock+0x353/0x7e0 [mdc] [16328.465676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16328.480274] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16328.484521] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16328.497231] lmv_intent_open+0x338/0x1150 [lmv] [16328.503566] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16328.505347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16328.522641] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16328.534455] ll_intent_lock+0xc5/0xaa0 [lustre] [16328.539446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16328.548550] ll_lookup_it+0xe0e/0x2270 [lustre] [16328.553145] ? do_raw_spin_unlock+0x75/0x190 [16328.555817] ? _raw_spin_unlock+0x12/0x30 [16328.557591] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16328.559326] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16328.565268] ll_atomic_open+0x36a/0xd70 [lustre] [16328.577412] lookup_open+0x356/0xb40 [16328.578747] do_last+0x313/0xfc0 [16328.579916] ? nd_jump_root+0xe5/0x160 [16328.586384] ? path_init+0x437/0x520 [16328.594019] path_openat+0xf7/0x500 [16328.595469] ? do_raw_spin_unlock+0x75/0x190 [16328.604386] do_filp_open+0x99/0x140 [16328.608126] ? getname_flags+0x6e/0x330 [16328.620034] ? __check_object_size+0xff/0x256 [16328.627373] ? do_raw_spin_unlock+0x75/0x190 [16328.637810] ? _raw_spin_unlock+0x12/0x30 [16328.645756] do_sys_openat2+0x2b4/0x410 [16328.655310] do_sys_open+0x73/0xa0 [16328.661561] __x64_sys_openat+0x24/0x30 [16328.668540] do_syscall_64+0xc1/0x440 [16328.675942] entry_SYSCALL_64_after_hwframe+0x49/0xae [16328.680641] RIP: 0033:0x7fa749552332 [16328.688206] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16328.716561] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16328.727660] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16328.743688] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16328.746110] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16328.753996] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16328.765629] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16331.744343] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14414s! [16331.752584] Showing busy workqueues and worker pools: [16331.763285] workqueue events: flags=0x0 [16331.775838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16331.790106] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16331.803179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16331.807338] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16331.825776] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.831402] pending: kfree_rcu_monitor [16331.836036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16331.839135] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16331.850214] workqueue events_long: flags=0x0 [16331.852678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.859083] pending: lu_objects_flush [obdclass] [16331.866843] workqueue events_unbound: flags=0x2 [16331.869364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16331.875914] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16331.889081] workqueue events_power_efficient: flags=0x80 [16331.894565] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.897990] pending: check_lifetime [16331.903909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.914104] pending: neigh_periodic_work [16331.918207] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.924338] pending: neigh_periodic_work [16331.929931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.935867] pending: do_cache_clean [16331.947892] workqueue mm_percpu_wq: flags=0x8 [16331.958754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.968987] pending: vmstat_update [16331.970370] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.982452] pending: vmstat_update [16331.987142] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16331.994096] pending: vmstat_update [16331.998037] workqueue writeback: flags=0x4a [16332.000960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16332.013308] pending: wb_workfn, wb_workfn [16332.023892] workqueue xprtiod: flags=0xa [16332.027567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16332.034288] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16332.044981] workqueue ipv6_addrconf: flags=0x40008 [16332.050106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16332.052304] pending: addrconf_verify_work [16332.057712] workqueue knbd0-recv: flags=0x1a [16332.059878] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16332.064445] in-flight: 651:recv_work [16332.064554] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16332.073086] workqueue ptlrpc_pinger: flags=0x2 [16332.073097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16332.073108] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16332.073382] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14414s workers=4 idle: 37 31846 28897 [16332.073395] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16321s workers=2 idle: 653 [16332.114985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16332.148770] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16332.155885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16332.167811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16332.171417] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16332.181241] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16332.185034] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16332.188640] RDX: 0000000000000eeb RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16332.192179] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16332.196449] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16332.199964] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16332.203444] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16332.207416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16332.210200] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16332.214088] Call Trace: [16332.215755] [16332.216983] ? show_regs.cold.9+0x22/0x2f [16332.219650] ? watchdog_timer_fn.cold.6+0x62/0x10a [16332.221708] ? __hrtimer_run_queues+0x1c2/0x4d0 [16332.224348] ? watchdog+0x40/0x40 [16332.226238] ? hrtimer_interrupt+0x128/0x2c0 [16332.228714] ? smp_apic_timer_interrupt+0xa6/0x270 [16332.232195] ? apic_timer_interrupt+0xf/0x20 [16332.234534] [16332.236077] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16332.240516] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16332.244042] do_raw_spin_lock+0x177/0x190 [16332.246787] _raw_spin_lock+0x21/0x30 [16332.248983] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16332.252793] obd_mod_free+0x46/0x140 [ptlrpc] [16332.256871] mdc_commit_open+0x91/0xb0 [mdc] [16332.258959] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16332.263544] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16332.266371] after_reply+0xe2c/0x1780 [ptlrpc] [16332.270524] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16332.273401] ? __next_timer_interrupt+0x160/0x160 [16332.275675] ptlrpcd+0xa82/0xe40 [ptlrpc] [16332.278660] ? do_wait_intr+0xf0/0xf0 [16332.280639] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16332.285671] kthread+0x1d1/0x200 [16332.288039] ? set_kthread_struct+0x70/0x70 [16332.290524] ret_from_fork+0x1f/0x30 [16336.060384] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16336.066694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16336.112482] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16336.125751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16336.134406] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16336.137641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16336.140634] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16336.148805] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16336.155813] RAX: 0000000000004588 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16336.159644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16336.169027] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16336.174428] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16336.181332] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16336.189457] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16336.197620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16336.203595] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16336.210343] Call Trace: [16336.214492] [16336.216396] ? show_regs.cold.9+0x22/0x2f [16336.218338] ? watchdog_timer_fn.cold.6+0x62/0x10a [16336.224320] ? __hrtimer_run_queues+0x1c2/0x4d0 [16336.230067] ? watchdog+0x40/0x40 [16336.231617] ? hrtimer_interrupt+0x128/0x2c0 [16336.236430] ? smp_apic_timer_interrupt+0xa6/0x270 [16336.244413] ? apic_timer_interrupt+0xf/0x20 [16336.248444] [16336.252148] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16336.255995] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16336.259894] do_raw_spin_lock+0x177/0x190 [16336.261318] _raw_spin_lock+0x21/0x30 [16336.265845] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16336.268358] process_one_work+0x2c8/0x700 [16336.273012] worker_thread+0x296/0x6e0 [16336.275331] ? rescuer_thread+0x570/0x570 [16336.280507] kthread+0x1d1/0x200 [16336.282258] ? set_kthread_struct+0x70/0x70 [16336.283985] ret_from_fork+0x1f/0x30 [16344.059567] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16344.065614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16344.092474] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16344.095620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16344.100174] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16344.105352] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16344.114668] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16344.119116] RAX: 0000000000000a27 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16344.123019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16344.127768] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16344.131100] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16344.136353] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16344.140835] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16344.145386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16344.149768] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16344.152847] Call Trace: [16344.154006] [16344.155037] ? show_regs.cold.9+0x22/0x2f [16344.163629] ? watchdog_timer_fn.cold.6+0x62/0x10a [16344.173680] ? __hrtimer_run_queues+0x1c2/0x4d0 [16344.175643] ? watchdog+0x40/0x40 [16344.176788] ? hrtimer_interrupt+0x128/0x2c0 [16344.178816] ? smp_apic_timer_interrupt+0xa6/0x270 [16344.180285] ? apic_timer_interrupt+0xf/0x20 [16344.182206] [16344.183094] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16344.185844] do_raw_spin_lock+0x177/0x190 [16344.187345] _raw_spin_lock+0x21/0x30 [16344.189315] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16344.191594] ? do_raw_spin_unlock+0x75/0x190 [16344.193273] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16344.195650] ? do_wait_intr+0xf0/0xf0 [16344.197389] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16344.200324] mdc_close+0x53b/0xd70 [mdc] [16344.202399] lmv_close+0x18d/0x490 [lmv] [16344.203569] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16344.206269] ll_md_real_close+0xa5/0x2a0 [lustre] [16344.210024] ll_md_close+0x274/0x800 [lustre] [16344.215780] ll_file_release+0x128/0xc50 [lustre] [16344.218750] __fput+0x10d/0x440 [16344.221197] ____fput+0x12/0x20 [16344.222604] task_work_run+0xc6/0x110 [16344.227380] exit_to_usermode_loop+0x1dd/0x1f0 [16344.229294] do_syscall_64+0x3d6/0x440 [16344.233201] entry_SYSCALL_64_after_hwframe+0x49/0xae [16344.237593] RIP: 0033:0x7fa749552cd8 [16344.240555] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16344.255879] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16344.263832] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16344.267972] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16344.273981] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16344.282221] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16344.290838] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16360.057305] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16360.058017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16360.066216] Modules linked in: [16360.069978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16360.070118] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16360.070184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16360.070187] RIP: 0010:native_safe_halt+0x2e/0x50 [16360.070200] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16360.070204] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16360.070209] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16360.070212] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16360.070214] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16360.070216] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16360.070218] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16360.070221] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16360.070224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16360.070226] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16360.070235] Call Trace: [16360.070241] [16360.070247] ? show_regs.cold.9+0x22/0x2f [16360.070257] ? watchdog_timer_fn.cold.6+0x62/0x10a [16360.070263] ? __hrtimer_run_queues+0x1c2/0x4d0 [16360.070267] ? watchdog+0x40/0x40 [16360.070272] ? hrtimer_interrupt+0x128/0x2c0 [16360.070276] ? smp_apic_timer_interrupt+0xa6/0x270 [16360.070283] ? apic_timer_interrupt+0xf/0x20 [16360.070287] [16360.070290] ? native_safe_halt+0x2e/0x50 [16360.070295] kvm_wait+0xcd/0xe0 [16360.070301] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16360.070308] do_raw_spin_lock+0x177/0x190 [16360.070312] _raw_spin_lock+0x21/0x30 [16360.070317] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16360.070602] obd_mod_free+0x46/0x140 [ptlrpc] [16360.070807] mdc_commit_open+0x91/0xb0 [mdc] [16360.070846] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16360.071185] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16360.071401] after_reply+0xe2c/0x1780 [ptlrpc] [16360.071624] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16360.071840] ? __next_timer_interrupt+0x160/0x160 [16360.071849] ptlrpcd+0xa82/0xe40 [ptlrpc] [16360.072211] ? do_wait_intr+0xf0/0xf0 [16360.072221] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16360.072461] kthread+0x1d1/0x200 [16360.072467] ? set_kthread_struct+0x70/0x70 [16360.072471] ret_from_fork+0x1f/0x30 [16360.196980] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16360.209235] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16360.212783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16360.216201] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16360.218533] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16360.225473] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16360.228025] RAX: 0000000000007f84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16360.230338] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16360.232303] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16360.234223] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16360.236311] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16360.238029] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16360.240985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16360.248300] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16360.250965] Call Trace: [16360.251998] [16360.254277] ? show_regs.cold.9+0x22/0x2f [16360.257952] ? watchdog_timer_fn.cold.6+0x62/0x10a [16360.277725] ? __hrtimer_run_queues+0x1c2/0x4d0 [16360.279380] ? watchdog+0x40/0x40 [16360.280439] ? hrtimer_interrupt+0x128/0x2c0 [16360.283360] ? smp_apic_timer_interrupt+0xa6/0x270 [16360.286978] ? apic_timer_interrupt+0xf/0x20 [16360.296253] [16360.299452] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16360.305904] do_raw_spin_lock+0x177/0x190 [16360.308206] _raw_spin_lock+0x21/0x30 [16360.314354] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16360.316324] ? do_raw_spin_unlock+0x75/0x190 [16360.322908] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16360.330496] ? do_wait_intr+0xf0/0xf0 [16360.334776] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16360.342371] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16360.345296] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16360.349850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.353686] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16360.359892] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16360.364526] mdc_intent_lock+0x353/0x7e0 [mdc] [16360.368428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.371764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16360.374938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16360.380977] lmv_intent_open+0x338/0x1150 [lmv] [16360.384914] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.389901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.394795] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16360.396923] ll_intent_lock+0xc5/0xaa0 [lustre] [16360.413307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.419871] ll_lookup_it+0xe0e/0x2270 [lustre] [16360.422084] ? do_raw_spin_unlock+0x75/0x190 [16360.423778] ? _raw_spin_unlock+0x12/0x30 [16360.426781] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16360.431732] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16360.434693] ll_atomic_open+0x36a/0xd70 [lustre] [16360.437244] lookup_open+0x356/0xb40 [16360.438719] do_last+0x313/0xfc0 [16360.440093] ? nd_jump_root+0xe5/0x160 [16360.441313] ? path_init+0x437/0x520 [16360.442516] path_openat+0xf7/0x500 [16360.443704] ? do_raw_spin_unlock+0x75/0x190 [16360.445198] do_filp_open+0x99/0x140 [16360.446523] ? getname_flags+0x6e/0x330 [16360.448217] ? __check_object_size+0xff/0x256 [16360.450198] ? do_raw_spin_unlock+0x75/0x190 [16360.452331] ? _raw_spin_unlock+0x12/0x30 [16360.453673] do_sys_openat2+0x2b4/0x410 [16360.455243] do_sys_open+0x73/0xa0 [16360.456082] __x64_sys_openat+0x24/0x30 [16360.457302] do_syscall_64+0xc1/0x440 [16360.463415] entry_SYSCALL_64_after_hwframe+0x49/0xae [16360.465078] RIP: 0033:0x7fa749552332 [16360.466311] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16360.478257] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16360.481524] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16360.488334] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16360.503580] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16360.517354] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16360.532961] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16362.466554] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14445s! [16362.476857] Showing busy workqueues and worker pools: [16362.483824] workqueue events: flags=0x0 [16362.487658] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16362.494788] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16362.504717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16362.511033] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16362.520654] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.527688] pending: kfree_rcu_monitor [16362.531116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16362.534516] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16362.540681] workqueue events_long: flags=0x0 [16362.542429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.546723] pending: lu_objects_flush [obdclass] [16362.550039] workqueue events_unbound: flags=0x2 [16362.552789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16362.555735] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16362.560664] workqueue events_power_efficient: flags=0x80 [16362.563517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.567111] pending: check_lifetime [16362.569124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.573525] pending: neigh_periodic_work [16362.575614] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.579702] pending: neigh_periodic_work [16362.581924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.585490] pending: do_cache_clean [16362.587500] workqueue mm_percpu_wq: flags=0x8 [16362.589776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.593592] pending: vmstat_update [16362.595318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.598346] pending: vmstat_update [16362.599921] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16362.603145] pending: vmstat_update [16362.605217] workqueue writeback: flags=0x4a [16362.606571] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16362.609234] pending: wb_workfn, wb_workfn [16362.611070] workqueue xprtiod: flags=0xa [16362.613048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16362.616198] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16362.619624] workqueue ipv6_addrconf: flags=0x40008 [16362.621493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16362.624141] pending: addrconf_verify_work [16362.626383] workqueue knbd0-recv: flags=0x1a [16362.627966] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16362.631460] in-flight: 651:recv_work [16362.633095] workqueue ptlrpc_pinger: flags=0x2 [16362.635172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16362.637767] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16362.640508] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14445s workers=4 idle: 37 31846 28897 [16362.643956] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16352s workers=2 idle: 653 [16364.060149] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16364.065643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16364.107100] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16364.114790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16364.119401] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16364.122500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16364.125326] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16364.136277] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16364.140649] RAX: 00000000000059c6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16364.146091] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16364.153474] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16364.158371] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16364.163368] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16364.168376] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16364.173672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.178349] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16364.182257] Call Trace: [16364.183500] [16364.184454] ? show_regs.cold.9+0x22/0x2f [16364.186657] ? watchdog_timer_fn.cold.6+0x62/0x10a [16364.189522] ? __hrtimer_run_queues+0x1c2/0x4d0 [16364.192039] ? watchdog+0x40/0x40 [16364.195546] ? hrtimer_interrupt+0x128/0x2c0 [16364.198202] ? smp_apic_timer_interrupt+0xa6/0x270 [16364.200826] ? apic_timer_interrupt+0xf/0x20 [16364.202787] [16364.203818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16364.206235] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16364.208686] do_raw_spin_lock+0x177/0x190 [16364.210711] _raw_spin_lock+0x21/0x30 [16364.212466] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16364.214803] process_one_work+0x2c8/0x700 [16364.216698] worker_thread+0x296/0x6e0 [16364.218541] ? rescuer_thread+0x570/0x570 [16364.223453] kthread+0x1d1/0x200 [16364.226775] ? set_kthread_struct+0x70/0x70 [16364.227939] ret_from_fork+0x1f/0x30 [16372.069228] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16372.074306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16372.111836] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16372.130593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16372.135766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16372.139633] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16372.153826] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16372.162956] RAX: 00000000000042ea RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16372.168758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16372.174541] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16372.179518] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16372.185781] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16372.189831] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16372.195945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16372.202424] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16372.206498] Call Trace: [16372.208477] [16372.209524] ? show_regs.cold.9+0x22/0x2f [16372.211536] ? watchdog_timer_fn.cold.6+0x62/0x10a [16372.213811] ? __hrtimer_run_queues+0x1c2/0x4d0 [16372.216562] ? watchdog+0x40/0x40 [16372.219352] ? hrtimer_interrupt+0x128/0x2c0 [16372.221678] ? smp_apic_timer_interrupt+0xa6/0x270 [16372.224725] ? apic_timer_interrupt+0xf/0x20 [16372.227152] [16372.227946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16372.230560] do_raw_spin_lock+0x177/0x190 [16372.232850] _raw_spin_lock+0x21/0x30 [16372.235451] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16372.239484] ? do_raw_spin_unlock+0x75/0x190 [16372.242121] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16372.245012] ? do_wait_intr+0xf0/0xf0 [16372.247466] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16372.249750] mdc_close+0x53b/0xd70 [mdc] [16372.259112] lmv_close+0x18d/0x490 [lmv] [16372.264449] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16372.268836] ll_md_real_close+0xa5/0x2a0 [lustre] [16372.271814] ll_md_close+0x274/0x800 [lustre] [16372.273589] ll_file_release+0x128/0xc50 [lustre] [16372.276187] __fput+0x10d/0x440 [16372.279596] ____fput+0x12/0x20 [16372.281962] task_work_run+0xc6/0x110 [16372.285699] exit_to_usermode_loop+0x1dd/0x1f0 [16372.288589] do_syscall_64+0x3d6/0x440 [16372.289741] entry_SYSCALL_64_after_hwframe+0x49/0xae [16372.291541] RIP: 0033:0x7fa749552cd8 [16372.293504] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16372.309050] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16372.318207] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16372.323508] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16372.328326] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16372.337599] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16372.340684] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16374.009184] rcu: INFO: rcu_sched self-detected stall on CPU [16374.012827] rcu: 2-...!: (8509334 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [16374.019117] (t=14460320 jiffies g=258701 q=5395) [16374.024202] rcu: rcu_sched kthread starved for 14457115 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [16374.032996] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16374.041048] rcu: RCU grace-period kthread stack dump: [16374.048609] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16374.052097] Call Trace: [16374.053263] __schedule+0x351/0xcb0 [16374.056291] schedule+0xc0/0x180 [16374.057751] schedule_timeout+0xb4/0x190 [16374.060122] ? do_raw_spin_unlock+0x75/0x190 [16374.062535] ? __next_timer_interrupt+0x160/0x160 [16374.064325] ? prepare_to_swait_event+0xf3/0x140 [16374.066459] rcu_gp_kthread+0x8dd/0x15b0 [16374.068242] ? force_qs_rnp+0x2c0/0x2c0 [16374.071151] kthread+0x1d1/0x200 [16374.072085] ? set_kthread_struct+0x70/0x70 [16374.075809] ret_from_fork+0x1f/0x30 [16374.079198] rcu: Stack dump where RCU GP kthread last ran: [16374.083805] NMI backtrace for cpu 2 [16374.087028] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16374.096116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16374.102515] Call Trace: [16374.103839] [16374.108620] ? dump_stack+0xbb/0x10e [16374.113348] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16374.119148] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16374.127973] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16374.136226] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16374.138439] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16374.143816] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [16374.148857] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [16374.151755] ? update_process_times+0x6d/0xc0 [16374.157858] ? tick_sched_handle.isra.18+0x27/0x90 [16374.160204] ? tick_sched_timer+0x53/0xb0 [16374.163067] ? __hrtimer_run_queues+0x1c2/0x4d0 [16374.174839] ? tick_sched_handle.isra.18+0x90/0x90 [16374.179223] ? hrtimer_interrupt+0x128/0x2c0 [16374.186859] ? smp_apic_timer_interrupt+0xa6/0x270 [16374.194839] ? apic_timer_interrupt+0xf/0x20 [16374.199686] [16374.200304] ? native_safe_halt+0x2e/0x50 [16374.210904] ? kvm_wait+0xcd/0xe0 [16374.216730] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16374.230848] ? do_raw_spin_lock+0x177/0x190 [16374.240042] ? _raw_spin_lock+0x21/0x30 [16374.245136] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16374.259112] ? obd_mod_free+0x46/0x140 [ptlrpc] [16374.265981] ? mdc_commit_open+0x91/0xb0 [mdc] [16374.269778] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16374.274278] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16374.281369] ? after_reply+0xe2c/0x1780 [ptlrpc] [16374.287746] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16374.293435] ? __next_timer_interrupt+0x160/0x160 [16374.296069] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16374.300491] ? do_wait_intr+0xf0/0xf0 [16374.302696] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16374.306351] ? kthread+0x1d1/0x200 [16374.310451] ? set_kthread_struct+0x70/0x70 [16374.312960] ? ret_from_fork+0x1f/0x30 [16374.316750] NMI backtrace for cpu 2 [16374.321077] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16374.333299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16374.347812] Call Trace: [16374.351762] [16374.354728] ? dump_stack+0xbb/0x10e [16374.356825] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16374.361064] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16374.363214] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16374.365884] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16374.369335] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16374.372242] ? rcu_dump_cpu_stacks+0x11b/0x171 [16374.375544] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16374.383148] ? update_process_times+0x6d/0xc0 [16374.386429] ? tick_sched_handle.isra.18+0x27/0x90 [16374.392411] ? tick_sched_timer+0x53/0xb0 [16374.393786] ? __hrtimer_run_queues+0x1c2/0x4d0 [16374.396726] ? tick_sched_handle.isra.18+0x90/0x90 [16374.398650] ? hrtimer_interrupt+0x128/0x2c0 [16374.401796] ? smp_apic_timer_interrupt+0xa6/0x270 [16374.405808] ? apic_timer_interrupt+0xf/0x20 [16374.408732] [16374.411004] ? native_safe_halt+0x2e/0x50 [16374.418313] ? kvm_wait+0xcd/0xe0 [16374.422818] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16374.431258] ? do_raw_spin_lock+0x177/0x190 [16374.444204] ? _raw_spin_lock+0x21/0x30 [16374.456627] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16374.465142] ? obd_mod_free+0x46/0x140 [ptlrpc] [16374.466811] ? mdc_commit_open+0x91/0xb0 [mdc] [16374.472755] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16374.477749] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16374.486507] ? after_reply+0xe2c/0x1780 [ptlrpc] [16374.497055] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16374.505545] ? __next_timer_interrupt+0x160/0x160 [16374.509227] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16374.512193] ? do_wait_intr+0xf0/0xf0 [16374.513374] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16374.518704] ? kthread+0x1d1/0x200 [16374.522718] ? set_kthread_struct+0x70/0x70 [16374.526821] ? ret_from_fork+0x1f/0x30 [16388.056201] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16388.060425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16388.099507] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16388.110143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16388.118759] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16388.125276] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16388.139936] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16388.144323] RAX: 0000000000007b8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16388.149232] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16388.158545] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16388.165225] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16388.168435] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16388.171361] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16388.176933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16388.180271] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16388.183736] Call Trace: [16388.184717] [16388.188662] ? show_regs.cold.9+0x22/0x2f [16388.190910] ? watchdog_timer_fn.cold.6+0x62/0x10a [16388.196736] ? __hrtimer_run_queues+0x1c2/0x4d0 [16388.200243] ? watchdog+0x40/0x40 [16388.204592] ? hrtimer_interrupt+0x128/0x2c0 [16388.208283] ? smp_apic_timer_interrupt+0xa6/0x270 [16388.212562] ? apic_timer_interrupt+0xf/0x20 [16388.215170] [16388.216035] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16388.218610] do_raw_spin_lock+0x177/0x190 [16388.220146] _raw_spin_lock+0x21/0x30 [16388.221704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16388.224318] ? do_raw_spin_unlock+0x75/0x190 [16388.226182] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16388.228253] ? do_wait_intr+0xf0/0xf0 [16388.229715] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16388.236285] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16388.238976] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16388.242759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16388.250161] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16388.254791] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16388.257127] mdc_intent_lock+0x353/0x7e0 [mdc] [16388.259934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16388.262495] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16388.265961] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16388.271166] lmv_intent_open+0x338/0x1150 [lmv] [16388.273832] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16388.276741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16388.279375] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16388.282143] ll_intent_lock+0xc5/0xaa0 [lustre] [16388.288002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16388.291759] ll_lookup_it+0xe0e/0x2270 [lustre] [16388.294005] ? do_raw_spin_unlock+0x75/0x190 [16388.295929] ? _raw_spin_unlock+0x12/0x30 [16388.297599] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16388.300763] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16388.303269] ll_atomic_open+0x36a/0xd70 [lustre] [16388.308511] lookup_open+0x356/0xb40 [16388.311788] do_last+0x313/0xfc0 [16388.313946] ? nd_jump_root+0xe5/0x160 [16388.316765] ? path_init+0x437/0x520 [16388.318973] path_openat+0xf7/0x500 [16388.320994] ? do_raw_spin_unlock+0x75/0x190 [16388.322713] do_filp_open+0x99/0x140 [16388.324575] ? getname_flags+0x6e/0x330 [16388.326413] ? __check_object_size+0xff/0x256 [16388.328723] ? do_raw_spin_unlock+0x75/0x190 [16388.331285] ? _raw_spin_unlock+0x12/0x30 [16388.333220] do_sys_openat2+0x2b4/0x410 [16388.336620] do_sys_open+0x73/0xa0 [16388.345968] __x64_sys_openat+0x24/0x30 [16388.347926] do_syscall_64+0xc1/0x440 [16388.350571] entry_SYSCALL_64_after_hwframe+0x49/0xae [16388.354422] RIP: 0033:0x7fa749552332 [16388.356005] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16388.365539] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16388.368445] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16388.371610] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16388.374869] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16388.378180] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16388.381710] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16392.058558] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16392.067839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16392.113932] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16392.121857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16392.127178] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16392.129675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16392.132322] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16392.147785] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16392.151408] RAX: 0000000000007a65 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16392.155248] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16392.162134] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16392.167261] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16392.174380] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16392.181829] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16392.186378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16392.189001] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16392.193341] Call Trace: [16392.194778] [16392.197933] ? show_regs.cold.9+0x22/0x2f [16392.199926] ? watchdog_timer_fn.cold.6+0x62/0x10a [16392.204797] ? __hrtimer_run_queues+0x1c2/0x4d0 [16392.211800] ? watchdog+0x40/0x40 [16392.214044] ? hrtimer_interrupt+0x128/0x2c0 [16392.217588] ? smp_apic_timer_interrupt+0xa6/0x270 [16392.219099] ? apic_timer_interrupt+0xf/0x20 [16392.222304] [16392.223348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16392.225831] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16392.227702] do_raw_spin_lock+0x177/0x190 [16392.229961] _raw_spin_lock+0x21/0x30 [16392.231912] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16392.235706] process_one_work+0x2c8/0x700 [16392.243673] worker_thread+0x296/0x6e0 [16392.251317] ? rescuer_thread+0x570/0x570 [16392.254727] kthread+0x1d1/0x200 [16392.257370] ? set_kthread_struct+0x70/0x70 [16392.263507] ret_from_fork+0x1f/0x30 [16392.673743] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14475s! [16392.680896] Showing busy workqueues and worker pools: [16392.685825] workqueue events: flags=0x0 [16392.689319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16392.693477] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16392.700572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16392.704989] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16392.711416] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.716328] pending: kfree_rcu_monitor [16392.719190] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16392.722919] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16392.727934] workqueue events_long: flags=0x0 [16392.730225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.733659] pending: lu_objects_flush [obdclass] [16392.736521] workqueue events_unbound: flags=0x2 [16392.739531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16392.742914] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16392.748077] workqueue events_power_efficient: flags=0x80 [16392.750889] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.754753] pending: check_lifetime [16392.756795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.760162] pending: neigh_periodic_work [16392.762288] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.765502] pending: neigh_periodic_work [16392.767498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.771020] pending: do_cache_clean [16392.773423] workqueue mm_percpu_wq: flags=0x8 [16392.775460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.779015] pending: vmstat_update [16392.781452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.785072] pending: vmstat_update [16392.787174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16392.790607] pending: vmstat_update [16392.792989] workqueue writeback: flags=0x4a [16392.797529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16392.804868] pending: wb_workfn, wb_workfn [16392.809168] workqueue xprtiod: flags=0xa [16392.813135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16392.818498] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16392.823395] workqueue ipv6_addrconf: flags=0x40008 [16392.826195] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16392.829727] pending: addrconf_verify_work [16392.832381] workqueue knbd0-recv: flags=0x1a [16392.834408] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16392.838187] in-flight: 651:recv_work [16392.840402] workqueue ptlrpc_pinger: flags=0x2 [16392.842294] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16392.846176] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16392.850946] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14475s workers=4 idle: 37 31846 28897 [16392.857078] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16382s workers=2 idle: 653 [16400.058059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16400.062221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16400.062610] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16400.094449] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16400.106287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16400.107087] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16400.107095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16400.107098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16400.107112] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16400.107116] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16400.107120] RAX: 000000000000388a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16400.107123] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16400.107125] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16400.107127] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16400.107130] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16400.107133] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16400.107136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16400.107138] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16400.107147] Call Trace: [16400.107152] [16400.107158] ? show_regs.cold.9+0x22/0x2f [16400.107166] ? watchdog_timer_fn.cold.6+0x62/0x10a [16400.107172] ? __hrtimer_run_queues+0x1c2/0x4d0 [16400.107176] ? watchdog+0x40/0x40 [16400.107180] ? hrtimer_interrupt+0x128/0x2c0 [16400.107184] ? smp_apic_timer_interrupt+0xa6/0x270 [16400.107191] ? apic_timer_interrupt+0xf/0x20 [16400.107196] [16400.107199] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16400.107206] do_raw_spin_lock+0x177/0x190 [16400.107210] _raw_spin_lock+0x21/0x30 [16400.107215] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16400.118272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16400.118363] RIP: 0010:native_safe_halt+0x2e/0x50 [16400.118378] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16400.118382] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16400.118388] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16400.118390] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16400.118393] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16400.118395] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16400.118398] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16400.118401] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16400.118404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16400.118406] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16400.118419] Call Trace: [16400.118486] [16400.118492] ? show_regs.cold.9+0x22/0x2f [16400.118509] ? watchdog_timer_fn.cold.6+0x62/0x10a [16400.118515] ? __hrtimer_run_queues+0x1c2/0x4d0 [16400.118519] ? watchdog+0x40/0x40 [16400.118523] ? hrtimer_interrupt+0x128/0x2c0 [16400.118526] ? smp_apic_timer_interrupt+0xa6/0x270 [16400.118532] ? apic_timer_interrupt+0xf/0x20 [16400.118536] [16400.118539] ? native_safe_halt+0x2e/0x50 [16400.118543] kvm_wait+0xcd/0xe0 [16400.118550] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16400.118652] do_raw_spin_lock+0x177/0x190 [16400.118657] _raw_spin_lock+0x21/0x30 [16400.118663] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16400.119878] obd_mod_free+0x46/0x140 [ptlrpc] [16400.157143] ? do_raw_spin_unlock+0x75/0x190 [16400.157163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16400.157608] ? do_wait_intr+0xf0/0xf0 [16400.157617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16400.161669] mdc_commit_open+0x91/0xb0 [mdc] [16400.161740] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16400.162468] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16400.162679] after_reply+0xe2c/0x1780 [ptlrpc] [16400.162902] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16400.165583] ? __next_timer_interrupt+0x160/0x160 [16400.165596] ptlrpcd+0xa82/0xe40 [ptlrpc] [16400.167062] ? do_wait_intr+0xf0/0xf0 [16400.167074] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16400.167336] kthread+0x1d1/0x200 [16400.167343] ? set_kthread_struct+0x70/0x70 [16400.167347] ret_from_fork+0x1f/0x30 [16400.380130] mdc_close+0x53b/0xd70 [mdc] [16400.381590] lmv_close+0x18d/0x490 [lmv] [16400.383042] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16400.385191] ll_md_real_close+0xa5/0x2a0 [lustre] [16400.387275] ll_md_close+0x274/0x800 [lustre] [16400.388864] ll_file_release+0x128/0xc50 [lustre] [16400.390742] __fput+0x10d/0x440 [16400.391814] ____fput+0x12/0x20 [16400.392910] task_work_run+0xc6/0x110 [16400.394386] exit_to_usermode_loop+0x1dd/0x1f0 [16400.396051] do_syscall_64+0x3d6/0x440 [16400.397659] entry_SYSCALL_64_after_hwframe+0x49/0xae [16400.399755] RIP: 0033:0x7fa749552cd8 [16400.405254] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16400.414420] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16400.421683] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16400.426981] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16400.430888] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16400.435038] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16400.442541] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16420.054377] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16420.060495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16420.065313] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16420.098129] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16420.100893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16420.101044] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16420.101051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16420.101057] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16420.101411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16420.101422] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16420.101426] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16420.101430] RAX: 0000000000006a2e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16420.101432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16420.101435] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16420.101437] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16420.101439] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16420.101452] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16420.101455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16420.101570] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16420.101577] Call Trace: [16420.101582] [16420.101586] ? show_regs.cold.9+0x22/0x2f [16420.101594] ? watchdog_timer_fn.cold.6+0x62/0x10a [16420.101598] ? __hrtimer_run_queues+0x1c2/0x4d0 [16420.101601] ? watchdog+0x40/0x40 [16420.101604] ? hrtimer_interrupt+0x128/0x2c0 [16420.101606] ? smp_apic_timer_interrupt+0xa6/0x270 [16420.101613] ? apic_timer_interrupt+0xf/0x20 [16420.101616] [16420.101618] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16420.101621] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16420.101625] do_raw_spin_lock+0x177/0x190 [16420.101628] _raw_spin_lock+0x21/0x30 [16420.101631] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16420.101812] process_one_work+0x2c8/0x700 [16420.101820] worker_thread+0x296/0x6e0 [16420.101825] ? rescuer_thread+0x570/0x570 [16420.101828] kthread+0x1d1/0x200 [16420.101832] ? set_kthread_struct+0x70/0x70 [16420.101836] ret_from_fork+0x1f/0x30 [16420.294551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16420.300823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16420.309806] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16420.326969] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16420.331530] RAX: 0000000000002409 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16420.335297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16420.338887] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16420.342555] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16420.346394] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16420.350013] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16420.353784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16420.356963] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16420.360682] Call Trace: [16420.361810] [16420.362764] ? show_regs.cold.9+0x22/0x2f [16420.364983] ? watchdog_timer_fn.cold.6+0x62/0x10a [16420.367769] ? __hrtimer_run_queues+0x1c2/0x4d0 [16420.370433] ? watchdog+0x40/0x40 [16420.371993] ? hrtimer_interrupt+0x128/0x2c0 [16420.374276] ? smp_apic_timer_interrupt+0xa6/0x270 [16420.376160] ? apic_timer_interrupt+0xf/0x20 [16420.378270] [16420.379200] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16420.381794] do_raw_spin_lock+0x177/0x190 [16420.384551] _raw_spin_lock+0x21/0x30 [16420.386321] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16420.388999] ? do_raw_spin_unlock+0x75/0x190 [16420.390934] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16420.393210] ? do_wait_intr+0xf0/0xf0 [16420.394164] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16420.396241] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16420.398302] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16420.401032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16420.403885] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16420.406027] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16420.408338] mdc_intent_lock+0x353/0x7e0 [mdc] [16420.410335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16420.412714] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16420.415811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16420.418279] lmv_intent_open+0x338/0x1150 [lmv] [16420.420644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16420.423275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16420.425953] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16420.427840] ll_intent_lock+0xc5/0xaa0 [lustre] [16420.429938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16420.432663] ll_lookup_it+0xe0e/0x2270 [lustre] [16420.434790] ? do_raw_spin_unlock+0x75/0x190 [16420.436481] ? _raw_spin_unlock+0x12/0x30 [16420.438577] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16420.441075] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16420.443452] ll_atomic_open+0x36a/0xd70 [lustre] [16420.445899] lookup_open+0x356/0xb40 [16420.447649] do_last+0x313/0xfc0 [16420.449276] ? nd_jump_root+0xe5/0x160 [16420.450950] ? path_init+0x437/0x520 [16420.453115] path_openat+0xf7/0x500 [16420.454802] ? do_raw_spin_unlock+0x75/0x190 [16420.457370] do_filp_open+0x99/0x140 [16420.459237] ? getname_flags+0x6e/0x330 [16420.461403] ? __check_object_size+0xff/0x256 [16420.463408] ? do_raw_spin_unlock+0x75/0x190 [16420.465206] ? _raw_spin_unlock+0x12/0x30 [16420.466974] do_sys_openat2+0x2b4/0x410 [16420.468902] do_sys_open+0x73/0xa0 [16420.470638] __x64_sys_openat+0x24/0x30 [16420.472317] do_syscall_64+0xc1/0x440 [16420.474319] entry_SYSCALL_64_after_hwframe+0x49/0xae [16420.476885] RIP: 0033:0x7fa749552332 [16420.478824] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16420.487441] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16420.490685] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16420.493946] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16420.497896] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16420.501389] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16420.504859] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16422.887144] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14505s! [16422.894596] Showing busy workqueues and worker pools: [16422.902155] workqueue events: flags=0x0 [16422.903792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16422.913256] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16422.933342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16422.941701] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16422.951576] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16422.956451] pending: kfree_rcu_monitor [16422.963894] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16422.971746] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16422.984111] workqueue events_long: flags=0x0 [16422.989339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16422.993260] pending: lu_objects_flush [obdclass] [16422.997245] workqueue events_unbound: flags=0x2 [16422.999687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16423.004301] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16423.012551] workqueue events_power_efficient: flags=0x80 [16423.015375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16423.023261] pending: check_lifetime [16423.027517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16423.033323] pending: neigh_periodic_work [16423.035474] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16423.043613] pending: neigh_periodic_work [16423.047492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16423.055721] pending: do_cache_clean [16423.060768] workqueue mm_percpu_wq: flags=0x8 [16423.065832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16423.074176] pending: vmstat_update [16423.082563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16423.091421] pending: vmstat_update [16423.093884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16423.101375] pending: vmstat_update [16423.105347] workqueue writeback: flags=0x4a [16423.108292] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16423.112829] pending: wb_workfn, wb_workfn [16423.121544] workqueue xprtiod: flags=0xa [16423.125322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16423.130246] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16423.136322] workqueue ipv6_addrconf: flags=0x40008 [16423.141383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16423.148957] pending: addrconf_verify_work [16423.154543] workqueue knbd0-recv: flags=0x1a [16423.157736] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16423.165922] in-flight: 651:recv_work [16423.167590] workqueue ptlrpc_pinger: flags=0x2 [16423.173240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16423.176096] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16423.180367] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14506s workers=4 idle: 37 31846 28897 [16423.184546] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16413s workers=2 idle: 653 [16428.059106] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16428.060748] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16428.061845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16428.061918] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16428.061926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16428.061929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16428.061943] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16428.061947] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16428.061952] RAX: 00000000000043b4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16428.061954] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16428.061956] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16428.061958] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16428.061960] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16428.061963] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16428.061966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16428.061969] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16428.061978] Call Trace: [16428.061984] [16428.061990] ? show_regs.cold.9+0x22/0x2f [16428.062000] ? watchdog_timer_fn.cold.6+0x62/0x10a [16428.062005] ? __hrtimer_run_queues+0x1c2/0x4d0 [16428.062009] ? watchdog+0x40/0x40 [16428.062013] ? hrtimer_interrupt+0x128/0x2c0 [16428.062018] ? smp_apic_timer_interrupt+0xa6/0x270 [16428.062025] ? apic_timer_interrupt+0xf/0x20 [16428.062029] [16428.062032] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16428.062038] do_raw_spin_lock+0x177/0x190 [16428.062043] _raw_spin_lock+0x21/0x30 [16428.062047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16428.062317] ? do_raw_spin_unlock+0x75/0x190 [16428.062322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16428.062514] ? do_wait_intr+0xf0/0xf0 [16428.062521] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16428.062665] mdc_close+0x53b/0xd70 [mdc] [16428.062692] lmv_close+0x18d/0x490 [lmv] [16428.062711] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16428.062794] ll_md_real_close+0xa5/0x2a0 [lustre] [16428.062857] ll_md_close+0x274/0x800 [lustre] [16428.062940] ll_file_release+0x128/0xc50 [lustre] [16428.063021] __fput+0x10d/0x440 [16428.063029] ____fput+0x12/0x20 [16428.063033] task_work_run+0xc6/0x110 [16428.063039] exit_to_usermode_loop+0x1dd/0x1f0 [16428.063044] do_syscall_64+0x3d6/0x440 [16428.063048] entry_SYSCALL_64_after_hwframe+0x49/0xae [16428.063054] RIP: 0033:0x7fa749552cd8 [16428.063060] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16428.063063] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16428.063067] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16428.063069] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16428.063071] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16428.063073] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16428.063075] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16428.308171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16428.327364] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16428.338268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16428.343867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16428.345527] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16428.359058] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16428.364427] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16428.371359] RDX: 0000000000002448 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16428.377477] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16428.380460] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16428.392760] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16428.399991] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16428.402041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16428.410113] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16428.416860] Call Trace: [16428.421323] [16428.422123] ? show_regs.cold.9+0x22/0x2f [16428.429718] ? watchdog_timer_fn.cold.6+0x62/0x10a [16428.433393] ? __hrtimer_run_queues+0x1c2/0x4d0 [16428.437337] ? watchdog+0x40/0x40 [16428.440233] ? hrtimer_interrupt+0x128/0x2c0 [16428.443432] ? smp_apic_timer_interrupt+0xa6/0x270 [16428.450868] ? apic_timer_interrupt+0xf/0x20 [16428.453491] [16428.454443] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16428.459495] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16428.468049] do_raw_spin_lock+0x177/0x190 [16428.469599] _raw_spin_lock+0x21/0x30 [16428.471017] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16428.480277] obd_mod_free+0x46/0x140 [ptlrpc] [16428.484767] mdc_commit_open+0x91/0xb0 [mdc] [16428.486227] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16428.490525] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16428.497625] after_reply+0xe2c/0x1780 [ptlrpc] [16428.500344] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16428.503476] ? __next_timer_interrupt+0x160/0x160 [16428.507487] ptlrpcd+0xa82/0xe40 [ptlrpc] [16428.511503] ? do_wait_intr+0xf0/0xf0 [16428.514522] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16428.521075] kthread+0x1d1/0x200 [16428.522240] ? set_kthread_struct+0x70/0x70 [16428.523717] ret_from_fork+0x1f/0x30 [16448.054023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16448.057029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16448.057385] Modules linked in: lustre(O) [16448.061706] Modules linked in: [16448.061711] osp(O) [16448.063928] lustre(O) osp(O) [16448.065501] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16448.065567] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16448.065645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16448.065648] RIP: 0010:native_safe_halt+0x2e/0x50 [16448.065663] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16448.065666] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16448.065671] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16448.065673] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16448.065675] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16448.065677] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16448.065679] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16448.065683] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16448.065686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16448.065688] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16448.065696] Call Trace: [16448.065702] [16448.065708] ? show_regs.cold.9+0x22/0x2f [16448.065718] ? watchdog_timer_fn.cold.6+0x62/0x10a [16448.065724] ? __hrtimer_run_queues+0x1c2/0x4d0 [16448.065729] ? watchdog+0x40/0x40 [16448.065733] ? hrtimer_interrupt+0x128/0x2c0 [16448.065736] ? smp_apic_timer_interrupt+0xa6/0x270 [16448.065743] ? apic_timer_interrupt+0xf/0x20 [16448.065747] [16448.065749] ? native_safe_halt+0x2e/0x50 [16448.065752] kvm_wait+0xcd/0xe0 [16448.065759] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16448.065768] do_raw_spin_lock+0x177/0x190 [16448.065773] _raw_spin_lock+0x21/0x30 [16448.065777] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16448.066258] ? do_raw_spin_unlock+0x75/0x190 [16448.066264] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16448.066474] ? do_wait_intr+0xf0/0xf0 [16448.066483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16448.066782] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16448.067104] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16448.067314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16448.067429] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16448.067469] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16448.067509] mdc_intent_lock+0x353/0x7e0 [mdc] [16448.067550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16448.067718] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16448.068858] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16448.068938] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16448.068944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16448.068951] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16448.069186] RIP: 0010:native_safe_halt+0x2e/0x50 [16448.069195] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16448.069199] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [16448.069204] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16448.069207] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16448.069209] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16448.069211] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16448.069214] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16448.069217] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16448.069220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16448.069222] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16448.069230] Call Trace: [16448.069235] [16448.069238] ? show_regs.cold.9+0x22/0x2f [16448.069248] ? watchdog_timer_fn.cold.6+0x62/0x10a [16448.069254] ? __hrtimer_run_queues+0x1c2/0x4d0 [16448.069258] ? watchdog+0x40/0x40 [16448.069263] ? hrtimer_interrupt+0x128/0x2c0 [16448.069267] ? smp_apic_timer_interrupt+0xa6/0x270 [16448.069275] ? apic_timer_interrupt+0xf/0x20 [16448.069280] [16448.069283] ? native_safe_halt+0x2e/0x50 [16448.069286] kvm_wait+0xcd/0xe0 [16448.069294] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16448.069303] do_raw_spin_lock+0x177/0x190 [16448.069307] _raw_spin_lock+0x21/0x30 [16448.069312] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16448.103188] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16448.103313] lmv_intent_open+0x338/0x1150 [lmv] [16448.103344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16448.103457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16448.103538] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16448.103560] ll_intent_lock+0xc5/0xaa0 [lustre] [16448.103766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16448.103851] ll_lookup_it+0xe0e/0x2270 [lustre] [16448.103994] ? do_raw_spin_unlock+0x75/0x190 [16448.104002] ? _raw_spin_unlock+0x12/0x30 [16448.104009] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16448.104039] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16448.104061] ll_atomic_open+0x36a/0xd70 [lustre] [16448.104147] lookup_open+0x356/0xb40 [16448.104157] do_last+0x313/0xfc0 [16448.104162] ? nd_jump_root+0xe5/0x160 [16448.104167] ? path_init+0x437/0x520 [16448.104171] path_openat+0xf7/0x500 [16448.104177] ? do_raw_spin_unlock+0x75/0x190 [16448.104181] do_filp_open+0x99/0x140 [16448.104187] ? getname_flags+0x6e/0x330 [16448.104192] ? __check_object_size+0xff/0x256 [16448.104200] ? do_raw_spin_unlock+0x75/0x190 [16448.104203] ? _raw_spin_unlock+0x12/0x30 [16448.104208] do_sys_openat2+0x2b4/0x410 [16448.104284] do_sys_open+0x73/0xa0 [16448.104289] __x64_sys_openat+0x24/0x30 [16448.104293] do_syscall_64+0xc1/0x440 [16448.104299] entry_SYSCALL_64_after_hwframe+0x49/0xae [16448.104306] RIP: 0033:0x7fa749552332 [16448.104313] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16448.104317] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16448.104322] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16448.104324] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16448.104327] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16448.104329] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16448.104331] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16448.524334] process_one_work+0x2c8/0x700 [16448.525534] worker_thread+0x296/0x6e0 [16448.526965] ? rescuer_thread+0x570/0x570 [16448.528316] kthread+0x1d1/0x200 [16448.529350] ? set_kthread_struct+0x70/0x70 [16448.532261] ret_from_fork+0x1f/0x30 [16453.606273] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14536s! [16453.617928] Showing busy workqueues and worker pools: [16453.622156] workqueue events: flags=0x0 [16453.627010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16453.630949] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16453.651101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16453.653169] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16453.661962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.668013] pending: kfree_rcu_monitor [16453.670850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16453.679688] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16453.689291] workqueue events_long: flags=0x0 [16453.692864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.697361] pending: lu_objects_flush [obdclass] [16453.700267] workqueue events_unbound: flags=0x2 [16453.703048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16453.710893] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16453.715192] workqueue events_power_efficient: flags=0x80 [16453.721746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.729798] pending: check_lifetime [16453.734945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.740080] pending: neigh_periodic_work [16453.742279] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.747805] pending: neigh_periodic_work [16453.754508] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.756366] pending: do_cache_clean [16453.757395] workqueue mm_percpu_wq: flags=0x8 [16453.768706] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.777937] pending: vmstat_update [16453.784383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.796399] pending: vmstat_update [16453.797755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16453.807023] pending: vmstat_update [16453.815602] workqueue writeback: flags=0x4a [16453.819284] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16453.829998] pending: wb_workfn, wb_workfn [16453.831385] workqueue xprtiod: flags=0xa [16453.843124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16453.856473] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16453.867970] workqueue ipv6_addrconf: flags=0x40008 [16453.869913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16453.874251] pending: addrconf_verify_work [16453.875713] workqueue knbd0-recv: flags=0x1a [16453.883753] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16453.893940] in-flight: 651:recv_work [16453.895906] workqueue ptlrpc_pinger: flags=0x2 [16453.899810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16453.911159] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16453.919386] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14536s workers=4 idle: 37 31846 28897 [16453.926796] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16443s workers=2 idle: 653 [16456.058214] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16456.061612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16456.061786] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16456.084844] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16456.084860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16456.084863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16456.084880] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16456.084884] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16456.084890] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16456.084893] RDX: 0000000000005829 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16456.084896] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16456.084898] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16456.084901] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16456.084904] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16456.084907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16456.084909] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16456.084917] Call Trace: [16456.084922] [16456.084927] ? show_regs.cold.9+0x22/0x2f [16456.084935] ? watchdog_timer_fn.cold.6+0x62/0x10a [16456.084941] ? __hrtimer_run_queues+0x1c2/0x4d0 [16456.084945] ? watchdog+0x40/0x40 [16456.084950] ? hrtimer_interrupt+0x128/0x2c0 [16456.084958] ? smp_apic_timer_interrupt+0xa6/0x270 [16456.084966] ? apic_timer_interrupt+0xf/0x20 [16456.084971] [16456.084974] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16456.084978] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16456.084985] do_raw_spin_lock+0x177/0x190 [16456.084990] _raw_spin_lock+0x21/0x30 [16456.084997] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16456.085367] obd_mod_free+0x46/0x140 [ptlrpc] [16456.085646] mdc_commit_open+0x91/0xb0 [mdc] [16456.085683] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16456.086023] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16456.086237] after_reply+0xe2c/0x1780 [ptlrpc] [16456.086633] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16456.086859] ? __next_timer_interrupt+0x160/0x160 [16456.086869] ptlrpcd+0xa82/0xe40 [ptlrpc] [16456.087098] ? do_wait_intr+0xf0/0xf0 [16456.087108] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16456.087333] kthread+0x1d1/0x200 [16456.087339] ? set_kthread_struct+0x70/0x70 [16456.087343] ret_from_fork+0x1f/0x30 [16456.209787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16456.239417] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16456.247265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16456.253156] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16456.256004] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16456.264378] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16456.268945] RAX: 0000000000000389 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16456.272195] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16456.275842] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16456.279399] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16456.283332] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16456.287696] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16456.290943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16456.293208] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16456.296557] Call Trace: [16456.297990] [16456.299661] ? show_regs.cold.9+0x22/0x2f [16456.301513] ? watchdog_timer_fn.cold.6+0x62/0x10a [16456.303970] ? __hrtimer_run_queues+0x1c2/0x4d0 [16456.306180] ? watchdog+0x40/0x40 [16456.307862] ? hrtimer_interrupt+0x128/0x2c0 [16456.309914] ? smp_apic_timer_interrupt+0xa6/0x270 [16456.312732] ? apic_timer_interrupt+0xf/0x20 [16456.314686] [16456.315667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16456.318101] do_raw_spin_lock+0x177/0x190 [16456.319562] _raw_spin_lock+0x21/0x30 [16456.320654] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16456.322934] ? do_raw_spin_unlock+0x75/0x190 [16456.324194] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16456.326781] ? do_wait_intr+0xf0/0xf0 [16456.328357] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16456.331359] mdc_close+0x53b/0xd70 [mdc] [16456.333596] lmv_close+0x18d/0x490 [lmv] [16456.335164] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16456.338099] ll_md_real_close+0xa5/0x2a0 [lustre] [16456.340822] ll_md_close+0x274/0x800 [lustre] [16456.342957] ll_file_release+0x128/0xc50 [lustre] [16456.345102] __fput+0x10d/0x440 [16456.346528] ____fput+0x12/0x20 [16456.348190] task_work_run+0xc6/0x110 [16456.349736] exit_to_usermode_loop+0x1dd/0x1f0 [16456.351633] do_syscall_64+0x3d6/0x440 [16456.353267] entry_SYSCALL_64_after_hwframe+0x49/0xae [16456.355367] RIP: 0033:0x7fa749552cd8 [16456.357212] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16456.364593] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16456.367792] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16456.371069] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16456.374007] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16456.376800] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16456.380069] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16476.064477] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16476.075142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16476.132278] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16476.154571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16476.169804] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16476.181343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16476.186197] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16476.211350] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16476.230192] RAX: 0000000000007f8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16476.240430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16476.253244] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16476.263242] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16476.273003] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16476.287986] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16476.296603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16476.301978] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16476.305565] Call Trace: [16476.310150] [16476.311342] ? show_regs.cold.9+0x22/0x2f [16476.316929] ? watchdog_timer_fn.cold.6+0x62/0x10a [16476.321350] ? __hrtimer_run_queues+0x1c2/0x4d0 [16476.326313] ? watchdog+0x40/0x40 [16476.328208] ? hrtimer_interrupt+0x128/0x2c0 [16476.330610] ? smp_apic_timer_interrupt+0xa6/0x270 [16476.332343] ? apic_timer_interrupt+0xf/0x20 [16476.334420] [16476.335195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16476.337939] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16476.340045] do_raw_spin_lock+0x177/0x190 [16476.344615] _raw_spin_lock+0x21/0x30 [16476.347337] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16476.350488] process_one_work+0x2c8/0x700 [16476.358294] worker_thread+0x296/0x6e0 [16476.363969] ? rescuer_thread+0x570/0x570 [16476.372787] kthread+0x1d1/0x200 [16476.384489] ? set_kthread_struct+0x70/0x70 [16476.390611] ret_from_fork+0x1f/0x30 [16480.054012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16480.057914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16480.079603] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16480.085411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16480.091474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16480.095094] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16480.103222] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16480.107011] RAX: 000000000000451e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16480.113062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16480.116966] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16480.119915] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16480.123012] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16480.126192] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16480.129447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16480.131815] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16480.134823] Call Trace: [16480.135750] [16480.136592] ? show_regs.cold.9+0x22/0x2f [16480.138439] ? watchdog_timer_fn.cold.6+0x62/0x10a [16480.140066] ? __hrtimer_run_queues+0x1c2/0x4d0 [16480.142501] ? watchdog+0x40/0x40 [16480.144193] ? hrtimer_interrupt+0x128/0x2c0 [16480.145962] ? smp_apic_timer_interrupt+0xa6/0x270 [16480.148411] ? apic_timer_interrupt+0xf/0x20 [16480.150262] [16480.151474] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16480.153394] do_raw_spin_lock+0x177/0x190 [16480.155302] _raw_spin_lock+0x21/0x30 [16480.157123] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16480.159377] ? do_raw_spin_unlock+0x75/0x190 [16480.161726] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16480.163817] ? do_wait_intr+0xf0/0xf0 [16480.165548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16480.168193] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16480.170739] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16480.173792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16480.176520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16480.179177] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16480.181263] mdc_intent_lock+0x353/0x7e0 [mdc] [16480.183606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16480.186179] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16480.189140] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16480.192175] lmv_intent_open+0x338/0x1150 [lmv] [16480.194135] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16480.196607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16480.199227] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16480.201235] ll_intent_lock+0xc5/0xaa0 [lustre] [16480.203476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16480.206179] ll_lookup_it+0xe0e/0x2270 [lustre] [16480.208803] ? do_raw_spin_unlock+0x75/0x190 [16480.210694] ? _raw_spin_unlock+0x12/0x30 [16480.213671] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16480.218433] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16480.221291] ll_atomic_open+0x36a/0xd70 [lustre] [16480.225359] lookup_open+0x356/0xb40 [16480.227793] do_last+0x313/0xfc0 [16480.229328] ? nd_jump_root+0xe5/0x160 [16480.231582] ? path_init+0x437/0x520 [16480.233244] path_openat+0xf7/0x500 [16480.234951] ? do_raw_spin_unlock+0x75/0x190 [16480.236908] do_filp_open+0x99/0x140 [16480.238837] ? getname_flags+0x6e/0x330 [16480.240996] ? __check_object_size+0xff/0x256 [16480.242915] ? do_raw_spin_unlock+0x75/0x190 [16480.245130] ? _raw_spin_unlock+0x12/0x30 [16480.247263] do_sys_openat2+0x2b4/0x410 [16480.249329] do_sys_open+0x73/0xa0 [16480.250939] __x64_sys_openat+0x24/0x30 [16480.252463] do_syscall_64+0xc1/0x440 [16480.254626] entry_SYSCALL_64_after_hwframe+0x49/0xae [16480.256706] RIP: 0033:0x7fa749552332 [16480.258302] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16480.266909] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16480.270158] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16480.273194] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16480.276454] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16480.279800] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16480.283491] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16484.058435] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16484.059064] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16484.069408] Modules linked in: [16484.073581] Modules linked in: [16484.073586] lustre(O) [16484.076086] lustre(O) [16484.077599] osp(O) [16484.079186] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16484.079272] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16484.079278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16484.079281] RIP: 0010:native_safe_halt+0x2e/0x50 [16484.079293] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16484.079298] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16484.079303] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16484.080587] ofd(O) [16484.082260] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16484.098786] lod(O) [16484.103451] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16484.103463] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16484.103465] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16484.103470] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16484.103473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16484.103475] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16484.103483] Call Trace: [16484.103489] [16484.103495] ? show_regs.cold.9+0x22/0x2f [16484.103509] ? watchdog_timer_fn.cold.6+0x62/0x10a [16484.103516] ? __hrtimer_run_queues+0x1c2/0x4d0 [16484.103520] ? watchdog+0x40/0x40 [16484.103524] ? hrtimer_interrupt+0x128/0x2c0 [16484.103529] ? smp_apic_timer_interrupt+0xa6/0x270 [16484.103538] ? apic_timer_interrupt+0xf/0x20 [16484.103543] [16484.103545] ? native_safe_halt+0x2e/0x50 [16484.103550] kvm_wait+0xcd/0xe0 [16484.103558] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16484.103567] do_raw_spin_lock+0x177/0x190 [16484.103572] _raw_spin_lock+0x21/0x30 [16484.103576] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16484.103888] ? do_raw_spin_unlock+0x75/0x190 [16484.103895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16484.108697] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16484.108772] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16484.108779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16484.108782] RIP: 0010:native_safe_halt+0x2e/0x50 [16484.108797] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16484.108802] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16484.108807] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16484.108809] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16484.108811] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16484.108813] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16484.108815] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16484.108818] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16484.108820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16484.108822] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16484.109171] Call Trace: [16484.113583] ? do_wait_intr+0xf0/0xf0 [16484.113606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16484.126229] [16484.126252] ? show_regs.cold.9+0x22/0x2f [16484.126265] ? watchdog_timer_fn.cold.6+0x62/0x10a [16484.126272] ? __hrtimer_run_queues+0x1c2/0x4d0 [16484.126276] ? watchdog+0x40/0x40 [16484.126279] ? hrtimer_interrupt+0x128/0x2c0 [16484.126283] ? smp_apic_timer_interrupt+0xa6/0x270 [16484.126291] ? apic_timer_interrupt+0xf/0x20 [16484.126295] [16484.126298] ? native_safe_halt+0x2e/0x50 [16484.126302] kvm_wait+0xcd/0xe0 [16484.126307] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16484.126313] do_raw_spin_lock+0x177/0x190 [16484.126317] _raw_spin_lock+0x21/0x30 [16484.126323] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16484.126577] obd_mod_free+0x46/0x140 [ptlrpc] [16484.126762] mdc_commit_open+0x91/0xb0 [mdc] [16484.126798] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16484.127191] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16484.130894] mdc_close+0x53b/0xd70 [mdc] [16484.133208] after_reply+0xe2c/0x1780 [ptlrpc] [16484.134121] lmv_close+0x18d/0x490 [lmv] [16484.134155] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16484.134284] ll_md_real_close+0xa5/0x2a0 [lustre] [16484.134364] ll_md_close+0x274/0x800 [lustre] [16484.134445] ll_file_release+0x128/0xc50 [lustre] [16484.134525] __fput+0x10d/0x440 [16484.134533] ____fput+0x12/0x20 [16484.134537] task_work_run+0xc6/0x110 [16484.134544] exit_to_usermode_loop+0x1dd/0x1f0 [16484.134551] do_syscall_64+0x3d6/0x440 [16484.134556] entry_SYSCALL_64_after_hwframe+0x49/0xae [16484.134565] RIP: 0033:0x7fa749552cd8 [16484.134573] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16484.134577] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16484.134582] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16484.134584] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16484.134587] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16484.134589] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16484.134591] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16484.443205] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16484.448215] ? __next_timer_interrupt+0x160/0x160 [16484.451910] ptlrpcd+0xa82/0xe40 [ptlrpc] [16484.457754] ? do_wait_intr+0xf0/0xf0 [16484.468424] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16484.476709] kthread+0x1d1/0x200 [16484.481138] ? set_kthread_struct+0x70/0x70 [16484.486277] ret_from_fork+0x1f/0x30 [16484.490040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14567s! [16484.494643] Showing busy workqueues and worker pools: [16484.499194] workqueue events: flags=0x0 [16484.500440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16484.507356] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16484.515145] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16484.518544] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16484.522555] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.525939] pending: kfree_rcu_monitor [16484.531854] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16484.535314] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16484.544282] workqueue events_long: flags=0x0 [16484.546120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.549811] pending: lu_objects_flush [obdclass] [16484.554422] workqueue events_unbound: flags=0x2 [16484.556291] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16484.562971] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16484.567851] workqueue events_power_efficient: flags=0x80 [16484.570396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.573871] pending: check_lifetime [16484.576154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.584168] pending: neigh_periodic_work [16484.585722] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.589934] pending: neigh_periodic_work [16484.592862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.597478] pending: do_cache_clean [16484.601639] workqueue mm_percpu_wq: flags=0x8 [16484.606032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.609448] pending: vmstat_update [16484.614682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.617329] pending: vmstat_update [16484.621297] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16484.626269] pending: vmstat_update [16484.631380] workqueue writeback: flags=0x4a [16484.633894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16484.636944] pending: wb_workfn, wb_workfn [16484.640428] workqueue xprtiod: flags=0xa [16484.641964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16484.645472] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16484.649346] workqueue ipv6_addrconf: flags=0x40008 [16484.651390] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16484.654446] pending: addrconf_verify_work [16484.656447] workqueue knbd0-recv: flags=0x1a [16484.658365] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16484.661327] in-flight: 651:recv_work [16484.663368] workqueue ptlrpc_pinger: flags=0x2 [16484.666561] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16484.669863] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16484.672468] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14567s workers=4 idle: 37 31846 28897 [16484.676082] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16474s workers=2 idle: 653 [16504.058622] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16504.066520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16504.104033] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16504.113839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16504.116597] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16504.126454] RIP: 0010:native_safe_halt+0x2e/0x50 [16504.134253] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16504.154738] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16504.167163] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16504.175951] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16504.182414] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16504.196811] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16504.206973] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16504.212963] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16504.228022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16504.232150] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16504.238093] Call Trace: [16504.240437] [16504.243657] ? show_regs.cold.9+0x22/0x2f [16504.245102] ? watchdog_timer_fn.cold.6+0x62/0x10a [16504.246766] ? __hrtimer_run_queues+0x1c2/0x4d0 [16504.257417] ? watchdog+0x40/0x40 [16504.264542] ? hrtimer_interrupt+0x128/0x2c0 [16504.265978] ? smp_apic_timer_interrupt+0xa6/0x270 [16504.274767] ? apic_timer_interrupt+0xf/0x20 [16504.282948] [16504.283784] ? native_safe_halt+0x2e/0x50 [16504.285587] kvm_wait+0xcd/0xe0 [16504.288579] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16504.294688] do_raw_spin_lock+0x177/0x190 [16504.296580] _raw_spin_lock+0x21/0x30 [16504.306930] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16504.310903] process_one_work+0x2c8/0x700 [16504.313002] worker_thread+0x296/0x6e0 [16504.318262] ? rescuer_thread+0x570/0x570 [16504.323847] kthread+0x1d1/0x200 [16504.333366] ? set_kthread_struct+0x70/0x70 [16504.335677] ret_from_fork+0x1f/0x30 [16512.055684] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16512.058023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16512.059025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16512.059033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16512.059110] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16512.059115] Modules linked in: [16512.059117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16512.059122] RIP: 0010:native_safe_halt+0x2e/0x50 [16512.059145] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16512.059150] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16512.059154] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16512.059225] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16512.059229] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16512.059232] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16512.059234] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16512.059238] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16512.059241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16512.059243] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16512.059251] Call Trace: [16512.059257] [16512.059262] ? show_regs.cold.9+0x22/0x2f [16512.059272] ? watchdog_timer_fn.cold.6+0x62/0x10a [16512.059284] ? __hrtimer_run_queues+0x1c2/0x4d0 [16512.059288] ? watchdog+0x40/0x40 [16512.059292] ? hrtimer_interrupt+0x128/0x2c0 [16512.059296] ? smp_apic_timer_interrupt+0xa6/0x270 [16512.059304] ? apic_timer_interrupt+0xf/0x20 [16512.059309] [16512.059313] ? native_safe_halt+0x2e/0x50 [16512.059317] kvm_wait+0xcd/0xe0 [16512.059442] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16512.059453] do_raw_spin_lock+0x177/0x190 [16512.059457] _raw_spin_lock+0x21/0x30 [16512.059462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16512.059731] ? do_raw_spin_unlock+0x75/0x190 [16512.059737] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16512.059948] ? do_wait_intr+0xf0/0xf0 [16512.060060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16512.060440] mdc_close+0x53b/0xd70 [mdc] [16512.060477] lmv_close+0x18d/0x490 [lmv] [16512.060505] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16512.060615] ll_md_real_close+0xa5/0x2a0 [lustre] [16512.060697] ll_md_close+0x274/0x800 [lustre] [16512.060796] ll_file_release+0x128/0xc50 [lustre] [16512.060882] __fput+0x10d/0x440 [16512.060890] ____fput+0x12/0x20 [16512.060895] task_work_run+0xc6/0x110 [16512.060904] exit_to_usermode_loop+0x1dd/0x1f0 [16512.060912] do_syscall_64+0x3d6/0x440 [16512.060916] entry_SYSCALL_64_after_hwframe+0x49/0xae [16512.060923] RIP: 0033:0x7fa749552cd8 [16512.060930] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16512.060933] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16512.060938] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16512.060940] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16512.060942] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16512.060945] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16512.060947] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16512.062913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs [16512.065247] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16512.065321] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16512.065328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16512.065332] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16512.065346] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16512.065351] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16512.065356] RAX: 00000000000001ad RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16512.065357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16512.065359] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16512.065361] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16512.065362] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16512.065365] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16512.065366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16512.065368] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16512.065375] Call Trace: [16512.065379] [16512.065384] ? show_regs.cold.9+0x22/0x2f [16512.065391] ? watchdog_timer_fn.cold.6+0x62/0x10a [16512.065395] ? __hrtimer_run_queues+0x1c2/0x4d0 [16512.065398] ? watchdog+0x40/0x40 [16512.065401] ? hrtimer_interrupt+0x128/0x2c0 [16512.065403] ? smp_apic_timer_interrupt+0xa6/0x270 [16512.065409] ? apic_timer_interrupt+0xf/0x20 [16512.065413] [16512.065415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16512.065419] do_raw_spin_lock+0x177/0x190 [16512.065422] _raw_spin_lock+0x21/0x30 [16512.065426] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16512.065625] ? do_raw_spin_unlock+0x75/0x190 [16512.065628] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16512.065767] ? do_wait_intr+0xf0/0xf0 [16512.065776] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16512.065970] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16512.066184] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16512.066395] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.066502] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16512.066538] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16512.066565] mdc_intent_lock+0x353/0x7e0 [mdc] [16512.066589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.066672] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16512.096058] ata_generic [16512.098975] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16512.099024] lmv_intent_open+0x338/0x1150 [lmv] [16512.099058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.109873] crct10dif_pclmul [16512.115566] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.125988] crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [16512.129855] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16512.137567] sha512_ssse3 sha512_generic [16512.137584] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16512.137591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16512.137593] RIP: 0010:native_safe_halt+0x2e/0x50 [16512.137668] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16512.137673] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16512.137678] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16512.137681] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16512.137683] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16512.137686] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16512.137688] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16512.137692] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16512.137695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16512.137698] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16512.137705] Call Trace: [16512.137711] [16512.137717] ? show_regs.cold.9+0x22/0x2f [16512.137726] ? watchdog_timer_fn.cold.6+0x62/0x10a [16512.137733] ? __hrtimer_run_queues+0x1c2/0x4d0 [16512.137738] ? watchdog+0x40/0x40 [16512.137742] ? hrtimer_interrupt+0x128/0x2c0 [16512.137756] ? smp_apic_timer_interrupt+0xa6/0x270 [16512.137764] ? apic_timer_interrupt+0xf/0x20 [16512.137769] [16512.137772] ? native_safe_halt+0x2e/0x50 [16512.137776] kvm_wait+0xcd/0xe0 [16512.137783] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16512.137790] do_raw_spin_lock+0x177/0x190 [16512.137795] _raw_spin_lock+0x21/0x30 [16512.137802] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16512.138059] obd_mod_free+0x46/0x140 [ptlrpc] [16512.143046] ll_intent_lock+0xc5/0xaa0 [lustre] [16512.154059] mdc_commit_open+0x91/0xb0 [mdc] [16512.156240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.160797] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16512.164566] ll_lookup_it+0xe0e/0x2270 [lustre] [16512.164716] ? do_raw_spin_unlock+0x75/0x190 [16512.164726] ? _raw_spin_unlock+0x12/0x30 [16512.164733] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16512.164780] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16512.164803] ll_atomic_open+0x36a/0xd70 [lustre] [16512.164905] lookup_open+0x356/0xb40 [16512.164914] do_last+0x313/0xfc0 [16512.164919] ? nd_jump_root+0xe5/0x160 [16512.164924] ? path_init+0x437/0x520 [16512.164928] path_openat+0xf7/0x500 [16512.164933] ? do_raw_spin_unlock+0x75/0x190 [16512.164938] do_filp_open+0x99/0x140 [16512.164945] ? getname_flags+0x6e/0x330 [16512.164949] ? __check_object_size+0xff/0x256 [16512.164956] ? do_raw_spin_unlock+0x75/0x190 [16512.164960] ? _raw_spin_unlock+0x12/0x30 [16512.164966] do_sys_openat2+0x2b4/0x410 [16512.164970] do_sys_open+0x73/0xa0 [16512.164974] __x64_sys_openat+0x24/0x30 [16512.164978] do_syscall_64+0xc1/0x440 [16512.164986] entry_SYSCALL_64_after_hwframe+0x49/0xae [16512.164992] RIP: 0033:0x7fa749552332 [16512.164999] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16512.165003] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16512.165009] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16512.165011] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16512.165014] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16512.165016] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16512.165018] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16512.777949] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16512.780644] after_reply+0xe2c/0x1780 [ptlrpc] [16512.783093] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16512.785910] ? __next_timer_interrupt+0x160/0x160 [16512.788089] ptlrpcd+0xa82/0xe40 [ptlrpc] [16512.790231] ? do_wait_intr+0xf0/0xf0 [16512.791890] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16512.794306] kthread+0x1d1/0x200 [16512.795586] ? set_kthread_struct+0x70/0x70 [16512.797159] ret_from_fork+0x1f/0x30 [16515.041042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14597s! [16515.043097] Showing busy workqueues and worker pools: [16515.044278] workqueue events: flags=0x0 [16515.045479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16515.047592] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16515.052048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16515.054091] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16515.056879] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.058885] pending: kfree_rcu_monitor [16515.060336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16515.062316] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16515.065029] workqueue events_long: flags=0x0 [16515.066259] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.068175] pending: lu_objects_flush [obdclass] [16515.069994] workqueue events_unbound: flags=0x2 [16515.071320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16515.073393] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16515.076482] workqueue events_power_efficient: flags=0x80 [16515.078236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.080227] pending: check_lifetime [16515.081332] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.083231] pending: neigh_periodic_work [16515.084513] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.086822] pending: neigh_periodic_work [16515.088321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.090734] pending: do_cache_clean [16515.091966] workqueue mm_percpu_wq: flags=0x8 [16515.093230] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.095481] pending: vmstat_update [16515.096863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.098924] pending: vmstat_update [16515.100018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16515.101972] pending: vmstat_update [16515.103017] workqueue writeback: flags=0x4a [16515.104119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16515.106185] pending: wb_workfn, wb_workfn [16515.107837] workqueue xprtiod: flags=0xa [16515.108960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16515.110876] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16515.113717] workqueue ipv6_addrconf: flags=0x40008 [16515.116224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16515.117904] pending: addrconf_verify_work [16515.119111] workqueue knbd0-recv: flags=0x1a [16515.120848] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16515.123064] in-flight: 651:recv_work [16515.124305] workqueue ptlrpc_pinger: flags=0x2 [16515.125713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16515.127868] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16515.129921] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14597s workers=4 idle: 37 31846 28897 [16515.132307] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16505s workers=2 idle: 653 [16532.058024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16532.062963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16532.087330] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16532.093216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16532.097760] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16532.101146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16532.104248] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16532.112973] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16532.116810] RAX: 0000000000007f85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16532.120069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16532.123074] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16532.126812] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16532.129984] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16532.133215] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16532.136870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16532.139907] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16532.143742] Call Trace: [16532.144666] [16532.145217] ? show_regs.cold.9+0x22/0x2f [16532.147660] ? watchdog_timer_fn.cold.6+0x62/0x10a [16532.149685] ? __hrtimer_run_queues+0x1c2/0x4d0 [16532.153124] ? watchdog+0x40/0x40 [16532.154728] ? hrtimer_interrupt+0x128/0x2c0 [16532.157173] ? smp_apic_timer_interrupt+0xa6/0x270 [16532.159646] ? apic_timer_interrupt+0xf/0x20 [16532.161793] [16532.163032] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16532.165736] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16532.169217] do_raw_spin_lock+0x177/0x190 [16532.173497] _raw_spin_lock+0x21/0x30 [16532.176010] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16532.179862] process_one_work+0x2c8/0x700 [16532.182499] worker_thread+0x296/0x6e0 [16532.188691] ? rescuer_thread+0x570/0x570 [16532.191538] kthread+0x1d1/0x200 [16532.193310] ? set_kthread_struct+0x70/0x70 [16532.196140] ret_from_fork+0x1f/0x30 [16540.054050] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16540.059039] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16540.062301] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16540.062313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16540.062396] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16540.062402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16540.062405] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16540.062419] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16540.062424] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16540.062429] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16540.062431] RDX: 0000000000001fcd RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16540.062434] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16540.062436] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16540.062439] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16540.062442] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16540.062445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16540.062448] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16540.062458] Call Trace: [16540.062463] [16540.062469] ? show_regs.cold.9+0x22/0x2f [16540.062478] ? watchdog_timer_fn.cold.6+0x62/0x10a [16540.062484] ? __hrtimer_run_queues+0x1c2/0x4d0 [16540.062489] ? watchdog+0x40/0x40 [16540.062494] ? hrtimer_interrupt+0x128/0x2c0 [16540.062498] ? smp_apic_timer_interrupt+0xa6/0x270 [16540.062507] ? apic_timer_interrupt+0xf/0x20 [16540.062512] [16540.062515] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16540.062520] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16540.062525] do_raw_spin_lock+0x177/0x190 [16540.062530] _raw_spin_lock+0x21/0x30 [16540.063383] Modules linked in: [16540.062536] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16540.065827] obd_mod_free+0x46/0x140 [ptlrpc] [16540.066788] Modules linked in: [16540.067803] mdc_commit_open+0x91/0xb0 [mdc] [16540.067839] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16540.069582] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16540.069793] after_reply+0xe2c/0x1780 [ptlrpc] [16540.072087] lustre(O) osp(O) [16540.073559] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16540.073816] ? __next_timer_interrupt+0x160/0x160 [16540.073827] ptlrpcd+0xa82/0xe40 [ptlrpc] [16540.074055] ? do_wait_intr+0xf0/0xf0 [16540.074065] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16540.075702] kthread+0x1d1/0x200 [16540.075709] ? set_kthread_struct+0x70/0x70 [16540.075714] ret_from_fork+0x1f/0x30 [16540.088904] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16540.089033] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16540.089041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16540.089045] RIP: 0010:native_safe_halt+0x2e/0x50 [16540.089057] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16540.089061] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16540.089065] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16540.089068] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16540.089070] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16540.089072] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16540.089074] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16540.089077] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16540.089080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16540.089082] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16540.089091] Call Trace: [16540.089097] [16540.089102] ? show_regs.cold.9+0x22/0x2f [16540.089111] ? watchdog_timer_fn.cold.6+0x62/0x10a [16540.089117] ? __hrtimer_run_queues+0x1c2/0x4d0 [16540.089122] ? watchdog+0x40/0x40 [16540.089127] ? hrtimer_interrupt+0x128/0x2c0 [16540.089131] ? smp_apic_timer_interrupt+0xa6/0x270 [16540.089138] ? apic_timer_interrupt+0xf/0x20 [16540.089142] [16540.089144] ? native_safe_halt+0x2e/0x50 [16540.089148] kvm_wait+0xcd/0xe0 [16540.089156] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16540.089164] do_raw_spin_lock+0x177/0x190 [16540.089169] _raw_spin_lock+0x21/0x30 [16540.089173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16540.089451] ? do_raw_spin_unlock+0x75/0x190 [16540.089457] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16540.089703] ? do_wait_intr+0xf0/0xf0 [16540.089710] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16540.089905] mdc_close+0x53b/0xd70 [mdc] [16540.089938] lmv_close+0x18d/0x490 [lmv] [16540.089967] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16540.090129] ll_md_real_close+0xa5/0x2a0 [lustre] [16540.090210] ll_md_close+0x274/0x800 [lustre] [16540.090304] ll_file_release+0x128/0xc50 [lustre] [16540.090388] __fput+0x10d/0x440 [16540.090396] ____fput+0x12/0x20 [16540.090400] task_work_run+0xc6/0x110 [16540.090406] exit_to_usermode_loop+0x1dd/0x1f0 [16540.090411] do_syscall_64+0x3d6/0x440 [16540.090415] entry_SYSCALL_64_after_hwframe+0x49/0xae [16540.090421] RIP: 0033:0x7fa749552cd8 [16540.090427] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16540.090431] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16540.090435] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16540.090437] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16540.090439] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16540.090442] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16540.090444] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16540.355541] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16540.412383] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16540.420093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16540.426225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16540.428840] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16540.457184] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16540.463290] RAX: 00000000000016b9 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16540.466585] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16540.473218] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16540.479902] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16540.492724] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16540.500528] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16540.510519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16540.515425] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16540.519571] Call Trace: [16540.521487] [16540.524294] ? show_regs.cold.9+0x22/0x2f [16540.525664] ? watchdog_timer_fn.cold.6+0x62/0x10a [16540.527713] ? __hrtimer_run_queues+0x1c2/0x4d0 [16540.536158] ? watchdog+0x40/0x40 [16540.537739] ? hrtimer_interrupt+0x128/0x2c0 [16540.539516] ? smp_apic_timer_interrupt+0xa6/0x270 [16540.541929] ? apic_timer_interrupt+0xf/0x20 [16540.543863] [16540.544750] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16540.546702] do_raw_spin_lock+0x177/0x190 [16540.548101] _raw_spin_lock+0x21/0x30 [16540.549164] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16540.551215] ? do_raw_spin_unlock+0x75/0x190 [16540.552549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16540.554472] ? do_wait_intr+0xf0/0xf0 [16540.555701] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16540.557611] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16540.559479] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16540.561318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.562909] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16540.564307] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16540.565659] mdc_intent_lock+0x353/0x7e0 [mdc] [16540.566873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.568463] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16540.570217] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16540.572423] lmv_intent_open+0x338/0x1150 [lmv] [16540.573908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.575666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.577338] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16540.578508] ll_intent_lock+0xc5/0xaa0 [lustre] [16540.580082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.581522] ll_lookup_it+0xe0e/0x2270 [lustre] [16540.582808] ? do_raw_spin_unlock+0x75/0x190 [16540.583909] ? _raw_spin_unlock+0x12/0x30 [16540.585013] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16540.586709] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16540.588490] ll_atomic_open+0x36a/0xd70 [lustre] [16540.590200] lookup_open+0x356/0xb40 [16540.591426] do_last+0x313/0xfc0 [16540.592478] ? nd_jump_root+0xe5/0x160 [16540.593819] ? path_init+0x437/0x520 [16540.595084] path_openat+0xf7/0x500 [16540.596373] ? do_raw_spin_unlock+0x75/0x190 [16540.597895] do_filp_open+0x99/0x140 [16540.599197] ? getname_flags+0x6e/0x330 [16540.600574] ? __check_object_size+0xff/0x256 [16540.603443] ? do_raw_spin_unlock+0x75/0x190 [16540.605883] ? _raw_spin_unlock+0x12/0x30 [16540.608378] do_sys_openat2+0x2b4/0x410 [16540.611546] do_sys_open+0x73/0xa0 [16540.614562] __x64_sys_openat+0x24/0x30 [16540.615728] do_syscall_64+0xc1/0x440 [16540.616667] entry_SYSCALL_64_after_hwframe+0x49/0xae [16540.623373] RIP: 0033:0x7fa749552332 [16540.631371] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16540.641138] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16540.643372] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16540.646609] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16540.650859] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16540.653490] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16540.656358] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16545.253022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14628s! [16545.256256] Showing busy workqueues and worker pools: [16545.259010] workqueue events: flags=0x0 [16545.261943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16545.263830] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16545.270747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16545.272716] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16545.275530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.277940] pending: kfree_rcu_monitor [16545.279489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16545.281447] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16545.284385] workqueue events_long: flags=0x0 [16545.285944] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.291188] pending: lu_objects_flush [obdclass] [16545.293594] workqueue events_unbound: flags=0x2 [16545.295131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16545.298343] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16545.301776] workqueue events_power_efficient: flags=0x80 [16545.303870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.306572] pending: check_lifetime [16545.307960] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.310467] pending: neigh_periodic_work [16545.312211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.314956] pending: neigh_periodic_work [16545.316405] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.318655] pending: do_cache_clean [16545.320133] workqueue mm_percpu_wq: flags=0x8 [16545.321854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.324529] pending: vmstat_update [16545.326131] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.328654] pending: vmstat_update [16545.330032] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16545.333069] pending: vmstat_update [16545.334493] workqueue writeback: flags=0x4a [16545.336265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16545.338760] pending: wb_workfn, wb_workfn [16545.346792] workqueue xprtiod: flags=0xa [16545.348735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16545.354957] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16545.358011] workqueue ipv6_addrconf: flags=0x40008 [16545.359767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16545.362574] pending: addrconf_verify_work [16545.364725] workqueue knbd0-recv: flags=0x1a [16545.367099] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16545.369501] in-flight: 651:recv_work [16545.371574] workqueue ptlrpc_pinger: flags=0x2 [16545.373305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16545.376207] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16545.380693] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14628s workers=4 idle: 37 31846 28897 [16545.384233] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16535s workers=2 idle: 653 [16554.012469] rcu: INFO: rcu_sched self-detected stall on CPU [16554.018206] rcu: 2-...!: (8626909 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [16554.024462] (t=14640323 jiffies g=258701 q=5422) [16554.027161] rcu: rcu_sched kthread starved for 14637118 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [16554.032499] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16554.039124] rcu: RCU grace-period kthread stack dump: [16554.045499] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16554.053834] Call Trace: [16554.054977] __schedule+0x351/0xcb0 [16554.056828] schedule+0xc0/0x180 [16554.058532] schedule_timeout+0xb4/0x190 [16554.059919] ? do_raw_spin_unlock+0x75/0x190 [16554.061983] ? __next_timer_interrupt+0x160/0x160 [16554.064492] ? prepare_to_swait_event+0xf3/0x140 [16554.066850] rcu_gp_kthread+0x8dd/0x15b0 [16554.069067] ? force_qs_rnp+0x2c0/0x2c0 [16554.071152] kthread+0x1d1/0x200 [16554.072529] ? set_kthread_struct+0x70/0x70 [16554.073793] ret_from_fork+0x1f/0x30 [16554.075340] rcu: Stack dump where RCU GP kthread last ran: [16554.077940] Sending NMI from CPU 2 to CPUs 1: [16554.093468] NMI backtrace for cpu 1 [16554.093476] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16554.093478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16554.093480] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16554.093483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16554.093485] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16554.093487] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 [16554.093489] RAX: 0000000000002014 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16554.093490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16554.093492] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16554.093493] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16554.093495] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16554.093497] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16554.093498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16554.093500] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16554.093501] Call Trace: [16554.093502] [16554.093503] ? show_regs.cold.9+0x22/0x2f [16554.093504] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [16554.093506] ? nmi_cpu_backtrace_handler+0x15/0x40 [16554.093507] ? nmi_handle+0xbd/0x210 [16554.093508] ? default_do_nmi+0x4b/0x1b0 [16554.093509] ? do_nmi+0x280/0x380 [16554.093510] ? end_repeat_nmi+0x16/0x1f [16554.093511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16554.093512] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16554.093513] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16554.093514] [16554.093516] do_raw_spin_lock+0x177/0x190 [16554.093516] _raw_spin_lock+0x21/0x30 [16554.093518] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16554.093519] process_one_work+0x2c8/0x700 [16554.093520] worker_thread+0x296/0x6e0 [16554.093521] ? rescuer_thread+0x570/0x570 [16554.093521] kthread+0x1d1/0x200 [16554.093522] ? set_kthread_struct+0x70/0x70 [16554.093523] ret_from_fork+0x1f/0x30 [16554.094425] NMI backtrace for cpu 2 [16554.285388] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16554.290299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16554.297172] Call Trace: [16554.307141] [16554.307994] ? dump_stack+0xbb/0x10e [16554.310069] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16554.316520] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16554.322157] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16554.324166] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16554.329289] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16554.336856] ? rcu_dump_cpu_stacks+0x11b/0x171 [16554.340554] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16554.344717] ? update_process_times+0x6d/0xc0 [16554.347844] ? tick_sched_handle.isra.18+0x27/0x90 [16554.350699] ? tick_sched_timer+0x53/0xb0 [16554.353192] ? __hrtimer_run_queues+0x1c2/0x4d0 [16554.356793] ? tick_sched_handle.isra.18+0x90/0x90 [16554.358199] ? hrtimer_interrupt+0x128/0x2c0 [16554.360656] ? smp_apic_timer_interrupt+0xa6/0x270 [16554.363182] ? apic_timer_interrupt+0xf/0x20 [16554.365370] [16554.366229] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16554.368390] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16554.370601] ? do_raw_spin_lock+0x177/0x190 [16554.372463] ? _raw_spin_lock+0x21/0x30 [16554.374448] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16554.377258] ? obd_mod_free+0x46/0x140 [ptlrpc] [16554.379575] ? mdc_commit_open+0x91/0xb0 [mdc] [16554.381290] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16554.383820] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16554.386185] ? after_reply+0xe2c/0x1780 [ptlrpc] [16554.388443] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16554.394028] ? __next_timer_interrupt+0x160/0x160 [16554.397123] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16554.398489] ? do_wait_intr+0xf0/0xf0 [16554.400482] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16554.402044] ? kthread+0x1d1/0x200 [16554.408063] ? set_kthread_struct+0x70/0x70 [16554.411654] ? ret_from_fork+0x1f/0x30 [16568.059449] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16568.067208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16568.120307] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16568.130877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16568.140487] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16568.146828] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16568.166976] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16568.176283] RAX: 0000000000007b85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16568.182808] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16568.188727] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16568.194372] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16568.199540] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16568.204737] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16568.210221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16568.215206] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16568.219063] Call Trace: [16568.221787] [16568.222827] ? show_regs.cold.9+0x22/0x2f [16568.228229] ? watchdog_timer_fn.cold.6+0x62/0x10a [16568.231365] ? __hrtimer_run_queues+0x1c2/0x4d0 [16568.234307] ? watchdog+0x40/0x40 [16568.245443] ? hrtimer_interrupt+0x128/0x2c0 [16568.250291] ? smp_apic_timer_interrupt+0xa6/0x270 [16568.254710] ? apic_timer_interrupt+0xf/0x20 [16568.257282] [16568.258825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16568.264690] do_raw_spin_lock+0x177/0x190 [16568.266135] _raw_spin_lock+0x21/0x30 [16568.267123] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16568.271303] ? do_raw_spin_unlock+0x75/0x190 [16568.274903] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16568.278242] ? do_wait_intr+0xf0/0xf0 [16568.283843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16568.288958] mdc_close+0x53b/0xd70 [mdc] [16568.297300] lmv_close+0x18d/0x490 [lmv] [16568.300581] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16568.305834] ll_md_real_close+0xa5/0x2a0 [lustre] [16568.307549] ll_md_close+0x274/0x800 [lustre] [16568.311220] ll_file_release+0x128/0xc50 [lustre] [16568.316225] __fput+0x10d/0x440 [16568.320400] ____fput+0x12/0x20 [16568.322574] task_work_run+0xc6/0x110 [16568.330286] exit_to_usermode_loop+0x1dd/0x1f0 [16568.336698] do_syscall_64+0x3d6/0x440 [16568.339224] entry_SYSCALL_64_after_hwframe+0x49/0xae [16568.344004] RIP: 0033:0x7fa749552cd8 [16568.345857] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16568.359102] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16568.362313] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16568.366822] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16568.369887] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16568.372679] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16568.375320] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16572.057081] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16572.077963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16572.133138] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16572.145661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16572.151055] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16572.153148] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16572.171015] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16572.173224] RAX: 0000000000005f4e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16572.185787] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16572.191150] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16572.196226] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16572.201637] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16572.204647] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16572.209788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16572.218556] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16572.221962] Call Trace: [16572.223177] [16572.224182] ? show_regs.cold.9+0x22/0x2f [16572.226090] ? watchdog_timer_fn.cold.6+0x62/0x10a [16572.229097] ? __hrtimer_run_queues+0x1c2/0x4d0 [16572.231166] ? watchdog+0x40/0x40 [16572.233118] ? hrtimer_interrupt+0x128/0x2c0 [16572.242915] ? smp_apic_timer_interrupt+0xa6/0x270 [16572.246007] ? apic_timer_interrupt+0xf/0x20 [16572.250117] [16572.250991] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16572.254370] do_raw_spin_lock+0x177/0x190 [16572.257863] _raw_spin_lock+0x21/0x30 [16572.260210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16572.265093] ? do_raw_spin_unlock+0x75/0x190 [16572.269357] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16572.278790] ? do_wait_intr+0xf0/0xf0 [16572.290550] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16572.300376] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16572.307957] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16572.313246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16572.321084] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16572.327634] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16572.332753] mdc_intent_lock+0x353/0x7e0 [mdc] [16572.336220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16572.347682] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16572.350792] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16572.354677] lmv_intent_open+0x338/0x1150 [lmv] [16572.357148] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16572.359658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16572.362150] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16572.364713] ll_intent_lock+0xc5/0xaa0 [lustre] [16572.373610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16572.378110] ll_lookup_it+0xe0e/0x2270 [lustre] [16572.384131] ? do_raw_spin_unlock+0x75/0x190 [16572.389156] ? _raw_spin_unlock+0x12/0x30 [16572.391857] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16572.398282] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16572.401923] ll_atomic_open+0x36a/0xd70 [lustre] [16572.408049] lookup_open+0x356/0xb40 [16572.412137] do_last+0x313/0xfc0 [16572.417805] ? nd_jump_root+0xe5/0x160 [16572.422624] ? path_init+0x437/0x520 [16572.425456] path_openat+0xf7/0x500 [16572.427765] ? do_raw_spin_unlock+0x75/0x190 [16572.430805] do_filp_open+0x99/0x140 [16572.433635] ? getname_flags+0x6e/0x330 [16572.441187] ? __check_object_size+0xff/0x256 [16572.446666] ? do_raw_spin_unlock+0x75/0x190 [16572.453141] ? _raw_spin_unlock+0x12/0x30 [16572.455951] do_sys_openat2+0x2b4/0x410 [16572.459073] do_sys_open+0x73/0xa0 [16572.461819] __x64_sys_openat+0x24/0x30 [16572.463707] do_syscall_64+0xc1/0x440 [16572.467234] entry_SYSCALL_64_after_hwframe+0x49/0xae [16572.470356] RIP: 0033:0x7fa749552332 [16572.472570] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16572.494210] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16572.502941] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16572.514136] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16572.523759] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16572.533073] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16572.542800] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16575.457461] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14658s! [16575.476505] Showing busy workqueues and worker pools: [16575.479465] workqueue events: flags=0x0 [16575.483913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16575.490050] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16575.498349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16575.506898] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16575.519210] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.539480] pending: kfree_rcu_monitor [16575.540885] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16575.553006] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16575.559783] workqueue events_long: flags=0x0 [16575.561182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.567583] pending: lu_objects_flush [obdclass] [16575.571259] workqueue events_unbound: flags=0x2 [16575.575360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16575.579143] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16575.586087] workqueue events_power_efficient: flags=0x80 [16575.595018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.597146] pending: check_lifetime [16575.604799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.609282] pending: neigh_periodic_work [16575.617132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.625823] pending: neigh_periodic_work [16575.628188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.634921] pending: do_cache_clean [16575.637161] workqueue mm_percpu_wq: flags=0x8 [16575.639087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.646394] pending: vmstat_update [16575.647547] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.659314] pending: vmstat_update [16575.660707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16575.669505] pending: vmstat_update [16575.675902] workqueue writeback: flags=0x4a [16575.680037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16575.684475] pending: wb_workfn, wb_workfn [16575.688582] workqueue xprtiod: flags=0xa [16575.691188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16575.697167] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16575.701835] workqueue ipv6_addrconf: flags=0x40008 [16575.707053] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16575.716119] pending: addrconf_verify_work [16575.717685] workqueue knbd0-recv: flags=0x1a [16575.730181] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16575.735308] in-flight: 651:recv_work [16575.739214] workqueue ptlrpc_pinger: flags=0x2 [16575.740312] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16575.742739] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16575.745910] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14658s workers=4 idle: 37 31846 28897 [16575.749886] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16565s workers=2 idle: 653 [16580.059805] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16580.063227] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16580.063617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16580.064134] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16580.064140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16580.064147] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16580.065377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16580.065391] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16580.065395] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [16580.065399] RAX: 0000000000000700 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16580.065406] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16580.065408] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16580.065410] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16580.065412] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16580.065416] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16580.065419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16580.065421] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16580.065430] Call Trace: [16580.065436] [16580.065441] ? show_regs.cold.9+0x22/0x2f [16580.065452] ? watchdog_timer_fn.cold.6+0x62/0x10a [16580.065458] ? __hrtimer_run_queues+0x1c2/0x4d0 [16580.065463] ? watchdog+0x40/0x40 [16580.065467] ? hrtimer_interrupt+0x128/0x2c0 [16580.065470] ? smp_apic_timer_interrupt+0xa6/0x270 [16580.065477] ? apic_timer_interrupt+0xf/0x20 [16580.065490] [16580.065778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16580.065786] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16580.065793] do_raw_spin_lock+0x177/0x190 [16580.065798] _raw_spin_lock+0x21/0x30 [16580.065807] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16580.066176] process_one_work+0x2c8/0x700 [16580.066184] worker_thread+0x296/0x6e0 [16580.066189] ? rescuer_thread+0x570/0x570 [16580.066193] kthread+0x1d1/0x200 [16580.066198] ? set_kthread_struct+0x70/0x70 [16580.066202] ret_from_fork+0x1f/0x30 [16580.217135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16580.253914] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16580.272842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16580.275325] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16580.284444] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16580.295849] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16580.301844] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16580.305138] RDX: 00000000000057d2 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16580.312661] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16580.316495] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16580.323071] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16580.326801] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16580.331769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16580.337143] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16580.341641] Call Trace: [16580.342895] [16580.344032] ? show_regs.cold.9+0x22/0x2f [16580.346132] ? watchdog_timer_fn.cold.6+0x62/0x10a [16580.347629] ? __hrtimer_run_queues+0x1c2/0x4d0 [16580.360010] ? watchdog+0x40/0x40 [16580.362006] ? hrtimer_interrupt+0x128/0x2c0 [16580.365998] ? smp_apic_timer_interrupt+0xa6/0x270 [16580.367864] ? apic_timer_interrupt+0xf/0x20 [16580.380059] [16580.380838] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16580.385492] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16580.388392] do_raw_spin_lock+0x177/0x190 [16580.392516] _raw_spin_lock+0x21/0x30 [16580.394179] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16580.396932] obd_mod_free+0x46/0x140 [ptlrpc] [16580.403097] mdc_commit_open+0x91/0xb0 [mdc] [16580.405450] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16580.409865] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16580.412982] after_reply+0xe2c/0x1780 [ptlrpc] [16580.414396] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16580.421740] ? __next_timer_interrupt+0x160/0x160 [16580.425212] ptlrpcd+0xa82/0xe40 [ptlrpc] [16580.427656] ? do_wait_intr+0xf0/0xf0 [16580.429888] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16580.433599] kthread+0x1d1/0x200 [16580.435596] ? set_kthread_struct+0x70/0x70 [16580.439724] ret_from_fork+0x1f/0x30 [16596.059045] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16596.067548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16596.126095] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16596.133507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16596.139986] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16596.144059] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16596.165768] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16596.171913] RAX: 0000000000002c42 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16596.178830] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16596.181590] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16596.188197] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16596.192289] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16596.197536] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16596.202539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16596.205461] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16596.213763] Call Trace: [16596.215031] [16596.217361] ? show_regs.cold.9+0x22/0x2f [16596.221645] ? watchdog_timer_fn.cold.6+0x62/0x10a [16596.223782] ? __hrtimer_run_queues+0x1c2/0x4d0 [16596.225366] ? watchdog+0x40/0x40 [16596.229454] ? hrtimer_interrupt+0x128/0x2c0 [16596.233729] ? smp_apic_timer_interrupt+0xa6/0x270 [16596.235798] ? apic_timer_interrupt+0xf/0x20 [16596.237258] [16596.239794] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16596.242001] do_raw_spin_lock+0x177/0x190 [16596.246189] _raw_spin_lock+0x21/0x30 [16596.247584] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16596.250223] ? do_raw_spin_unlock+0x75/0x190 [16596.254569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16596.257920] ? do_wait_intr+0xf0/0xf0 [16596.261529] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16596.265562] mdc_close+0x53b/0xd70 [mdc] [16596.267708] lmv_close+0x18d/0x490 [lmv] [16596.270110] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16596.275631] ll_md_real_close+0xa5/0x2a0 [lustre] [16596.277864] ll_md_close+0x274/0x800 [lustre] [16596.280287] ll_file_release+0x128/0xc50 [lustre] [16596.282397] __fput+0x10d/0x440 [16596.283243] ____fput+0x12/0x20 [16596.285331] task_work_run+0xc6/0x110 [16596.287731] exit_to_usermode_loop+0x1dd/0x1f0 [16596.293133] do_syscall_64+0x3d6/0x440 [16596.295028] entry_SYSCALL_64_after_hwframe+0x49/0xae [16596.299685] RIP: 0033:0x7fa749552cd8 [16596.300693] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16596.309963] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16596.313618] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16596.318113] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16596.321107] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16596.324675] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16596.328329] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16600.054262] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16600.059738] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16600.084414] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16600.090461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16600.094226] RIP: 0010:native_safe_halt+0x2e/0x50 [16600.095890] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16600.101784] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16600.104127] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16600.106269] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16600.108321] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16600.110395] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16600.112418] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16600.114478] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16600.116819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16600.119050] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16600.121466] Call Trace: [16600.122290] [16600.122848] ? show_regs.cold.9+0x22/0x2f [16600.124298] ? watchdog_timer_fn.cold.6+0x62/0x10a [16600.125728] ? __hrtimer_run_queues+0x1c2/0x4d0 [16600.127304] ? watchdog+0x40/0x40 [16600.128350] ? hrtimer_interrupt+0x128/0x2c0 [16600.129616] ? smp_apic_timer_interrupt+0xa6/0x270 [16600.131173] ? apic_timer_interrupt+0xf/0x20 [16600.132458] [16600.133217] ? native_safe_halt+0x2e/0x50 [16600.134412] kvm_wait+0xcd/0xe0 [16600.135397] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16600.136971] do_raw_spin_lock+0x177/0x190 [16600.138319] _raw_spin_lock+0x21/0x30 [16600.139378] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16600.141340] ? do_raw_spin_unlock+0x75/0x190 [16600.142609] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16600.144764] ? do_wait_intr+0xf0/0xf0 [16600.146118] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16600.147993] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16600.150157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16600.152333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16600.154202] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16600.156129] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16600.157556] mdc_intent_lock+0x353/0x7e0 [mdc] [16600.160126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16600.162252] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16600.164448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16600.166375] lmv_intent_open+0x338/0x1150 [lmv] [16600.167903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16600.170118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16600.171873] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16600.173407] ll_intent_lock+0xc5/0xaa0 [lustre] [16600.174995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16600.176804] ll_lookup_it+0xe0e/0x2270 [lustre] [16600.178435] ? do_raw_spin_unlock+0x75/0x190 [16600.179742] ? _raw_spin_unlock+0x12/0x30 [16600.181359] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16600.182809] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16600.184644] ll_atomic_open+0x36a/0xd70 [lustre] [16600.186424] lookup_open+0x356/0xb40 [16600.187686] do_last+0x313/0xfc0 [16600.188893] ? nd_jump_root+0xe5/0x160 [16600.190554] ? path_init+0x437/0x520 [16600.191728] path_openat+0xf7/0x500 [16600.192906] ? do_raw_spin_unlock+0x75/0x190 [16600.195170] do_filp_open+0x99/0x140 [16600.196547] ? getname_flags+0x6e/0x330 [16600.197885] ? __check_object_size+0xff/0x256 [16600.199616] ? do_raw_spin_unlock+0x75/0x190 [16600.201299] ? _raw_spin_unlock+0x12/0x30 [16600.203171] do_sys_openat2+0x2b4/0x410 [16600.204994] do_sys_open+0x73/0xa0 [16600.206364] __x64_sys_openat+0x24/0x30 [16600.207980] do_syscall_64+0xc1/0x440 [16600.209436] entry_SYSCALL_64_after_hwframe+0x49/0xae [16600.211873] RIP: 0033:0x7fa749552332 [16600.213344] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16600.221439] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16600.223782] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16600.225905] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16600.228610] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16600.230825] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16600.232754] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16606.186174] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14689s! [16606.192137] Showing busy workqueues and worker pools: [16606.194889] workqueue events: flags=0x0 [16606.197165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16606.205508] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16606.212541] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16606.217721] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16606.226033] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.234110] pending: kfree_rcu_monitor [16606.239865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16606.245802] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16606.251698] workqueue events_long: flags=0x0 [16606.254621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.260406] pending: lu_objects_flush [obdclass] [16606.263543] workqueue events_unbound: flags=0x2 [16606.271925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16606.276968] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16606.287801] workqueue events_power_efficient: flags=0x80 [16606.294362] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.305914] pending: check_lifetime [16606.306947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.314313] pending: neigh_periodic_work [16606.328843] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.338028] pending: neigh_periodic_work [16606.339583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.344851] pending: do_cache_clean [16606.347923] workqueue mm_percpu_wq: flags=0x8 [16606.352387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.358134] pending: vmstat_update [16606.371877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.379163] pending: vmstat_update [16606.382010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16606.385728] pending: vmstat_update [16606.399140] workqueue writeback: flags=0x4a [16606.410219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16606.413837] pending: wb_workfn, wb_workfn [16606.418686] workqueue xprtiod: flags=0xa [16606.424418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16606.427761] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16606.436378] workqueue ipv6_addrconf: flags=0x40008 [16606.437786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16606.442579] pending: addrconf_verify_work [16606.447063] workqueue knbd0-recv: flags=0x1a [16606.450175] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16606.457140] in-flight: 651:recv_work [16606.465266] workqueue ptlrpc_pinger: flags=0x2 [16606.469179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16606.475680] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16606.479986] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14689s workers=4 idle: 37 31846 28897 [16606.484141] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16596s workers=2 idle: 653 [16608.058028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16608.061428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16608.063546] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16608.078484] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16608.078497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16608.078500] RIP: 0010:native_safe_halt+0x2e/0x50 [16608.078513] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16608.078517] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16608.078523] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16608.078526] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16608.078528] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16608.078530] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16608.078532] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16608.078536] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16608.078539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16608.078542] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16608.078549] Call Trace: [16608.078578] [16608.078585] ? show_regs.cold.9+0x22/0x2f [16608.078595] ? watchdog_timer_fn.cold.6+0x62/0x10a [16608.078601] ? __hrtimer_run_queues+0x1c2/0x4d0 [16608.078606] ? watchdog+0x40/0x40 [16608.078610] ? hrtimer_interrupt+0x128/0x2c0 [16608.078614] ? smp_apic_timer_interrupt+0xa6/0x270 [16608.081978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16608.082063] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16608.082086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16608.082091] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16608.082382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16608.082394] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16608.082398] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16608.082403] RAX: 0000000000000490 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16608.082405] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16608.082408] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16608.082410] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16608.082412] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16608.082416] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16608.082419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16608.082422] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16608.082431] Call Trace: [16608.082437] [16608.082442] ? show_regs.cold.9+0x22/0x2f [16608.082452] ? watchdog_timer_fn.cold.6+0x62/0x10a [16608.082460] ? __hrtimer_run_queues+0x1c2/0x4d0 [16608.082464] ? watchdog+0x40/0x40 [16608.082468] ? hrtimer_interrupt+0x128/0x2c0 [16608.082473] ? smp_apic_timer_interrupt+0xa6/0x270 [16608.082483] ? apic_timer_interrupt+0xf/0x20 [16608.082487] [16608.082490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16608.082495] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16608.082500] do_raw_spin_lock+0x177/0x190 [16608.082504] _raw_spin_lock+0x21/0x30 [16608.082510] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16608.082713] process_one_work+0x2c8/0x700 [16608.082719] worker_thread+0x296/0x6e0 [16608.082723] ? rescuer_thread+0x570/0x570 [16608.082727] kthread+0x1d1/0x200 [16608.082731] ? set_kthread_struct+0x70/0x70 [16608.082735] ret_from_fork+0x1f/0x30 [16608.538354] ? apic_timer_interrupt+0xf/0x20 [16608.543796] [16608.549993] ? native_safe_halt+0x2e/0x50 [16608.551155] kvm_wait+0xcd/0xe0 [16608.551952] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16608.560455] do_raw_spin_lock+0x177/0x190 [16608.561987] _raw_spin_lock+0x21/0x30 [16608.569576] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16608.582137] obd_mod_free+0x46/0x140 [ptlrpc] [16608.589265] mdc_commit_open+0x91/0xb0 [mdc] [16608.593534] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16608.596438] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16608.599419] after_reply+0xe2c/0x1780 [ptlrpc] [16608.601720] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16608.604654] ? __next_timer_interrupt+0x160/0x160 [16608.605955] ptlrpcd+0xa82/0xe40 [ptlrpc] [16608.609699] ? do_wait_intr+0xf0/0xf0 [16608.611428] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16608.617257] kthread+0x1d1/0x200 [16608.618509] ? set_kthread_struct+0x70/0x70 [16608.622257] ret_from_fork+0x1f/0x30 [16624.059302] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16624.065194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16624.083451] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16624.087064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16624.089569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16624.091404] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16624.098496] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16624.102024] RAX: 0000000000007888 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16624.104480] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16624.107166] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16624.109440] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16624.112004] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16624.121810] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16624.127629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16624.129951] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16624.132247] Call Trace: [16624.133141] [16624.134112] ? show_regs.cold.9+0x22/0x2f [16624.135380] ? watchdog_timer_fn.cold.6+0x62/0x10a [16624.137019] ? __hrtimer_run_queues+0x1c2/0x4d0 [16624.138374] ? watchdog+0x40/0x40 [16624.139498] ? hrtimer_interrupt+0x128/0x2c0 [16624.140855] ? smp_apic_timer_interrupt+0xa6/0x270 [16624.146218] ? apic_timer_interrupt+0xf/0x20 [16624.153627] [16624.154346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16624.156007] do_raw_spin_lock+0x177/0x190 [16624.159187] _raw_spin_lock+0x21/0x30 [16624.161195] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16624.165021] ? do_raw_spin_unlock+0x75/0x190 [16624.166695] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16624.169318] ? do_wait_intr+0xf0/0xf0 [16624.170964] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16624.173474] mdc_close+0x53b/0xd70 [mdc] [16624.175114] lmv_close+0x18d/0x490 [lmv] [16624.176597] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16624.184022] ll_md_real_close+0xa5/0x2a0 [lustre] [16624.186266] ll_md_close+0x274/0x800 [lustre] [16624.191308] ll_file_release+0x128/0xc50 [lustre] [16624.192872] __fput+0x10d/0x440 [16624.195098] ____fput+0x12/0x20 [16624.196386] task_work_run+0xc6/0x110 [16624.201757] exit_to_usermode_loop+0x1dd/0x1f0 [16624.207341] do_syscall_64+0x3d6/0x440 [16624.208492] entry_SYSCALL_64_after_hwframe+0x49/0xae [16624.213322] RIP: 0033:0x7fa749552cd8 [16624.215295] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16624.230176] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16624.236382] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16624.242156] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16624.246146] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16624.250108] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16624.258059] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16632.054047] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16632.056963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16632.079125] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16632.084114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16632.087340] RIP: 0010:native_safe_halt+0x2e/0x50 [16632.088958] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16632.095871] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16632.098994] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16632.102089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16632.106080] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16632.112863] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16632.122908] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16632.129132] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16632.133250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16632.137330] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16632.147204] Call Trace: [16632.149192] [16632.150342] ? show_regs.cold.9+0x22/0x2f [16632.153762] ? watchdog_timer_fn.cold.6+0x62/0x10a [16632.161883] ? __hrtimer_run_queues+0x1c2/0x4d0 [16632.167366] ? watchdog+0x40/0x40 [16632.168976] ? hrtimer_interrupt+0x128/0x2c0 [16632.171526] ? smp_apic_timer_interrupt+0xa6/0x270 [16632.173586] ? apic_timer_interrupt+0xf/0x20 [16632.175693] [16632.176505] ? native_safe_halt+0x2e/0x50 [16632.178751] kvm_wait+0xcd/0xe0 [16632.179971] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16632.182482] do_raw_spin_lock+0x177/0x190 [16632.184573] _raw_spin_lock+0x21/0x30 [16632.191162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16632.194036] ? do_raw_spin_unlock+0x75/0x190 [16632.196338] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16632.200065] ? do_wait_intr+0xf0/0xf0 [16632.202877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16632.206561] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16632.209867] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16632.213430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16632.215987] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16632.218781] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16632.221432] mdc_intent_lock+0x353/0x7e0 [mdc] [16632.223612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16632.226279] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16632.229743] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16632.232395] lmv_intent_open+0x338/0x1150 [lmv] [16632.235276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16632.239185] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16632.242405] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16632.245015] ll_intent_lock+0xc5/0xaa0 [lustre] [16632.247963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16632.251762] ll_lookup_it+0xe0e/0x2270 [lustre] [16632.254468] ? do_raw_spin_unlock+0x75/0x190 [16632.256909] ? _raw_spin_unlock+0x12/0x30 [16632.259006] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16632.262101] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16632.265001] ll_atomic_open+0x36a/0xd70 [lustre] [16632.267992] lookup_open+0x356/0xb40 [16632.270695] do_last+0x313/0xfc0 [16632.272209] ? nd_jump_root+0xe5/0x160 [16632.274535] ? path_init+0x437/0x520 [16632.275770] path_openat+0xf7/0x500 [16632.278091] ? do_raw_spin_unlock+0x75/0x190 [16632.280975] do_filp_open+0x99/0x140 [16632.283080] ? getname_flags+0x6e/0x330 [16632.285973] ? __check_object_size+0xff/0x256 [16632.288349] ? do_raw_spin_unlock+0x75/0x190 [16632.291176] ? _raw_spin_unlock+0x12/0x30 [16632.294304] do_sys_openat2+0x2b4/0x410 [16632.297195] do_sys_open+0x73/0xa0 [16632.299239] __x64_sys_openat+0x24/0x30 [16632.301326] do_syscall_64+0xc1/0x440 [16632.304020] entry_SYSCALL_64_after_hwframe+0x49/0xae [16632.306123] RIP: 0033:0x7fa749552332 [16632.307720] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16632.317107] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16632.320745] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16632.324468] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16632.329673] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16632.333175] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16632.336173] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16636.060683] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16636.062161] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16636.067968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16636.068045] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16636.068054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16636.068057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16636.068071] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16636.068075] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16636.068080] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16636.068082] RDX: 00000000000041da RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16636.068085] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16636.068087] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16636.068089] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16636.068093] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16636.068096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16636.068098] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16636.068108] Call Trace: [16636.068114] [16636.068120] ? show_regs.cold.9+0x22/0x2f [16636.068128] ? watchdog_timer_fn.cold.6+0x62/0x10a [16636.068135] ? __hrtimer_run_queues+0x1c2/0x4d0 [16636.068139] ? watchdog+0x40/0x40 [16636.068143] ? hrtimer_interrupt+0x128/0x2c0 [16636.068147] ? smp_apic_timer_interrupt+0xa6/0x270 [16636.068155] ? apic_timer_interrupt+0xf/0x20 [16636.068160] [16636.068164] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16636.068169] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16636.068173] do_raw_spin_lock+0x177/0x190 [16636.068178] _raw_spin_lock+0x21/0x30 [16636.068183] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16636.068445] obd_mod_free+0x46/0x140 [ptlrpc] [16636.069314] mdc_commit_open+0x91/0xb0 [mdc] [16636.069350] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16636.069535] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16636.069793] after_reply+0xe2c/0x1780 [ptlrpc] [16636.070026] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16636.070218] ? __next_timer_interrupt+0x160/0x160 [16636.070228] ptlrpcd+0xa82/0xe40 [ptlrpc] [16636.070490] ? do_wait_intr+0xf0/0xf0 [16636.070910] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16636.071119] kthread+0x1d1/0x200 [16636.071125] ? set_kthread_struct+0x70/0x70 [16636.071129] ret_from_fork+0x1f/0x30 [16636.268646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16636.303213] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16636.309739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16636.315666] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16636.320122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16636.328162] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16636.347506] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16636.353667] RAX: 0000000000007586 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16636.363036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16636.366152] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16636.369768] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16636.378329] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16636.380421] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16636.386155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16636.394474] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16636.405339] Call Trace: [16636.406112] [16636.406633] ? show_regs.cold.9+0x22/0x2f [16636.408035] ? watchdog_timer_fn.cold.6+0x62/0x10a [16636.410846] ? __hrtimer_run_queues+0x1c2/0x4d0 [16636.414119] ? watchdog+0x40/0x40 [16636.415716] ? hrtimer_interrupt+0x128/0x2c0 [16636.417744] ? smp_apic_timer_interrupt+0xa6/0x270 [16636.420266] ? apic_timer_interrupt+0xf/0x20 [16636.423605] [16636.424265] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16636.429625] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16636.433372] do_raw_spin_lock+0x177/0x190 [16636.437340] _raw_spin_lock+0x21/0x30 [16636.438550] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16636.440506] process_one_work+0x2c8/0x700 [16636.446030] worker_thread+0x296/0x6e0 [16636.448552] ? rescuer_thread+0x570/0x570 [16636.452260] kthread+0x1d1/0x200 [16636.453801] ? set_kthread_struct+0x70/0x70 [16636.455814] ret_from_fork+0x1f/0x30 [16636.896072] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14719s! [16636.899803] Showing busy workqueues and worker pools: [16636.903597] workqueue events: flags=0x0 [16636.905499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16636.908802] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16636.915756] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16636.922962] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16636.926780] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.929655] pending: kfree_rcu_monitor [16636.931174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16636.934575] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16636.941938] workqueue events_long: flags=0x0 [16636.944668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.949858] pending: lu_objects_flush [obdclass] [16636.953596] workqueue events_unbound: flags=0x2 [16636.957223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16636.960694] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16636.965839] workqueue events_power_efficient: flags=0x80 [16636.967797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.970678] pending: check_lifetime [16636.972845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.975646] pending: neigh_periodic_work [16636.977547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.980419] pending: neigh_periodic_work [16636.982114] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.985115] pending: do_cache_clean [16636.986868] workqueue mm_percpu_wq: flags=0x8 [16636.988665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.991216] pending: vmstat_update [16636.992742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.995205] pending: vmstat_update [16636.996580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16636.999375] pending: vmstat_update [16637.000920] workqueue writeback: flags=0x4a [16637.002735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16637.006908] pending: wb_workfn, wb_workfn [16637.011408] workqueue xprtiod: flags=0xa [16637.012601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16637.014040] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16637.019340] workqueue ipv6_addrconf: flags=0x40008 [16637.022114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16637.024996] pending: addrconf_verify_work [16637.028993] workqueue knbd0-recv: flags=0x1a [16637.031389] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16637.035165] in-flight: 651:recv_work [16637.036995] workqueue ptlrpc_pinger: flags=0x2 [16637.038832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16637.040850] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16637.043214] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14719s workers=4 idle: 37 31846 28897 [16637.047056] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16626s workers=2 idle: 653 [16652.059011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16652.062893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16652.086343] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16652.092179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16652.098087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16652.101881] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16652.123463] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16652.131671] RAX: 00000000000005e3 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16652.135965] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16652.139293] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16652.144913] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16652.153942] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16652.158720] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16652.171953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16652.182116] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16652.188149] Call Trace: [16652.189869] [16652.193930] ? show_regs.cold.9+0x22/0x2f [16652.195955] ? watchdog_timer_fn.cold.6+0x62/0x10a [16652.202968] ? __hrtimer_run_queues+0x1c2/0x4d0 [16652.208974] ? watchdog+0x40/0x40 [16652.210805] ? hrtimer_interrupt+0x128/0x2c0 [16652.221837] ? smp_apic_timer_interrupt+0xa6/0x270 [16652.228064] ? apic_timer_interrupt+0xf/0x20 [16652.232748] [16652.233534] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16652.237766] do_raw_spin_lock+0x177/0x190 [16652.242866] _raw_spin_lock+0x21/0x30 [16652.248135] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16652.256395] ? do_raw_spin_unlock+0x75/0x190 [16652.261905] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16652.265747] ? do_wait_intr+0xf0/0xf0 [16652.272378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16652.276448] mdc_close+0x53b/0xd70 [mdc] [16652.280823] lmv_close+0x18d/0x490 [lmv] [16652.286422] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16652.298004] ll_md_real_close+0xa5/0x2a0 [lustre] [16652.300226] ll_md_close+0x274/0x800 [lustre] [16652.302111] ll_file_release+0x128/0xc50 [lustre] [16652.308879] __fput+0x10d/0x440 [16652.310905] ____fput+0x12/0x20 [16652.312456] task_work_run+0xc6/0x110 [16652.317618] exit_to_usermode_loop+0x1dd/0x1f0 [16652.319205] do_syscall_64+0x3d6/0x440 [16652.322134] entry_SYSCALL_64_after_hwframe+0x49/0xae [16652.326924] RIP: 0033:0x7fa749552cd8 [16652.329246] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16652.358850] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16652.372913] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16652.386471] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16652.405786] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16652.413414] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16652.421988] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16664.054018] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16664.058217] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16664.058507] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16664.058514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16664.058583] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16664.058590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16664.058596] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16664.058865] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16664.058876] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16664.058880] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16664.058884] RAX: 000000000000628f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16664.058887] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16664.058889] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16664.058891] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16664.058894] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16664.058897] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16664.058900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16664.058903] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16664.059538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16664.059604] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16664.059610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16664.059613] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16664.059622] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16664.059626] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16664.059630] RAX: 000000000000282a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16664.059633] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16664.059635] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16664.059637] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16664.059639] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16664.059649] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16664.059655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16664.059658] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16664.059680] Call Trace: [16664.059686] [16664.059692] ? show_regs.cold.9+0x22/0x2f [16664.059702] ? watchdog_timer_fn.cold.6+0x62/0x10a [16664.059718] ? __hrtimer_run_queues+0x1c2/0x4d0 [16664.059722] ? watchdog+0x40/0x40 [16664.059727] ? hrtimer_interrupt+0x128/0x2c0 [16664.059731] ? smp_apic_timer_interrupt+0xa6/0x270 [16664.059739] ? apic_timer_interrupt+0xf/0x20 [16664.059744] [16664.059748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16664.059754] do_raw_spin_lock+0x177/0x190 [16664.059759] _raw_spin_lock+0x21/0x30 [16664.059764] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16664.059946] Call Trace: [16664.059951] [16664.059954] ? show_regs.cold.9+0x22/0x2f [16664.059962] ? watchdog_timer_fn.cold.6+0x62/0x10a [16664.059967] ? __hrtimer_run_queues+0x1c2/0x4d0 [16664.059970] ? watchdog+0x40/0x40 [16664.059974] ? hrtimer_interrupt+0x128/0x2c0 [16664.059977] ? smp_apic_timer_interrupt+0xa6/0x270 [16664.059982] ? apic_timer_interrupt+0xf/0x20 [16664.059988] [16664.059990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16664.059995] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16664.060000] do_raw_spin_lock+0x177/0x190 [16664.060004] _raw_spin_lock+0x21/0x30 [16664.060186] ? do_raw_spin_unlock+0x75/0x190 [16664.060008] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16664.060518] process_one_work+0x2c8/0x700 [16664.060526] worker_thread+0x296/0x6e0 [16664.060530] ? rescuer_thread+0x570/0x570 [16664.060533] kthread+0x1d1/0x200 [16664.060539] ? set_kthread_struct+0x70/0x70 [16664.060193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16664.060543] ret_from_fork+0x1f/0x30 [16664.060655] ? do_wait_intr+0xf0/0xf0 [16664.060664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16664.060952] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16664.061384] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16664.061952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.062078] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16664.062139] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16664.062180] mdc_intent_lock+0x353/0x7e0 [mdc] [16664.062218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.062593] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16664.062968] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16664.063001] lmv_intent_open+0x338/0x1150 [lmv] [16664.063037] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.063155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.063259] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16664.063294] ll_intent_lock+0xc5/0xaa0 [lustre] [16664.063402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.063508] ll_lookup_it+0xe0e/0x2270 [lustre] [16664.063612] ? do_raw_spin_unlock+0x75/0x190 [16664.063620] ? _raw_spin_unlock+0x12/0x30 [16664.063632] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16664.063661] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16664.063686] ll_atomic_open+0x36a/0xd70 [lustre] [16664.063807] lookup_open+0x356/0xb40 [16664.063821] do_last+0x313/0xfc0 [16664.063826] ? nd_jump_root+0xe5/0x160 [16664.063841] ? path_init+0x437/0x520 [16664.063846] path_openat+0xf7/0x500 [16664.063855] ? do_raw_spin_unlock+0x75/0x190 [16664.063860] do_filp_open+0x99/0x140 [16664.063866] ? getname_flags+0x6e/0x330 [16664.063870] ? __check_object_size+0xff/0x256 [16664.063879] ? do_raw_spin_unlock+0x75/0x190 [16664.063885] ? _raw_spin_unlock+0x12/0x30 [16664.063889] do_sys_openat2+0x2b4/0x410 [16664.063893] do_sys_open+0x73/0xa0 [16664.063897] __x64_sys_openat+0x24/0x30 [16664.063900] do_syscall_64+0xc1/0x440 [16664.063906] entry_SYSCALL_64_after_hwframe+0x49/0xae [16664.063912] RIP: 0033:0x7fa749552332 [16664.063919] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16664.063923] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16664.063928] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16664.063931] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16664.063933] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16664.063935] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16664.063938] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16664.066789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16664.067398] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16664.067406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16664.067409] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16664.067421] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16664.067426] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16664.067430] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16664.067433] RDX: 000000000000580f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16664.067435] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16664.067437] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16664.067440] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16664.067443] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16664.067446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16664.067448] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16664.067456] Call Trace: [16664.067462] [16664.067533] ? show_regs.cold.9+0x22/0x2f [16664.067545] ? watchdog_timer_fn.cold.6+0x62/0x10a [16664.067552] ? __hrtimer_run_queues+0x1c2/0x4d0 [16664.067556] ? watchdog+0x40/0x40 [16664.067561] ? hrtimer_interrupt+0x128/0x2c0 [16664.067565] ? smp_apic_timer_interrupt+0xa6/0x270 [16664.067573] ? apic_timer_interrupt+0xf/0x20 [16664.067578] [16664.067581] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16664.067586] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16664.067590] do_raw_spin_lock+0x177/0x190 [16664.067595] _raw_spin_lock+0x21/0x30 [16664.067601] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16664.067888] obd_mod_free+0x46/0x140 [ptlrpc] [16664.068446] mdc_commit_open+0x91/0xb0 [mdc] [16664.069162] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16664.069992] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16664.070963] after_reply+0xe2c/0x1780 [ptlrpc] [16664.071201] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16664.977987] ? __next_timer_interrupt+0x160/0x160 [16664.982446] ptlrpcd+0xa82/0xe40 [ptlrpc] [16664.987042] ? do_wait_intr+0xf0/0xf0 [16664.992031] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16664.998779] kthread+0x1d1/0x200 [16664.999946] ? set_kthread_struct+0x70/0x70 [16665.001536] ret_from_fork+0x1f/0x30 [16667.113863] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14749s! [16667.123864] Showing busy workqueues and worker pools: [16667.137285] workqueue events: flags=0x0 [16667.145233] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16667.150727] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16667.172402] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16667.179644] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16667.185217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.195055] pending: kfree_rcu_monitor [16667.202419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16667.208132] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16667.217151] workqueue events_long: flags=0x0 [16667.219099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.222643] pending: lu_objects_flush [obdclass] [16667.227284] workqueue events_unbound: flags=0x2 [16667.230989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16667.233900] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16667.238780] workqueue events_power_efficient: flags=0x80 [16667.242929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.247596] pending: check_lifetime [16667.250912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.253818] pending: neigh_periodic_work [16667.255387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.259473] pending: neigh_periodic_work [16667.261586] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.265864] pending: do_cache_clean [16667.280421] workqueue mm_percpu_wq: flags=0x8 [16667.281681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.289460] pending: vmstat_update [16667.295500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.305318] pending: vmstat_update [16667.307635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16667.314626] pending: vmstat_update [16667.319109] workqueue writeback: flags=0x4a [16667.323446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16667.331338] pending: wb_workfn, wb_workfn [16667.336547] workqueue xprtiod: flags=0xa [16667.337913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16667.347129] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16667.352240] workqueue ipv6_addrconf: flags=0x40008 [16667.354834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16667.358931] pending: addrconf_verify_work [16667.360782] workqueue knbd0-recv: flags=0x1a [16667.363305] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16667.366807] in-flight: 651:recv_work [16667.371977] workqueue ptlrpc_pinger: flags=0x2 [16667.377803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16667.384997] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16667.389044] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14750s workers=4 idle: 37 31846 28897 [16667.394072] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16657s workers=2 idle: 653 [16680.060954] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16680.068088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16680.101469] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16680.116202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16680.124532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16680.128787] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16680.144857] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16680.147520] RAX: 000000000000780d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16680.156465] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16680.162268] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16680.167166] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16680.171083] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16680.182437] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16680.196725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16680.200029] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16680.216424] Call Trace: [16680.217313] [16680.218017] ? show_regs.cold.9+0x22/0x2f [16680.222876] ? watchdog_timer_fn.cold.6+0x62/0x10a [16680.225591] ? __hrtimer_run_queues+0x1c2/0x4d0 [16680.228563] ? watchdog+0x40/0x40 [16680.232429] ? hrtimer_interrupt+0x128/0x2c0 [16680.237069] ? smp_apic_timer_interrupt+0xa6/0x270 [16680.248469] ? apic_timer_interrupt+0xf/0x20 [16680.251049] [16680.256260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16680.262400] do_raw_spin_lock+0x177/0x190 [16680.269954] _raw_spin_lock+0x21/0x30 [16680.272238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16680.275623] ? do_raw_spin_unlock+0x75/0x190 [16680.277182] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16680.286005] ? do_wait_intr+0xf0/0xf0 [16680.292821] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16680.297120] mdc_close+0x53b/0xd70 [mdc] [16680.300593] lmv_close+0x18d/0x490 [lmv] [16680.307139] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16680.311919] ll_md_real_close+0xa5/0x2a0 [lustre] [16680.316176] ll_md_close+0x274/0x800 [lustre] [16680.318360] ll_file_release+0x128/0xc50 [lustre] [16680.323198] __fput+0x10d/0x440 [16680.324425] ____fput+0x12/0x20 [16680.330158] task_work_run+0xc6/0x110 [16680.331676] exit_to_usermode_loop+0x1dd/0x1f0 [16680.339266] do_syscall_64+0x3d6/0x440 [16680.342733] entry_SYSCALL_64_after_hwframe+0x49/0xae [16680.348419] RIP: 0033:0x7fa749552cd8 [16680.357159] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16680.391373] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16680.406009] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16680.408627] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16680.426378] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16680.443477] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16680.446700] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16692.056054] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16692.060835] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16692.063538] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16692.063553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16692.063632] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16692.063638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16692.063641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16692.063656] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16692.063661] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16692.063665] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16692.063667] RDX: 0000000000002e0d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16692.063669] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16692.063671] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16692.063673] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16692.063677] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16692.063679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16692.063682] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16692.063690] Call Trace: [16692.063697] [16692.063705] ? show_regs.cold.9+0x22/0x2f [16692.063714] ? watchdog_timer_fn.cold.6+0x62/0x10a [16692.063721] ? __hrtimer_run_queues+0x1c2/0x4d0 [16692.063725] ? watchdog+0x40/0x40 [16692.063728] ? hrtimer_interrupt+0x128/0x2c0 [16692.063732] ? smp_apic_timer_interrupt+0xa6/0x270 [16692.063743] ? apic_timer_interrupt+0xf/0x20 [16692.063748] [16692.063750] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16692.063755] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16692.063760] do_raw_spin_lock+0x177/0x190 [16692.063764] _raw_spin_lock+0x21/0x30 [16692.063769] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16692.064946] obd_mod_free+0x46/0x140 [ptlrpc] [16692.065717] mdc_commit_open+0x91/0xb0 [mdc] [16692.065764] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16692.065964] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16692.066172] after_reply+0xe2c/0x1780 [ptlrpc] [16692.066372] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16692.066567] ? __next_timer_interrupt+0x160/0x160 [16692.066576] ptlrpcd+0xa82/0xe40 [ptlrpc] [16692.066795] ? do_wait_intr+0xf0/0xf0 [16692.066803] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16692.066996] kthread+0x1d1/0x200 [16692.067002] ? set_kthread_struct+0x70/0x70 [16692.067006] ret_from_fork+0x1f/0x30 [16692.071825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16692.071918] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16692.071927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16692.071929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16692.071945] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16692.071949] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16692.071954] RAX: 0000000000007d6d RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16692.071957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16692.071959] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16692.071961] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16692.071964] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16692.071967] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16692.071970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16692.071973] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16692.071982] Call Trace: [16692.071987] [16692.071993] ? show_regs.cold.9+0x22/0x2f [16692.072014] ? watchdog_timer_fn.cold.6+0x62/0x10a [16692.072021] ? __hrtimer_run_queues+0x1c2/0x4d0 [16692.072026] ? watchdog+0x40/0x40 [16692.072030] ? hrtimer_interrupt+0x128/0x2c0 [16692.072034] ? smp_apic_timer_interrupt+0xa6/0x270 [16692.072043] ? apic_timer_interrupt+0xf/0x20 [16692.072048] [16692.072050] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16692.072057] do_raw_spin_lock+0x177/0x190 [16692.072062] _raw_spin_lock+0x21/0x30 [16692.072066] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16692.072367] ? do_raw_spin_unlock+0x75/0x190 [16692.072374] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16692.072840] ? do_wait_intr+0xf0/0xf0 [16692.072849] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16692.073247] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16692.073857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16692.074614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.075133] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16692.075177] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16692.075208] mdc_intent_lock+0x353/0x7e0 [mdc] [16692.075236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.075539] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16692.088487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16692.088570] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16692.088577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16692.088584] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16692.089566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16692.089580] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16692.089584] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16692.089589] RAX: 0000000000006085 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16692.089592] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16692.089594] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16692.089608] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16692.089611] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16692.089615] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16692.089618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16692.089621] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16692.089628] Call Trace: [16692.089633] [16692.089639] ? show_regs.cold.9+0x22/0x2f [16692.089650] ? watchdog_timer_fn.cold.6+0x62/0x10a [16692.089657] ? __hrtimer_run_queues+0x1c2/0x4d0 [16692.089662] ? watchdog+0x40/0x40 [16692.089667] ? hrtimer_interrupt+0x128/0x2c0 [16692.089672] ? smp_apic_timer_interrupt+0xa6/0x270 [16692.089816] ? apic_timer_interrupt+0xf/0x20 [16692.089821] [16692.089825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16692.089830] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16692.089836] do_raw_spin_lock+0x177/0x190 [16692.089841] _raw_spin_lock+0x21/0x30 [16692.089847] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16692.090096] process_one_work+0x2c8/0x700 [16692.090103] worker_thread+0x296/0x6e0 [16692.090107] ? rescuer_thread+0x570/0x570 [16692.090110] kthread+0x1d1/0x200 [16692.090115] ? set_kthread_struct+0x70/0x70 [16692.090119] ret_from_fork+0x1f/0x30 [16692.912262] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16692.917386] lmv_intent_open+0x338/0x1150 [lmv] [16692.921886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.931664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.936103] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16692.938505] ll_intent_lock+0xc5/0xaa0 [lustre] [16692.942153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.945806] ll_lookup_it+0xe0e/0x2270 [lustre] [16692.949187] ? do_raw_spin_unlock+0x75/0x190 [16692.954664] ? _raw_spin_unlock+0x12/0x30 [16692.957070] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16692.959858] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16692.962890] ll_atomic_open+0x36a/0xd70 [lustre] [16692.966911] lookup_open+0x356/0xb40 [16692.970679] do_last+0x313/0xfc0 [16692.974052] ? nd_jump_root+0xe5/0x160 [16692.978255] ? path_init+0x437/0x520 [16692.980563] path_openat+0xf7/0x500 [16692.982946] ? do_raw_spin_unlock+0x75/0x190 [16692.992125] do_filp_open+0x99/0x140 [16692.996136] ? getname_flags+0x6e/0x330 [16693.001915] ? __check_object_size+0xff/0x256 [16693.006647] ? do_raw_spin_unlock+0x75/0x190 [16693.010845] ? _raw_spin_unlock+0x12/0x30 [16693.014799] do_sys_openat2+0x2b4/0x410 [16693.018893] do_sys_open+0x73/0xa0 [16693.020701] __x64_sys_openat+0x24/0x30 [16693.023860] do_syscall_64+0xc1/0x440 [16693.032828] entry_SYSCALL_64_after_hwframe+0x49/0xae [16693.034385] RIP: 0033:0x7fa749552332 [16693.043157] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16693.061832] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16693.083500] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16693.095904] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16693.101355] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16693.106768] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16693.111469] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16697.824354] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14780s! [16697.832106] Showing busy workqueues and worker pools: [16697.835944] workqueue events: flags=0x0 [16697.838603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16697.847410] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16697.858363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16697.868310] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16697.877737] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16697.883064] pending: kfree_rcu_monitor [16697.886400] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16697.893831] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16697.905402] workqueue events_long: flags=0x0 [16697.909677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16697.916734] pending: lu_objects_flush [obdclass] [16697.923270] workqueue events_unbound: flags=0x2 [16697.926322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16697.930591] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16697.940776] workqueue events_power_efficient: flags=0x80 [16697.948089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16697.953435] pending: check_lifetime [16697.958868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16697.966441] pending: neigh_periodic_work [16697.969065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16697.972543] pending: neigh_periodic_work [16697.975009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16697.979877] pending: do_cache_clean [16697.981578] workqueue mm_percpu_wq: flags=0x8 [16697.985959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16697.993332] pending: vmstat_update [16697.995405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16698.003756] pending: vmstat_update [16698.009759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16698.019262] pending: vmstat_update [16698.021878] workqueue writeback: flags=0x4a [16698.027780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16698.034124] pending: wb_workfn, wb_workfn [16698.038328] workqueue xprtiod: flags=0xa [16698.040084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16698.043377] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16698.053685] workqueue ipv6_addrconf: flags=0x40008 [16698.057561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16698.064769] pending: addrconf_verify_work [16698.066675] workqueue knbd0-recv: flags=0x1a [16698.068642] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16698.071404] in-flight: 651:recv_work [16698.072819] workqueue ptlrpc_pinger: flags=0x2 [16698.075401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16698.077678] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16698.081433] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14780s workers=4 idle: 37 31846 28897 [16698.087517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16687s workers=2 idle: 653 [16708.061358] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16708.063692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16708.101080] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16708.106963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16708.115604] RIP: 0010:native_safe_halt+0x2e/0x50 [16708.117993] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16708.132785] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16708.135768] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16708.139175] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16708.147134] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16708.149841] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16708.152691] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16708.155910] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16708.159525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16708.162910] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16708.165829] Call Trace: [16708.166726] [16708.167443] ? show_regs.cold.9+0x22/0x2f [16708.168973] ? watchdog_timer_fn.cold.6+0x62/0x10a [16708.171335] ? __hrtimer_run_queues+0x1c2/0x4d0 [16708.173259] ? watchdog+0x40/0x40 [16708.174455] ? hrtimer_interrupt+0x128/0x2c0 [16708.175965] ? smp_apic_timer_interrupt+0xa6/0x270 [16708.177653] ? apic_timer_interrupt+0xf/0x20 [16708.179542] [16708.180382] ? native_safe_halt+0x2e/0x50 [16708.181786] kvm_wait+0xcd/0xe0 [16708.182836] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16708.184726] do_raw_spin_lock+0x177/0x190 [16708.186448] _raw_spin_lock+0x21/0x30 [16708.188147] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16708.190380] ? do_raw_spin_unlock+0x75/0x190 [16708.194443] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16708.196825] ? do_wait_intr+0xf0/0xf0 [16708.199594] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16708.201713] mdc_close+0x53b/0xd70 [mdc] [16708.204543] lmv_close+0x18d/0x490 [lmv] [16708.205855] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16708.211174] ll_md_real_close+0xa5/0x2a0 [lustre] [16708.212781] ll_md_close+0x274/0x800 [lustre] [16708.216186] ll_file_release+0x128/0xc50 [lustre] [16708.217840] __fput+0x10d/0x440 [16708.218971] ____fput+0x12/0x20 [16708.220012] task_work_run+0xc6/0x110 [16708.221362] exit_to_usermode_loop+0x1dd/0x1f0 [16708.224240] do_syscall_64+0x3d6/0x440 [16708.225380] entry_SYSCALL_64_after_hwframe+0x49/0xae [16708.228253] RIP: 0033:0x7fa749552cd8 [16708.229829] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16708.238564] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16708.241636] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16708.245736] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16708.248128] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16708.252347] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16708.257520] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16720.057030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16720.058055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16720.059466] Modules linked in: [16720.062230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16720.062307] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16720.062313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16720.062315] RIP: 0010:native_safe_halt+0x2e/0x50 [16720.062328] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16720.062332] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [16720.062336] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16720.062339] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16720.062341] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16720.062343] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16720.062345] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16720.062348] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16720.062351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16720.062353] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16720.062360] Call Trace: [16720.062365] [16720.062370] ? show_regs.cold.9+0x22/0x2f [16720.062378] ? watchdog_timer_fn.cold.6+0x62/0x10a [16720.062383] ? __hrtimer_run_queues+0x1c2/0x4d0 [16720.062387] ? watchdog+0x40/0x40 [16720.062390] ? hrtimer_interrupt+0x128/0x2c0 [16720.062394] ? smp_apic_timer_interrupt+0xa6/0x270 [16720.062400] ? apic_timer_interrupt+0xf/0x20 [16720.062404] [16720.062407] ? native_safe_halt+0x2e/0x50 [16720.062410] kvm_wait+0xcd/0xe0 [16720.062417] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16720.062423] do_raw_spin_lock+0x177/0x190 [16720.062427] _raw_spin_lock+0x21/0x30 [16720.062432] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16720.062694] obd_mod_free+0x46/0x140 [ptlrpc] [16720.068285] lustre(O) [16720.088469] mdc_commit_open+0x91/0xb0 [mdc] [16720.088520] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16720.096390] osp(O) [16720.103242] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16720.103548] after_reply+0xe2c/0x1780 [ptlrpc] [16720.105706] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16720.105774] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16720.105780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16720.105786] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16720.106198] RIP: 0010:native_safe_halt+0x2e/0x50 [16720.106208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16720.106212] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16720.106216] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16720.106218] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16720.106220] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16720.106222] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16720.106225] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16720.106228] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16720.106230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16720.106233] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16720.106241] Call Trace: [16720.106247] [16720.106252] ? show_regs.cold.9+0x22/0x2f [16720.106263] ? watchdog_timer_fn.cold.6+0x62/0x10a [16720.106270] ? __hrtimer_run_queues+0x1c2/0x4d0 [16720.106275] ? watchdog+0x40/0x40 [16720.106279] ? hrtimer_interrupt+0x128/0x2c0 [16720.106283] ? smp_apic_timer_interrupt+0xa6/0x270 [16720.106291] ? apic_timer_interrupt+0xf/0x20 [16720.106295] [16720.106299] ? native_safe_halt+0x2e/0x50 [16720.106303] kvm_wait+0xcd/0xe0 [16720.106311] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16720.106320] do_raw_spin_lock+0x177/0x190 [16720.106324] _raw_spin_lock+0x21/0x30 [16720.106328] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16720.106526] process_one_work+0x2c8/0x700 [16720.106533] worker_thread+0x296/0x6e0 [16720.106537] ? rescuer_thread+0x570/0x570 [16720.106541] kthread+0x1d1/0x200 [16720.106545] ? set_kthread_struct+0x70/0x70 [16720.106549] ret_from_fork+0x1f/0x30 [16720.620611] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16720.624752] ? __next_timer_interrupt+0x160/0x160 [16720.627704] ptlrpcd+0xa82/0xe40 [ptlrpc] [16720.631480] ? do_wait_intr+0xf0/0xf0 [16720.635612] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16720.640388] kthread+0x1d1/0x200 [16720.642421] ? set_kthread_struct+0x70/0x70 [16720.644563] ret_from_fork+0x1f/0x30 [16724.054022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16724.057526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16724.114258] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16724.119566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16724.133854] RIP: 0010:native_safe_halt+0x2e/0x50 [16724.144236] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16724.163417] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16724.176949] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16724.180706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16724.183307] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16724.186392] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16724.191012] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16724.194238] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16724.201308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16724.209936] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16724.216246] Call Trace: [16724.217856] [16724.219108] ? show_regs.cold.9+0x22/0x2f [16724.221596] ? watchdog_timer_fn.cold.6+0x62/0x10a [16724.223998] ? __hrtimer_run_queues+0x1c2/0x4d0 [16724.227319] ? watchdog+0x40/0x40 [16724.232413] ? hrtimer_interrupt+0x128/0x2c0 [16724.233784] ? smp_apic_timer_interrupt+0xa6/0x270 [16724.240166] ? apic_timer_interrupt+0xf/0x20 [16724.247234] [16724.252696] ? native_safe_halt+0x2e/0x50 [16724.259900] kvm_wait+0xcd/0xe0 [16724.261961] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16724.264691] do_raw_spin_lock+0x177/0x190 [16724.268357] _raw_spin_lock+0x21/0x30 [16724.270003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16724.273361] ? do_raw_spin_unlock+0x75/0x190 [16724.276493] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16724.281676] ? do_wait_intr+0xf0/0xf0 [16724.285531] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16724.296501] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16724.302345] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16724.308719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16724.317529] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16724.326456] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16724.328054] mdc_intent_lock+0x353/0x7e0 [mdc] [16724.337384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16724.340635] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16724.348149] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16724.359136] lmv_intent_open+0x338/0x1150 [lmv] [16724.368491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16724.370668] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16724.384866] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16724.389233] ll_intent_lock+0xc5/0xaa0 [lustre] [16724.391901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16724.396256] ll_lookup_it+0xe0e/0x2270 [lustre] [16724.398776] ? do_raw_spin_unlock+0x75/0x190 [16724.401300] ? _raw_spin_unlock+0x12/0x30 [16724.403461] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16724.405448] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16724.411883] ll_atomic_open+0x36a/0xd70 [lustre] [16724.416220] lookup_open+0x356/0xb40 [16724.419975] do_last+0x313/0xfc0 [16724.421474] ? nd_jump_root+0xe5/0x160 [16724.427362] ? path_init+0x437/0x520 [16724.430417] path_openat+0xf7/0x500 [16724.438203] ? do_raw_spin_unlock+0x75/0x190 [16724.442861] do_filp_open+0x99/0x140 [16724.446977] ? getname_flags+0x6e/0x330 [16724.448338] ? __check_object_size+0xff/0x256 [16724.450295] ? do_raw_spin_unlock+0x75/0x190 [16724.453685] ? _raw_spin_unlock+0x12/0x30 [16724.456173] do_sys_openat2+0x2b4/0x410 [16724.463001] do_sys_open+0x73/0xa0 [16724.465398] __x64_sys_openat+0x24/0x30 [16724.469087] do_syscall_64+0xc1/0x440 [16724.473321] entry_SYSCALL_64_after_hwframe+0x49/0xae [16724.479785] RIP: 0033:0x7fa749552332 [16724.483735] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16724.498257] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16724.505317] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16724.509756] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16724.517738] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16724.521482] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16724.526140] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16728.551265] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14811s! [16728.561329] Showing busy workqueues and worker pools: [16728.564187] workqueue events: flags=0x0 [16728.570336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16728.579011] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16728.592847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16728.600248] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16728.612933] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.620003] pending: kfree_rcu_monitor [16728.621951] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16728.624660] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16728.628524] workqueue events_long: flags=0x0 [16728.630471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.633088] pending: lu_objects_flush [obdclass] [16728.634896] workqueue events_unbound: flags=0x2 [16728.636448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16728.639124] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16728.643106] workqueue events_power_efficient: flags=0x80 [16728.645251] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.648032] pending: check_lifetime [16728.649459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.651571] pending: neigh_periodic_work [16728.653763] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.656239] pending: neigh_periodic_work [16728.658506] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.662131] pending: do_cache_clean [16728.665316] workqueue mm_percpu_wq: flags=0x8 [16728.667249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.670835] pending: vmstat_update [16728.673379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.677192] pending: vmstat_update [16728.678736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16728.681190] pending: vmstat_update [16728.682815] workqueue writeback: flags=0x4a [16728.684193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16728.686923] pending: wb_workfn, wb_workfn [16728.688859] workqueue xprtiod: flags=0xa [16728.690461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16728.693015] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16728.695906] workqueue ipv6_addrconf: flags=0x40008 [16728.698274] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16728.702301] pending: addrconf_verify_work [16728.706451] workqueue knbd0-recv: flags=0x1a [16728.711567] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16728.713555] in-flight: 651:recv_work [16728.717193] workqueue ptlrpc_pinger: flags=0x2 [16728.719488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16728.722993] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16728.725650] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14811s workers=4 idle: 37 31846 28897 [16728.729325] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16718s workers=2 idle: 653 [16734.015011] rcu: INFO: rcu_sched self-detected stall on CPU [16734.018860] rcu: 2-...!: (8733362 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [16734.024573] (t=14820326 jiffies g=258701 q=5468) [16734.026530] rcu: rcu_sched kthread starved for 14817121 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [16734.031676] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16734.036622] rcu: RCU grace-period kthread stack dump: [16734.038881] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16734.049153] Call Trace: [16734.052409] __schedule+0x351/0xcb0 [16734.054845] schedule+0xc0/0x180 [16734.057581] schedule_timeout+0xb4/0x190 [16734.061517] ? do_raw_spin_unlock+0x75/0x190 [16734.068030] ? __next_timer_interrupt+0x160/0x160 [16734.069492] ? prepare_to_swait_event+0xf3/0x140 [16734.075236] rcu_gp_kthread+0x8dd/0x15b0 [16734.077145] ? force_qs_rnp+0x2c0/0x2c0 [16734.078738] kthread+0x1d1/0x200 [16734.080788] ? set_kthread_struct+0x70/0x70 [16734.086558] ret_from_fork+0x1f/0x30 [16734.088483] rcu: Stack dump where RCU GP kthread last ran: [16734.090320] Sending NMI from CPU 2 to CPUs 1: [16734.099833] NMI backtrace for cpu 1 [16734.099840] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16734.099842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16734.099844] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16734.099847] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16734.099849] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16734.099851] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 [16734.099853] RAX: 0000000000004ab4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16734.099854] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16734.099856] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16734.099857] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16734.099859] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16734.099860] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16734.100014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16734.100015] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16734.100016] Call Trace: [16734.100017] [16734.100018] ? show_regs.cold.9+0x22/0x2f [16734.100019] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [16734.100020] ? nmi_cpu_backtrace_handler+0x15/0x40 [16734.100021] ? nmi_handle+0xbd/0x210 [16734.100022] ? default_do_nmi+0x4b/0x1b0 [16734.100023] ? do_nmi+0x280/0x380 [16734.100024] ? end_repeat_nmi+0x16/0x1f [16734.100025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16734.100026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16734.100027] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16734.100029] [16734.100029] do_raw_spin_lock+0x177/0x190 [16734.100030] _raw_spin_lock+0x21/0x30 [16734.100031] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16734.100032] process_one_work+0x2c8/0x700 [16734.100034] worker_thread+0x296/0x6e0 [16734.100035] ? rescuer_thread+0x570/0x570 [16734.100035] kthread+0x1d1/0x200 [16734.100037] ? set_kthread_struct+0x70/0x70 [16734.100038] ret_from_fork+0x1f/0x30 [16734.100850] NMI backtrace for cpu 2 [16734.285282] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16734.293307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16734.302169] Call Trace: [16734.306676] [16734.314186] ? dump_stack+0xbb/0x10e [16734.316191] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16734.324406] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16734.337724] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16734.340517] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16734.342729] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16734.344508] ? rcu_dump_cpu_stacks+0x11b/0x171 [16734.345810] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16734.349850] ? update_process_times+0x6d/0xc0 [16734.352054] ? tick_sched_handle.isra.18+0x27/0x90 [16734.355218] ? tick_sched_timer+0x53/0xb0 [16734.357517] ? __hrtimer_run_queues+0x1c2/0x4d0 [16734.360039] ? tick_sched_handle.isra.18+0x90/0x90 [16734.362230] ? hrtimer_interrupt+0x128/0x2c0 [16734.363993] ? smp_apic_timer_interrupt+0xa6/0x270 [16734.365958] ? apic_timer_interrupt+0xf/0x20 [16734.370073] [16734.371016] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16734.373934] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16734.377551] ? do_raw_spin_lock+0x177/0x190 [16734.380410] ? _raw_spin_lock+0x21/0x30 [16734.384104] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16734.388343] ? obd_mod_free+0x46/0x140 [ptlrpc] [16734.392465] ? mdc_commit_open+0x91/0xb0 [mdc] [16734.396347] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16734.401056] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16734.408358] ? after_reply+0xe2c/0x1780 [ptlrpc] [16734.412192] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16734.417413] ? __next_timer_interrupt+0x160/0x160 [16734.420526] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16734.424177] ? do_wait_intr+0xf0/0xf0 [16734.425872] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16734.428395] ? kthread+0x1d1/0x200 [16734.429724] ? set_kthread_struct+0x70/0x70 [16734.431137] ? ret_from_fork+0x1f/0x30 [16736.059005] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16736.061458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16736.083757] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16736.096120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16736.100064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16736.102430] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16736.110754] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16736.113703] RAX: 0000000000005af6 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16736.121321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16736.130633] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16736.137503] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16736.145916] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16736.156526] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16736.162281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16736.165399] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16736.168745] Call Trace: [16736.170299] [16736.171182] ? show_regs.cold.9+0x22/0x2f [16736.172295] ? watchdog_timer_fn.cold.6+0x62/0x10a [16736.175206] ? __hrtimer_run_queues+0x1c2/0x4d0 [16736.177478] ? watchdog+0x40/0x40 [16736.179105] ? hrtimer_interrupt+0x128/0x2c0 [16736.180885] ? smp_apic_timer_interrupt+0xa6/0x270 [16736.182317] ? apic_timer_interrupt+0xf/0x20 [16736.183811] [16736.185775] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16736.188472] do_raw_spin_lock+0x177/0x190 [16736.190622] _raw_spin_lock+0x21/0x30 [16736.192513] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16736.195135] ? do_raw_spin_unlock+0x75/0x190 [16736.197163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16736.199296] ? do_wait_intr+0xf0/0xf0 [16736.201435] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16736.203977] mdc_close+0x53b/0xd70 [mdc] [16736.206225] lmv_close+0x18d/0x490 [lmv] [16736.208111] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16736.213786] ll_md_real_close+0xa5/0x2a0 [lustre] [16736.215309] ll_md_close+0x274/0x800 [lustre] [16736.217348] ll_file_release+0x128/0xc50 [lustre] [16736.219862] __fput+0x10d/0x440 [16736.221314] ____fput+0x12/0x20 [16736.222725] task_work_run+0xc6/0x110 [16736.224255] exit_to_usermode_loop+0x1dd/0x1f0 [16736.225718] do_syscall_64+0x3d6/0x440 [16736.228137] entry_SYSCALL_64_after_hwframe+0x49/0xae [16736.230345] RIP: 0033:0x7fa749552cd8 [16736.232154] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16736.241206] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16736.244692] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16736.247990] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16736.251200] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16736.253374] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16736.257357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16756.055640] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16756.083985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16756.144444] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16756.167327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16756.179725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16756.193850] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16756.217804] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16756.228775] RAX: 0000000000007e87 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16756.235775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16756.243421] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16756.247280] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16756.250015] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16756.256766] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16756.262841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16756.266253] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16756.273114] Call Trace: [16756.277906] [16756.279072] ? show_regs.cold.9+0x22/0x2f [16756.282155] ? watchdog_timer_fn.cold.6+0x62/0x10a [16756.284957] ? __hrtimer_run_queues+0x1c2/0x4d0 [16756.286948] ? watchdog+0x40/0x40 [16756.288456] ? hrtimer_interrupt+0x128/0x2c0 [16756.290620] ? smp_apic_timer_interrupt+0xa6/0x270 [16756.292990] ? apic_timer_interrupt+0xf/0x20 [16756.294863] [16756.295845] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16756.298221] do_raw_spin_lock+0x177/0x190 [16756.300225] _raw_spin_lock+0x21/0x30 [16756.302272] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16756.304888] ? do_raw_spin_unlock+0x75/0x190 [16756.309795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16756.314921] ? do_wait_intr+0xf0/0xf0 [16756.316749] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16756.319365] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16756.321989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16756.324944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16756.328426] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16756.333294] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16756.335020] mdc_intent_lock+0x353/0x7e0 [mdc] [16756.337028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16756.339692] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16756.342682] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16756.345985] lmv_intent_open+0x338/0x1150 [lmv] [16756.348695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16756.356454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16756.360488] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16756.363840] ll_intent_lock+0xc5/0xaa0 [lustre] [16756.368450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16756.372607] ll_lookup_it+0xe0e/0x2270 [lustre] [16756.374401] ? do_raw_spin_unlock+0x75/0x190 [16756.377444] ? _raw_spin_unlock+0x12/0x30 [16756.380026] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16756.382159] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16756.385137] ll_atomic_open+0x36a/0xd70 [lustre] [16756.387366] lookup_open+0x356/0xb40 [16756.389345] do_last+0x313/0xfc0 [16756.390662] ? nd_jump_root+0xe5/0x160 [16756.392237] ? path_init+0x437/0x520 [16756.394615] path_openat+0xf7/0x500 [16756.396440] ? do_raw_spin_unlock+0x75/0x190 [16756.399850] do_filp_open+0x99/0x140 [16756.402081] ? getname_flags+0x6e/0x330 [16756.403654] ? __check_object_size+0xff/0x256 [16756.405836] ? do_raw_spin_unlock+0x75/0x190 [16756.407848] ? _raw_spin_unlock+0x12/0x30 [16756.409487] do_sys_openat2+0x2b4/0x410 [16756.411371] do_sys_open+0x73/0xa0 [16756.412612] __x64_sys_openat+0x24/0x30 [16756.414247] do_syscall_64+0xc1/0x440 [16756.415976] entry_SYSCALL_64_after_hwframe+0x49/0xae [16756.417947] RIP: 0033:0x7fa749552332 [16756.419813] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16756.435423] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16756.438563] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16756.441806] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16756.445022] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16756.448214] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16756.451361] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16758.753291] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14841s! [16758.756768] Showing busy workqueues and worker pools: [16758.763660] workqueue events: flags=0x0 [16758.766692] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16758.769858] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16758.774658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16758.777374] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16758.781832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.803370] pending: kfree_rcu_monitor [16758.809518] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16758.821486] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16758.832929] workqueue events_long: flags=0x0 [16758.839888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.853031] pending: lu_objects_flush [obdclass] [16758.855779] workqueue events_unbound: flags=0x2 [16758.858980] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16758.863392] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16758.872059] workqueue events_power_efficient: flags=0x80 [16758.875715] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.890072] pending: check_lifetime [16758.894072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.897642] pending: neigh_periodic_work [16758.905499] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.913713] pending: neigh_periodic_work [16758.919817] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.923601] pending: do_cache_clean [16758.924968] workqueue mm_percpu_wq: flags=0x8 [16758.927886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.933250] pending: vmstat_update [16758.936118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.940245] pending: vmstat_update [16758.942135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16758.946793] pending: vmstat_update [16758.949838] workqueue writeback: flags=0x4a [16758.953047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16758.961896] pending: wb_workfn, wb_workfn [16758.965455] workqueue xprtiod: flags=0xa [16758.967710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16758.974006] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16758.976392] workqueue ipv6_addrconf: flags=0x40008 [16758.978014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16758.980943] pending: addrconf_verify_work [16758.986570] workqueue knbd0-recv: flags=0x1a [16758.988526] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16758.992681] in-flight: 651:recv_work [16758.994156] workqueue ptlrpc_pinger: flags=0x2 [16758.996057] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16759.001560] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16759.005751] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14841s workers=4 idle: 37 31846 28897 [16759.009413] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16748s workers=2 idle: 653 [16760.057018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16760.058276] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16760.059831] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16760.059977] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16760.059984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16760.059994] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16760.060266] RIP: 0010:native_safe_halt+0x2e/0x50 [16760.060276] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16760.060280] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16760.060284] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16760.060286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16760.060289] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16760.060290] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16760.060292] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16760.060295] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16760.060298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16760.060300] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16760.060307] Call Trace: [16760.060313] [16760.060318] ? show_regs.cold.9+0x22/0x2f [16760.060328] ? watchdog_timer_fn.cold.6+0x62/0x10a [16760.060335] ? __hrtimer_run_queues+0x1c2/0x4d0 [16760.060340] ? watchdog+0x40/0x40 [16760.060344] ? hrtimer_interrupt+0x128/0x2c0 [16760.060349] ? smp_apic_timer_interrupt+0xa6/0x270 [16760.060356] ? apic_timer_interrupt+0xf/0x20 [16760.060360] [16760.060368] ? native_safe_halt+0x2e/0x50 [16760.060373] kvm_wait+0xcd/0xe0 [16760.060381] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16760.060389] do_raw_spin_lock+0x177/0x190 [16760.060394] _raw_spin_lock+0x21/0x30 [16760.060398] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16760.060619] process_one_work+0x2c8/0x700 [16760.060626] worker_thread+0x296/0x6e0 [16760.060632] ? rescuer_thread+0x570/0x570 [16760.060636] kthread+0x1d1/0x200 [16760.060640] ? set_kthread_struct+0x70/0x70 [16760.060644] ret_from_fork+0x1f/0x30 [16760.160330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16760.177678] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16760.181633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16760.184704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16760.186886] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16760.193848] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16760.196427] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16760.199432] RDX: 000000000000408e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16760.202318] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16760.205027] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16760.207861] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16760.211002] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16760.214172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16760.216238] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16760.219462] Call Trace: [16760.220400] [16760.221523] ? show_regs.cold.9+0x22/0x2f [16760.223262] ? watchdog_timer_fn.cold.6+0x62/0x10a [16760.225251] ? __hrtimer_run_queues+0x1c2/0x4d0 [16760.227108] ? watchdog+0x40/0x40 [16760.228458] ? hrtimer_interrupt+0x128/0x2c0 [16760.230101] ? smp_apic_timer_interrupt+0xa6/0x270 [16760.232394] ? apic_timer_interrupt+0xf/0x20 [16760.234173] [16760.235166] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16760.237332] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16760.239581] do_raw_spin_lock+0x177/0x190 [16760.241226] _raw_spin_lock+0x21/0x30 [16760.242683] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16760.244940] obd_mod_free+0x46/0x140 [ptlrpc] [16760.247187] mdc_commit_open+0x91/0xb0 [mdc] [16760.249479] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16760.251782] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16760.254401] after_reply+0xe2c/0x1780 [ptlrpc] [16760.256666] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16760.259054] ? __next_timer_interrupt+0x160/0x160 [16760.260844] ptlrpcd+0xa82/0xe40 [ptlrpc] [16760.262717] ? do_wait_intr+0xf0/0xf0 [16760.264741] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16760.266994] kthread+0x1d1/0x200 [16760.268259] ? set_kthread_struct+0x70/0x70 [16760.270087] ret_from_fork+0x1f/0x30 [16764.059204] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16764.062665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16764.082021] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16764.086427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16764.089905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [16764.092549] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [16764.099559] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16764.102533] RAX: 00000000000041ba RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16764.105533] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16764.107937] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16764.110745] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16764.113283] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16764.115941] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16764.119254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16764.121528] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16764.124915] Call Trace: [16764.125887] [16764.126572] ? show_regs.cold.9+0x22/0x2f [16764.127663] ? watchdog_timer_fn.cold.6+0x62/0x10a [16764.128907] ? __hrtimer_run_queues+0x1c2/0x4d0 [16764.130668] ? watchdog+0x40/0x40 [16764.132425] ? hrtimer_interrupt+0x128/0x2c0 [16764.133846] ? smp_apic_timer_interrupt+0xa6/0x270 [16764.135387] ? apic_timer_interrupt+0xf/0x20 [16764.136851] [16764.137804] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [16764.140146] do_raw_spin_lock+0x177/0x190 [16764.141657] _raw_spin_lock+0x21/0x30 [16764.143263] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16764.145657] ? do_raw_spin_unlock+0x75/0x190 [16764.147372] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16764.149672] ? do_wait_intr+0xf0/0xf0 [16764.151407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16764.153926] mdc_close+0x53b/0xd70 [mdc] [16764.155580] lmv_close+0x18d/0x490 [lmv] [16764.157450] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16764.159688] ll_md_real_close+0xa5/0x2a0 [lustre] [16764.161702] ll_md_close+0x274/0x800 [lustre] [16764.163903] ll_file_release+0x128/0xc50 [lustre] [16764.166149] __fput+0x10d/0x440 [16764.167206] ____fput+0x12/0x20 [16764.168443] task_work_run+0xc6/0x110 [16764.169937] exit_to_usermode_loop+0x1dd/0x1f0 [16764.171288] do_syscall_64+0x3d6/0x440 [16764.172855] entry_SYSCALL_64_after_hwframe+0x49/0xae [16764.174782] RIP: 0033:0x7fa749552cd8 [16764.176387] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16764.183511] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16764.186616] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16764.189287] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16764.191926] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16764.194569] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16764.196826] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16784.054027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16784.057060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16784.123028] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16784.131522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16784.145976] RIP: 0010:native_safe_halt+0x2e/0x50 [16784.152614] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16784.182626] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16784.199207] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16784.213939] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16784.218530] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16784.232830] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16784.241821] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16784.249901] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16784.261109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16784.265228] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16784.270272] Call Trace: [16784.273082] [16784.277933] ? show_regs.cold.9+0x22/0x2f [16784.293652] ? watchdog_timer_fn.cold.6+0x62/0x10a [16784.295298] ? __hrtimer_run_queues+0x1c2/0x4d0 [16784.311451] ? watchdog+0x40/0x40 [16784.318179] ? hrtimer_interrupt+0x128/0x2c0 [16784.323577] ? smp_apic_timer_interrupt+0xa6/0x270 [16784.325303] ? apic_timer_interrupt+0xf/0x20 [16784.337899] [16784.338727] ? native_safe_halt+0x2e/0x50 [16784.352849] kvm_wait+0xcd/0xe0 [16784.354066] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16784.355815] do_raw_spin_lock+0x177/0x190 [16784.385008] _raw_spin_lock+0x21/0x30 [16784.386348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16784.410248] ? do_raw_spin_unlock+0x75/0x190 [16784.420778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16784.422765] ? do_wait_intr+0xf0/0xf0 [16784.435259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16784.441911] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16784.457383] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16784.467889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16784.469824] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16784.480347] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16784.495427] mdc_intent_lock+0x353/0x7e0 [mdc] [16784.497180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16784.511153] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16784.540785] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16784.550075] lmv_intent_open+0x338/0x1150 [lmv] [16784.571575] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16784.575638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16784.582422] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16784.586877] ll_intent_lock+0xc5/0xaa0 [lustre] [16784.594238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16784.599687] ll_lookup_it+0xe0e/0x2270 [lustre] [16784.603352] ? do_raw_spin_unlock+0x75/0x190 [16784.606667] ? _raw_spin_unlock+0x12/0x30 [16784.611023] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16784.614943] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16784.620508] ll_atomic_open+0x36a/0xd70 [lustre] [16784.626066] lookup_open+0x356/0xb40 [16784.628825] do_last+0x313/0xfc0 [16784.631558] ? nd_jump_root+0xe5/0x160 [16784.635817] ? path_init+0x437/0x520 [16784.644527] path_openat+0xf7/0x500 [16784.648950] ? do_raw_spin_unlock+0x75/0x190 [16784.655268] do_filp_open+0x99/0x140 [16784.665580] ? getname_flags+0x6e/0x330 [16784.669037] ? __check_object_size+0xff/0x256 [16784.675695] ? do_raw_spin_unlock+0x75/0x190 [16784.679973] ? _raw_spin_unlock+0x12/0x30 [16784.684999] do_sys_openat2+0x2b4/0x410 [16784.692109] do_sys_open+0x73/0xa0 [16784.698569] __x64_sys_openat+0x24/0x30 [16784.706575] do_syscall_64+0xc1/0x440 [16784.708007] entry_SYSCALL_64_after_hwframe+0x49/0xae [16784.709918] RIP: 0033:0x7fa749552332 [16784.721647] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16784.744588] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16784.749454] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16784.755821] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16784.763115] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16784.768669] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16784.773406] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16788.057022] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16788.059871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16788.060763] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16788.084697] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16788.084710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16788.084717] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16788.087486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16788.087563] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16788.087571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16788.087575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16788.087588] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16788.087592] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16788.087597] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16788.087599] RDX: 000000000000063a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16788.087602] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16788.087604] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16788.087607] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16788.087610] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16788.087613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16788.087615] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16788.087624] Call Trace: [16788.087630] [16788.087636] ? show_regs.cold.9+0x22/0x2f [16788.087647] ? watchdog_timer_fn.cold.6+0x62/0x10a [16788.087655] ? __hrtimer_run_queues+0x1c2/0x4d0 [16788.087660] ? watchdog+0x40/0x40 [16788.087664] ? hrtimer_interrupt+0x128/0x2c0 [16788.087669] ? smp_apic_timer_interrupt+0xa6/0x270 [16788.087678] ? apic_timer_interrupt+0xf/0x20 [16788.087683] [16788.087686] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16788.087691] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16788.087696] do_raw_spin_lock+0x177/0x190 [16788.087701] _raw_spin_lock+0x21/0x30 [16788.087706] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16788.087982] obd_mod_free+0x46/0x140 [ptlrpc] [16788.088180] mdc_commit_open+0x91/0xb0 [mdc] [16788.088215] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16788.088415] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16788.088617] after_reply+0xe2c/0x1780 [ptlrpc] [16788.088815] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16788.093442] [16788.093452] RIP: 0010:native_safe_halt+0x2e/0x50 [16788.093468] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16788.093473] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16788.093478] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16788.093481] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16788.093483] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16788.093485] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16788.093488] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16788.093491] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16788.093494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16788.093496] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16788.093504] Call Trace: [16788.093510] [16788.093516] ? show_regs.cold.9+0x22/0x2f [16788.093528] ? watchdog_timer_fn.cold.6+0x62/0x10a [16788.093534] ? __hrtimer_run_queues+0x1c2/0x4d0 [16788.093538] ? watchdog+0x40/0x40 [16788.093543] ? hrtimer_interrupt+0x128/0x2c0 [16788.093547] ? smp_apic_timer_interrupt+0xa6/0x270 [16788.093553] ? apic_timer_interrupt+0xf/0x20 [16788.093558] [16788.093561] ? native_safe_halt+0x2e/0x50 [16788.093565] kvm_wait+0xcd/0xe0 [16788.093572] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16788.093582] do_raw_spin_lock+0x177/0x190 [16788.093586] _raw_spin_lock+0x21/0x30 [16788.093590] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16788.097388] ? __next_timer_interrupt+0x160/0x160 [16788.097409] ptlrpcd+0xa82/0xe40 [ptlrpc] [16788.097679] ? do_wait_intr+0xf0/0xf0 [16788.097690] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16788.097891] kthread+0x1d1/0x200 [16788.097898] ? set_kthread_struct+0x70/0x70 [16788.097902] ret_from_fork+0x1f/0x30 [16788.284491] process_one_work+0x2c8/0x700 [16788.285699] worker_thread+0x296/0x6e0 [16788.288208] ? rescuer_thread+0x570/0x570 [16788.289875] kthread+0x1d1/0x200 [16788.292327] ? set_kthread_struct+0x70/0x70 [16788.294247] ret_from_fork+0x1f/0x30 [16789.478567] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14872s! [16789.483615] Showing busy workqueues and worker pools: [16789.485842] workqueue events: flags=0x0 [16789.488344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16789.492249] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16789.499335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16789.503945] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16789.509267] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.513141] pending: kfree_rcu_monitor [16789.515095] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16789.518524] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16789.523497] workqueue events_long: flags=0x0 [16789.526167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.529375] pending: lu_objects_flush [obdclass] [16789.532196] workqueue events_unbound: flags=0x2 [16789.534431] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16789.537572] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16789.542504] workqueue events_power_efficient: flags=0x80 [16789.545352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.549070] pending: check_lifetime [16789.550915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.553990] pending: neigh_periodic_work [16789.556042] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.559980] pending: neigh_periodic_work [16789.562275] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.565812] pending: do_cache_clean [16789.567769] workqueue mm_percpu_wq: flags=0x8 [16789.569799] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.573061] pending: vmstat_update [16789.574753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.577794] pending: vmstat_update [16789.579205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16789.580747] pending: vmstat_update [16789.582582] workqueue writeback: flags=0x4a [16789.584723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16789.587383] pending: wb_workfn, wb_workfn [16789.589522] workqueue xprtiod: flags=0xa [16789.591401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16789.594394] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16789.598768] workqueue ipv6_addrconf: flags=0x40008 [16789.601070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16789.604105] pending: addrconf_verify_work [16789.606582] workqueue knbd0-recv: flags=0x1a [16789.608565] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16789.612226] in-flight: 651:recv_work [16789.614373] workqueue ptlrpc_pinger: flags=0x2 [16789.617133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16789.620329] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16789.622917] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14872s workers=4 idle: 37 31846 28897 [16789.627864] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16779s workers=2 idle: 653 [16792.059112] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16792.062553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16792.098293] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16792.107862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16792.116269] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16792.121424] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16792.141108] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16792.152253] RAX: 000000000000354f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16792.160356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16792.176649] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16792.183016] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16792.189753] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16792.192240] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16792.204386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16792.209332] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16792.218307] Call Trace: [16792.220255] [16792.221437] ? show_regs.cold.9+0x22/0x2f [16792.224949] ? watchdog_timer_fn.cold.6+0x62/0x10a [16792.228773] ? __hrtimer_run_queues+0x1c2/0x4d0 [16792.231534] ? watchdog+0x40/0x40 [16792.233513] ? hrtimer_interrupt+0x128/0x2c0 [16792.235083] ? smp_apic_timer_interrupt+0xa6/0x270 [16792.238394] ? apic_timer_interrupt+0xf/0x20 [16792.240968] [16792.241934] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16792.244268] do_raw_spin_lock+0x177/0x190 [16792.252784] _raw_spin_lock+0x21/0x30 [16792.258301] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16792.263987] ? do_raw_spin_unlock+0x75/0x190 [16792.265469] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16792.274987] ? do_wait_intr+0xf0/0xf0 [16792.282503] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16792.290506] mdc_close+0x53b/0xd70 [mdc] [16792.295368] lmv_close+0x18d/0x490 [lmv] [16792.297090] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16792.300665] ll_md_real_close+0xa5/0x2a0 [lustre] [16792.303316] ll_md_close+0x274/0x800 [lustre] [16792.308129] ll_file_release+0x128/0xc50 [lustre] [16792.312504] __fput+0x10d/0x440 [16792.313990] ____fput+0x12/0x20 [16792.316439] task_work_run+0xc6/0x110 [16792.317821] exit_to_usermode_loop+0x1dd/0x1f0 [16792.320331] do_syscall_64+0x3d6/0x440 [16792.325162] entry_SYSCALL_64_after_hwframe+0x49/0xae [16792.327314] RIP: 0033:0x7fa749552cd8 [16792.331014] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16792.350726] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16792.358510] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16792.361446] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16792.366424] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16792.370404] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16792.379064] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16816.055946] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16816.057040] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16816.062461] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16816.062475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16816.062547] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16816.062553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16816.062556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16816.062570] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16816.062574] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16816.062579] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16816.062581] RDX: 0000000000007e42 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16816.062584] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16816.062586] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16816.062589] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16816.062592] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16816.062596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16816.062598] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16816.062608] Call Trace: [16816.062613] [16816.062620] ? show_regs.cold.9+0x22/0x2f [16816.062630] ? watchdog_timer_fn.cold.6+0x62/0x10a [16816.062636] ? __hrtimer_run_queues+0x1c2/0x4d0 [16816.062640] ? watchdog+0x40/0x40 [16816.062644] ? hrtimer_interrupt+0x128/0x2c0 [16816.062648] ? smp_apic_timer_interrupt+0xa6/0x270 [16816.062658] ? apic_timer_interrupt+0xf/0x20 [16816.062663] [16816.062665] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16816.062775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16816.062836] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16816.062841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16816.062845] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16816.062852] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16816.062856] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16816.062860] RAX: 000000000000048b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16816.062863] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16816.062865] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16816.062867] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16816.062869] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16816.062872] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16816.062874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16816.062876] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16816.062883] Call Trace: [16816.062886] [16816.062888] ? show_regs.cold.9+0x22/0x2f [16816.062895] ? watchdog_timer_fn.cold.6+0x62/0x10a [16816.062900] ? __hrtimer_run_queues+0x1c2/0x4d0 [16816.062904] ? watchdog+0x40/0x40 [16816.062908] ? hrtimer_interrupt+0x128/0x2c0 [16816.062912] ? smp_apic_timer_interrupt+0xa6/0x270 [16816.062917] ? apic_timer_interrupt+0xf/0x20 [16816.062921] [16816.062924] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16816.062929] do_raw_spin_lock+0x177/0x190 [16816.062934] _raw_spin_lock+0x21/0x30 [16816.062939] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16816.063764] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16816.063936] do_raw_spin_lock+0x177/0x190 [16816.063941] _raw_spin_lock+0x21/0x30 [16816.063946] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16816.064977] obd_mod_free+0x46/0x140 [ptlrpc] [16816.066733] mdc_commit_open+0x91/0xb0 [mdc] [16816.066849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [16816.066767] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16816.066880] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16816.066927] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16816.066935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16816.067197] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16816.066941] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16816.067643] after_reply+0xe2c/0x1780 [ptlrpc] [16816.069235] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16816.069458] ? __next_timer_interrupt+0x160/0x160 [16816.069468] ptlrpcd+0xa82/0xe40 [ptlrpc] [16816.072700] ? do_wait_intr+0xf0/0xf0 [16816.072714] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16816.073001] kthread+0x1d1/0x200 [16816.073881] ? set_kthread_struct+0x70/0x70 [16816.073888] ret_from_fork+0x1f/0x30 [16816.076985] ? do_raw_spin_unlock+0x75/0x190 [16816.101993] [16816.102002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16816.102019] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16816.102028] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16816.102032] RAX: 000000000000558a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16816.102035] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16816.102037] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16816.102039] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16816.102041] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16816.102046] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16816.102049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16816.102051] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16816.102059] Call Trace: [16816.102064] [16816.102069] ? show_regs.cold.9+0x22/0x2f [16816.102079] ? watchdog_timer_fn.cold.6+0x62/0x10a [16816.102085] ? __hrtimer_run_queues+0x1c2/0x4d0 [16816.102089] ? watchdog+0x40/0x40 [16816.102094] ? hrtimer_interrupt+0x128/0x2c0 [16816.102098] ? smp_apic_timer_interrupt+0xa6/0x270 [16816.102105] ? apic_timer_interrupt+0xf/0x20 [16816.102110] [16816.102113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16816.102119] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16816.102127] do_raw_spin_lock+0x177/0x190 [16816.102131] _raw_spin_lock+0x21/0x30 [16816.102136] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16816.102444] process_one_work+0x2c8/0x700 [16816.102452] worker_thread+0x296/0x6e0 [16816.102457] ? rescuer_thread+0x570/0x570 [16816.102460] kthread+0x1d1/0x200 [16816.102465] ? set_kthread_struct+0x70/0x70 [16816.102468] ret_from_fork+0x1f/0x30 [16816.475111] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16816.478218] ? do_wait_intr+0xf0/0xf0 [16816.480790] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16816.483669] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16816.486236] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16816.489404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.492319] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16816.494918] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16816.497052] mdc_intent_lock+0x353/0x7e0 [mdc] [16816.499628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.502189] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16816.505103] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16816.507884] lmv_intent_open+0x338/0x1150 [lmv] [16816.511306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.513896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.517035] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16816.519429] ll_intent_lock+0xc5/0xaa0 [lustre] [16816.522716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.525727] ll_lookup_it+0xe0e/0x2270 [lustre] [16816.528873] ? do_raw_spin_unlock+0x75/0x190 [16816.531242] ? _raw_spin_unlock+0x12/0x30 [16816.533582] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16816.536182] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16816.539310] ll_atomic_open+0x36a/0xd70 [lustre] [16816.542214] lookup_open+0x356/0xb40 [16816.544333] do_last+0x313/0xfc0 [16816.545846] ? nd_jump_root+0xe5/0x160 [16816.548071] ? path_init+0x437/0x520 [16816.549748] path_openat+0xf7/0x500 [16816.551687] ? do_raw_spin_unlock+0x75/0x190 [16816.553723] do_filp_open+0x99/0x140 [16816.555643] ? getname_flags+0x6e/0x330 [16816.557797] ? __check_object_size+0xff/0x256 [16816.560778] ? do_raw_spin_unlock+0x75/0x190 [16816.563193] ? _raw_spin_unlock+0x12/0x30 [16816.565379] do_sys_openat2+0x2b4/0x410 [16816.568203] do_sys_open+0x73/0xa0 [16816.569882] __x64_sys_openat+0x24/0x30 [16816.572582] do_syscall_64+0xc1/0x440 [16816.574358] entry_SYSCALL_64_after_hwframe+0x49/0xae [16816.577282] RIP: 0033:0x7fa749552332 [16816.579635] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16816.589026] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16816.593224] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16816.597047] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16816.600539] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16816.604335] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16816.607900] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16819.682672] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14902s! [16819.687963] Showing busy workqueues and worker pools: [16819.690227] workqueue events: flags=0x0 [16819.695755] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16819.700577] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16819.711095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16819.718695] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16819.725062] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.728532] pending: kfree_rcu_monitor [16819.730898] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16819.737713] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16819.752636] workqueue events_long: flags=0x0 [16819.754920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.757913] pending: lu_objects_flush [obdclass] [16819.761936] workqueue events_unbound: flags=0x2 [16819.764467] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16819.768329] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16819.777154] workqueue events_power_efficient: flags=0x80 [16819.778402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.781624] pending: check_lifetime [16819.786687] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.791620] pending: neigh_periodic_work [16819.794048] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.802309] pending: neigh_periodic_work [16819.809810] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.819142] pending: do_cache_clean [16819.820904] workqueue mm_percpu_wq: flags=0x8 [16819.823454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.826879] pending: vmstat_update [16819.828988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.832552] pending: vmstat_update [16819.834270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16819.837208] pending: vmstat_update [16819.838974] workqueue writeback: flags=0x4a [16819.850124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16819.854650] pending: wb_workfn, wb_workfn [16819.862838] workqueue xprtiod: flags=0xa [16819.864306] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16819.871583] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16819.880345] workqueue ipv6_addrconf: flags=0x40008 [16819.883930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16819.890041] pending: addrconf_verify_work [16819.892680] workqueue knbd0-recv: flags=0x1a [16819.899409] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16819.905207] in-flight: 651:recv_work [16819.909042] workqueue ptlrpc_pinger: flags=0x2 [16819.910921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16819.913729] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16819.916245] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14902s workers=4 idle: 37 31846 28897 [16819.921986] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16809s workers=2 idle: 653 [16820.059064] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16820.062757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16820.108649] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16820.116817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16820.130520] RIP: 0010:native_safe_halt+0x2e/0x50 [16820.133286] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16820.166746] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16820.182557] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16820.195598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16820.204672] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16820.209301] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16820.218606] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16820.226197] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16820.230041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16820.232841] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16820.237419] Call Trace: [16820.239375] [16820.240400] ? show_regs.cold.9+0x22/0x2f [16820.243429] ? watchdog_timer_fn.cold.6+0x62/0x10a [16820.250490] ? __hrtimer_run_queues+0x1c2/0x4d0 [16820.262459] ? watchdog+0x40/0x40 [16820.266649] ? hrtimer_interrupt+0x128/0x2c0 [16820.270650] ? smp_apic_timer_interrupt+0xa6/0x270 [16820.277437] ? apic_timer_interrupt+0xf/0x20 [16820.280395] [16820.281582] ? native_safe_halt+0x2e/0x50 [16820.283797] kvm_wait+0xcd/0xe0 [16820.285804] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16820.289614] do_raw_spin_lock+0x177/0x190 [16820.294893] _raw_spin_lock+0x21/0x30 [16820.301571] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16820.310331] ? do_raw_spin_unlock+0x75/0x190 [16820.317442] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16820.321675] ? do_wait_intr+0xf0/0xf0 [16820.324567] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16820.328621] mdc_close+0x53b/0xd70 [mdc] [16820.331585] lmv_close+0x18d/0x490 [lmv] [16820.338546] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16820.345545] ll_md_real_close+0xa5/0x2a0 [lustre] [16820.352703] ll_md_close+0x274/0x800 [lustre] [16820.361594] ll_file_release+0x128/0xc50 [lustre] [16820.365895] __fput+0x10d/0x440 [16820.368564] ____fput+0x12/0x20 [16820.370521] task_work_run+0xc6/0x110 [16820.372681] exit_to_usermode_loop+0x1dd/0x1f0 [16820.375438] do_syscall_64+0x3d6/0x440 [16820.377638] entry_SYSCALL_64_after_hwframe+0x49/0xae [16820.380403] RIP: 0033:0x7fa749552cd8 [16820.382136] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16820.392363] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16820.396472] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16820.402348] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16820.408784] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16820.412715] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16820.417366] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16844.054057] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16844.057022] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16844.059926] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16844.059950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16844.060029] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16844.060035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16844.060037] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16844.060048] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16844.060051] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16844.060055] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16844.060057] RDX: 0000000000007e0f RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16844.060058] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16844.060060] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16844.060062] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16844.060064] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16844.060066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16844.060067] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16844.060074] Call Trace: [16844.060078] [16844.060083] ? show_regs.cold.9+0x22/0x2f [16844.060090] ? watchdog_timer_fn.cold.6+0x62/0x10a [16844.060094] ? __hrtimer_run_queues+0x1c2/0x4d0 [16844.060097] ? watchdog+0x40/0x40 [16844.060100] ? hrtimer_interrupt+0x128/0x2c0 [16844.060102] ? smp_apic_timer_interrupt+0xa6/0x270 [16844.060108] ? apic_timer_interrupt+0xf/0x20 [16844.060112] [16844.060113] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16844.060116] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16844.060119] do_raw_spin_lock+0x177/0x190 [16844.060122] _raw_spin_lock+0x21/0x30 [16844.060127] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16844.060348] obd_mod_free+0x46/0x140 [ptlrpc] [16844.060467] Modules linked in: lustre(O) [16844.060483] mdc_commit_open+0x91/0xb0 [mdc] [16844.060519] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16844.061608] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16844.061823] after_reply+0xe2c/0x1780 [ptlrpc] [16844.062061] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16844.062302] ? __next_timer_interrupt+0x160/0x160 [16844.062313] ptlrpcd+0xa82/0xe40 [ptlrpc] [16844.062569] ? do_wait_intr+0xf0/0xf0 [16844.062579] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16844.062814] kthread+0x1d1/0x200 [16844.062821] ? set_kthread_struct+0x70/0x70 [16844.062824] ret_from_fork+0x1f/0x30 [16844.067735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16844.067813] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16844.067820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16844.067827] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16844.069024] RIP: 0010:native_safe_halt+0x2e/0x50 [16844.069037] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16844.069042] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16844.069351] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16844.069354] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16844.069356] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16844.069358] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16844.069361] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16844.069364] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16844.069367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16844.069370] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16844.069378] Call Trace: [16844.069384] [16844.069389] ? show_regs.cold.9+0x22/0x2f [16844.069399] ? watchdog_timer_fn.cold.6+0x62/0x10a [16844.069407] ? __hrtimer_run_queues+0x1c2/0x4d0 [16844.069411] ? watchdog+0x40/0x40 [16844.069415] ? hrtimer_interrupt+0x128/0x2c0 [16844.069418] ? smp_apic_timer_interrupt+0xa6/0x270 [16844.069425] ? apic_timer_interrupt+0xf/0x20 [16844.069430] [16844.069432] ? native_safe_halt+0x2e/0x50 [16844.069436] kvm_wait+0xcd/0xe0 [16844.069443] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16844.069451] do_raw_spin_lock+0x177/0x190 [16844.069455] _raw_spin_lock+0x21/0x30 [16844.069459] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16844.073043] process_one_work+0x2c8/0x700 [16844.073057] worker_thread+0x296/0x6e0 [16844.073067] ? rescuer_thread+0x570/0x570 [16844.073072] kthread+0x1d1/0x200 [16844.073077] ? set_kthread_struct+0x70/0x70 [16844.073080] ret_from_fork+0x1f/0x30 [16844.385450] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16844.421792] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16844.428539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16844.436277] RIP: 0010:native_safe_halt+0x2e/0x50 [16844.440977] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16844.460449] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16844.462663] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16844.464901] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [16844.468728] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16844.472110] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16844.475016] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16844.478804] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16844.482582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16844.485447] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16844.488544] Call Trace: [16844.489727] [16844.490617] ? show_regs.cold.9+0x22/0x2f [16844.492473] ? watchdog_timer_fn.cold.6+0x62/0x10a [16844.494542] ? __hrtimer_run_queues+0x1c2/0x4d0 [16844.497005] ? watchdog+0x40/0x40 [16844.499373] ? hrtimer_interrupt+0x128/0x2c0 [16844.501419] ? smp_apic_timer_interrupt+0xa6/0x270 [16844.504513] ? apic_timer_interrupt+0xf/0x20 [16844.506694] [16844.507669] ? native_safe_halt+0x2e/0x50 [16844.509771] kvm_wait+0xcd/0xe0 [16844.511018] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16844.513037] do_raw_spin_lock+0x177/0x190 [16844.514385] _raw_spin_lock+0x21/0x30 [16844.516320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16844.518396] ? do_raw_spin_unlock+0x75/0x190 [16844.519910] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16844.522544] ? do_wait_intr+0xf0/0xf0 [16844.524452] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16844.527132] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16844.529639] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16844.532771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.535360] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16844.538519] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16844.540961] mdc_intent_lock+0x353/0x7e0 [mdc] [16844.543164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.545840] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16844.548984] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16844.553922] lmv_intent_open+0x338/0x1150 [lmv] [16844.557345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.560248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.562781] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16844.564897] ll_intent_lock+0xc5/0xaa0 [lustre] [16844.569384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.572605] ll_lookup_it+0xe0e/0x2270 [lustre] [16844.574280] ? do_raw_spin_unlock+0x75/0x190 [16844.580913] ? _raw_spin_unlock+0x12/0x30 [16844.582924] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16844.585365] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16844.589758] ll_atomic_open+0x36a/0xd70 [lustre] [16844.591955] lookup_open+0x356/0xb40 [16844.593271] do_last+0x313/0xfc0 [16844.594324] ? nd_jump_root+0xe5/0x160 [16844.595711] ? path_init+0x437/0x520 [16844.601188] path_openat+0xf7/0x500 [16844.603968] ? do_raw_spin_unlock+0x75/0x190 [16844.609474] do_filp_open+0x99/0x140 [16844.611828] ? getname_flags+0x6e/0x330 [16844.617467] ? __check_object_size+0xff/0x256 [16844.619982] ? do_raw_spin_unlock+0x75/0x190 [16844.622942] ? _raw_spin_unlock+0x12/0x30 [16844.624474] do_sys_openat2+0x2b4/0x410 [16844.626151] do_sys_open+0x73/0xa0 [16844.627336] __x64_sys_openat+0x24/0x30 [16844.629480] do_syscall_64+0xc1/0x440 [16844.631144] entry_SYSCALL_64_after_hwframe+0x49/0xae [16844.633749] RIP: 0033:0x7fa749552332 [16844.635541] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16844.656636] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16844.660026] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16844.664821] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16844.669499] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16844.681412] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16844.686833] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16848.060015] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16848.068068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16848.112030] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16848.118069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16848.130340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16848.133958] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16848.146153] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16848.150032] RAX: 0000000000007c0a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16848.158052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16848.165140] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16848.170005] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16848.173094] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16848.176924] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16848.180977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16848.184002] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16848.187826] Call Trace: [16848.188937] [16848.190104] ? show_regs.cold.9+0x22/0x2f [16848.192195] ? watchdog_timer_fn.cold.6+0x62/0x10a [16848.194782] ? __hrtimer_run_queues+0x1c2/0x4d0 [16848.196911] ? watchdog+0x40/0x40 [16848.198647] ? hrtimer_interrupt+0x128/0x2c0 [16848.200954] ? smp_apic_timer_interrupt+0xa6/0x270 [16848.202852] ? apic_timer_interrupt+0xf/0x20 [16848.204477] [16848.205393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16848.208023] do_raw_spin_lock+0x177/0x190 [16848.210682] _raw_spin_lock+0x21/0x30 [16848.212958] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16848.221166] ? do_raw_spin_unlock+0x75/0x190 [16848.223883] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16848.231278] ? do_wait_intr+0xf0/0xf0 [16848.240141] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16848.250079] mdc_close+0x53b/0xd70 [mdc] [16848.256272] lmv_close+0x18d/0x490 [lmv] [16848.260642] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16848.268420] ll_md_real_close+0xa5/0x2a0 [lustre] [16848.282262] ll_md_close+0x274/0x800 [lustre] [16848.289905] ll_file_release+0x128/0xc50 [lustre] [16848.299676] __fput+0x10d/0x440 [16848.304985] ____fput+0x12/0x20 [16848.308933] task_work_run+0xc6/0x110 [16848.310308] exit_to_usermode_loop+0x1dd/0x1f0 [16848.311850] do_syscall_64+0x3d6/0x440 [16848.320623] entry_SYSCALL_64_after_hwframe+0x49/0xae [16848.325987] RIP: 0033:0x7fa749552cd8 [16848.331842] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16848.350267] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16848.357996] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16848.367324] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16848.373997] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16848.379653] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16848.390853] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16850.401331] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14933s! [16850.411595] Showing busy workqueues and worker pools: [16850.413482] workqueue events: flags=0x0 [16850.414667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16850.432946] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16850.447192] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16850.458996] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16850.469990] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.473520] pending: kfree_rcu_monitor [16850.475914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16850.478750] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16850.483191] workqueue events_long: flags=0x0 [16850.486095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.489516] pending: lu_objects_flush [obdclass] [16850.493300] workqueue events_unbound: flags=0x2 [16850.498135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16850.505501] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16850.512067] workqueue events_power_efficient: flags=0x80 [16850.515428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.518117] pending: check_lifetime [16850.519470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.522096] pending: neigh_periodic_work [16850.523749] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.526416] pending: neigh_periodic_work [16850.528024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.530348] pending: do_cache_clean [16850.532512] workqueue mm_percpu_wq: flags=0x8 [16850.535456] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.540155] pending: vmstat_update [16850.541809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.544722] pending: vmstat_update [16850.546409] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16850.549636] pending: vmstat_update [16850.550880] workqueue writeback: flags=0x4a [16850.553393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16850.554923] pending: wb_workfn, wb_workfn [16850.555964] workqueue xprtiod: flags=0xa [16850.559644] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16850.563696] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16850.568946] workqueue ipv6_addrconf: flags=0x40008 [16850.572201] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16850.577121] pending: addrconf_verify_work [16850.579736] workqueue knbd0-recv: flags=0x1a [16850.583447] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16850.585682] in-flight: 651:recv_work [16850.587747] workqueue ptlrpc_pinger: flags=0x2 [16850.591076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16850.596771] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16850.601052] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14933s workers=4 idle: 37 31846 28897 [16850.612069] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16840s workers=2 idle: 653 [16872.057006] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [16872.058099] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [16872.061028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [16872.065128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16872.065524] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16872.065532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16872.065535] RIP: 0010:native_safe_halt+0x2e/0x50 [16872.065549] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16872.065553] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16872.065558] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16872.065561] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16872.065563] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16872.065565] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16872.065568] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16872.065571] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16872.065574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16872.065577] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16872.065586] Call Trace: [16872.065592] [16872.065598] ? show_regs.cold.9+0x22/0x2f [16872.065609] ? watchdog_timer_fn.cold.6+0x62/0x10a [16872.065616] ? __hrtimer_run_queues+0x1c2/0x4d0 [16872.065621] ? watchdog+0x40/0x40 [16872.065625] ? hrtimer_interrupt+0x128/0x2c0 [16872.065635] ? smp_apic_timer_interrupt+0xa6/0x270 [16872.065643] ? apic_timer_interrupt+0xf/0x20 [16872.065647] [16872.065650] ? native_safe_halt+0x2e/0x50 [16872.065654] kvm_wait+0xcd/0xe0 [16872.065663] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16872.065671] do_raw_spin_lock+0x177/0x190 [16872.065676] _raw_spin_lock+0x21/0x30 [16872.065682] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16872.066140] obd_mod_free+0x46/0x140 [ptlrpc] [16872.066370] mdc_commit_open+0x91/0xb0 [mdc] [16872.066406] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16872.099144] dm_mod sha512_ssse3 [16872.147617] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16872.163335] sha512_generic [16872.163348] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16872.163355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16872.163362] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16872.170186] after_reply+0xe2c/0x1780 [ptlrpc] [16872.170566] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16872.170780] ? __next_timer_interrupt+0x160/0x160 [16872.170793] ptlrpcd+0xa82/0xe40 [ptlrpc] [16872.171114] ? do_wait_intr+0xf0/0xf0 [16872.171125] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16872.174261] [16872.174270] RIP: 0010:native_safe_halt+0x2e/0x50 [16872.174286] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16872.174291] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16872.174296] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16872.174298] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16872.174300] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16872.174302] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16872.174304] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16872.174307] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16872.174310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16872.174312] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16872.174321] Call Trace: [16872.174326] [16872.174331] ? show_regs.cold.9+0x22/0x2f [16872.174341] ? watchdog_timer_fn.cold.6+0x62/0x10a [16872.174354] ? __hrtimer_run_queues+0x1c2/0x4d0 [16872.174358] ? watchdog+0x40/0x40 [16872.174362] ? hrtimer_interrupt+0x128/0x2c0 [16872.174366] ? smp_apic_timer_interrupt+0xa6/0x270 [16872.174372] ? apic_timer_interrupt+0xf/0x20 [16872.174377] [16872.174380] ? native_safe_halt+0x2e/0x50 [16872.174387] kvm_wait+0xcd/0xe0 [16872.174395] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16872.174405] do_raw_spin_lock+0x177/0x190 [16872.174409] _raw_spin_lock+0x21/0x30 [16872.174413] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16872.174727] process_one_work+0x2c8/0x700 [16872.174734] worker_thread+0x296/0x6e0 [16872.174738] ? rescuer_thread+0x570/0x570 [16872.174741] kthread+0x1d1/0x200 [16872.174746] ? set_kthread_struct+0x70/0x70 [16872.174750] ret_from_fork+0x1f/0x30 [16872.446082] kthread+0x1d1/0x200 [16872.446970] ? set_kthread_struct+0x70/0x70 [16872.459474] ret_from_fork+0x1f/0x30 [16876.055439] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [16876.060281] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16876.069193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 [16876.072829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16876.072916] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16876.072924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16876.072927] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16876.072944] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16876.072948] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16876.072953] RAX: 0000000000001932 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16876.072956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16876.072959] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16876.073420] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16876.073424] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16876.073431] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16876.073434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16876.073437] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16876.073445] Call Trace: [16876.073450] [16876.073461] ? show_regs.cold.9+0x22/0x2f [16876.073473] ? watchdog_timer_fn.cold.6+0x62/0x10a [16876.073481] ? __hrtimer_run_queues+0x1c2/0x4d0 [16876.073489] ? watchdog+0x40/0x40 [16876.073494] ? hrtimer_interrupt+0x128/0x2c0 [16876.073498] ? smp_apic_timer_interrupt+0xa6/0x270 [16876.073511] ? apic_timer_interrupt+0xf/0x20 [16876.073516] [16876.073519] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16876.073529] do_raw_spin_lock+0x177/0x190 [16876.073534] _raw_spin_lock+0x21/0x30 [16876.073539] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16876.073942] ? do_raw_spin_unlock+0x75/0x190 [16876.073953] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16876.074431] ? do_wait_intr+0xf0/0xf0 [16876.074440] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16876.074679] mdc_close+0x53b/0xd70 [mdc] [16876.074723] lmv_close+0x18d/0x490 [lmv] [16876.074755] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16876.074884] ll_md_real_close+0xa5/0x2a0 [lustre] [16876.074985] ll_md_close+0x274/0x800 [lustre] [16876.075085] ll_file_release+0x128/0xc50 [lustre] [16876.119023] pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16876.119055] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16876.119064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16876.119067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16876.140238] __fput+0x10d/0x440 [16876.140258] ____fput+0x12/0x20 [16876.140263] task_work_run+0xc6/0x110 [16876.140270] exit_to_usermode_loop+0x1dd/0x1f0 [16876.140277] do_syscall_64+0x3d6/0x440 [16876.140282] entry_SYSCALL_64_after_hwframe+0x49/0xae [16876.140291] RIP: 0033:0x7fa749552cd8 [16876.140300] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16876.140303] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16876.140310] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16876.140312] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16876.140314] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16876.140316] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16876.140318] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16876.599597] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16876.613959] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16876.623285] RAX: 0000000000007d35 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16876.630521] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16876.641196] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16876.647011] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16876.654410] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16876.661936] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16876.673388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16876.682032] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16876.688928] Call Trace: [16876.691801] [16876.692781] ? show_regs.cold.9+0x22/0x2f [16876.697756] ? watchdog_timer_fn.cold.6+0x62/0x10a [16876.706576] ? __hrtimer_run_queues+0x1c2/0x4d0 [16876.716053] ? watchdog+0x40/0x40 [16876.728196] ? hrtimer_interrupt+0x128/0x2c0 [16876.734538] ? smp_apic_timer_interrupt+0xa6/0x270 [16876.739959] ? apic_timer_interrupt+0xf/0x20 [16876.748499] [16876.750466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16876.757818] do_raw_spin_lock+0x177/0x190 [16876.762767] _raw_spin_lock+0x21/0x30 [16876.769296] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16876.778967] ? do_raw_spin_unlock+0x75/0x190 [16876.786215] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16876.794056] ? do_wait_intr+0xf0/0xf0 [16876.797973] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16876.801633] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16876.813954] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16876.820561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16876.829417] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16876.837997] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16876.848346] mdc_intent_lock+0x353/0x7e0 [mdc] [16876.856998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16876.862629] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16876.874423] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16876.883534] lmv_intent_open+0x338/0x1150 [lmv] [16876.896336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16876.902254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16876.910219] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16876.915402] ll_intent_lock+0xc5/0xaa0 [lustre] [16876.919878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16876.924902] ll_lookup_it+0xe0e/0x2270 [lustre] [16876.927578] ? do_raw_spin_unlock+0x75/0x190 [16876.929024] ? _raw_spin_unlock+0x12/0x30 [16876.931470] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16876.933820] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16876.935720] ll_atomic_open+0x36a/0xd70 [lustre] [16876.938631] lookup_open+0x356/0xb40 [16876.940218] do_last+0x313/0xfc0 [16876.941534] ? nd_jump_root+0xe5/0x160 [16876.944618] ? path_init+0x437/0x520 [16876.946316] path_openat+0xf7/0x500 [16876.947935] ? do_raw_spin_unlock+0x75/0x190 [16876.949423] do_filp_open+0x99/0x140 [16876.958647] ? getname_flags+0x6e/0x330 [16876.964637] ? __check_object_size+0xff/0x256 [16876.974928] ? do_raw_spin_unlock+0x75/0x190 [16876.981163] ? _raw_spin_unlock+0x12/0x30 [16876.985888] do_sys_openat2+0x2b4/0x410 [16876.987117] do_sys_open+0x73/0xa0 [16876.988175] __x64_sys_openat+0x24/0x30 [16877.005816] do_syscall_64+0xc1/0x440 [16877.013910] entry_SYSCALL_64_after_hwframe+0x49/0xae [16877.018533] RIP: 0033:0x7fa749552332 [16877.031523] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16877.071686] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16877.084603] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16877.090150] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16877.107403] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16877.116410] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16877.140474] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16881.123330] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14963s! [16881.139800] Showing busy workqueues and worker pools: [16881.151251] workqueue events: flags=0x0 [16881.154496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16881.161836] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16881.175450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16881.184472] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16881.194587] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.199837] pending: kfree_rcu_monitor [16881.209557] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16881.227443] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16881.239391] workqueue events_long: flags=0x0 [16881.240650] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.248224] pending: lu_objects_flush [obdclass] [16881.256396] workqueue events_unbound: flags=0x2 [16881.262084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16881.267522] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16881.285444] workqueue events_power_efficient: flags=0x80 [16881.288668] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.295450] pending: check_lifetime [16881.297993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.303300] pending: neigh_periodic_work [16881.306607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.309739] pending: neigh_periodic_work [16881.312031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.315930] pending: do_cache_clean [16881.317136] workqueue mm_percpu_wq: flags=0x8 [16881.318848] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.322557] pending: vmstat_update [16881.324330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.332415] pending: vmstat_update [16881.334324] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16881.339815] pending: vmstat_update [16881.342384] workqueue writeback: flags=0x4a [16881.343793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16881.346327] pending: wb_workfn, wb_workfn [16881.348240] workqueue xprtiod: flags=0xa [16881.349810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16881.352393] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16881.355470] workqueue ipv6_addrconf: flags=0x40008 [16881.357411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16881.359751] pending: addrconf_verify_work [16881.361774] workqueue knbd0-recv: flags=0x1a [16881.364365] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16881.367162] in-flight: 651:recv_work [16881.368928] workqueue ptlrpc_pinger: flags=0x2 [16881.370963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16881.373769] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16881.376257] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14964s workers=4 idle: 37 31846 28897 [16881.378932] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16871s workers=2 idle: 653 [16900.060411] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16900.060584] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16900.077943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16900.078033] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16900.078040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16900.078047] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16900.078318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16900.078402] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16900.078408] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16900.078413] RAX: 0000000000001199 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16900.078415] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16900.078418] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16900.078420] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16900.078423] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16900.078426] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16900.078429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16900.078431] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16900.078439] Call Trace: [16900.078444] [16900.078449] ? show_regs.cold.9+0x22/0x2f [16900.078459] ? watchdog_timer_fn.cold.6+0x62/0x10a [16900.078465] ? __hrtimer_run_queues+0x1c2/0x4d0 [16900.078469] ? watchdog+0x40/0x40 [16900.078473] ? hrtimer_interrupt+0x128/0x2c0 [16900.078484] ? smp_apic_timer_interrupt+0xa6/0x270 [16900.078493] ? apic_timer_interrupt+0xf/0x20 [16900.078502] [16900.078505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16900.078514] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16900.078519] do_raw_spin_lock+0x177/0x190 [16900.078524] _raw_spin_lock+0x21/0x30 [16900.078529] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16900.079475] process_one_work+0x2c8/0x700 [16900.079483] worker_thread+0x296/0x6e0 [16900.079488] ? rescuer_thread+0x570/0x570 [16900.079492] kthread+0x1d1/0x200 [16900.079496] ? set_kthread_struct+0x70/0x70 [16900.079505] ret_from_fork+0x1f/0x30 [16900.350005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16900.399132] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16900.415913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16900.433521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16900.441240] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16900.469037] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16900.481258] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16900.494471] RDX: 0000000000006388 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16900.503329] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16900.509810] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16900.515488] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16900.519046] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16900.526369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16900.528235] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16900.536224] Call Trace: [16900.538087] [16900.539250] ? show_regs.cold.9+0x22/0x2f [16900.550947] ? watchdog_timer_fn.cold.6+0x62/0x10a [16900.553639] ? __hrtimer_run_queues+0x1c2/0x4d0 [16900.560057] ? watchdog+0x40/0x40 [16900.562160] ? hrtimer_interrupt+0x128/0x2c0 [16900.566622] ? smp_apic_timer_interrupt+0xa6/0x270 [16900.572458] ? apic_timer_interrupt+0xf/0x20 [16900.581765] [16900.582528] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16900.596718] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16900.598655] do_raw_spin_lock+0x177/0x190 [16900.603262] _raw_spin_lock+0x21/0x30 [16900.604549] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16900.610351] obd_mod_free+0x46/0x140 [ptlrpc] [16900.617879] mdc_commit_open+0x91/0xb0 [mdc] [16900.622020] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16900.633409] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16900.639501] after_reply+0xe2c/0x1780 [ptlrpc] [16900.643322] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16900.649463] ? __next_timer_interrupt+0x160/0x160 [16900.658768] ptlrpcd+0xa82/0xe40 [ptlrpc] [16900.666135] ? do_wait_intr+0xf0/0xf0 [16900.668178] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16900.676279] kthread+0x1d1/0x200 [16900.681393] ? set_kthread_struct+0x70/0x70 [16900.686312] ret_from_fork+0x1f/0x30 [16904.064126] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [16904.075788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16904.128661] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16904.145182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16904.160325] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16904.174781] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16904.193378] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16904.197761] RAX: 0000000000005813 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16904.202499] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16904.220247] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16904.224190] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16904.232641] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16904.240684] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16904.252627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16904.266844] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16904.280092] Call Trace: [16904.281911] [16904.285268] ? show_regs.cold.9+0x22/0x2f [16904.290966] ? watchdog_timer_fn.cold.6+0x62/0x10a [16904.296420] ? __hrtimer_run_queues+0x1c2/0x4d0 [16904.301038] ? watchdog+0x40/0x40 [16904.303287] ? hrtimer_interrupt+0x128/0x2c0 [16904.310587] ? smp_apic_timer_interrupt+0xa6/0x270 [16904.316475] ? apic_timer_interrupt+0xf/0x20 [16904.320192] [16904.329088] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16904.331412] do_raw_spin_lock+0x177/0x190 [16904.333819] _raw_spin_lock+0x21/0x30 [16904.335503] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16904.339239] ? do_raw_spin_unlock+0x75/0x190 [16904.342158] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16904.344753] ? do_wait_intr+0xf0/0xf0 [16904.347251] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16904.349449] mdc_close+0x53b/0xd70 [mdc] [16904.364165] lmv_close+0x18d/0x490 [lmv] [16904.366277] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16904.373105] ll_md_real_close+0xa5/0x2a0 [lustre] [16904.380852] ll_md_close+0x274/0x800 [lustre] [16904.382420] ll_file_release+0x128/0xc50 [lustre] [16904.389568] __fput+0x10d/0x440 [16904.393052] ____fput+0x12/0x20 [16904.402476] task_work_run+0xc6/0x110 [16904.404232] exit_to_usermode_loop+0x1dd/0x1f0 [16904.417017] do_syscall_64+0x3d6/0x440 [16904.418698] entry_SYSCALL_64_after_hwframe+0x49/0xae [16904.432007] RIP: 0033:0x7fa749552cd8 [16904.433442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16904.463819] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16904.468486] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16904.477022] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16904.484149] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16904.493841] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16904.507986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16908.056686] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16908.065894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16908.104500] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16908.110102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16908.114872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16908.117114] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16908.123507] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16908.126066] RAX: 0000000000004b9c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16908.128417] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16908.130846] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16908.133443] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16908.136442] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16908.139050] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16908.142457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16908.144905] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16908.147504] Call Trace: [16908.148742] [16908.149645] ? show_regs.cold.9+0x22/0x2f [16908.150789] ? watchdog_timer_fn.cold.6+0x62/0x10a [16908.152175] ? __hrtimer_run_queues+0x1c2/0x4d0 [16908.153801] ? watchdog+0x40/0x40 [16908.154920] ? hrtimer_interrupt+0x128/0x2c0 [16908.156305] ? smp_apic_timer_interrupt+0xa6/0x270 [16908.157896] ? apic_timer_interrupt+0xf/0x20 [16908.162810] [16908.165038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16908.166993] do_raw_spin_lock+0x177/0x190 [16908.168434] _raw_spin_lock+0x21/0x30 [16908.169884] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16908.172067] ? do_raw_spin_unlock+0x75/0x190 [16908.173790] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16908.175695] ? do_wait_intr+0xf0/0xf0 [16908.177190] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16908.184951] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16908.190693] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16908.194922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16908.200853] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16908.204062] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16908.206775] mdc_intent_lock+0x353/0x7e0 [mdc] [16908.209752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16908.216179] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16908.223775] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16908.227999] lmv_intent_open+0x338/0x1150 [lmv] [16908.232880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16908.234626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16908.241125] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16908.249899] ll_intent_lock+0xc5/0xaa0 [lustre] [16908.256118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16908.259028] ll_lookup_it+0xe0e/0x2270 [lustre] [16908.264800] ? do_raw_spin_unlock+0x75/0x190 [16908.269048] ? _raw_spin_unlock+0x12/0x30 [16908.271599] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16908.275014] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16908.278648] ll_atomic_open+0x36a/0xd70 [lustre] [16908.281946] lookup_open+0x356/0xb40 [16908.284911] do_last+0x313/0xfc0 [16908.289627] ? nd_jump_root+0xe5/0x160 [16908.291348] ? path_init+0x437/0x520 [16908.296043] path_openat+0xf7/0x500 [16908.299583] ? do_raw_spin_unlock+0x75/0x190 [16908.302692] do_filp_open+0x99/0x140 [16908.305085] ? getname_flags+0x6e/0x330 [16908.309795] ? __check_object_size+0xff/0x256 [16908.317860] ? do_raw_spin_unlock+0x75/0x190 [16908.324084] ? _raw_spin_unlock+0x12/0x30 [16908.329770] do_sys_openat2+0x2b4/0x410 [16908.333036] do_sys_open+0x73/0xa0 [16908.334366] __x64_sys_openat+0x24/0x30 [16908.338567] do_syscall_64+0xc1/0x440 [16908.340730] entry_SYSCALL_64_after_hwframe+0x49/0xae [16908.345663] RIP: 0033:0x7fa749552332 [16908.350618] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16908.371494] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16908.378275] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16908.386714] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16908.398112] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16908.406000] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16908.412943] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16911.850913] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14994s! [16911.859658] Showing busy workqueues and worker pools: [16911.865801] workqueue events: flags=0x0 [16911.869881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16911.879234] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16911.894188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16911.904124] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16911.913835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16911.921622] pending: kfree_rcu_monitor [16911.925073] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16911.933825] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16911.942010] workqueue events_long: flags=0x0 [16911.944387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16911.952277] pending: lu_objects_flush [obdclass] [16911.960258] workqueue events_unbound: flags=0x2 [16911.961715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16911.968910] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16911.975583] workqueue events_power_efficient: flags=0x80 [16911.981250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16911.983494] pending: check_lifetime [16911.986781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16911.989952] pending: neigh_periodic_work [16911.993826] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16912.003731] pending: neigh_periodic_work [16912.006708] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16912.011736] pending: do_cache_clean [16912.017208] workqueue mm_percpu_wq: flags=0x8 [16912.020247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16912.027695] pending: vmstat_update [16912.030700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16912.042000] pending: vmstat_update [16912.044948] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16912.058525] pending: vmstat_update [16912.067051] workqueue writeback: flags=0x4a [16912.068427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16912.076286] pending: wb_workfn, wb_workfn [16912.080665] workqueue xprtiod: flags=0xa [16912.082408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16912.086164] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16912.097755] workqueue ipv6_addrconf: flags=0x40008 [16912.105162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16912.118979] pending: addrconf_verify_work [16912.120642] workqueue knbd0-recv: flags=0x1a [16912.128015] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16912.135827] in-flight: 651:recv_work [16912.140055] workqueue ptlrpc_pinger: flags=0x2 [16912.146367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16912.149736] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16912.153137] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14994s workers=4 idle: 37 31846 28897 [16912.157421] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16902s workers=2 idle: 653 [16914.018708] rcu: INFO: rcu_sched self-detected stall on CPU [16914.023373] rcu: 2-...!: (8829117 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [16914.032248] (t=15000329 jiffies g=258701 q=5563) [16914.034476] rcu: rcu_sched kthread starved for 14997124 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [16914.041973] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16914.050207] rcu: RCU grace-period kthread stack dump: [16914.055870] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16914.068071] Call Trace: [16914.070934] __schedule+0x351/0xcb0 [16914.074199] schedule+0xc0/0x180 [16914.076073] schedule_timeout+0xb4/0x190 [16914.078074] ? do_raw_spin_unlock+0x75/0x190 [16914.080908] ? __next_timer_interrupt+0x160/0x160 [16914.082971] ? prepare_to_swait_event+0xf3/0x140 [16914.084616] rcu_gp_kthread+0x8dd/0x15b0 [16914.085860] ? force_qs_rnp+0x2c0/0x2c0 [16914.087972] kthread+0x1d1/0x200 [16914.089790] ? set_kthread_struct+0x70/0x70 [16914.092016] ret_from_fork+0x1f/0x30 [16914.093784] rcu: Stack dump where RCU GP kthread last ran: [16914.096549] Sending NMI from CPU 2 to CPUs 3: [16914.098869] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [16914.101218] NMI backtrace for cpu 2 [16914.111004] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16914.123685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16914.129549] Call Trace: [16914.134217] [16914.135261] ? dump_stack+0xbb/0x10e [16914.136139] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16914.142320] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16914.149757] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16914.153122] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16914.159910] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16914.168246] ? rcu_dump_cpu_stacks+0x11b/0x171 [16914.170790] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16914.174621] ? update_process_times+0x6d/0xc0 [16914.176272] ? tick_sched_handle.isra.18+0x27/0x90 [16914.182972] ? tick_sched_timer+0x53/0xb0 [16914.186856] ? __hrtimer_run_queues+0x1c2/0x4d0 [16914.193856] ? tick_sched_handle.isra.18+0x90/0x90 [16914.198833] ? hrtimer_interrupt+0x128/0x2c0 [16914.201194] ? smp_apic_timer_interrupt+0xa6/0x270 [16914.204186] ? apic_timer_interrupt+0xf/0x20 [16914.206221] [16914.207332] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16914.209304] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16914.211153] ? do_raw_spin_lock+0x177/0x190 [16914.213676] ? _raw_spin_lock+0x21/0x30 [16914.217746] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16914.219877] ? obd_mod_free+0x46/0x140 [ptlrpc] [16914.223671] ? mdc_commit_open+0x91/0xb0 [mdc] [16914.225150] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16914.227483] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16914.229960] ? after_reply+0xe2c/0x1780 [ptlrpc] [16914.231784] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16914.238985] ? __next_timer_interrupt+0x160/0x160 [16914.241577] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16914.244338] ? do_wait_intr+0xf0/0xf0 [16914.246240] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16914.248647] ? kthread+0x1d1/0x200 [16914.250293] ? set_kthread_struct+0x70/0x70 [16914.252676] ? ret_from_fork+0x1f/0x30 [16928.058091] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16928.075951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16928.110104] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16928.115559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16928.127692] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16928.131651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16928.137603] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16928.155962] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16928.162595] RAX: 0000000000005f8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16928.168144] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16928.182168] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16928.188302] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16928.197958] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16928.204137] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16928.215610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16928.221329] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16928.228670] Call Trace: [16928.231637] [16928.234045] ? show_regs.cold.9+0x22/0x2f [16928.237966] ? watchdog_timer_fn.cold.6+0x62/0x10a [16928.240585] ? __hrtimer_run_queues+0x1c2/0x4d0 [16928.244163] ? watchdog+0x40/0x40 [16928.247705] ? hrtimer_interrupt+0x128/0x2c0 [16928.252898] ? smp_apic_timer_interrupt+0xa6/0x270 [16928.259055] ? apic_timer_interrupt+0xf/0x20 [16928.263121] [16928.265106] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16928.271429] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16928.274998] do_raw_spin_lock+0x177/0x190 [16928.278152] _raw_spin_lock+0x21/0x30 [16928.280191] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16928.288055] process_one_work+0x2c8/0x700 [16928.290665] worker_thread+0x296/0x6e0 [16928.292800] ? rescuer_thread+0x570/0x570 [16928.294068] kthread+0x1d1/0x200 [16928.296406] ? set_kthread_struct+0x70/0x70 [16928.299213] ret_from_fork+0x1f/0x30 [16940.054183] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16940.059791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16940.062868] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16940.065275] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16940.065289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16940.065363] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16940.065369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16940.065372] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16940.065387] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16940.066046] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16940.066051] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16940.066054] RDX: 00000000000014e0 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16940.066056] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16940.066059] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16940.066061] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16940.066065] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16940.066068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16940.066070] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16940.066078] Call Trace: [16940.066082] [16940.066087] ? show_regs.cold.9+0x22/0x2f [16940.066096] ? watchdog_timer_fn.cold.6+0x62/0x10a [16940.066102] ? __hrtimer_run_queues+0x1c2/0x4d0 [16940.066105] ? watchdog+0x40/0x40 [16940.066110] ? hrtimer_interrupt+0x128/0x2c0 [16940.066114] ? smp_apic_timer_interrupt+0xa6/0x270 [16940.066122] ? apic_timer_interrupt+0xf/0x20 [16940.066126] [16940.066195] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16940.066200] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16940.066205] do_raw_spin_lock+0x177/0x190 [16940.066210] _raw_spin_lock+0x21/0x30 [16940.066217] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16940.066711] obd_mod_free+0x46/0x140 [ptlrpc] [16940.066938] mdc_commit_open+0x91/0xb0 [mdc] [16940.066975] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16940.068929] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16940.069704] after_reply+0xe2c/0x1780 [ptlrpc] [16940.069909] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16940.070112] ? __next_timer_interrupt+0x160/0x160 [16940.070121] ptlrpcd+0xa82/0xe40 [ptlrpc] [16940.072627] ? do_wait_intr+0xf0/0xf0 [16940.072638] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16940.072848] kthread+0x1d1/0x200 [16940.072854] ? set_kthread_struct+0x70/0x70 [16940.072858] ret_from_fork+0x1f/0x30 [16940.098579] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16940.098594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16940.098597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16940.098611] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16940.098616] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16940.098621] RAX: 000000000000148e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16940.098624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16940.098626] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16940.098628] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16940.098631] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16940.098634] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16940.098637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16940.098639] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16940.098649] Call Trace: [16940.098654] [16940.098660] ? show_regs.cold.9+0x22/0x2f [16940.098669] ? watchdog_timer_fn.cold.6+0x62/0x10a [16940.098675] ? __hrtimer_run_queues+0x1c2/0x4d0 [16940.098679] ? watchdog+0x40/0x40 [16940.098684] ? hrtimer_interrupt+0x128/0x2c0 [16940.098689] ? smp_apic_timer_interrupt+0xa6/0x270 [16940.098697] ? apic_timer_interrupt+0xf/0x20 [16940.098702] [16940.098705] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16940.098713] do_raw_spin_lock+0x177/0x190 [16940.098718] _raw_spin_lock+0x21/0x30 [16940.098723] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16940.101729] ? do_raw_spin_unlock+0x75/0x190 [16940.101739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16940.101955] ? do_wait_intr+0xf0/0xf0 [16940.101965] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16940.102317] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16940.102808] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16940.103218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.104071] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16940.104121] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16940.104153] mdc_intent_lock+0x353/0x7e0 [mdc] [16940.104181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.104629] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16940.105484] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16940.105519] lmv_intent_open+0x338/0x1150 [lmv] [16940.105562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.105659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.105741] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16940.105765] ll_intent_lock+0xc5/0xaa0 [lustre] [16940.105856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.105940] ll_lookup_it+0xe0e/0x2270 [lustre] [16940.106093] ? do_raw_spin_unlock+0x75/0x190 [16940.106113] ? _raw_spin_unlock+0x12/0x30 [16940.106121] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16940.106150] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16940.128975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16940.130077] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16940.130084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16940.130087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16940.130100] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16940.130105] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16940.130109] RAX: 000000000000430c RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16940.130112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16940.130114] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16940.130117] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16940.130119] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16940.130123] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16940.130126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16940.130129] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16940.130242] Call Trace: [16940.130250] [16940.130264] ? show_regs.cold.9+0x22/0x2f [16940.130274] ? watchdog_timer_fn.cold.6+0x62/0x10a [16940.130280] ? __hrtimer_run_queues+0x1c2/0x4d0 [16940.148398] ll_atomic_open+0x36a/0xd70 [lustre] [16940.196747] ? watchdog+0x40/0x40 [16940.196774] ? hrtimer_interrupt+0x128/0x2c0 [16940.196780] ? smp_apic_timer_interrupt+0xa6/0x270 [16940.196793] ? apic_timer_interrupt+0xf/0x20 [16940.196797] [16940.196801] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16940.196815] do_raw_spin_lock+0x177/0x190 [16940.196823] _raw_spin_lock+0x21/0x30 [16940.196827] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16940.197334] ? do_raw_spin_unlock+0x75/0x190 [16940.197341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16940.197736] ? do_wait_intr+0xf0/0xf0 [16940.197745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16940.198724] mdc_close+0x53b/0xd70 [mdc] [16940.198763] lmv_close+0x18d/0x490 [lmv] [16940.198792] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16940.199335] ll_md_real_close+0xa5/0x2a0 [lustre] [16940.199429] ll_md_close+0x274/0x800 [lustre] [16940.199644] ll_file_release+0x128/0xc50 [lustre] [16940.199731] __fput+0x10d/0x440 [16940.199741] ____fput+0x12/0x20 [16940.199746] task_work_run+0xc6/0x110 [16940.199754] exit_to_usermode_loop+0x1dd/0x1f0 [16940.199761] do_syscall_64+0x3d6/0x440 [16940.199766] entry_SYSCALL_64_after_hwframe+0x49/0xae [16940.199775] RIP: 0033:0x7fa749552cd8 [16940.199783] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16940.199788] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16940.199793] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16940.199795] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16940.199797] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16940.199799] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16940.199802] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16940.950746] lookup_open+0x356/0xb40 [16940.960964] do_last+0x313/0xfc0 [16940.967643] ? nd_jump_root+0xe5/0x160 [16940.972861] ? path_init+0x437/0x520 [16940.977591] path_openat+0xf7/0x500 [16940.981905] ? do_raw_spin_unlock+0x75/0x190 [16940.988201] do_filp_open+0x99/0x140 [16940.992808] ? getname_flags+0x6e/0x330 [16940.995914] ? __check_object_size+0xff/0x256 [16941.001180] ? do_raw_spin_unlock+0x75/0x190 [16941.003646] ? _raw_spin_unlock+0x12/0x30 [16941.005891] do_sys_openat2+0x2b4/0x410 [16941.007201] do_sys_open+0x73/0xa0 [16941.008274] __x64_sys_openat+0x24/0x30 [16941.020439] do_syscall_64+0xc1/0x440 [16941.021794] entry_SYSCALL_64_after_hwframe+0x49/0xae [16941.031460] RIP: 0033:0x7fa749552332 [16941.033661] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16941.059940] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16941.062765] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16941.071799] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16941.077497] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16941.085576] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16941.089747] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16942.562459] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15025s! [16942.577672] Showing busy workqueues and worker pools: [16942.583708] workqueue events: flags=0x0 [16942.590319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16942.602031] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16942.618028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16942.630749] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16942.642856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.652516] pending: kfree_rcu_monitor [16942.662484] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16942.674238] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16942.703560] workqueue events_long: flags=0x0 [16942.706773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.729345] pending: lu_objects_flush [obdclass] [16942.738757] workqueue events_unbound: flags=0x2 [16942.740354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16942.750398] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16942.768624] workqueue events_power_efficient: flags=0x80 [16942.784820] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.787152] pending: check_lifetime [16942.796468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.800682] pending: neigh_periodic_work [16942.802714] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.807694] pending: neigh_periodic_work [16942.810135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.815190] pending: do_cache_clean [16942.816603] workqueue mm_percpu_wq: flags=0x8 [16942.819380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.822540] pending: vmstat_update [16942.824680] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.830292] pending: vmstat_update [16942.834597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16942.838898] pending: vmstat_update [16942.840675] workqueue writeback: flags=0x4a [16942.845444] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16942.848988] pending: wb_workfn, wb_workfn [16942.850570] workqueue xprtiod: flags=0xa [16942.853741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16942.857204] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16942.861585] workqueue ipv6_addrconf: flags=0x40008 [16942.863743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16942.867450] pending: addrconf_verify_work [16942.870765] workqueue knbd0-recv: flags=0x1a [16942.874589] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16942.876202] in-flight: 651:recv_work [16942.883380] workqueue ptlrpc_pinger: flags=0x2 [16942.886129] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16942.890469] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16942.899930] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15025s workers=4 idle: 37 31846 28897 [16942.907367] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16932s workers=2 idle: 653 [16956.057429] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16956.066247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16956.121023] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16956.126143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16956.130164] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16956.133295] RIP: 0010:native_safe_halt+0x2e/0x50 [16956.135532] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16956.143764] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16956.146830] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16956.149951] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16956.153165] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16956.156664] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16956.159918] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16956.163083] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16956.166918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16956.169452] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16956.172412] Call Trace: [16956.173987] [16956.175128] ? show_regs.cold.9+0x22/0x2f [16956.176452] ? watchdog_timer_fn.cold.6+0x62/0x10a [16956.178927] ? __hrtimer_run_queues+0x1c2/0x4d0 [16956.180946] ? watchdog+0x40/0x40 [16956.182662] ? hrtimer_interrupt+0x128/0x2c0 [16956.184831] ? smp_apic_timer_interrupt+0xa6/0x270 [16956.187126] ? apic_timer_interrupt+0xf/0x20 [16956.189259] [16956.190321] ? native_safe_halt+0x2e/0x50 [16956.192012] kvm_wait+0xcd/0xe0 [16956.193057] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16956.195227] do_raw_spin_lock+0x177/0x190 [16956.196871] _raw_spin_lock+0x21/0x30 [16956.198125] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16956.200362] process_one_work+0x2c8/0x700 [16956.201862] worker_thread+0x296/0x6e0 [16956.203130] ? rescuer_thread+0x570/0x570 [16956.204672] kthread+0x1d1/0x200 [16956.205954] ? set_kthread_struct+0x70/0x70 [16956.207442] ret_from_fork+0x1f/0x30 [16968.058085] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16968.061831] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16968.064762] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [16968.064784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16968.064860] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16968.064864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16968.064866] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16968.064878] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16968.064881] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16968.064884] RAX: 00000000000003ed RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16968.064886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [16968.064887] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [16968.064888] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [16968.064890] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [16968.064893] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [16968.064895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16968.064896] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [16968.064904] Call Trace: [16968.064907] [16968.064912] ? show_regs.cold.9+0x22/0x2f [16968.064919] ? watchdog_timer_fn.cold.6+0x62/0x10a [16968.064923] ? __hrtimer_run_queues+0x1c2/0x4d0 [16968.064926] ? watchdog+0x40/0x40 [16968.064929] ? hrtimer_interrupt+0x128/0x2c0 [16968.064931] ? smp_apic_timer_interrupt+0xa6/0x270 [16968.064938] ? apic_timer_interrupt+0xf/0x20 [16968.064941] [16968.064943] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16968.064947] do_raw_spin_lock+0x177/0x190 [16968.064956] _raw_spin_lock+0x21/0x30 [16968.064959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16968.066052] ? do_raw_spin_unlock+0x75/0x190 [16968.066058] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16968.066656] ? do_wait_intr+0xf0/0xf0 [16968.066663] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16968.066926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16968.067005] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16968.067011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16968.067014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16968.067023] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16968.067027] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16968.067031] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16968.067033] RDX: 00000000000008c6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16968.067036] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16968.067038] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16968.067041] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16968.067044] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16968.067047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16968.067049] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16968.067057] Call Trace: [16968.067061] [16968.067065] ? show_regs.cold.9+0x22/0x2f [16968.067074] ? watchdog_timer_fn.cold.6+0x62/0x10a [16968.067080] ? __hrtimer_run_queues+0x1c2/0x4d0 [16968.067085] ? watchdog+0x40/0x40 [16968.067089] ? hrtimer_interrupt+0x128/0x2c0 [16968.067093] ? smp_apic_timer_interrupt+0xa6/0x270 [16968.067101] ? apic_timer_interrupt+0xf/0x20 [16968.067106] [16968.067109] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16968.067115] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16968.067120] do_raw_spin_lock+0x177/0x190 [16968.067125] _raw_spin_lock+0x21/0x30 [16968.067130] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16968.067411] obd_mod_free+0x46/0x140 [ptlrpc] [16968.067613] mdc_commit_open+0x91/0xb0 [mdc] [16968.067655] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16968.067873] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16968.068592] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16968.068839] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16968.069066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.069192] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16968.069231] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16968.069262] mdc_intent_lock+0x353/0x7e0 [mdc] [16968.069291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.069391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16968.069614] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16968.069643] lmv_intent_open+0x338/0x1150 [lmv] [16968.069672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.069760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.069855] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16968.069879] ll_intent_lock+0xc5/0xaa0 [lustre] [16968.069969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.070056] ll_lookup_it+0xe0e/0x2270 [lustre] [16968.070141] ? do_raw_spin_unlock+0x75/0x190 [16968.070149] ? _raw_spin_unlock+0x12/0x30 [16968.070155] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16968.070183] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16968.070203] ll_atomic_open+0x36a/0xd70 [lustre] [16968.070289] lookup_open+0x356/0xb40 [16968.070299] do_last+0x313/0xfc0 [16968.070304] ? nd_jump_root+0xe5/0x160 [16968.070309] ? path_init+0x437/0x520 [16968.070314] path_openat+0xf7/0x500 [16968.070318] ? do_raw_spin_unlock+0x75/0x190 [16968.070323] do_filp_open+0x99/0x140 [16968.070328] ? getname_flags+0x6e/0x330 [16968.070333] ? __check_object_size+0xff/0x256 [16968.070339] ? do_raw_spin_unlock+0x75/0x190 [16968.070342] ? _raw_spin_unlock+0x12/0x30 [16968.070347] do_sys_openat2+0x2b4/0x410 [16968.070350] do_sys_open+0x73/0xa0 [16968.070354] __x64_sys_openat+0x24/0x30 [16968.070358] do_syscall_64+0xc1/0x440 [16968.070365] entry_SYSCALL_64_after_hwframe+0x49/0xae [16968.070370] RIP: 0033:0x7fa749552332 [16968.070424] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16968.070429] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16968.070435] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [16968.070438] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [16968.070440] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [16968.070442] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [16968.070445] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [16968.074072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16968.074149] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16968.074225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16968.074228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16968.074242] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16968.074247] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16968.074252] RAX: 0000000000000a8e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16968.074254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16968.074257] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16968.074259] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16968.074262] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16968.074266] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16968.074269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16968.074272] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16968.074280] Call Trace: [16968.074285] [16968.074292] ? show_regs.cold.9+0x22/0x2f [16968.074302] ? watchdog_timer_fn.cold.6+0x62/0x10a [16968.074309] ? __hrtimer_run_queues+0x1c2/0x4d0 [16968.074313] ? watchdog+0x40/0x40 [16968.074318] ? hrtimer_interrupt+0x128/0x2c0 [16968.074323] ? smp_apic_timer_interrupt+0xa6/0x270 [16968.074332] ? apic_timer_interrupt+0xf/0x20 [16968.074338] [16968.074341] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16968.074349] do_raw_spin_lock+0x177/0x190 [16968.074354] _raw_spin_lock+0x21/0x30 [16968.074699] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16968.075160] ? do_raw_spin_unlock+0x75/0x190 [16968.075169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16968.075622] ? do_wait_intr+0xf0/0xf0 [16968.075630] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16968.075857] mdc_close+0x53b/0xd70 [mdc] [16968.084815] after_reply+0xe2c/0x1780 [ptlrpc] [16968.106818] lmv_close+0x18d/0x490 [lmv] [16968.106861] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16968.107069] ll_md_real_close+0xa5/0x2a0 [lustre] [16968.107203] ll_md_close+0x274/0x800 [lustre] [16968.107299] ll_file_release+0x128/0xc50 [lustre] [16968.107432] __fput+0x10d/0x440 [16968.107443] ____fput+0x12/0x20 [16968.107447] task_work_run+0xc6/0x110 [16968.107457] exit_to_usermode_loop+0x1dd/0x1f0 [16968.107464] do_syscall_64+0x3d6/0x440 [16968.107469] entry_SYSCALL_64_after_hwframe+0x49/0xae [16968.107477] RIP: 0033:0x7fa749552cd8 [16968.107536] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16968.107541] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16968.107547] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16968.107549] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16968.107551] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16968.107554] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16968.107556] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16968.637217] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16968.640080] ? __next_timer_interrupt+0x160/0x160 [16968.642119] ptlrpcd+0xa82/0xe40 [ptlrpc] [16968.644538] ? do_wait_intr+0xf0/0xf0 [16968.646298] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16968.649420] kthread+0x1d1/0x200 [16968.651124] ? set_kthread_struct+0x70/0x70 [16968.653382] ret_from_fork+0x1f/0x30 [16973.282787] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15056s! [16973.290270] Showing busy workqueues and worker pools: [16973.293691] workqueue events: flags=0x0 [16973.301091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16973.315180] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [16973.322704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16973.330050] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [16973.337945] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.340265] pending: kfree_rcu_monitor [16973.346129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16973.349923] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [16973.356057] workqueue events_long: flags=0x0 [16973.358590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.362167] pending: lu_objects_flush [obdclass] [16973.365709] workqueue events_unbound: flags=0x2 [16973.368101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [16973.372064] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16973.376788] workqueue events_power_efficient: flags=0x80 [16973.378917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.381998] pending: check_lifetime [16973.384619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.388092] pending: neigh_periodic_work [16973.390921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.397598] pending: neigh_periodic_work [16973.404903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.408725] pending: do_cache_clean [16973.411169] workqueue mm_percpu_wq: flags=0x8 [16973.414034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.426365] pending: vmstat_update [16973.435431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.441437] pending: vmstat_update [16973.444682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16973.449084] pending: vmstat_update [16973.451160] workqueue writeback: flags=0x4a [16973.457102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16973.461344] pending: wb_workfn, wb_workfn [16973.465971] workqueue xprtiod: flags=0xa [16973.468206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16973.473914] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [16973.490707] workqueue ipv6_addrconf: flags=0x40008 [16973.496186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16973.504127] pending: addrconf_verify_work [16973.511229] workqueue knbd0-recv: flags=0x1a [16973.521025] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16973.524310] in-flight: 651:recv_work [16973.526393] workqueue ptlrpc_pinger: flags=0x2 [16973.529169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16973.539235] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [16973.545927] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15056s workers=4 idle: 37 31846 28897 [16973.550532] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16963s workers=2 idle: 653 [16984.057022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [16984.060557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16984.085576] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16984.098853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16984.106615] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16984.110841] RIP: 0010:native_safe_halt+0x2e/0x50 [16984.112875] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16984.122021] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16984.128514] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16984.131503] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [16984.134179] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [16984.138335] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [16984.142521] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [16984.146907] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [16984.150932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16984.153871] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [16984.167120] Call Trace: [16984.169034] [16984.174333] ? show_regs.cold.9+0x22/0x2f [16984.184353] ? watchdog_timer_fn.cold.6+0x62/0x10a [16984.188195] ? __hrtimer_run_queues+0x1c2/0x4d0 [16984.192591] ? watchdog+0x40/0x40 [16984.197044] ? hrtimer_interrupt+0x128/0x2c0 [16984.199622] ? smp_apic_timer_interrupt+0xa6/0x270 [16984.208638] ? apic_timer_interrupt+0xf/0x20 [16984.214332] [16984.215676] ? native_safe_halt+0x2e/0x50 [16984.220978] kvm_wait+0xcd/0xe0 [16984.222703] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16984.228098] do_raw_spin_lock+0x177/0x190 [16984.231287] _raw_spin_lock+0x21/0x30 [16984.233367] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16984.236350] process_one_work+0x2c8/0x700 [16984.238054] worker_thread+0x296/0x6e0 [16984.243515] ? rescuer_thread+0x570/0x570 [16984.244704] kthread+0x1d1/0x200 [16984.249663] ? set_kthread_struct+0x70/0x70 [16984.251120] ret_from_fork+0x1f/0x30 [16996.059778] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [16996.071006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16996.074721] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [16996.117263] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16996.117287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16996.117290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16996.117304] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16996.117308] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16996.117314] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [16996.117316] RDX: 0000000000006be7 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [16996.117319] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [16996.117321] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [16996.117323] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16996.117326] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [16996.117329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16996.117331] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [16996.117340] Call Trace: [16996.117346] [16996.117352] ? show_regs.cold.9+0x22/0x2f [16996.117371] ? watchdog_timer_fn.cold.6+0x62/0x10a [16996.117377] ? __hrtimer_run_queues+0x1c2/0x4d0 [16996.117381] ? watchdog+0x40/0x40 [16996.117384] ? hrtimer_interrupt+0x128/0x2c0 [16996.117388] ? smp_apic_timer_interrupt+0xa6/0x270 [16996.117396] ? apic_timer_interrupt+0xf/0x20 [16996.117401] [16996.117404] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16996.117409] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16996.117413] do_raw_spin_lock+0x177/0x190 [16996.117417] _raw_spin_lock+0x21/0x30 [16996.117423] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16996.118079] obd_mod_free+0x46/0x140 [ptlrpc] [16996.118301] mdc_commit_open+0x91/0xb0 [mdc] [16996.118340] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16996.130087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16996.130171] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16996.130179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16996.130183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16996.130198] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16996.130202] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16996.130206] RAX: 0000000000004189 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [16996.130208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [16996.130211] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [16996.130213] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [16996.130215] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [16996.130219] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [16996.130221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16996.130223] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [16996.130232] Call Trace: [16996.130236] [16996.130241] ? show_regs.cold.9+0x22/0x2f [16996.130250] ? watchdog_timer_fn.cold.6+0x62/0x10a [16996.130256] ? __hrtimer_run_queues+0x1c2/0x4d0 [16996.130260] ? watchdog+0x40/0x40 [16996.130264] ? hrtimer_interrupt+0x128/0x2c0 [16996.130268] ? smp_apic_timer_interrupt+0xa6/0x270 [16996.130288] ? apic_timer_interrupt+0xf/0x20 [16996.130294] [16996.130298] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16996.130305] do_raw_spin_lock+0x177/0x190 [16996.138254] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16996.138535] after_reply+0xe2c/0x1780 [ptlrpc] [16996.138747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16996.138951] ? __next_timer_interrupt+0x160/0x160 [16996.138961] ptlrpcd+0xa82/0xe40 [ptlrpc] [16996.139168] ? do_wait_intr+0xf0/0xf0 [16996.139177] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16996.145011] _raw_spin_lock+0x21/0x30 [16996.145033] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16996.165195] kthread+0x1d1/0x200 [16996.165216] ? set_kthread_struct+0x70/0x70 [16996.165221] ret_from_fork+0x1f/0x30 [16996.670826] ? do_raw_spin_unlock+0x75/0x190 [16996.681984] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16996.687936] ? do_wait_intr+0xf0/0xf0 [16996.692065] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16996.700196] mdc_close+0x53b/0xd70 [mdc] [16996.701740] lmv_close+0x18d/0x490 [lmv] [16996.707234] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16996.712557] ll_md_real_close+0xa5/0x2a0 [lustre] [16996.715046] ll_md_close+0x274/0x800 [lustre] [16996.717389] ll_file_release+0x128/0xc50 [lustre] [16996.719732] __fput+0x10d/0x440 [16996.721276] ____fput+0x12/0x20 [16996.723207] task_work_run+0xc6/0x110 [16996.725173] exit_to_usermode_loop+0x1dd/0x1f0 [16996.727378] do_syscall_64+0x3d6/0x440 [16996.729148] entry_SYSCALL_64_after_hwframe+0x49/0xae [16996.731766] RIP: 0033:0x7fa749552cd8 [16996.733211] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16996.742878] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16996.747081] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [16996.750925] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [16996.757838] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [16996.769577] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [16996.784471] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17000.054832] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17000.066757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17000.103765] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17000.115306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17000.130379] RIP: 0010:native_safe_halt+0x2e/0x50 [17000.140596] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17000.154341] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17000.158813] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17000.164100] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17000.169261] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17000.176016] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17000.182428] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17000.185664] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17000.193119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17000.198529] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17000.209089] Call Trace: [17000.216114] [17000.217136] ? show_regs.cold.9+0x22/0x2f [17000.221124] ? watchdog_timer_fn.cold.6+0x62/0x10a [17000.223381] ? __hrtimer_run_queues+0x1c2/0x4d0 [17000.225691] ? watchdog+0x40/0x40 [17000.231603] ? hrtimer_interrupt+0x128/0x2c0 [17000.234197] ? smp_apic_timer_interrupt+0xa6/0x270 [17000.237282] ? apic_timer_interrupt+0xf/0x20 [17000.244817] [17000.247212] ? native_safe_halt+0x2e/0x50 [17000.250130] kvm_wait+0xcd/0xe0 [17000.251687] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17000.254691] do_raw_spin_lock+0x177/0x190 [17000.257753] _raw_spin_lock+0x21/0x30 [17000.261479] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17000.265446] ? do_raw_spin_unlock+0x75/0x190 [17000.266920] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17000.278889] ? do_wait_intr+0xf0/0xf0 [17000.280182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17000.290921] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17000.293400] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17000.295522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17000.299822] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17000.302115] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17000.307393] mdc_intent_lock+0x353/0x7e0 [mdc] [17000.311814] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17000.315663] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17000.319412] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17000.326236] lmv_intent_open+0x338/0x1150 [lmv] [17000.327848] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17000.331341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17000.338596] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17000.340118] ll_intent_lock+0xc5/0xaa0 [lustre] [17000.350672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17000.355090] ll_lookup_it+0xe0e/0x2270 [lustre] [17000.359642] ? do_raw_spin_unlock+0x75/0x190 [17000.363290] ? _raw_spin_unlock+0x12/0x30 [17000.364721] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17000.367082] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17000.373517] ll_atomic_open+0x36a/0xd70 [lustre] [17000.378146] lookup_open+0x356/0xb40 [17000.383325] do_last+0x313/0xfc0 [17000.384616] ? nd_jump_root+0xe5/0x160 [17000.388197] ? path_init+0x437/0x520 [17000.392340] path_openat+0xf7/0x500 [17000.397272] ? do_raw_spin_unlock+0x75/0x190 [17000.401611] do_filp_open+0x99/0x140 [17000.402606] ? getname_flags+0x6e/0x330 [17000.403865] ? __check_object_size+0xff/0x256 [17000.413780] ? do_raw_spin_unlock+0x75/0x190 [17000.418430] ? _raw_spin_unlock+0x12/0x30 [17000.422687] do_sys_openat2+0x2b4/0x410 [17000.423944] do_sys_open+0x73/0xa0 [17000.427175] __x64_sys_openat+0x24/0x30 [17000.431877] do_syscall_64+0xc1/0x440 [17000.437248] entry_SYSCALL_64_after_hwframe+0x49/0xae [17000.442741] RIP: 0033:0x7fa749552332 [17000.446191] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17000.465050] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17000.474330] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17000.481864] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17000.490848] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17000.494643] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17000.498601] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17004.002171] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15086s! [17004.012394] Showing busy workqueues and worker pools: [17004.028575] workqueue events: flags=0x0 [17004.046338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17004.053289] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17004.064450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17004.069818] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17004.088569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.101617] pending: kfree_rcu_monitor [17004.112662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17004.120095] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17004.132153] workqueue events_long: flags=0x0 [17004.135558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.151721] pending: lu_objects_flush [obdclass] [17004.160843] workqueue events_unbound: flags=0x2 [17004.170821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17004.178521] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17004.187713] workqueue events_power_efficient: flags=0x80 [17004.199280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.211148] pending: check_lifetime [17004.216420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.220307] pending: neigh_periodic_work [17004.225758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.234600] pending: neigh_periodic_work [17004.236596] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.239717] pending: do_cache_clean [17004.241574] workqueue mm_percpu_wq: flags=0x8 [17004.243641] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.246204] pending: vmstat_update [17004.248119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.249946] pending: vmstat_update [17004.251601] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17004.259181] pending: vmstat_update [17004.263570] workqueue writeback: flags=0x4a [17004.274302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17004.282117] pending: wb_workfn, wb_workfn [17004.288284] workqueue xprtiod: flags=0xa [17004.290155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17004.296962] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17004.304819] workqueue ipv6_addrconf: flags=0x40008 [17004.319488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17004.329821] pending: addrconf_verify_work [17004.334931] workqueue knbd0-recv: flags=0x1a [17004.350261] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17004.357408] in-flight: 651:recv_work [17004.364223] workqueue ptlrpc_pinger: flags=0x2 [17004.368151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17004.371464] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17004.373934] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15087s workers=4 idle: 37 31846 28897 [17004.381391] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16994s workers=2 idle: 653 [17012.057089] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17012.059630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17012.086703] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17012.092380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17012.098183] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17012.102831] RIP: 0010:native_safe_halt+0x2e/0x50 [17012.105307] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17012.115966] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17012.120165] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17012.123626] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17012.127029] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17012.130276] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17012.133147] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17012.136648] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17012.140589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17012.142918] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17012.146435] Call Trace: [17012.147509] [17012.148691] ? show_regs.cold.9+0x22/0x2f [17012.150558] ? watchdog_timer_fn.cold.6+0x62/0x10a [17012.153573] ? __hrtimer_run_queues+0x1c2/0x4d0 [17012.155588] ? watchdog+0x40/0x40 [17012.157318] ? hrtimer_interrupt+0x128/0x2c0 [17012.160531] ? smp_apic_timer_interrupt+0xa6/0x270 [17012.164649] ? apic_timer_interrupt+0xf/0x20 [17012.167458] [17012.168147] ? native_safe_halt+0x2e/0x50 [17012.171266] kvm_wait+0xcd/0xe0 [17012.172358] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17012.173777] do_raw_spin_lock+0x177/0x190 [17012.179140] _raw_spin_lock+0x21/0x30 [17012.184747] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17012.187347] process_one_work+0x2c8/0x700 [17012.189565] worker_thread+0x296/0x6e0 [17012.192166] ? rescuer_thread+0x570/0x570 [17012.196080] kthread+0x1d1/0x200 [17012.197670] ? set_kthread_struct+0x70/0x70 [17012.199928] ret_from_fork+0x1f/0x30 [17024.058830] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17024.059247] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17024.063724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17024.063816] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17024.063824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17024.063827] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17024.063841] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17024.063845] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17024.063849] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17024.063852] RDX: 00000000000068d6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17024.063854] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17024.063856] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17024.063859] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17024.063861] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17024.063864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17024.063866] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17024.063875] Call Trace: [17024.063880] [17024.063886] ? show_regs.cold.9+0x22/0x2f [17024.063895] ? watchdog_timer_fn.cold.6+0x62/0x10a [17024.063901] ? __hrtimer_run_queues+0x1c2/0x4d0 [17024.063906] ? watchdog+0x40/0x40 [17024.063911] ? hrtimer_interrupt+0x128/0x2c0 [17024.063915] ? smp_apic_timer_interrupt+0xa6/0x270 [17024.063922] ? apic_timer_interrupt+0xf/0x20 [17024.063927] [17024.063936] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17024.063942] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17024.063947] do_raw_spin_lock+0x177/0x190 [17024.063951] _raw_spin_lock+0x21/0x30 [17024.063957] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17024.064232] obd_mod_free+0x46/0x140 [ptlrpc] [17024.064770] mdc_commit_open+0x91/0xb0 [mdc] [17024.064827] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17024.065032] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17024.065229] after_reply+0xe2c/0x1780 [ptlrpc] [17024.076111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17024.076187] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17024.076195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17024.076197] RIP: 0010:native_safe_halt+0x2e/0x50 [17024.076210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17024.076215] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17024.076220] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17024.076222] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17024.076224] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17024.076226] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17024.076228] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17024.076231] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17024.076233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17024.076236] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17024.076245] Call Trace: [17024.076251] [17024.076256] ? show_regs.cold.9+0x22/0x2f [17024.076265] ? watchdog_timer_fn.cold.6+0x62/0x10a [17024.076271] ? __hrtimer_run_queues+0x1c2/0x4d0 [17024.076275] ? watchdog+0x40/0x40 [17024.076279] ? hrtimer_interrupt+0x128/0x2c0 [17024.076284] ? smp_apic_timer_interrupt+0xa6/0x270 [17024.076291] ? apic_timer_interrupt+0xf/0x20 [17024.076295] [17024.076305] ? native_safe_halt+0x2e/0x50 [17024.076309] kvm_wait+0xcd/0xe0 [17024.076316] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17024.076324] do_raw_spin_lock+0x177/0x190 [17024.076328] _raw_spin_lock+0x21/0x30 [17024.076332] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17024.076667] ? do_raw_spin_unlock+0x75/0x190 [17024.076999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17024.077304] ? do_wait_intr+0xf0/0xf0 [17024.077312] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17024.077582] mdc_close+0x53b/0xd70 [mdc] [17024.077629] lmv_close+0x18d/0x490 [lmv] [17024.077660] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17024.113995] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17024.114272] ? __next_timer_interrupt+0x160/0x160 [17024.114284] ptlrpcd+0xa82/0xe40 [ptlrpc] [17024.114710] ? do_wait_intr+0xf0/0xf0 [17024.114720] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17024.121087] ll_md_real_close+0xa5/0x2a0 [lustre] [17024.121357] ll_md_close+0x274/0x800 [lustre] [17024.121467] ll_file_release+0x128/0xc50 [lustre] [17024.121609] __fput+0x10d/0x440 [17024.121620] ____fput+0x12/0x20 [17024.121625] task_work_run+0xc6/0x110 [17024.121632] exit_to_usermode_loop+0x1dd/0x1f0 [17024.121639] do_syscall_64+0x3d6/0x440 [17024.121644] entry_SYSCALL_64_after_hwframe+0x49/0xae [17024.121653] RIP: 0033:0x7fa749552cd8 [17024.121661] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17024.121665] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17024.121671] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17024.121673] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17024.121675] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17024.121678] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17024.121680] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17024.490646] kthread+0x1d1/0x200 [17024.491909] ? set_kthread_struct+0x70/0x70 [17024.496207] ret_from_fork+0x1f/0x30 [17032.054040] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [17032.056401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17032.075244] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17032.078544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17032.081296] RIP: 0010:native_safe_halt+0x2e/0x50 [17032.082795] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17032.089088] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17032.091591] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17032.094115] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [17032.096613] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17032.099023] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17032.105516] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17032.111889] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17032.117009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17032.120437] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17032.123023] Call Trace: [17032.123739] [17032.124283] ? show_regs.cold.9+0x22/0x2f [17032.125732] ? watchdog_timer_fn.cold.6+0x62/0x10a [17032.128143] ? __hrtimer_run_queues+0x1c2/0x4d0 [17032.130993] ? watchdog+0x40/0x40 [17032.132599] ? hrtimer_interrupt+0x128/0x2c0 [17032.134821] ? smp_apic_timer_interrupt+0xa6/0x270 [17032.138428] ? apic_timer_interrupt+0xf/0x20 [17032.140112] [17032.142958] ? native_safe_halt+0x2e/0x50 [17032.145385] kvm_wait+0xcd/0xe0 [17032.148006] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17032.150362] do_raw_spin_lock+0x177/0x190 [17032.153070] _raw_spin_lock+0x21/0x30 [17032.155098] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17032.157030] ? do_raw_spin_unlock+0x75/0x190 [17032.158991] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17032.162659] ? do_wait_intr+0xf0/0xf0 [17032.164950] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17032.167987] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17032.170105] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17032.173338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17032.177707] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17032.179209] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17032.180525] mdc_intent_lock+0x353/0x7e0 [mdc] [17032.182077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17032.183692] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17032.185586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17032.187831] lmv_intent_open+0x338/0x1150 [lmv] [17032.190096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17032.192464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17032.195601] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17032.198186] ll_intent_lock+0xc5/0xaa0 [lustre] [17032.201309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17032.203372] ll_lookup_it+0xe0e/0x2270 [lustre] [17032.204978] ? do_raw_spin_unlock+0x75/0x190 [17032.206312] ? _raw_spin_unlock+0x12/0x30 [17032.207594] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17032.209241] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17032.210815] ll_atomic_open+0x36a/0xd70 [lustre] [17032.212419] lookup_open+0x356/0xb40 [17032.213502] do_last+0x313/0xfc0 [17032.214485] ? nd_jump_root+0xe5/0x160 [17032.215793] ? path_init+0x437/0x520 [17032.216894] path_openat+0xf7/0x500 [17032.218032] ? do_raw_spin_unlock+0x75/0x190 [17032.219343] do_filp_open+0x99/0x140 [17032.220507] ? getname_flags+0x6e/0x330 [17032.221846] ? __check_object_size+0xff/0x256 [17032.223246] ? do_raw_spin_unlock+0x75/0x190 [17032.224557] ? _raw_spin_unlock+0x12/0x30 [17032.225753] do_sys_openat2+0x2b4/0x410 [17032.227027] do_sys_open+0x73/0xa0 [17032.228084] __x64_sys_openat+0x24/0x30 [17032.229222] do_syscall_64+0xc1/0x440 [17032.230407] entry_SYSCALL_64_after_hwframe+0x49/0xae [17032.231954] RIP: 0033:0x7fa749552332 [17032.233117] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17032.238969] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17032.241537] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17032.243633] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17032.246081] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17032.248529] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17032.250842] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17034.724645] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15117s! [17034.730304] Showing busy workqueues and worker pools: [17034.733404] workqueue events: flags=0x0 [17034.737718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17034.741714] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17034.751258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17034.754554] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17034.758891] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.761539] pending: kfree_rcu_monitor [17034.763045] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17034.765941] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17034.770018] workqueue events_long: flags=0x0 [17034.773046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.775254] pending: lu_objects_flush [obdclass] [17034.777289] workqueue events_unbound: flags=0x2 [17034.779043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17034.781064] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17034.791116] workqueue events_power_efficient: flags=0x80 [17034.792494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.799614] pending: check_lifetime [17034.804932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.808709] pending: neigh_periodic_work [17034.812516] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.819864] pending: neigh_periodic_work [17034.822942] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.825904] pending: do_cache_clean [17034.827421] workqueue mm_percpu_wq: flags=0x8 [17034.829056] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.831664] pending: vmstat_update [17034.832972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.835189] pending: vmstat_update [17034.837189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17034.839708] pending: vmstat_update [17034.841121] workqueue writeback: flags=0x4a [17034.842561] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17034.844714] pending: wb_workfn, wb_workfn [17034.846136] workqueue xprtiod: flags=0xa [17034.847305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17034.850005] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17034.853400] workqueue ipv6_addrconf: flags=0x40008 [17034.854979] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17034.857352] pending: addrconf_verify_work [17034.863303] workqueue knbd0-recv: flags=0x1a [17034.865107] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17034.872267] in-flight: 651:recv_work [17034.874838] workqueue ptlrpc_pinger: flags=0x2 [17034.876097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17034.879900] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17034.883948] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15117s workers=4 idle: 37 31846 28897 [17034.896639] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17024s workers=2 idle: 653 [17040.057527] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17040.063413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17040.096194] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17040.110407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17040.119428] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17040.122675] RIP: 0010:native_safe_halt+0x2e/0x50 [17040.126207] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17040.142119] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17040.146583] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17040.150587] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17040.156061] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17040.171793] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17040.177044] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17040.187908] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17040.191629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17040.193866] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17040.198815] Call Trace: [17040.200531] [17040.201470] ? show_regs.cold.9+0x22/0x2f [17040.205517] ? watchdog_timer_fn.cold.6+0x62/0x10a [17040.210901] ? __hrtimer_run_queues+0x1c2/0x4d0 [17040.214008] ? watchdog+0x40/0x40 [17040.217301] ? hrtimer_interrupt+0x128/0x2c0 [17040.219630] ? smp_apic_timer_interrupt+0xa6/0x270 [17040.224830] ? apic_timer_interrupt+0xf/0x20 [17040.229665] [17040.230645] ? native_safe_halt+0x2e/0x50 [17040.231981] kvm_wait+0xcd/0xe0 [17040.236757] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17040.240623] do_raw_spin_lock+0x177/0x190 [17040.243468] _raw_spin_lock+0x21/0x30 [17040.247617] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17040.257378] process_one_work+0x2c8/0x700 [17040.262294] worker_thread+0x296/0x6e0 [17040.263469] ? rescuer_thread+0x570/0x570 [17040.264714] kthread+0x1d1/0x200 [17040.266746] ? set_kthread_struct+0x70/0x70 [17040.270322] ret_from_fork+0x1f/0x30 [17052.058223] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17052.059073] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17052.063494] Modules linked in: [17052.066864] Modules linked in: [17052.066876] lustre(O) [17052.068224] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17052.068314] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17052.068322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17052.068325] RIP: 0010:native_safe_halt+0x2e/0x50 [17052.068338] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17052.068342] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17052.068347] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17052.068349] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17052.068352] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17052.068354] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17052.068357] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17052.068361] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17052.068364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17052.068367] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17052.068374] Call Trace: [17052.068379] [17052.068384] ? show_regs.cold.9+0x22/0x2f [17052.068393] ? watchdog_timer_fn.cold.6+0x62/0x10a [17052.068399] ? __hrtimer_run_queues+0x1c2/0x4d0 [17052.068404] ? watchdog+0x40/0x40 [17052.068408] ? hrtimer_interrupt+0x128/0x2c0 [17052.068413] ? smp_apic_timer_interrupt+0xa6/0x270 [17052.068420] ? apic_timer_interrupt+0xf/0x20 [17052.068424] [17052.068427] ? native_safe_halt+0x2e/0x50 [17052.068431] kvm_wait+0xcd/0xe0 [17052.068437] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17052.068444] do_raw_spin_lock+0x177/0x190 [17052.068449] _raw_spin_lock+0x21/0x30 [17052.068453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17052.071002] osp(O) [17052.071906] ? do_raw_spin_unlock+0x75/0x190 [17052.071921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17052.093752] ofd(O) [17052.098168] ? do_wait_intr+0xf0/0xf0 [17052.098189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17052.101902] lod(O) [17052.103785] mdc_close+0x53b/0xd70 [mdc] [17052.111710] mdt(O) [17052.115090] lmv_close+0x18d/0x490 [lmv] [17052.118215] mdd(O) [17052.122249] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17052.124921] mgs(O) [17052.129010] ll_md_real_close+0xa5/0x2a0 [lustre] [17052.131994] lquota(O) [17052.135786] ll_md_close+0x274/0x800 [lustre] [17052.138811] lfsck(O) [17052.141658] ll_file_release+0x128/0xc50 [lustre] [17052.141773] __fput+0x10d/0x440 [17052.141783] ____fput+0x12/0x20 [17052.141789] task_work_run+0xc6/0x110 [17052.141796] exit_to_usermode_loop+0x1dd/0x1f0 [17052.141804] do_syscall_64+0x3d6/0x440 [17052.141808] entry_SYSCALL_64_after_hwframe+0x49/0xae [17052.141816] RIP: 0033:0x7fa749552cd8 [17052.141824] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17052.141827] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17052.141832] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17052.141835] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17052.141837] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17052.141839] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17052.141841] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17052.236875] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17052.252786] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17052.258244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17052.261987] RIP: 0010:native_safe_halt+0x2e/0x50 [17052.263688] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17052.270373] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17052.273920] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17052.276890] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17052.279691] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17052.282710] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17052.286112] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17052.289541] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17052.293972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17052.296461] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17052.300041] Call Trace: [17052.301302] [17052.302291] ? show_regs.cold.9+0x22/0x2f [17052.304354] ? watchdog_timer_fn.cold.6+0x62/0x10a [17052.306876] ? __hrtimer_run_queues+0x1c2/0x4d0 [17052.309300] ? watchdog+0x40/0x40 [17052.311458] ? hrtimer_interrupt+0x128/0x2c0 [17052.314360] ? smp_apic_timer_interrupt+0xa6/0x270 [17052.316717] ? apic_timer_interrupt+0xf/0x20 [17052.318466] [17052.319348] ? native_safe_halt+0x2e/0x50 [17052.321483] kvm_wait+0xcd/0xe0 [17052.323072] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17052.325355] do_raw_spin_lock+0x177/0x190 [17052.327272] _raw_spin_lock+0x21/0x30 [17052.328954] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17052.330926] obd_mod_free+0x46/0x140 [ptlrpc] [17052.333262] mdc_commit_open+0x91/0xb0 [mdc] [17052.335866] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17052.338689] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17052.341579] after_reply+0xe2c/0x1780 [ptlrpc] [17052.345002] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17052.347942] ? __next_timer_interrupt+0x160/0x160 [17052.349907] ptlrpcd+0xa82/0xe40 [ptlrpc] [17052.351836] ? do_wait_intr+0xf0/0xf0 [17052.353370] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17052.356263] kthread+0x1d1/0x200 [17052.357786] ? set_kthread_struct+0x70/0x70 [17052.359695] ret_from_fork+0x1f/0x30 [17060.058493] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17060.067482] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17060.112590] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17060.120470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17060.129000] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17060.132362] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17060.146725] RSP: 0018:ffffb893c1493500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17060.154055] RAX: 0000000000005684 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17060.159782] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17060.162630] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17060.167665] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17060.175126] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17060.179536] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17060.186055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17060.188616] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17060.193628] Call Trace: [17060.195737] [17060.197852] ? show_regs.cold.9+0x22/0x2f [17060.199734] ? watchdog_timer_fn.cold.6+0x62/0x10a [17060.203288] ? __hrtimer_run_queues+0x1c2/0x4d0 [17060.205586] ? watchdog+0x40/0x40 [17060.207730] ? hrtimer_interrupt+0x128/0x2c0 [17060.212377] ? smp_apic_timer_interrupt+0xa6/0x270 [17060.215704] ? apic_timer_interrupt+0xf/0x20 [17060.219423] [17060.221252] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17060.225266] do_raw_spin_lock+0x177/0x190 [17060.226505] _raw_spin_lock+0x21/0x30 [17060.232597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17060.238497] ? do_raw_spin_unlock+0x75/0x190 [17060.240838] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17060.247565] ? do_wait_intr+0xf0/0xf0 [17060.251768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17060.256718] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17060.262685] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17060.270538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17060.275904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17060.277665] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17060.280615] mdc_intent_lock+0x353/0x7e0 [mdc] [17060.281700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17060.283706] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17060.286777] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17060.292431] lmv_intent_open+0x338/0x1150 [lmv] [17060.294043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17060.300285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17060.303760] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17060.305243] ll_intent_lock+0xc5/0xaa0 [lustre] [17060.312828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17060.316712] ll_lookup_it+0xe0e/0x2270 [lustre] [17060.321510] ? do_raw_spin_unlock+0x75/0x190 [17060.324720] ? _raw_spin_unlock+0x12/0x30 [17060.327466] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17060.333370] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17060.335276] ll_atomic_open+0x36a/0xd70 [lustre] [17060.341338] lookup_open+0x356/0xb40 [17060.342445] do_last+0x313/0xfc0 [17060.343859] ? nd_jump_root+0xe5/0x160 [17060.347600] ? path_init+0x437/0x520 [17060.350532] path_openat+0xf7/0x500 [17060.352722] ? do_raw_spin_unlock+0x75/0x190 [17060.357126] do_filp_open+0x99/0x140 [17060.358854] ? getname_flags+0x6e/0x330 [17060.363458] ? __check_object_size+0xff/0x256 [17060.365442] ? do_raw_spin_unlock+0x75/0x190 [17060.371395] ? _raw_spin_unlock+0x12/0x30 [17060.374223] do_sys_openat2+0x2b4/0x410 [17060.376372] do_sys_open+0x73/0xa0 [17060.379408] __x64_sys_openat+0x24/0x30 [17060.381484] do_syscall_64+0xc1/0x440 [17060.384391] entry_SYSCALL_64_after_hwframe+0x49/0xae [17060.388655] RIP: 0033:0x7fa749552332 [17060.391666] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17060.404792] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17060.410341] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17060.413637] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17060.419677] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17060.426646] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17060.430266] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17064.928050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15147s! [17064.934803] Showing busy workqueues and worker pools: [17064.937929] workqueue events: flags=0x0 [17064.940091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17064.942581] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17064.947365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17064.951420] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17064.955963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17064.958997] pending: kfree_rcu_monitor [17064.960640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17064.964372] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17064.973969] workqueue events_long: flags=0x0 [17064.976234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17064.979365] pending: lu_objects_flush [obdclass] [17064.983447] workqueue events_unbound: flags=0x2 [17064.985630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17064.988434] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17064.992669] workqueue events_power_efficient: flags=0x80 [17064.997293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17065.002484] pending: check_lifetime [17065.005153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17065.012369] pending: neigh_periodic_work [17065.015145] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17065.019490] pending: neigh_periodic_work [17065.021378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17065.024985] pending: do_cache_clean [17065.026525] workqueue mm_percpu_wq: flags=0x8 [17065.031202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17065.034747] pending: vmstat_update [17065.037913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17065.043048] pending: vmstat_update [17065.046579] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17065.050410] pending: vmstat_update [17065.052648] workqueue writeback: flags=0x4a [17065.055491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17065.059523] pending: wb_workfn, wb_workfn [17065.062889] workqueue xprtiod: flags=0xa [17065.065276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17065.068572] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17065.074014] workqueue ipv6_addrconf: flags=0x40008 [17065.078060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17065.083322] pending: addrconf_verify_work [17065.085406] workqueue knbd0-recv: flags=0x1a [17065.087360] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17065.090273] in-flight: 651:recv_work [17065.093356] workqueue ptlrpc_pinger: flags=0x2 [17065.095688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17065.098555] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17065.101226] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15147s workers=4 idle: 37 31846 28897 [17065.104569] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17055s workers=2 idle: 653 [17068.057096] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17068.063204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17068.112403] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17068.125494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17068.130608] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17068.134519] RIP: 0010:native_safe_halt+0x2e/0x50 [17068.136757] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17068.157833] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17068.166691] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17068.178588] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17068.189567] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17068.199342] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17068.202616] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17068.204642] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17068.207348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17068.209670] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17068.212644] Call Trace: [17068.213574] [17068.214195] ? show_regs.cold.9+0x22/0x2f [17068.215263] ? watchdog_timer_fn.cold.6+0x62/0x10a [17068.216907] ? __hrtimer_run_queues+0x1c2/0x4d0 [17068.218360] ? watchdog+0x40/0x40 [17068.219628] ? hrtimer_interrupt+0x128/0x2c0 [17068.221330] ? smp_apic_timer_interrupt+0xa6/0x270 [17068.223278] ? apic_timer_interrupt+0xf/0x20 [17068.225122] [17068.225915] ? native_safe_halt+0x2e/0x50 [17068.231322] kvm_wait+0xcd/0xe0 [17068.233647] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17068.239521] do_raw_spin_lock+0x177/0x190 [17068.244393] _raw_spin_lock+0x21/0x30 [17068.245637] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17068.249563] process_one_work+0x2c8/0x700 [17068.250631] worker_thread+0x296/0x6e0 [17068.251564] ? rescuer_thread+0x570/0x570 [17068.255153] kthread+0x1d1/0x200 [17068.258514] ? set_kthread_struct+0x70/0x70 [17068.264431] ret_from_fork+0x1f/0x30 [17080.058042] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [17080.061467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr [17080.061683] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17080.061693] Modules linked in: [17080.074333] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17080.074364] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17080.074371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17080.074374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17080.074388] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17080.074391] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 [17080.078154] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17080.078232] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17080.078240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17080.078243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17080.079728] ORIG_RAX: ffffffffffffff13 [17080.079736] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17080.079739] RDX: 000000000000217a RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17080.079742] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17080.087615] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17080.092159] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17080.092170] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17080.092173] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17080.092177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17080.092179] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17080.092188] Call Trace: [17080.092253] [17080.092261] ? show_regs.cold.9+0x22/0x2f [17080.092273] ? watchdog_timer_fn.cold.6+0x62/0x10a [17080.092280] ? __hrtimer_run_queues+0x1c2/0x4d0 [17080.092284] ? watchdog+0x40/0x40 [17080.092289] ? hrtimer_interrupt+0x128/0x2c0 [17080.092293] ? smp_apic_timer_interrupt+0xa6/0x270 [17080.092302] ? apic_timer_interrupt+0xf/0x20 [17080.092306] [17080.092309] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17080.092316] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17080.092321] do_raw_spin_lock+0x177/0x190 [17080.092325] _raw_spin_lock+0x21/0x30 [17080.092332] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17080.092608] obd_mod_free+0x46/0x140 [ptlrpc] [17080.092839] mdc_commit_open+0x91/0xb0 [mdc] [17080.093032] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17080.093310] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17080.093518] after_reply+0xe2c/0x1780 [ptlrpc] [17080.093739] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17080.094027] ? __next_timer_interrupt+0x160/0x160 [17080.094037] ptlrpcd+0xa82/0xe40 [ptlrpc] [17080.094319] ? do_wait_intr+0xf0/0xf0 [17080.094329] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17080.094526] kthread+0x1d1/0x200 [17080.094532] ? set_kthread_struct+0x70/0x70 [17080.094537] ret_from_fork+0x1f/0x30 [17080.309277] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17080.313475] RAX: 0000000000004c8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17080.317099] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17080.320386] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17080.324316] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17080.328023] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17080.335491] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17080.343000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17080.347078] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17080.351093] Call Trace: [17080.352366] [17080.353356] ? show_regs.cold.9+0x22/0x2f [17080.355856] ? watchdog_timer_fn.cold.6+0x62/0x10a [17080.358985] ? __hrtimer_run_queues+0x1c2/0x4d0 [17080.361402] ? watchdog+0x40/0x40 [17080.363504] ? hrtimer_interrupt+0x128/0x2c0 [17080.365200] ? smp_apic_timer_interrupt+0xa6/0x270 [17080.368328] ? apic_timer_interrupt+0xf/0x20 [17080.370818] [17080.372948] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17080.376295] do_raw_spin_lock+0x177/0x190 [17080.379101] _raw_spin_lock+0x21/0x30 [17080.381320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17080.384148] ? do_raw_spin_unlock+0x75/0x190 [17080.388545] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17080.392588] ? do_wait_intr+0xf0/0xf0 [17080.394082] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17080.398256] mdc_close+0x53b/0xd70 [mdc] [17080.399969] lmv_close+0x18d/0x490 [lmv] [17080.402614] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17080.405273] ll_md_real_close+0xa5/0x2a0 [lustre] [17080.407698] ll_md_close+0x274/0x800 [lustre] [17080.410207] ll_file_release+0x128/0xc50 [lustre] [17080.412331] __fput+0x10d/0x440 [17080.414044] ____fput+0x12/0x20 [17080.416026] task_work_run+0xc6/0x110 [17080.418691] exit_to_usermode_loop+0x1dd/0x1f0 [17080.420706] do_syscall_64+0x3d6/0x440 [17080.422836] entry_SYSCALL_64_after_hwframe+0x49/0xae [17080.425365] RIP: 0033:0x7fa749552cd8 [17080.427139] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17080.438155] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17080.440887] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17080.444879] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17080.450452] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17080.455067] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17080.458380] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17092.054135] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [17092.065231] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17092.115032] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17092.120998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17092.125149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17092.128795] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17092.141118] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17092.143908] RAX: 0000000000007f88 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17092.147052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17092.150664] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17092.156982] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17092.165538] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17092.171036] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17092.178489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17092.183674] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17092.189378] Call Trace: [17092.190155] [17092.192460] ? show_regs.cold.9+0x22/0x2f [17092.193941] ? watchdog_timer_fn.cold.6+0x62/0x10a [17092.198447] ? __hrtimer_run_queues+0x1c2/0x4d0 [17092.202052] ? watchdog+0x40/0x40 [17092.206552] ? hrtimer_interrupt+0x128/0x2c0 [17092.210587] ? smp_apic_timer_interrupt+0xa6/0x270 [17092.214085] ? apic_timer_interrupt+0xf/0x20 [17092.217718] [17092.219782] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17092.221747] do_raw_spin_lock+0x177/0x190 [17092.223437] _raw_spin_lock+0x21/0x30 [17092.224664] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17092.227781] ? do_raw_spin_unlock+0x75/0x190 [17092.235262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17092.241053] ? do_wait_intr+0xf0/0xf0 [17092.246573] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17092.255788] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17092.260100] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17092.271019] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17092.272801] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17092.285724] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17092.287304] mdc_intent_lock+0x353/0x7e0 [mdc] [17092.299701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17092.308538] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17092.318045] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17092.328086] lmv_intent_open+0x338/0x1150 [lmv] [17092.336648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17092.341880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17092.348557] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17092.353686] ll_intent_lock+0xc5/0xaa0 [lustre] [17092.358788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17092.365986] ll_lookup_it+0xe0e/0x2270 [lustre] [17092.372218] ? do_raw_spin_unlock+0x75/0x190 [17092.373384] ? _raw_spin_unlock+0x12/0x30 [17092.381779] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17092.385810] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17092.389865] ll_atomic_open+0x36a/0xd70 [lustre] [17092.395666] lookup_open+0x356/0xb40 [17092.399376] do_last+0x313/0xfc0 [17092.400663] ? nd_jump_root+0xe5/0x160 [17092.401784] ? path_init+0x437/0x520 [17092.408029] path_openat+0xf7/0x500 [17092.411801] ? do_raw_spin_unlock+0x75/0x190 [17092.416950] do_filp_open+0x99/0x140 [17092.420403] ? getname_flags+0x6e/0x330 [17092.424826] ? __check_object_size+0xff/0x256 [17092.430808] ? do_raw_spin_unlock+0x75/0x190 [17092.434160] ? _raw_spin_unlock+0x12/0x30 [17092.437648] do_sys_openat2+0x2b4/0x410 [17092.438731] do_sys_open+0x73/0xa0 [17092.439630] __x64_sys_openat+0x24/0x30 [17092.446684] do_syscall_64+0xc1/0x440 [17092.447994] entry_SYSCALL_64_after_hwframe+0x49/0xae [17092.457015] RIP: 0033:0x7fa749552332 [17092.458188] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17092.479126] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17092.481532] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17092.495339] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17092.503576] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17092.506846] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17092.514780] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17094.021224] rcu: INFO: rcu_sched self-detected stall on CPU [17094.025050] rcu: 2-...!: (8943417 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [17094.039363] (t=15180332 jiffies g=258701 q=5647) [17094.042397] rcu: rcu_sched kthread starved for 15177127 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [17094.054252] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17094.060572] rcu: RCU grace-period kthread stack dump: [17094.063337] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17094.077532] Call Trace: [17094.078737] __schedule+0x351/0xcb0 [17094.080744] schedule+0xc0/0x180 [17094.086615] schedule_timeout+0xb4/0x190 [17094.089597] ? do_raw_spin_unlock+0x75/0x190 [17094.091019] ? __next_timer_interrupt+0x160/0x160 [17094.094210] ? prepare_to_swait_event+0xf3/0x140 [17094.096238] rcu_gp_kthread+0x8dd/0x15b0 [17094.099412] ? force_qs_rnp+0x2c0/0x2c0 [17094.102508] kthread+0x1d1/0x200 [17094.107063] ? set_kthread_struct+0x70/0x70 [17094.110093] ret_from_fork+0x1f/0x30 [17094.111356] rcu: Stack dump where RCU GP kthread last ran: [17094.114967] Sending NMI from CPU 2 to CPUs 1: [17094.119100] NMI backtrace for cpu 1 [17094.119108] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17094.119110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17094.119112] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17094.119115] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17094.119117] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17094.119119] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 [17094.119120] RAX: 000000000000512b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17094.119122] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17094.119124] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17094.119125] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17094.119126] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17094.119127] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17094.119129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17094.119130] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17094.119131] Call Trace: [17094.119132] [17094.119132] ? show_regs.cold.9+0x22/0x2f [17094.119133] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17094.119134] ? nmi_cpu_backtrace_handler+0x15/0x40 [17094.119135] ? nmi_handle+0xbd/0x210 [17094.119136] ? default_do_nmi+0x4b/0x1b0 [17094.119137] ? do_nmi+0x280/0x380 [17094.119139] ? end_repeat_nmi+0x16/0x1f [17094.119140] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17094.119141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17094.119142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17094.119143] [17094.119144] do_raw_spin_lock+0x177/0x190 [17094.119145] _raw_spin_lock+0x21/0x30 [17094.119146] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17094.119147] process_one_work+0x2c8/0x700 [17094.119148] worker_thread+0x296/0x6e0 [17094.119149] ? rescuer_thread+0x570/0x570 [17094.119150] kthread+0x1d1/0x200 [17094.119151] ? set_kthread_struct+0x70/0x70 [17094.119289] ret_from_fork+0x1f/0x30 [17094.121651] NMI backtrace for cpu 2 [17094.280836] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17094.287334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17094.291803] Call Trace: [17094.293036] [17094.294030] ? dump_stack+0xbb/0x10e [17094.296060] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17094.302367] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17094.312043] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17094.315728] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17094.320804] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17094.324031] ? rcu_dump_cpu_stacks+0x11b/0x171 [17094.327099] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17094.330373] ? update_process_times+0x6d/0xc0 [17094.333034] ? tick_sched_handle.isra.18+0x27/0x90 [17094.339413] ? tick_sched_timer+0x53/0xb0 [17094.340743] ? __hrtimer_run_queues+0x1c2/0x4d0 [17094.345437] ? tick_sched_handle.isra.18+0x90/0x90 [17094.352122] ? hrtimer_interrupt+0x128/0x2c0 [17094.356021] ? smp_apic_timer_interrupt+0xa6/0x270 [17094.360702] ? apic_timer_interrupt+0xf/0x20 [17094.365787] [17094.367254] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17094.371024] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17094.375157] ? do_raw_spin_lock+0x177/0x190 [17094.379196] ? _raw_spin_lock+0x21/0x30 [17094.382475] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17094.388287] ? obd_mod_free+0x46/0x140 [ptlrpc] [17094.399849] ? mdc_commit_open+0x91/0xb0 [mdc] [17094.409046] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17094.413995] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17094.419397] ? after_reply+0xe2c/0x1780 [ptlrpc] [17094.425003] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17094.433054] ? __next_timer_interrupt+0x160/0x160 [17094.439998] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17094.442408] ? do_wait_intr+0xf0/0xf0 [17094.447793] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17094.451823] ? kthread+0x1d1/0x200 [17094.454304] ? set_kthread_struct+0x70/0x70 [17094.456446] ? ret_from_fork+0x1f/0x30 [17095.139812] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15177s! [17095.146667] Showing busy workqueues and worker pools: [17095.152695] workqueue events: flags=0x0 [17095.155373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17095.162552] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17095.175655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17095.183202] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17095.192453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.197839] pending: kfree_rcu_monitor [17095.199740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17095.206017] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17095.215609] workqueue events_long: flags=0x0 [17095.218471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.229177] pending: lu_objects_flush [obdclass] [17095.233615] workqueue events_unbound: flags=0x2 [17095.242953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17095.252715] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17095.280508] workqueue events_power_efficient: flags=0x80 [17095.293713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.311170] pending: check_lifetime [17095.318086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.329211] pending: neigh_periodic_work [17095.334643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.338922] pending: neigh_periodic_work [17095.342646] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.347213] pending: do_cache_clean [17095.350221] workqueue mm_percpu_wq: flags=0x8 [17095.352467] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.357193] pending: vmstat_update [17095.367668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.375868] pending: vmstat_update [17095.378654] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17095.382504] pending: vmstat_update [17095.389693] workqueue writeback: flags=0x4a [17095.399529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17095.403581] pending: wb_workfn, wb_workfn [17095.407063] workqueue xprtiod: flags=0xa [17095.412874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17095.417593] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17095.422844] workqueue ipv6_addrconf: flags=0x40008 [17095.426029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17095.433507] pending: addrconf_verify_work [17095.436086] workqueue knbd0-recv: flags=0x1a [17095.442470] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17095.446501] in-flight: 651:recv_work [17095.450320] workqueue ptlrpc_pinger: flags=0x2 [17095.453868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17095.459504] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17095.466065] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15178s workers=4 idle: 37 31846 28897 [17095.480407] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17085s workers=2 idle: 653 [17108.059050] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17108.062787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17108.087126] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17108.093240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17108.097746] RIP: 0010:native_safe_halt+0x2e/0x50 [17108.103594] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17108.119016] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17108.122889] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17108.127131] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17108.131201] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17108.135527] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17108.139508] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17108.142614] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17108.152679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17108.156430] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17108.161257] Call Trace: [17108.162794] [17108.164419] ? show_regs.cold.9+0x22/0x2f [17108.165955] ? watchdog_timer_fn.cold.6+0x62/0x10a [17108.168749] ? __hrtimer_run_queues+0x1c2/0x4d0 [17108.171522] ? watchdog+0x40/0x40 [17108.173383] ? hrtimer_interrupt+0x128/0x2c0 [17108.176626] ? smp_apic_timer_interrupt+0xa6/0x270 [17108.178414] ? apic_timer_interrupt+0xf/0x20 [17108.179773] [17108.180690] ? native_safe_halt+0x2e/0x50 [17108.184277] kvm_wait+0xcd/0xe0 [17108.186533] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17108.189639] do_raw_spin_lock+0x177/0x190 [17108.193082] _raw_spin_lock+0x21/0x30 [17108.194670] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17108.197262] ? do_raw_spin_unlock+0x75/0x190 [17108.198892] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17108.202593] ? do_wait_intr+0xf0/0xf0 [17108.204207] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17108.207869] mdc_close+0x53b/0xd70 [mdc] [17108.212882] lmv_close+0x18d/0x490 [lmv] [17108.214332] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17108.219028] ll_md_real_close+0xa5/0x2a0 [lustre] [17108.220428] ll_md_close+0x274/0x800 [lustre] [17108.222574] ll_file_release+0x128/0xc50 [lustre] [17108.225215] __fput+0x10d/0x440 [17108.234969] ____fput+0x12/0x20 [17108.239303] task_work_run+0xc6/0x110 [17108.242254] exit_to_usermode_loop+0x1dd/0x1f0 [17108.244952] do_syscall_64+0x3d6/0x440 [17108.250227] entry_SYSCALL_64_after_hwframe+0x49/0xae [17108.258477] RIP: 0033:0x7fa749552cd8 [17108.260345] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17108.271590] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17108.274857] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17108.280051] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17108.284008] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17108.289572] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17108.293579] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17120.054048] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [17120.057021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [17120.058020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [17120.058028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17120.058117] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17120.058123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17120.058126] RIP: 0010:native_safe_halt+0x2e/0x50 [17120.058140] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17120.058145] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17120.058150] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17120.058153] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17120.058156] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17120.058158] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17120.058161] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17120.058164] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17120.058168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17120.058171] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17120.058179] Call Trace: [17120.058184] [17120.058190] ? show_regs.cold.9+0x22/0x2f [17120.058201] ? watchdog_timer_fn.cold.6+0x62/0x10a [17120.058207] ? __hrtimer_run_queues+0x1c2/0x4d0 [17120.058211] ? watchdog+0x40/0x40 [17120.058216] ? hrtimer_interrupt+0x128/0x2c0 [17120.058220] ? smp_apic_timer_interrupt+0xa6/0x270 [17120.058228] ? apic_timer_interrupt+0xf/0x20 [17120.058232] [17120.058235] ? native_safe_halt+0x2e/0x50 [17120.058239] kvm_wait+0xcd/0xe0 [17120.058247] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17120.058254] do_raw_spin_lock+0x177/0x190 [17120.058259] _raw_spin_lock+0x21/0x30 [17120.058265] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17120.058598] obd_mod_free+0x46/0x140 [ptlrpc] [17120.058788] mdc_commit_open+0x91/0xb0 [mdc] [17120.058829] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17120.059034] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17120.059237] after_reply+0xe2c/0x1780 [ptlrpc] [17120.059554] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17120.059751] ? __next_timer_interrupt+0x160/0x160 [17120.059761] ptlrpcd+0xa82/0xe40 [ptlrpc] [17120.060028] ? do_wait_intr+0xf0/0xf0 [17120.060038] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17120.060262] kthread+0x1d1/0x200 [17120.060268] ? set_kthread_struct+0x70/0x70 [17120.060273] ret_from_fork+0x1f/0x30 [17120.061751] Modules linked in: [17120.065429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17120.065516] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17120.065531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17120.065537] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17120.066015] RIP: 0010:native_safe_halt+0x2e/0x50 [17120.066027] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17120.066032] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17120.066036] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17120.066038] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17120.066041] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17120.066043] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17120.066046] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17120.066049] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17120.066051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17120.066053] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17120.066063] Call Trace: [17120.066068] [17120.066073] ? show_regs.cold.9+0x22/0x2f [17120.066083] ? watchdog_timer_fn.cold.6+0x62/0x10a [17120.066090] ? __hrtimer_run_queues+0x1c2/0x4d0 [17120.066094] ? watchdog+0x40/0x40 [17120.066099] ? hrtimer_interrupt+0x128/0x2c0 [17120.066103] ? smp_apic_timer_interrupt+0xa6/0x270 [17120.066121] ? apic_timer_interrupt+0xf/0x20 [17120.066125] [17120.066128] ? native_safe_halt+0x2e/0x50 [17120.066132] kvm_wait+0xcd/0xe0 [17120.066139] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17120.066146] do_raw_spin_lock+0x177/0x190 [17120.066150] _raw_spin_lock+0x21/0x30 [17120.066154] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17120.066351] process_one_work+0x2c8/0x700 [17120.066359] worker_thread+0x296/0x6e0 [17120.066364] ? rescuer_thread+0x570/0x570 [17120.066367] kthread+0x1d1/0x200 [17120.066372] ? set_kthread_struct+0x70/0x70 [17120.066375] ret_from_fork+0x1f/0x30 [17120.459053] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17120.478636] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17120.483007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17120.487154] RIP: 0010:native_safe_halt+0x2e/0x50 [17120.490804] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17120.500596] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17120.512763] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17120.522042] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [17120.528132] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17120.532259] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17120.540408] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17120.548345] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17120.552799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17120.557689] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17120.562537] Call Trace: [17120.563940] [17120.568827] ? show_regs.cold.9+0x22/0x2f [17120.570985] ? watchdog_timer_fn.cold.6+0x62/0x10a [17120.575877] ? __hrtimer_run_queues+0x1c2/0x4d0 [17120.577423] ? watchdog+0x40/0x40 [17120.580338] ? hrtimer_interrupt+0x128/0x2c0 [17120.582787] ? smp_apic_timer_interrupt+0xa6/0x270 [17120.586000] ? apic_timer_interrupt+0xf/0x20 [17120.592068] [17120.593314] ? native_safe_halt+0x2e/0x50 [17120.595742] kvm_wait+0xcd/0xe0 [17120.604761] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17120.610765] do_raw_spin_lock+0x177/0x190 [17120.612704] _raw_spin_lock+0x21/0x30 [17120.615737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17120.623042] ? do_raw_spin_unlock+0x75/0x190 [17120.626002] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17120.629310] ? do_wait_intr+0xf0/0xf0 [17120.631944] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17120.633964] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17120.641005] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17120.645409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.649117] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17120.653917] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17120.661166] mdc_intent_lock+0x353/0x7e0 [mdc] [17120.664767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.673963] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17120.681107] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17120.685173] lmv_intent_open+0x338/0x1150 [lmv] [17120.688460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.695174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.701116] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17120.705778] ll_intent_lock+0xc5/0xaa0 [lustre] [17120.709359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.715085] ll_lookup_it+0xe0e/0x2270 [lustre] [17120.717519] ? do_raw_spin_unlock+0x75/0x190 [17120.719417] ? _raw_spin_unlock+0x12/0x30 [17120.721765] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17120.731561] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17120.737671] ll_atomic_open+0x36a/0xd70 [lustre] [17120.742967] lookup_open+0x356/0xb40 [17120.745007] do_last+0x313/0xfc0 [17120.746714] ? nd_jump_root+0xe5/0x160 [17120.752320] ? path_init+0x437/0x520 [17120.756039] path_openat+0xf7/0x500 [17120.758583] ? do_raw_spin_unlock+0x75/0x190 [17120.761173] do_filp_open+0x99/0x140 [17120.763917] ? getname_flags+0x6e/0x330 [17120.768024] ? __check_object_size+0xff/0x256 [17120.771105] ? do_raw_spin_unlock+0x75/0x190 [17120.776706] ? _raw_spin_unlock+0x12/0x30 [17120.780038] do_sys_openat2+0x2b4/0x410 [17120.781424] do_sys_open+0x73/0xa0 [17120.787571] __x64_sys_openat+0x24/0x30 [17120.789919] do_syscall_64+0xc1/0x440 [17120.792065] entry_SYSCALL_64_after_hwframe+0x49/0xae [17120.794678] RIP: 0033:0x7fa749552332 [17120.797543] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17120.812241] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17120.815451] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17120.821338] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17120.824747] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17120.827703] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17120.830499] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17125.857043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15208s! [17125.859543] Showing busy workqueues and worker pools: [17125.861425] workqueue events: flags=0x0 [17125.863572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17125.868484] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17125.874981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17125.878242] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17125.883246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17125.886821] pending: kfree_rcu_monitor [17125.889302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17125.892471] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17125.897177] workqueue events_long: flags=0x0 [17125.899397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17125.902946] pending: lu_objects_flush [obdclass] [17125.905454] workqueue events_unbound: flags=0x2 [17125.909245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17125.911413] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17125.920367] workqueue events_power_efficient: flags=0x80 [17125.926592] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17125.935084] pending: check_lifetime [17125.939606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17125.948830] pending: neigh_periodic_work [17125.952362] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17125.956378] pending: neigh_periodic_work [17125.960472] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17125.965474] pending: do_cache_clean [17125.968345] workqueue mm_percpu_wq: flags=0x8 [17125.969920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17125.980696] pending: vmstat_update [17125.982865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17125.989593] pending: vmstat_update [17125.995320] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17126.001982] pending: vmstat_update [17126.006281] workqueue writeback: flags=0x4a [17126.008566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17126.010710] pending: wb_workfn, wb_workfn [17126.017148] workqueue xprtiod: flags=0xa [17126.021428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17126.028145] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17126.035532] workqueue ipv6_addrconf: flags=0x40008 [17126.045738] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17126.048226] pending: addrconf_verify_work [17126.056358] workqueue knbd0-recv: flags=0x1a [17126.059235] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17126.067011] in-flight: 651:recv_work [17126.071738] workqueue ptlrpc_pinger: flags=0x2 [17126.073345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17126.078067] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17126.085166] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15208s workers=4 idle: 37 31846 28897 [17126.093372] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17115s workers=2 idle: 653 [17136.059019] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17136.061255] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17136.104364] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17136.120726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17136.133515] RIP: 0010:native_safe_halt+0x2e/0x50 [17136.139341] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17136.161590] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17136.171836] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17136.184992] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17136.193972] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17136.203525] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17136.216862] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17136.229472] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17136.244835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17136.253619] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17136.262626] Call Trace: [17136.266836] [17136.270276] ? show_regs.cold.9+0x22/0x2f [17136.272248] ? watchdog_timer_fn.cold.6+0x62/0x10a [17136.274842] ? __hrtimer_run_queues+0x1c2/0x4d0 [17136.276763] ? watchdog+0x40/0x40 [17136.278040] ? hrtimer_interrupt+0x128/0x2c0 [17136.280153] ? smp_apic_timer_interrupt+0xa6/0x270 [17136.282469] ? apic_timer_interrupt+0xf/0x20 [17136.284774] [17136.285849] ? native_safe_halt+0x2e/0x50 [17136.287648] kvm_wait+0xcd/0xe0 [17136.289475] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17136.292005] do_raw_spin_lock+0x177/0x190 [17136.294299] _raw_spin_lock+0x21/0x30 [17136.296031] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17136.300197] ? do_raw_spin_unlock+0x75/0x190 [17136.302334] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17136.305278] ? do_wait_intr+0xf0/0xf0 [17136.306942] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17136.310227] mdc_close+0x53b/0xd70 [mdc] [17136.312487] lmv_close+0x18d/0x490 [lmv] [17136.314626] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17136.317372] ll_md_real_close+0xa5/0x2a0 [lustre] [17136.319442] ll_md_close+0x274/0x800 [lustre] [17136.321510] ll_file_release+0x128/0xc50 [lustre] [17136.323927] __fput+0x10d/0x440 [17136.325317] ____fput+0x12/0x20 [17136.326555] task_work_run+0xc6/0x110 [17136.328069] exit_to_usermode_loop+0x1dd/0x1f0 [17136.329969] do_syscall_64+0x3d6/0x440 [17136.331544] entry_SYSCALL_64_after_hwframe+0x49/0xae [17136.333738] RIP: 0033:0x7fa749552cd8 [17136.335229] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17136.342527] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17136.345996] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17136.348527] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17136.351720] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17136.354699] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17136.357384] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17148.058614] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17148.060823] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17148.069708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17148.069987] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17148.069994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17148.070005] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17148.070373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17148.070385] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17148.070390] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17148.070395] RAX: 00000000000078e2 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17148.070397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17148.070400] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17148.070402] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17148.070405] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17148.070409] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17148.070412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17148.070414] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17148.070423] Call Trace: [17148.070428] [17148.070433] ? show_regs.cold.9+0x22/0x2f [17148.070441] ? watchdog_timer_fn.cold.6+0x62/0x10a [17148.070448] ? __hrtimer_run_queues+0x1c2/0x4d0 [17148.070452] ? watchdog+0x40/0x40 [17148.070456] ? hrtimer_interrupt+0x128/0x2c0 [17148.070460] ? smp_apic_timer_interrupt+0xa6/0x270 [17148.070468] ? apic_timer_interrupt+0xf/0x20 [17148.070472] [17148.070475] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17148.070479] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17148.070483] do_raw_spin_lock+0x177/0x190 [17148.070487] _raw_spin_lock+0x21/0x30 [17148.070492] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17148.071040] process_one_work+0x2c8/0x700 [17148.071048] worker_thread+0x296/0x6e0 [17148.071059] ? rescuer_thread+0x570/0x570 [17148.071064] kthread+0x1d1/0x200 [17148.071073] ? set_kthread_struct+0x70/0x70 [17148.071077] ret_from_fork+0x1f/0x30 [17148.213743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17148.237021] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17148.241721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17148.245529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17148.247512] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17148.253905] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17148.256889] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17148.259867] RDX: 0000000000005e9e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17148.262722] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17148.265777] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17148.268755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17148.271737] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17148.274642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17148.276207] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17148.278833] Call Trace: [17148.279702] [17148.280385] ? show_regs.cold.9+0x22/0x2f [17148.281882] ? watchdog_timer_fn.cold.6+0x62/0x10a [17148.283509] ? __hrtimer_run_queues+0x1c2/0x4d0 [17148.284985] ? watchdog+0x40/0x40 [17148.286484] ? hrtimer_interrupt+0x128/0x2c0 [17148.287934] ? smp_apic_timer_interrupt+0xa6/0x270 [17148.290926] ? apic_timer_interrupt+0xf/0x20 [17148.292493] [17148.293382] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17148.295735] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17148.297774] do_raw_spin_lock+0x177/0x190 [17148.299140] _raw_spin_lock+0x21/0x30 [17148.300667] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17148.303186] obd_mod_free+0x46/0x140 [ptlrpc] [17148.305418] mdc_commit_open+0x91/0xb0 [mdc] [17148.307095] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17148.310738] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17148.313734] after_reply+0xe2c/0x1780 [ptlrpc] [17148.317199] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17148.322452] ? __next_timer_interrupt+0x160/0x160 [17148.324145] ptlrpcd+0xa82/0xe40 [ptlrpc] [17148.326200] ? do_wait_intr+0xf0/0xf0 [17148.327872] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17148.329420] kthread+0x1d1/0x200 [17148.332809] ? set_kthread_struct+0x70/0x70 [17148.334291] ret_from_fork+0x1f/0x30 [17152.061907] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [17152.069500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17152.103834] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17152.108373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17152.130056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17152.133066] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17152.148142] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17152.153086] RAX: 000000000000497a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17152.159979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17152.168805] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17152.174562] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17152.182384] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17152.188649] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17152.191073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17152.198052] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17152.203852] Call Trace: [17152.205314] [17152.207448] ? show_regs.cold.9+0x22/0x2f [17152.209869] ? watchdog_timer_fn.cold.6+0x62/0x10a [17152.213966] ? __hrtimer_run_queues+0x1c2/0x4d0 [17152.216467] ? watchdog+0x40/0x40 [17152.218492] ? hrtimer_interrupt+0x128/0x2c0 [17152.221840] ? smp_apic_timer_interrupt+0xa6/0x270 [17152.227345] ? apic_timer_interrupt+0xf/0x20 [17152.230101] [17152.231089] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17152.235676] do_raw_spin_lock+0x177/0x190 [17152.240978] _raw_spin_lock+0x21/0x30 [17152.247024] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17152.253271] ? do_raw_spin_unlock+0x75/0x190 [17152.255390] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17152.262929] ? do_wait_intr+0xf0/0xf0 [17152.266480] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17152.273088] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17152.275375] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17152.287411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17152.292755] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17152.296932] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17152.299525] mdc_intent_lock+0x353/0x7e0 [mdc] [17152.302224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17152.306733] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17152.311992] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17152.315060] lmv_intent_open+0x338/0x1150 [lmv] [17152.319565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17152.324028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17152.326849] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17152.329616] ll_intent_lock+0xc5/0xaa0 [lustre] [17152.333925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17152.336301] ll_lookup_it+0xe0e/0x2270 [lustre] [17152.341127] ? do_raw_spin_unlock+0x75/0x190 [17152.345628] ? _raw_spin_unlock+0x12/0x30 [17152.349119] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17152.354498] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17152.364960] ll_atomic_open+0x36a/0xd70 [lustre] [17152.373250] lookup_open+0x356/0xb40 [17152.376142] do_last+0x313/0xfc0 [17152.378604] ? nd_jump_root+0xe5/0x160 [17152.380840] ? path_init+0x437/0x520 [17152.383485] path_openat+0xf7/0x500 [17152.387608] ? do_raw_spin_unlock+0x75/0x190 [17152.389500] do_filp_open+0x99/0x140 [17152.394734] ? getname_flags+0x6e/0x330 [17152.397061] ? __check_object_size+0xff/0x256 [17152.398787] ? do_raw_spin_unlock+0x75/0x190 [17152.399850] ? _raw_spin_unlock+0x12/0x30 [17152.401137] do_sys_openat2+0x2b4/0x410 [17152.411570] do_sys_open+0x73/0xa0 [17152.413733] __x64_sys_openat+0x24/0x30 [17152.415728] do_syscall_64+0xc1/0x440 [17152.418445] entry_SYSCALL_64_after_hwframe+0x49/0xae [17152.428483] RIP: 0033:0x7fa749552332 [17152.430627] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17152.441697] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17152.445979] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17152.451128] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17152.462521] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17152.465325] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17152.474897] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17156.581722] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15239s! [17156.587967] Showing busy workqueues and worker pools: [17156.589460] workqueue events: flags=0x0 [17156.596271] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17156.600918] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17156.618340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17156.623571] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17156.630796] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.634527] pending: kfree_rcu_monitor [17156.636247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17156.638887] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17156.642396] workqueue events_long: flags=0x0 [17156.644433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.646989] pending: lu_objects_flush [obdclass] [17156.649595] workqueue events_unbound: flags=0x2 [17156.651601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17156.654345] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17156.659424] workqueue events_power_efficient: flags=0x80 [17156.662879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.665895] pending: check_lifetime [17156.667690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.670573] pending: neigh_periodic_work [17156.672429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.675492] pending: neigh_periodic_work [17156.677609] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.680904] pending: do_cache_clean [17156.682751] workqueue mm_percpu_wq: flags=0x8 [17156.685018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.688533] pending: vmstat_update [17156.690105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.693488] pending: vmstat_update [17156.695135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17156.702849] pending: vmstat_update [17156.704907] workqueue writeback: flags=0x4a [17156.706804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17156.709478] pending: wb_workfn, wb_workfn [17156.711453] workqueue xprtiod: flags=0xa [17156.713385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17156.715963] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17156.719539] workqueue ipv6_addrconf: flags=0x40008 [17156.721057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17156.724028] pending: addrconf_verify_work [17156.725911] workqueue knbd0-recv: flags=0x1a [17156.727645] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17156.730909] in-flight: 651:recv_work [17156.732936] workqueue ptlrpc_pinger: flags=0x2 [17156.734911] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17156.737829] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17156.740461] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15239s workers=4 idle: 37 31846 28897 [17156.743781] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17146s workers=2 idle: 653 [17164.061038] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17164.065843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17164.107759] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17164.117016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17164.125389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17164.131440] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17164.147516] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17164.156003] RAX: 0000000000007e08 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17164.159558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17164.164711] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17164.173775] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17164.181676] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17164.186360] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17164.194351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17164.200638] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17164.206599] Call Trace: [17164.211908] [17164.213560] ? show_regs.cold.9+0x22/0x2f [17164.216220] ? watchdog_timer_fn.cold.6+0x62/0x10a [17164.218767] ? __hrtimer_run_queues+0x1c2/0x4d0 [17164.223824] ? watchdog+0x40/0x40 [17164.226766] ? hrtimer_interrupt+0x128/0x2c0 [17164.232957] ? smp_apic_timer_interrupt+0xa6/0x270 [17164.236850] ? apic_timer_interrupt+0xf/0x20 [17164.238881] [17164.241442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17164.243894] do_raw_spin_lock+0x177/0x190 [17164.245622] _raw_spin_lock+0x21/0x30 [17164.247961] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17164.250629] ? do_raw_spin_unlock+0x75/0x190 [17164.252505] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17164.258422] ? do_wait_intr+0xf0/0xf0 [17164.262645] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17164.266414] mdc_close+0x53b/0xd70 [mdc] [17164.268944] lmv_close+0x18d/0x490 [lmv] [17164.270770] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17164.273729] ll_md_real_close+0xa5/0x2a0 [lustre] [17164.276817] ll_md_close+0x274/0x800 [lustre] [17164.279815] ll_file_release+0x128/0xc50 [lustre] [17164.282421] __fput+0x10d/0x440 [17164.283819] ____fput+0x12/0x20 [17164.285396] task_work_run+0xc6/0x110 [17164.287755] exit_to_usermode_loop+0x1dd/0x1f0 [17164.289902] do_syscall_64+0x3d6/0x440 [17164.291859] entry_SYSCALL_64_after_hwframe+0x49/0xae [17164.294894] RIP: 0033:0x7fa749552cd8 [17164.297753] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17164.309712] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17164.315569] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17164.319780] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17164.325565] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17164.330796] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17164.336519] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17176.057234] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17176.060720] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17176.064277] Modules linked in: lustre(O) osp(O) [17176.068071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [17176.073081] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17176.073373] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17176.073380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17176.073387] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17176.073698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17176.073726] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17176.073730] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17176.073735] RAX: 0000000000001e0b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17176.073738] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17176.073741] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17176.073743] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17176.073745] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17176.073748] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17176.073751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17176.073753] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17176.073762] Call Trace: [17176.073943] [17176.073950] ? show_regs.cold.9+0x22/0x2f [17176.073960] ? watchdog_timer_fn.cold.6+0x62/0x10a [17176.073968] ? __hrtimer_run_queues+0x1c2/0x4d0 [17176.073972] ? watchdog+0x40/0x40 [17176.073977] ? hrtimer_interrupt+0x128/0x2c0 [17176.073981] ? smp_apic_timer_interrupt+0xa6/0x270 [17176.073990] ? apic_timer_interrupt+0xf/0x20 [17176.073996] [17176.073999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17176.074004] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17176.074009] do_raw_spin_lock+0x177/0x190 [17176.074015] _raw_spin_lock+0x21/0x30 [17176.074020] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17176.074442] process_one_work+0x2c8/0x700 [17176.074450] worker_thread+0x296/0x6e0 [17176.074453] ? rescuer_thread+0x570/0x570 [17176.074457] kthread+0x1d1/0x200 [17176.074461] ? set_kthread_struct+0x70/0x70 [17176.074465] ret_from_fork+0x1f/0x30 [17176.229759] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17176.250948] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17176.256423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17176.260463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17176.263559] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17176.271859] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17176.275649] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17176.278104] RDX: 0000000000007377 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17176.281109] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17176.284518] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17176.287699] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17176.290662] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17176.294551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17176.297161] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17176.300383] Call Trace: [17176.302054] [17176.302740] ? show_regs.cold.9+0x22/0x2f [17176.304755] ? watchdog_timer_fn.cold.6+0x62/0x10a [17176.306661] ? __hrtimer_run_queues+0x1c2/0x4d0 [17176.308885] ? watchdog+0x40/0x40 [17176.310322] ? hrtimer_interrupt+0x128/0x2c0 [17176.312051] ? smp_apic_timer_interrupt+0xa6/0x270 [17176.314387] ? apic_timer_interrupt+0xf/0x20 [17176.316428] [17176.317499] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17176.320111] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17176.323195] do_raw_spin_lock+0x177/0x190 [17176.325035] _raw_spin_lock+0x21/0x30 [17176.326528] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17176.328858] obd_mod_free+0x46/0x140 [ptlrpc] [17176.331354] mdc_commit_open+0x91/0xb0 [mdc] [17176.333333] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17176.335797] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17176.338627] after_reply+0xe2c/0x1780 [ptlrpc] [17176.340686] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17176.343229] ? __next_timer_interrupt+0x160/0x160 [17176.344762] ptlrpcd+0xa82/0xe40 [ptlrpc] [17176.346816] ? do_wait_intr+0xf0/0xf0 [17176.348567] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17176.351245] kthread+0x1d1/0x200 [17176.352588] ? set_kthread_struct+0x70/0x70 [17176.354494] ret_from_fork+0x1f/0x30 [17184.054300] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17184.063555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17184.094154] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17184.098921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17184.102396] RIP: 0010:native_safe_halt+0x2e/0x50 [17184.104465] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17184.113831] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17184.117097] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17184.119669] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [17184.124933] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17184.130368] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17184.136314] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17184.144150] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17184.156435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17184.165791] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17184.174214] Call Trace: [17184.179059] [17184.182469] ? show_regs.cold.9+0x22/0x2f [17184.188069] ? watchdog_timer_fn.cold.6+0x62/0x10a [17184.192758] ? __hrtimer_run_queues+0x1c2/0x4d0 [17184.195673] ? watchdog+0x40/0x40 [17184.197561] ? hrtimer_interrupt+0x128/0x2c0 [17184.208486] ? smp_apic_timer_interrupt+0xa6/0x270 [17184.213674] ? apic_timer_interrupt+0xf/0x20 [17184.225366] [17184.226052] ? native_safe_halt+0x2e/0x50 [17184.232976] kvm_wait+0xcd/0xe0 [17184.237907] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17184.243551] do_raw_spin_lock+0x177/0x190 [17184.247664] _raw_spin_lock+0x21/0x30 [17184.253105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17184.259095] ? do_raw_spin_unlock+0x75/0x190 [17184.261994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17184.270191] ? do_wait_intr+0xf0/0xf0 [17184.273170] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17184.282019] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17184.288076] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17184.291886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17184.295577] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17184.304267] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17184.312260] mdc_intent_lock+0x353/0x7e0 [mdc] [17184.314273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17184.324934] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17184.332503] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17184.334502] lmv_intent_open+0x338/0x1150 [lmv] [17184.343509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17184.351625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17184.357786] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17184.359759] ll_intent_lock+0xc5/0xaa0 [lustre] [17184.363183] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17184.369122] ll_lookup_it+0xe0e/0x2270 [lustre] [17184.370364] ? do_raw_spin_unlock+0x75/0x190 [17184.375769] ? _raw_spin_unlock+0x12/0x30 [17184.378679] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17184.390948] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17184.394164] ll_atomic_open+0x36a/0xd70 [lustre] [17184.400293] lookup_open+0x356/0xb40 [17184.408994] do_last+0x313/0xfc0 [17184.414293] ? nd_jump_root+0xe5/0x160 [17184.417664] ? path_init+0x437/0x520 [17184.421963] path_openat+0xf7/0x500 [17184.426110] ? do_raw_spin_unlock+0x75/0x190 [17184.428794] do_filp_open+0x99/0x140 [17184.430382] ? getname_flags+0x6e/0x330 [17184.432277] ? __check_object_size+0xff/0x256 [17184.433755] ? do_raw_spin_unlock+0x75/0x190 [17184.436160] ? _raw_spin_unlock+0x12/0x30 [17184.437916] do_sys_openat2+0x2b4/0x410 [17184.439812] do_sys_open+0x73/0xa0 [17184.441494] __x64_sys_openat+0x24/0x30 [17184.443368] do_syscall_64+0xc1/0x440 [17184.445441] entry_SYSCALL_64_after_hwframe+0x49/0xae [17184.448491] RIP: 0033:0x7fa749552332 [17184.450648] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17184.467438] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17184.474306] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17184.478163] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17184.484568] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17184.492634] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17184.501143] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17186.784108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15269s! [17186.802724] Showing busy workqueues and worker pools: [17186.805519] workqueue events: flags=0x0 [17186.813628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17186.826071] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17186.834057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17186.839789] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17186.850616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.856832] pending: kfree_rcu_monitor [17186.860895] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17186.865159] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17186.872890] workqueue events_long: flags=0x0 [17186.874560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.877088] pending: lu_objects_flush [obdclass] [17186.879507] workqueue events_unbound: flags=0x2 [17186.881150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17186.887886] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17186.891838] workqueue events_power_efficient: flags=0x80 [17186.894035] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.896776] pending: check_lifetime [17186.898252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.901042] pending: neigh_periodic_work [17186.903024] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.906020] pending: neigh_periodic_work [17186.907947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.910631] pending: do_cache_clean [17186.912276] workqueue mm_percpu_wq: flags=0x8 [17186.913898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.916011] pending: vmstat_update [17186.917365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.919997] pending: vmstat_update [17186.921356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17186.923815] pending: vmstat_update [17186.924787] workqueue writeback: flags=0x4a [17186.925831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17186.929855] pending: wb_workfn, wb_workfn [17186.933488] workqueue xprtiod: flags=0xa [17186.935654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17186.940317] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17186.945100] workqueue ipv6_addrconf: flags=0x40008 [17186.948560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17186.950155] pending: addrconf_verify_work [17186.951309] workqueue knbd0-recv: flags=0x1a [17186.952819] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17186.955084] in-flight: 651:recv_work [17186.956019] workqueue ptlrpc_pinger: flags=0x2 [17186.957844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17186.959531] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17186.962081] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15269s workers=4 idle: 37 31846 28897 [17186.965263] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17176s workers=2 idle: 653 [17192.059048] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17192.061763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17192.081341] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17192.085820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17192.089862] RIP: 0010:native_safe_halt+0x2e/0x50 [17192.091523] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17192.099784] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17192.102782] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17192.106452] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17192.109130] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17192.111719] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17192.114588] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17192.117134] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17192.120376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17192.122627] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17192.125147] Call Trace: [17192.125857] [17192.126660] ? show_regs.cold.9+0x22/0x2f [17192.128204] ? watchdog_timer_fn.cold.6+0x62/0x10a [17192.132482] ? __hrtimer_run_queues+0x1c2/0x4d0 [17192.135430] ? watchdog+0x40/0x40 [17192.136927] ? hrtimer_interrupt+0x128/0x2c0 [17192.138802] ? smp_apic_timer_interrupt+0xa6/0x270 [17192.140880] ? apic_timer_interrupt+0xf/0x20 [17192.142352] [17192.142954] ? native_safe_halt+0x2e/0x50 [17192.144975] kvm_wait+0xcd/0xe0 [17192.145689] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17192.146901] do_raw_spin_lock+0x177/0x190 [17192.148249] _raw_spin_lock+0x21/0x30 [17192.149498] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17192.151449] ? do_raw_spin_unlock+0x75/0x190 [17192.152697] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17192.155198] ? do_wait_intr+0xf0/0xf0 [17192.156440] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17192.158274] mdc_close+0x53b/0xd70 [mdc] [17192.159801] lmv_close+0x18d/0x490 [lmv] [17192.161503] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17192.163323] ll_md_real_close+0xa5/0x2a0 [lustre] [17192.165082] ll_md_close+0x274/0x800 [lustre] [17192.166822] ll_file_release+0x128/0xc50 [lustre] [17192.168889] __fput+0x10d/0x440 [17192.170178] ____fput+0x12/0x20 [17192.171150] task_work_run+0xc6/0x110 [17192.172247] exit_to_usermode_loop+0x1dd/0x1f0 [17192.173730] do_syscall_64+0x3d6/0x440 [17192.176297] entry_SYSCALL_64_after_hwframe+0x49/0xae [17192.178105] RIP: 0033:0x7fa749552cd8 [17192.179711] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17192.187935] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17192.190701] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17192.193380] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17192.195839] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17192.198480] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17192.200301] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17204.057251] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17204.060489] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17204.062479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17204.062559] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17204.062566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17204.062572] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17204.063178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17204.063189] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17204.063193] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17204.063197] RAX: 0000000000007f85 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17204.063200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17204.063202] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17204.063204] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17204.063207] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17204.063210] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17204.063213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17204.063215] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17204.063223] Call Trace: [17204.063228] [17204.063233] ? show_regs.cold.9+0x22/0x2f [17204.063241] ? watchdog_timer_fn.cold.6+0x62/0x10a [17204.063247] ? __hrtimer_run_queues+0x1c2/0x4d0 [17204.063252] ? watchdog+0x40/0x40 [17204.063256] ? hrtimer_interrupt+0x128/0x2c0 [17204.063261] ? smp_apic_timer_interrupt+0xa6/0x270 [17204.063268] ? apic_timer_interrupt+0xf/0x20 [17204.063283] [17204.063286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17204.063291] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17204.063296] do_raw_spin_lock+0x177/0x190 [17204.063301] _raw_spin_lock+0x21/0x30 [17204.063305] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17204.063766] process_one_work+0x2c8/0x700 [17204.063774] worker_thread+0x296/0x6e0 [17204.063779] ? rescuer_thread+0x570/0x570 [17204.063782] kthread+0x1d1/0x200 [17204.063787] ? set_kthread_struct+0x70/0x70 [17204.063791] ret_from_fork+0x1f/0x30 [17204.303853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17204.349669] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17204.358586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17204.370765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17204.375683] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17204.394275] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17204.403953] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17204.409232] RDX: 0000000000007f80 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17204.414883] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17204.424669] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17204.429848] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17204.435176] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17204.444259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17204.455280] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17204.467737] Call Trace: [17204.468856] [17204.476173] ? show_regs.cold.9+0x22/0x2f [17204.477803] ? watchdog_timer_fn.cold.6+0x62/0x10a [17204.488293] ? __hrtimer_run_queues+0x1c2/0x4d0 [17204.491893] ? watchdog+0x40/0x40 [17204.496414] ? hrtimer_interrupt+0x128/0x2c0 [17204.499752] ? smp_apic_timer_interrupt+0xa6/0x270 [17204.501491] ? apic_timer_interrupt+0xf/0x20 [17204.504648] [17204.506762] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17204.513235] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17204.522818] do_raw_spin_lock+0x177/0x190 [17204.528684] _raw_spin_lock+0x21/0x30 [17204.534703] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17204.538858] obd_mod_free+0x46/0x140 [ptlrpc] [17204.547231] mdc_commit_open+0x91/0xb0 [mdc] [17204.552262] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17204.558992] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17204.567211] after_reply+0xe2c/0x1780 [ptlrpc] [17204.573717] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17204.578453] ? __next_timer_interrupt+0x160/0x160 [17204.583301] ptlrpcd+0xa82/0xe40 [ptlrpc] [17204.591477] ? do_wait_intr+0xf0/0xf0 [17204.594153] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17204.603662] kthread+0x1d1/0x200 [17204.607752] ? set_kthread_struct+0x70/0x70 [17204.619476] ret_from_fork+0x1f/0x30 [17212.054383] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17212.066032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17212.112872] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17212.124405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17212.136673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17212.144927] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17212.159980] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17212.166550] RAX: 0000000000006b84 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17212.174058] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17212.179669] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17212.187709] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17212.191167] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17212.196464] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17212.203079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17212.211376] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17212.217864] Call Trace: [17212.220027] [17212.220870] ? show_regs.cold.9+0x22/0x2f [17212.223621] ? watchdog_timer_fn.cold.6+0x62/0x10a [17212.228331] ? __hrtimer_run_queues+0x1c2/0x4d0 [17212.233104] ? watchdog+0x40/0x40 [17212.236008] ? hrtimer_interrupt+0x128/0x2c0 [17212.240963] ? smp_apic_timer_interrupt+0xa6/0x270 [17212.246612] ? apic_timer_interrupt+0xf/0x20 [17212.250779] [17212.252083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17212.257621] do_raw_spin_lock+0x177/0x190 [17212.264086] _raw_spin_lock+0x21/0x30 [17212.270655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17212.279000] ? do_raw_spin_unlock+0x75/0x190 [17212.286017] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17212.294927] ? do_wait_intr+0xf0/0xf0 [17212.298299] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17212.304490] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17212.315072] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17212.326149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17212.337166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17212.339161] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17212.348764] mdc_intent_lock+0x353/0x7e0 [mdc] [17212.350457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17212.353341] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17212.356195] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17212.368634] lmv_intent_open+0x338/0x1150 [lmv] [17212.371625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17212.378669] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17212.384686] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17212.390151] ll_intent_lock+0xc5/0xaa0 [lustre] [17212.396044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17212.403041] ll_lookup_it+0xe0e/0x2270 [lustre] [17212.408390] ? do_raw_spin_unlock+0x75/0x190 [17212.412253] ? _raw_spin_unlock+0x12/0x30 [17212.416186] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17212.421851] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17212.428935] ll_atomic_open+0x36a/0xd70 [lustre] [17212.433473] lookup_open+0x356/0xb40 [17212.435611] do_last+0x313/0xfc0 [17212.438932] ? nd_jump_root+0xe5/0x160 [17212.443689] ? path_init+0x437/0x520 [17212.449147] path_openat+0xf7/0x500 [17212.452591] ? do_raw_spin_unlock+0x75/0x190 [17212.454085] do_filp_open+0x99/0x140 [17212.455259] ? getname_flags+0x6e/0x330 [17212.463456] ? __check_object_size+0xff/0x256 [17212.467902] ? do_raw_spin_unlock+0x75/0x190 [17212.473920] ? _raw_spin_unlock+0x12/0x30 [17212.475202] do_sys_openat2+0x2b4/0x410 [17212.489078] do_sys_open+0x73/0xa0 [17212.494936] __x64_sys_openat+0x24/0x30 [17212.496516] do_syscall_64+0xc1/0x440 [17212.497790] entry_SYSCALL_64_after_hwframe+0x49/0xae [17212.506926] RIP: 0033:0x7fa749552332 [17212.515298] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17212.544110] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17212.553000] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17212.557335] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17212.563098] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17212.571948] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17212.584991] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17216.993298] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15299s! [17216.999596] Showing busy workqueues and worker pools: [17217.004973] workqueue events: flags=0x0 [17217.011817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17217.017915] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17217.022893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17217.027919] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17217.035814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.040916] pending: kfree_rcu_monitor [17217.044768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17217.053246] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17217.061855] workqueue events_long: flags=0x0 [17217.063343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.081448] pending: lu_objects_flush [obdclass] [17217.087090] workqueue events_unbound: flags=0x2 [17217.094703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17217.096618] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17217.107221] workqueue events_power_efficient: flags=0x80 [17217.111562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.115951] pending: check_lifetime [17217.118957] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.123852] pending: neigh_periodic_work [17217.127330] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.135787] pending: neigh_periodic_work [17217.139069] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.143830] pending: do_cache_clean [17217.146514] workqueue mm_percpu_wq: flags=0x8 [17217.148714] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.151247] pending: vmstat_update [17217.157528] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.163919] pending: vmstat_update [17217.170966] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17217.179006] pending: vmstat_update [17217.184980] workqueue writeback: flags=0x4a [17217.194142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17217.202063] pending: wb_workfn, wb_workfn [17217.205030] workqueue xprtiod: flags=0xa [17217.209531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17217.213253] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17217.217951] workqueue ipv6_addrconf: flags=0x40008 [17217.219876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17217.223078] pending: addrconf_verify_work [17217.225112] workqueue knbd0-recv: flags=0x1a [17217.227011] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17217.229710] in-flight: 651:recv_work [17217.231522] workqueue ptlrpc_pinger: flags=0x2 [17217.233769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17217.235947] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17217.238578] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15300s workers=4 idle: 37 31846 28897 [17217.241737] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17207s workers=2 idle: 653 [17220.066920] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17220.074664] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17220.117602] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17220.128322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17220.134295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17220.142878] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17220.156172] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17220.166896] RAX: 0000000000005802 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17220.172231] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17220.179029] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17220.182633] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17220.185765] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17220.194833] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17220.201217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17220.206507] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17220.212044] Call Trace: [17220.213017] [17220.213789] ? show_regs.cold.9+0x22/0x2f [17220.215060] ? watchdog_timer_fn.cold.6+0x62/0x10a [17220.217012] ? __hrtimer_run_queues+0x1c2/0x4d0 [17220.218970] ? watchdog+0x40/0x40 [17220.220978] ? hrtimer_interrupt+0x128/0x2c0 [17220.222340] ? smp_apic_timer_interrupt+0xa6/0x270 [17220.224288] ? apic_timer_interrupt+0xf/0x20 [17220.227809] [17220.228846] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17220.231772] do_raw_spin_lock+0x177/0x190 [17220.234694] _raw_spin_lock+0x21/0x30 [17220.236051] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17220.238034] ? do_raw_spin_unlock+0x75/0x190 [17220.240048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17220.242615] ? do_wait_intr+0xf0/0xf0 [17220.244657] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17220.247253] mdc_close+0x53b/0xd70 [mdc] [17220.249376] lmv_close+0x18d/0x490 [lmv] [17220.252022] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17220.254613] ll_md_real_close+0xa5/0x2a0 [lustre] [17220.257788] ll_md_close+0x274/0x800 [lustre] [17220.262635] ll_file_release+0x128/0xc50 [lustre] [17220.266324] __fput+0x10d/0x440 [17220.268495] ____fput+0x12/0x20 [17220.273240] task_work_run+0xc6/0x110 [17220.274795] exit_to_usermode_loop+0x1dd/0x1f0 [17220.276962] do_syscall_64+0x3d6/0x440 [17220.278339] entry_SYSCALL_64_after_hwframe+0x49/0xae [17220.281090] RIP: 0033:0x7fa749552cd8 [17220.282902] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17220.291719] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17220.296421] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17220.303801] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17220.306609] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17220.313799] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17220.317999] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17232.057015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17232.058027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17232.060357] Modules linked in: [17232.063723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17232.063816] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17232.063826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17232.063829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17232.063847] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17232.063851] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17232.063856] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17232.063859] RDX: 00000000000000b6 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17232.063861] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17232.063864] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17232.063866] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17232.063870] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17232.063873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17232.063875] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17232.063884] Call Trace: [17232.063892] [17232.063900] ? show_regs.cold.9+0x22/0x2f [17232.063910] ? watchdog_timer_fn.cold.6+0x62/0x10a [17232.063919] ? __hrtimer_run_queues+0x1c2/0x4d0 [17232.063923] ? watchdog+0x40/0x40 [17232.063928] ? hrtimer_interrupt+0x128/0x2c0 [17232.063932] ? smp_apic_timer_interrupt+0xa6/0x270 [17232.063943] ? apic_timer_interrupt+0xf/0x20 [17232.063947] [17232.063950] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17232.063955] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17232.063960] do_raw_spin_lock+0x177/0x190 [17232.063965] _raw_spin_lock+0x21/0x30 [17232.063973] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17232.064286] obd_mod_free+0x46/0x140 [ptlrpc] [17232.064483] mdc_commit_open+0x91/0xb0 [mdc] [17232.064529] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17232.064720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17232.064912] after_reply+0xe2c/0x1780 [ptlrpc] [17232.065111] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17232.069210] ? __next_timer_interrupt+0x160/0x160 [17232.069223] ptlrpcd+0xa82/0xe40 [ptlrpc] [17232.069587] ? do_wait_intr+0xf0/0xf0 [17232.069600] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17232.071508] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17232.071583] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17232.071590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17232.071598] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17232.072083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17232.072096] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17232.072100] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17232.072105] RAX: 0000000000006886 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17232.072107] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17232.072109] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17232.072111] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17232.072114] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17232.072118] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17232.072120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17232.072123] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17232.072131] Call Trace: [17232.072138] [17232.072158] ? show_regs.cold.9+0x22/0x2f [17232.072170] ? watchdog_timer_fn.cold.6+0x62/0x10a [17232.072176] ? __hrtimer_run_queues+0x1c2/0x4d0 [17232.072181] ? watchdog+0x40/0x40 [17232.072185] ? hrtimer_interrupt+0x128/0x2c0 [17232.072190] ? smp_apic_timer_interrupt+0xa6/0x270 [17232.072199] ? apic_timer_interrupt+0xf/0x20 [17232.072205] [17232.072208] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17232.072213] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17232.072219] do_raw_spin_lock+0x177/0x190 [17232.072224] _raw_spin_lock+0x21/0x30 [17232.072229] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17232.072481] process_one_work+0x2c8/0x700 [17232.072489] worker_thread+0x296/0x6e0 [17232.072494] ? rescuer_thread+0x570/0x570 [17232.072497] kthread+0x1d1/0x200 [17232.072502] ? set_kthread_struct+0x70/0x70 [17232.072507] ret_from_fork+0x1f/0x30 [17232.492678] kthread+0x1d1/0x200 [17232.495962] ? set_kthread_struct+0x70/0x70 [17232.500623] ret_from_fork+0x1f/0x30 [17244.054077] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17244.056235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17244.082859] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17244.088000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17244.090816] RIP: 0010:native_safe_halt+0x2e/0x50 [17244.092547] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17244.106658] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17244.109840] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17244.114448] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [17244.118014] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17244.123787] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17244.126324] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17244.129208] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17244.131925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17244.134465] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17244.139620] Call Trace: [17244.140444] [17244.141464] ? show_regs.cold.9+0x22/0x2f [17244.143225] ? watchdog_timer_fn.cold.6+0x62/0x10a [17244.146889] ? __hrtimer_run_queues+0x1c2/0x4d0 [17244.149546] ? watchdog+0x40/0x40 [17244.151993] ? hrtimer_interrupt+0x128/0x2c0 [17244.153490] ? smp_apic_timer_interrupt+0xa6/0x270 [17244.155467] ? apic_timer_interrupt+0xf/0x20 [17244.157283] [17244.158373] ? native_safe_halt+0x2e/0x50 [17244.160200] kvm_wait+0xcd/0xe0 [17244.161200] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17244.163702] do_raw_spin_lock+0x177/0x190 [17244.165477] _raw_spin_lock+0x21/0x30 [17244.167900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17244.175402] ? do_raw_spin_unlock+0x75/0x190 [17244.177894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17244.181010] ? do_wait_intr+0xf0/0xf0 [17244.182853] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17244.185495] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17244.187827] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17244.191237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17244.194308] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17244.196909] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17244.200424] mdc_intent_lock+0x353/0x7e0 [mdc] [17244.205040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17244.207635] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17244.213339] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17244.216393] lmv_intent_open+0x338/0x1150 [lmv] [17244.218371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17244.221141] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17244.224201] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17244.226411] ll_intent_lock+0xc5/0xaa0 [lustre] [17244.230738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17244.233478] ll_lookup_it+0xe0e/0x2270 [lustre] [17244.235890] ? do_raw_spin_unlock+0x75/0x190 [17244.238776] ? _raw_spin_unlock+0x12/0x30 [17244.243395] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17244.246609] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17244.251846] ll_atomic_open+0x36a/0xd70 [lustre] [17244.255450] lookup_open+0x356/0xb40 [17244.259887] do_last+0x313/0xfc0 [17244.261442] ? nd_jump_root+0xe5/0x160 [17244.263253] ? path_init+0x437/0x520 [17244.266795] path_openat+0xf7/0x500 [17244.270001] ? do_raw_spin_unlock+0x75/0x190 [17244.272560] do_filp_open+0x99/0x140 [17244.275662] ? getname_flags+0x6e/0x330 [17244.278456] ? __check_object_size+0xff/0x256 [17244.280323] ? do_raw_spin_unlock+0x75/0x190 [17244.282569] ? _raw_spin_unlock+0x12/0x30 [17244.286574] do_sys_openat2+0x2b4/0x410 [17244.288444] do_sys_open+0x73/0xa0 [17244.290612] __x64_sys_openat+0x24/0x30 [17244.292280] do_syscall_64+0xc1/0x440 [17244.295626] entry_SYSCALL_64_after_hwframe+0x49/0xae [17244.299419] RIP: 0033:0x7fa749552332 [17244.300928] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17244.310786] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17244.316997] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17244.320253] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17244.323811] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17244.327397] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17244.331328] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17247.727989] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15330s! [17247.735916] Showing busy workqueues and worker pools: [17247.739328] workqueue events: flags=0x0 [17247.740544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17247.756156] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17247.769113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17247.776301] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17247.781196] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.789298] pending: kfree_rcu_monitor [17247.798152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17247.812294] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17247.836015] workqueue events_long: flags=0x0 [17247.842962] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.845770] pending: lu_objects_flush [obdclass] [17247.849296] workqueue events_unbound: flags=0x2 [17247.852913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17247.864469] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17247.883087] workqueue events_power_efficient: flags=0x80 [17247.887739] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.891676] pending: check_lifetime [17247.903746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.910448] pending: neigh_periodic_work [17247.915087] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.920258] pending: neigh_periodic_work [17247.924102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.929052] pending: do_cache_clean [17247.936164] workqueue mm_percpu_wq: flags=0x8 [17247.938443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.948098] pending: vmstat_update [17247.949482] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.953846] pending: vmstat_update [17247.958235] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17247.963151] pending: vmstat_update [17247.966046] workqueue writeback: flags=0x4a [17247.974019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17247.983104] pending: wb_workfn, wb_workfn [17247.984678] workqueue xprtiod: flags=0xa [17247.994086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17248.003620] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17248.015510] workqueue ipv6_addrconf: flags=0x40008 [17248.024278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17248.030333] pending: addrconf_verify_work [17248.041107] workqueue knbd0-recv: flags=0x1a [17248.050843] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17248.059471] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17248.060743] in-flight: 651:recv_work [17248.060831] workqueue ptlrpc_pinger: flags=0x2 [17248.060837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17248.060847] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17248.066247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17248.066340] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17248.066349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17248.066352] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17248.066367] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17248.066371] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17248.066381] RAX: 0000000000003150 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17248.066384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17248.066386] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17248.066389] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17248.066391] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17248.066395] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17248.066398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17248.066400] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17248.066409] Call Trace: [17248.066414] [17248.066420] ? show_regs.cold.9+0x22/0x2f [17248.066429] ? watchdog_timer_fn.cold.6+0x62/0x10a [17248.066436] ? __hrtimer_run_queues+0x1c2/0x4d0 [17248.066441] ? watchdog+0x40/0x40 [17248.066445] ? hrtimer_interrupt+0x128/0x2c0 [17248.066450] ? smp_apic_timer_interrupt+0xa6/0x270 [17248.066458] ? apic_timer_interrupt+0xf/0x20 [17248.066463] [17248.066466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17248.066473] do_raw_spin_lock+0x177/0x190 [17248.066478] _raw_spin_lock+0x21/0x30 [17248.066483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17248.067246] ? do_raw_spin_unlock+0x75/0x190 [17248.067252] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17248.067540] ? do_wait_intr+0xf0/0xf0 [17248.067546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17248.067934] mdc_close+0x53b/0xd70 [mdc] [17248.067979] lmv_close+0x18d/0x490 [lmv] [17248.068093] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17248.068228] ll_md_real_close+0xa5/0x2a0 [lustre] [17248.068312] ll_md_close+0x274/0x800 [lustre] [17248.068450] ll_file_release+0x128/0xc50 [lustre] [17248.068534] __fput+0x10d/0x440 [17248.068543] ____fput+0x12/0x20 [17248.068558] task_work_run+0xc6/0x110 [17248.068567] exit_to_usermode_loop+0x1dd/0x1f0 [17248.068576] do_syscall_64+0x3d6/0x440 [17248.068580] entry_SYSCALL_64_after_hwframe+0x49/0xae [17248.068641] RIP: 0033:0x7fa749552cd8 [17248.068683] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17248.068688] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17248.068698] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17248.068700] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17248.068702] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17248.068705] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17248.068707] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17248.505274] [17248.506052] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15331s workers=4 idle: 37 31846 28897 [17248.509494] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17238s workers=2 idle: 653 [17260.057020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17260.059705] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17260.061139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17260.061211] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17260.061217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17260.061223] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17260.061716] RIP: 0010:native_safe_halt+0x2e/0x50 [17260.061726] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17260.061730] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17260.061734] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17260.061737] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17260.061739] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17260.061741] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17260.061743] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17260.061746] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17260.061749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17260.061751] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17260.061761] Call Trace: [17260.061766] [17260.061772] ? show_regs.cold.9+0x22/0x2f [17260.061780] ? watchdog_timer_fn.cold.6+0x62/0x10a [17260.061785] ? __hrtimer_run_queues+0x1c2/0x4d0 [17260.061789] ? watchdog+0x40/0x40 [17260.061793] ? hrtimer_interrupt+0x128/0x2c0 [17260.061796] ? smp_apic_timer_interrupt+0xa6/0x270 [17260.061802] ? apic_timer_interrupt+0xf/0x20 [17260.061806] [17260.061808] ? native_safe_halt+0x2e/0x50 [17260.061812] kvm_wait+0xcd/0xe0 [17260.061818] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17260.061824] do_raw_spin_lock+0x177/0x190 [17260.061828] _raw_spin_lock+0x21/0x30 [17260.061832] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17260.062250] process_one_work+0x2c8/0x700 [17260.062257] worker_thread+0x296/0x6e0 [17260.062262] ? rescuer_thread+0x570/0x570 [17260.062265] kthread+0x1d1/0x200 [17260.062270] ? set_kthread_struct+0x70/0x70 [17260.062273] ret_from_fork+0x1f/0x30 [17260.435898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17260.481863] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17260.497908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17260.517132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17260.520293] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17260.538728] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17260.549224] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17260.559285] RDX: 00000000000036fb RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17260.568224] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17260.588446] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17260.600987] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17260.608726] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17260.611995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17260.617356] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17260.622192] Call Trace: [17260.623099] [17260.625962] ? show_regs.cold.9+0x22/0x2f [17260.635601] ? watchdog_timer_fn.cold.6+0x62/0x10a [17260.645025] ? __hrtimer_run_queues+0x1c2/0x4d0 [17260.653099] ? watchdog+0x40/0x40 [17260.668667] ? hrtimer_interrupt+0x128/0x2c0 [17260.673923] ? smp_apic_timer_interrupt+0xa6/0x270 [17260.687054] ? apic_timer_interrupt+0xf/0x20 [17260.698607] [17260.699341] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17260.701272] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17260.708704] do_raw_spin_lock+0x177/0x190 [17260.710021] _raw_spin_lock+0x21/0x30 [17260.721448] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17260.728023] obd_mod_free+0x46/0x140 [ptlrpc] [17260.734163] mdc_commit_open+0x91/0xb0 [mdc] [17260.738786] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17260.742157] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17260.754052] after_reply+0xe2c/0x1780 [ptlrpc] [17260.760697] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17260.770979] ? __next_timer_interrupt+0x160/0x160 [17260.780061] ptlrpcd+0xa82/0xe40 [ptlrpc] [17260.792807] ? do_wait_intr+0xf0/0xf0 [17260.803895] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17260.806814] kthread+0x1d1/0x200 [17260.811740] ? set_kthread_struct+0x70/0x70 [17260.814799] ret_from_fork+0x1f/0x30 [17274.024028] rcu: INFO: rcu_sched self-detected stall on CPU [17274.026661] rcu: 2-...!: (9055290 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [17274.031380] (t=15360335 jiffies g=258701 q=5693) [17274.033347] rcu: rcu_sched kthread starved for 15357130 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [17274.037650] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17274.041894] rcu: RCU grace-period kthread stack dump: [17274.044437] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17274.048901] Call Trace: [17274.049979] __schedule+0x351/0xcb0 [17274.051481] schedule+0xc0/0x180 [17274.052827] schedule_timeout+0xb4/0x190 [17274.054528] ? do_raw_spin_unlock+0x75/0x190 [17274.056321] ? __next_timer_interrupt+0x160/0x160 [17274.058530] ? prepare_to_swait_event+0xf3/0x140 [17274.060357] rcu_gp_kthread+0x8dd/0x15b0 [17274.061871] ? force_qs_rnp+0x2c0/0x2c0 [17274.063463] kthread+0x1d1/0x200 [17274.064942] ? set_kthread_struct+0x70/0x70 [17274.066846] ret_from_fork+0x1f/0x30 [17274.068510] rcu: Stack dump where RCU GP kthread last ran: [17274.070714] Sending NMI from CPU 2 to CPUs 3: [17274.072402] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [17274.073378] NMI backtrace for cpu 2 [17274.075646] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17274.079934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17274.084481] Call Trace: [17274.085571] [17274.086361] ? dump_stack+0xbb/0x10e [17274.087508] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17274.089446] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17274.091676] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17274.093960] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17274.095937] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17274.098752] ? rcu_dump_cpu_stacks+0x11b/0x171 [17274.100270] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17274.102449] ? update_process_times+0x6d/0xc0 [17274.103990] ? tick_sched_handle.isra.18+0x27/0x90 [17274.106314] ? tick_sched_timer+0x53/0xb0 [17274.107677] ? __hrtimer_run_queues+0x1c2/0x4d0 [17274.109653] ? tick_sched_handle.isra.18+0x90/0x90 [17274.111801] ? hrtimer_interrupt+0x128/0x2c0 [17274.113930] ? smp_apic_timer_interrupt+0xa6/0x270 [17274.116731] ? apic_timer_interrupt+0xf/0x20 [17274.118438] [17274.119336] ? native_safe_halt+0x2e/0x50 [17274.120675] ? kvm_wait+0xcd/0xe0 [17274.121769] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17274.123668] ? do_raw_spin_lock+0x177/0x190 [17274.125774] ? _raw_spin_lock+0x21/0x30 [17274.127443] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17274.130261] ? obd_mod_free+0x46/0x140 [ptlrpc] [17274.132359] ? mdc_commit_open+0x91/0xb0 [mdc] [17274.133997] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17274.136729] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17274.138842] ? after_reply+0xe2c/0x1780 [ptlrpc] [17274.141342] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17274.143931] ? __next_timer_interrupt+0x160/0x160 [17274.145660] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17274.148065] ? do_wait_intr+0xf0/0xf0 [17274.149655] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17274.151709] ? kthread+0x1d1/0x200 [17274.152855] ? set_kthread_struct+0x70/0x70 [17274.155253] ? ret_from_fork+0x1f/0x30 [17276.054074] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17276.057876] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17276.093864] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17276.104887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17276.112420] RIP: 0010:native_safe_halt+0x2e/0x50 [17276.115373] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17276.128936] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17276.134069] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17276.138255] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [17276.143275] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17276.147739] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17276.150947] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17276.155918] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17276.160904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17276.162825] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17276.165080] Call Trace: [17276.168690] [17276.169709] ? show_regs.cold.9+0x22/0x2f [17276.171652] ? watchdog_timer_fn.cold.6+0x62/0x10a [17276.180330] ? __hrtimer_run_queues+0x1c2/0x4d0 [17276.181746] ? watchdog+0x40/0x40 [17276.185802] ? hrtimer_interrupt+0x128/0x2c0 [17276.188848] ? smp_apic_timer_interrupt+0xa6/0x270 [17276.191332] ? apic_timer_interrupt+0xf/0x20 [17276.192842] [17276.193547] ? native_safe_halt+0x2e/0x50 [17276.195733] kvm_wait+0xcd/0xe0 [17276.197231] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17276.200564] do_raw_spin_lock+0x177/0x190 [17276.202820] _raw_spin_lock+0x21/0x30 [17276.208853] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17276.217239] ? do_raw_spin_unlock+0x75/0x190 [17276.222747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17276.231949] ? do_wait_intr+0xf0/0xf0 [17276.236999] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17276.241655] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17276.248463] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17276.253292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17276.257809] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17276.261885] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17276.265791] mdc_intent_lock+0x353/0x7e0 [mdc] [17276.269765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17276.271771] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17276.278632] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17276.282326] lmv_intent_open+0x338/0x1150 [lmv] [17276.284435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17276.286934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17276.292554] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17276.294767] ll_intent_lock+0xc5/0xaa0 [lustre] [17276.296279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17276.300813] ll_lookup_it+0xe0e/0x2270 [lustre] [17276.302856] ? do_raw_spin_unlock+0x75/0x190 [17276.309944] ? _raw_spin_unlock+0x12/0x30 [17276.314337] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17276.318883] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17276.325584] ll_atomic_open+0x36a/0xd70 [lustre] [17276.327647] lookup_open+0x356/0xb40 [17276.332604] do_last+0x313/0xfc0 [17276.335566] ? nd_jump_root+0xe5/0x160 [17276.339852] ? path_init+0x437/0x520 [17276.343363] path_openat+0xf7/0x500 [17276.345385] ? do_raw_spin_unlock+0x75/0x190 [17276.347592] do_filp_open+0x99/0x140 [17276.349353] ? getname_flags+0x6e/0x330 [17276.351306] ? __check_object_size+0xff/0x256 [17276.353667] ? do_raw_spin_unlock+0x75/0x190 [17276.355878] ? _raw_spin_unlock+0x12/0x30 [17276.357902] do_sys_openat2+0x2b4/0x410 [17276.359836] do_sys_open+0x73/0xa0 [17276.361529] __x64_sys_openat+0x24/0x30 [17276.363523] do_syscall_64+0xc1/0x440 [17276.365493] entry_SYSCALL_64_after_hwframe+0x49/0xae [17276.368421] RIP: 0033:0x7fa749552332 [17276.371537] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17276.384425] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17276.391650] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17276.402282] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17276.405657] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17276.409906] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17276.415674] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17278.946578] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15361s! [17278.949863] Showing busy workqueues and worker pools: [17278.962591] workqueue events: flags=0x0 [17278.964020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17278.970798] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17278.978150] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17278.982826] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17278.990784] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.001440] pending: kfree_rcu_monitor [17279.002981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17279.010341] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17279.016184] workqueue events_long: flags=0x0 [17279.024275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.029462] pending: lu_objects_flush [obdclass] [17279.039988] workqueue events_unbound: flags=0x2 [17279.045934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17279.052316] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17279.064891] workqueue events_power_efficient: flags=0x80 [17279.070268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.071899] pending: check_lifetime [17279.075257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.078623] pending: neigh_periodic_work [17279.080113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.084600] pending: neigh_periodic_work [17279.086823] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.099501] pending: do_cache_clean [17279.107305] workqueue mm_percpu_wq: flags=0x8 [17279.110732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.112513] pending: vmstat_update [17279.117545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.120011] pending: vmstat_update [17279.123976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17279.129286] pending: vmstat_update [17279.135803] workqueue writeback: flags=0x4a [17279.142825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17279.145083] pending: wb_workfn, wb_workfn [17279.155816] workqueue xprtiod: flags=0xa [17279.157243] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17279.167903] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17279.179167] workqueue ipv6_addrconf: flags=0x40008 [17279.186106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17279.191303] pending: addrconf_verify_work [17279.195265] workqueue knbd0-recv: flags=0x1a [17279.199634] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17279.205495] in-flight: 651:recv_work [17279.210257] workqueue ptlrpc_pinger: flags=0x2 [17279.213739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17279.217755] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17279.223315] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15362s workers=4 idle: 37 31846 28897 [17279.229136] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17269s workers=2 idle: 653 [17288.059894] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17288.068856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17288.102506] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17288.109963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17288.116794] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17288.119953] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17288.125590] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17288.144057] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17288.154053] RAX: 0000000000007e8f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17288.157753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17288.161286] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17288.164577] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17288.168112] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17288.171585] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17288.175713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17288.178588] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17288.185307] Call Trace: [17288.187267] [17288.188486] ? show_regs.cold.9+0x22/0x2f [17288.191942] ? watchdog_timer_fn.cold.6+0x62/0x10a [17288.195981] ? __hrtimer_run_queues+0x1c2/0x4d0 [17288.200208] ? watchdog+0x40/0x40 [17288.203491] ? hrtimer_interrupt+0x128/0x2c0 [17288.207673] ? smp_apic_timer_interrupt+0xa6/0x270 [17288.210757] ? apic_timer_interrupt+0xf/0x20 [17288.214305] [17288.214819] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17288.220977] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17288.226560] do_raw_spin_lock+0x177/0x190 [17288.229547] _raw_spin_lock+0x21/0x30 [17288.232839] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17288.236863] process_one_work+0x2c8/0x700 [17288.240997] worker_thread+0x296/0x6e0 [17288.245432] ? rescuer_thread+0x570/0x570 [17288.250070] kthread+0x1d1/0x200 [17288.251959] ? set_kthread_struct+0x70/0x70 [17288.254556] ret_from_fork+0x1f/0x30 [17300.059046] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17300.059292] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17300.062804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17300.062906] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17300.062913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17300.062916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17300.062933] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17300.062940] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17300.062945] RAX: 0000000000005a19 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17300.062948] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17300.062951] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17300.062953] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17300.062967] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17300.062971] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17300.062975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17300.062977] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17300.062994] Call Trace: [17300.062999] [17300.063005] ? show_regs.cold.9+0x22/0x2f [17300.063015] ? watchdog_timer_fn.cold.6+0x62/0x10a [17300.063021] ? __hrtimer_run_queues+0x1c2/0x4d0 [17300.063025] ? watchdog+0x40/0x40 [17300.063030] ? hrtimer_interrupt+0x128/0x2c0 [17300.063039] ? smp_apic_timer_interrupt+0xa6/0x270 [17300.063051] ? apic_timer_interrupt+0xf/0x20 [17300.063056] [17300.063059] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17300.063066] do_raw_spin_lock+0x177/0x190 [17300.063070] _raw_spin_lock+0x21/0x30 [17300.063076] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17300.063919] ? do_raw_spin_unlock+0x75/0x190 [17300.063926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17300.064421] ? do_wait_intr+0xf0/0xf0 [17300.064430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17300.069913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17300.070077] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17300.070089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17300.070092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17300.070111] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17300.070115] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17300.070120] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17300.070122] RDX: 0000000000000f9d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17300.070124] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17300.070130] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17300.070133] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17300.070137] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17300.070140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17300.070142] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17300.070151] Call Trace: [17300.070156] [17300.070166] ? show_regs.cold.9+0x22/0x2f [17300.070180] ? watchdog_timer_fn.cold.6+0x62/0x10a [17300.070186] ? __hrtimer_run_queues+0x1c2/0x4d0 [17300.070190] ? watchdog+0x40/0x40 [17300.070198] ? hrtimer_interrupt+0x128/0x2c0 [17300.070203] ? smp_apic_timer_interrupt+0xa6/0x270 [17300.070210] ? apic_timer_interrupt+0xf/0x20 [17300.070219] [17300.070222] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17300.070227] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17300.070232] do_raw_spin_lock+0x177/0x190 [17300.070254] _raw_spin_lock+0x21/0x30 [17300.070264] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17300.071005] obd_mod_free+0x46/0x140 [ptlrpc] [17300.071265] mdc_commit_open+0x91/0xb0 [mdc] [17300.100342] mdc_close+0x53b/0xd70 [mdc] [17300.100400] lmv_close+0x18d/0x490 [lmv] [17300.100438] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17300.100603] ll_md_real_close+0xa5/0x2a0 [lustre] [17300.100690] ll_md_close+0x274/0x800 [lustre] [17300.100776] ll_file_release+0x128/0xc50 [lustre] [17300.100867] __fput+0x10d/0x440 [17300.100878] ____fput+0x12/0x20 [17300.100883] task_work_run+0xc6/0x110 [17300.100891] exit_to_usermode_loop+0x1dd/0x1f0 [17300.100898] do_syscall_64+0x3d6/0x440 [17300.100903] entry_SYSCALL_64_after_hwframe+0x49/0xae [17300.100912] RIP: 0033:0x7fa749552cd8 [17300.100925] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17300.100929] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17300.100943] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17300.100945] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17300.100947] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17300.100949] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17300.100951] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17300.376519] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17300.379401] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17300.382786] after_reply+0xe2c/0x1780 [ptlrpc] [17300.385396] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17300.386809] ? __next_timer_interrupt+0x160/0x160 [17300.390861] ptlrpcd+0xa82/0xe40 [ptlrpc] [17300.393248] ? do_wait_intr+0xf0/0xf0 [17300.395846] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17300.399052] kthread+0x1d1/0x200 [17300.400479] ? set_kthread_struct+0x70/0x70 [17300.402821] ret_from_fork+0x1f/0x30 [17304.054439] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17304.062131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17304.085816] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17304.091905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17304.097086] RIP: 0010:native_safe_halt+0x2e/0x50 [17304.099087] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17304.112151] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17304.116247] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17304.118856] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d7382034154 [17304.122447] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17304.126644] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17304.130169] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17304.134299] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17304.138595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17304.142621] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17304.146113] Call Trace: [17304.147299] [17304.148344] ? show_regs.cold.9+0x22/0x2f [17304.151000] ? watchdog_timer_fn.cold.6+0x62/0x10a [17304.154378] ? __hrtimer_run_queues+0x1c2/0x4d0 [17304.157211] ? watchdog+0x40/0x40 [17304.158984] ? hrtimer_interrupt+0x128/0x2c0 [17304.160991] ? smp_apic_timer_interrupt+0xa6/0x270 [17304.162302] ? apic_timer_interrupt+0xf/0x20 [17304.164966] [17304.166063] ? native_safe_halt+0x2e/0x50 [17304.168680] kvm_wait+0xcd/0xe0 [17304.170099] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17304.171536] do_raw_spin_lock+0x177/0x190 [17304.174667] _raw_spin_lock+0x21/0x30 [17304.177462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17304.180793] ? do_raw_spin_unlock+0x75/0x190 [17304.184700] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17304.188093] ? do_wait_intr+0xf0/0xf0 [17304.190633] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17304.197370] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17304.203847] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17304.206133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17304.210120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17304.214111] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17304.218973] mdc_intent_lock+0x353/0x7e0 [mdc] [17304.224059] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17304.226050] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17304.235491] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17304.237591] lmv_intent_open+0x338/0x1150 [lmv] [17304.243006] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17304.246193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17304.251394] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17304.253330] ll_intent_lock+0xc5/0xaa0 [lustre] [17304.255921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17304.257623] ll_lookup_it+0xe0e/0x2270 [lustre] [17304.260498] ? do_raw_spin_unlock+0x75/0x190 [17304.262305] ? _raw_spin_unlock+0x12/0x30 [17304.266794] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17304.268396] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17304.274336] ll_atomic_open+0x36a/0xd70 [lustre] [17304.286035] lookup_open+0x356/0xb40 [17304.287370] do_last+0x313/0xfc0 [17304.288295] ? nd_jump_root+0xe5/0x160 [17304.294542] ? path_init+0x437/0x520 [17304.296758] path_openat+0xf7/0x500 [17304.297737] ? do_raw_spin_unlock+0x75/0x190 [17304.301326] do_filp_open+0x99/0x140 [17304.303216] ? getname_flags+0x6e/0x330 [17304.307243] ? __check_object_size+0xff/0x256 [17304.308446] ? do_raw_spin_unlock+0x75/0x190 [17304.311965] ? _raw_spin_unlock+0x12/0x30 [17304.313376] do_sys_openat2+0x2b4/0x410 [17304.319057] do_sys_open+0x73/0xa0 [17304.320715] __x64_sys_openat+0x24/0x30 [17304.325589] do_syscall_64+0xc1/0x440 [17304.327058] entry_SYSCALL_64_after_hwframe+0x49/0xae [17304.329048] RIP: 0033:0x7fa749552332 [17304.330448] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17304.341972] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17304.348065] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17304.351361] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17304.354061] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17304.358848] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17304.365905] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17309.667525] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15392s! [17309.671219] Showing busy workqueues and worker pools: [17309.673721] workqueue events: flags=0x0 [17309.675298] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17309.680845] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17309.686207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17309.691209] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17309.695910] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.698961] pending: kfree_rcu_monitor [17309.701150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17309.704210] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17309.709031] workqueue events_long: flags=0x0 [17309.710859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.713912] pending: lu_objects_flush [obdclass] [17309.717923] workqueue events_unbound: flags=0x2 [17309.719931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17309.722936] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17309.730436] workqueue events_power_efficient: flags=0x80 [17309.738300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.745211] pending: check_lifetime [17309.749255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.752463] pending: neigh_periodic_work [17309.754424] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.759748] pending: neigh_periodic_work [17309.761800] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.765022] pending: do_cache_clean [17309.766595] workqueue mm_percpu_wq: flags=0x8 [17309.769292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.772327] pending: vmstat_update [17309.774318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.778728] pending: vmstat_update [17309.779912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17309.782267] pending: vmstat_update [17309.783821] workqueue writeback: flags=0x4a [17309.785345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17309.787982] pending: wb_workfn, wb_workfn [17309.789918] workqueue xprtiod: flags=0xa [17309.791481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17309.796949] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17309.800206] workqueue ipv6_addrconf: flags=0x40008 [17309.802030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17309.805119] pending: addrconf_verify_work [17309.807117] workqueue knbd0-recv: flags=0x1a [17309.809271] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17309.812248] in-flight: 651:recv_work [17309.814061] workqueue ptlrpc_pinger: flags=0x2 [17309.816383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17309.819209] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17309.822818] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15392s workers=4 idle: 37 31846 28897 [17309.825985] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17299s workers=2 idle: 653 [17316.057461] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17316.064691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17316.100923] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17316.106514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17316.115074] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17316.123134] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17316.127182] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17316.141542] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17316.145136] RAX: 00000000000060bf RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17316.148241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17316.152349] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17316.155514] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17316.158825] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17316.162115] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17316.166801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17316.169191] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17316.172749] Call Trace: [17316.174534] [17316.175564] ? show_regs.cold.9+0x22/0x2f [17316.177381] ? watchdog_timer_fn.cold.6+0x62/0x10a [17316.182147] ? __hrtimer_run_queues+0x1c2/0x4d0 [17316.184444] ? watchdog+0x40/0x40 [17316.185635] ? hrtimer_interrupt+0x128/0x2c0 [17316.186900] ? smp_apic_timer_interrupt+0xa6/0x270 [17316.191233] ? apic_timer_interrupt+0xf/0x20 [17316.195009] [17316.196174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17316.199503] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17316.202504] do_raw_spin_lock+0x177/0x190 [17316.204232] _raw_spin_lock+0x21/0x30 [17316.206002] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17316.208230] process_one_work+0x2c8/0x700 [17316.212815] worker_thread+0x296/0x6e0 [17316.215556] ? rescuer_thread+0x570/0x570 [17316.218483] kthread+0x1d1/0x200 [17316.221453] ? set_kthread_struct+0x70/0x70 [17316.224376] ret_from_fork+0x1f/0x30 [17328.058032] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [17328.061576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17328.066712] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17328.079372] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17328.079387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17328.079390] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17328.079405] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17328.079409] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17328.079415] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17328.079417] RDX: 000000000000432b RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17328.079420] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17328.079422] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17328.079425] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17328.079428] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17328.079431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17328.079433] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17328.079442] Call Trace: [17328.079447] [17328.079454] ? show_regs.cold.9+0x22/0x2f [17328.079463] ? watchdog_timer_fn.cold.6+0x62/0x10a [17328.079481] ? __hrtimer_run_queues+0x1c2/0x4d0 [17328.079486] ? watchdog+0x40/0x40 [17328.079490] ? hrtimer_interrupt+0x128/0x2c0 [17328.079494] ? smp_apic_timer_interrupt+0xa6/0x270 [17328.079503] ? apic_timer_interrupt+0xf/0x20 [17328.079508] [17328.079512] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17328.079517] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17328.079521] do_raw_spin_lock+0x177/0x190 [17328.079525] _raw_spin_lock+0x21/0x30 [17328.079533] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17328.079900] obd_mod_free+0x46/0x140 [ptlrpc] [17328.080120] mdc_commit_open+0x91/0xb0 [mdc] [17328.080292] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17328.080508] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17328.080779] after_reply+0xe2c/0x1780 [ptlrpc] [17328.080988] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17328.081255] ? __next_timer_interrupt+0x160/0x160 [17328.081266] ptlrpcd+0xa82/0xe40 [ptlrpc] [17328.081490] ? do_wait_intr+0xf0/0xf0 [17328.081499] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17328.081781] kthread+0x1d1/0x200 [17328.081787] ? set_kthread_struct+0x70/0x70 [17328.081790] ret_from_fork+0x1f/0x30 [17328.193966] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17328.220637] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17328.225888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17328.229379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17328.232311] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17328.240200] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17328.242849] RAX: 0000000000003785 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17328.246219] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17328.249162] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17328.252719] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17328.254636] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17328.257136] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17328.260790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17328.263852] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17328.267390] Call Trace: [17328.268673] [17328.270409] ? show_regs.cold.9+0x22/0x2f [17328.272861] ? watchdog_timer_fn.cold.6+0x62/0x10a [17328.275783] ? __hrtimer_run_queues+0x1c2/0x4d0 [17328.277457] ? watchdog+0x40/0x40 [17328.279761] ? hrtimer_interrupt+0x128/0x2c0 [17328.281521] ? smp_apic_timer_interrupt+0xa6/0x270 [17328.284253] ? apic_timer_interrupt+0xf/0x20 [17328.286016] [17328.288477] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17328.292081] do_raw_spin_lock+0x177/0x190 [17328.293656] _raw_spin_lock+0x21/0x30 [17328.295815] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17328.299448] ? do_raw_spin_unlock+0x75/0x190 [17328.301051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17328.305086] ? do_wait_intr+0xf0/0xf0 [17328.307616] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17328.315421] mdc_close+0x53b/0xd70 [mdc] [17328.317945] lmv_close+0x18d/0x490 [lmv] [17328.320991] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17328.326849] ll_md_real_close+0xa5/0x2a0 [lustre] [17328.328622] ll_md_close+0x274/0x800 [lustre] [17328.331041] ll_file_release+0x128/0xc50 [lustre] [17328.333714] __fput+0x10d/0x440 [17328.335082] ____fput+0x12/0x20 [17328.336257] task_work_run+0xc6/0x110 [17328.340035] exit_to_usermode_loop+0x1dd/0x1f0 [17328.342566] do_syscall_64+0x3d6/0x440 [17328.344773] entry_SYSCALL_64_after_hwframe+0x49/0xae [17328.346686] RIP: 0033:0x7fa749552cd8 [17328.348318] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17328.369007] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17328.372260] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17328.375981] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17328.378913] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17328.386164] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17328.391106] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17336.058625] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17336.063475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17336.105242] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17336.110977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17336.115683] RIP: 0010:native_safe_halt+0x2e/0x50 [17336.117590] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17336.128801] RSP: 0018:ffffb893c14934f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17336.138668] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17336.145317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17336.151403] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17336.155622] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17336.159200] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17336.166390] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17336.170846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17336.175585] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17336.179642] Call Trace: [17336.181027] [17336.182012] ? show_regs.cold.9+0x22/0x2f [17336.183251] ? watchdog_timer_fn.cold.6+0x62/0x10a [17336.185010] ? __hrtimer_run_queues+0x1c2/0x4d0 [17336.188277] ? watchdog+0x40/0x40 [17336.189657] ? hrtimer_interrupt+0x128/0x2c0 [17336.191464] ? smp_apic_timer_interrupt+0xa6/0x270 [17336.193580] ? apic_timer_interrupt+0xf/0x20 [17336.195496] [17336.196285] ? native_safe_halt+0x2e/0x50 [17336.198019] kvm_wait+0xcd/0xe0 [17336.199466] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17336.201676] do_raw_spin_lock+0x177/0x190 [17336.203519] _raw_spin_lock+0x21/0x30 [17336.204958] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17336.207613] ? do_raw_spin_unlock+0x75/0x190 [17336.209416] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17336.211673] ? do_wait_intr+0xf0/0xf0 [17336.213445] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17336.215900] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17336.222683] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17336.225912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17336.229333] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17336.234366] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17336.238331] mdc_intent_lock+0x353/0x7e0 [mdc] [17336.240466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17336.243082] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17336.250354] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17336.252219] lmv_intent_open+0x338/0x1150 [lmv] [17336.260263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17336.264956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17336.269215] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17336.271181] ll_intent_lock+0xc5/0xaa0 [lustre] [17336.272911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17336.277938] ll_lookup_it+0xe0e/0x2270 [lustre] [17336.279651] ? do_raw_spin_unlock+0x75/0x190 [17336.285255] ? _raw_spin_unlock+0x12/0x30 [17336.293409] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17336.303472] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17336.310834] ll_atomic_open+0x36a/0xd70 [lustre] [17336.316349] lookup_open+0x356/0xb40 [17336.322477] do_last+0x313/0xfc0 [17336.323870] ? nd_jump_root+0xe5/0x160 [17336.328171] ? path_init+0x437/0x520 [17336.329504] path_openat+0xf7/0x500 [17336.330719] ? do_raw_spin_unlock+0x75/0x190 [17336.336180] do_filp_open+0x99/0x140 [17336.340659] ? getname_flags+0x6e/0x330 [17336.343708] ? __check_object_size+0xff/0x256 [17336.349699] ? do_raw_spin_unlock+0x75/0x190 [17336.351622] ? _raw_spin_unlock+0x12/0x30 [17336.353634] do_sys_openat2+0x2b4/0x410 [17336.361043] do_sys_open+0x73/0xa0 [17336.362269] __x64_sys_openat+0x24/0x30 [17336.372491] do_syscall_64+0xc1/0x440 [17336.374279] entry_SYSCALL_64_after_hwframe+0x49/0xae [17336.388303] RIP: 0033:0x7fa749552332 [17336.391617] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17336.410253] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17336.418886] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17336.427286] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17336.433580] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17336.438202] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17336.442829] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17339.873044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15422s! [17339.876231] Showing busy workqueues and worker pools: [17339.878183] workqueue events: flags=0x0 [17339.881411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17339.883479] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17339.891640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17339.898836] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17339.907970] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17339.912661] pending: kfree_rcu_monitor [17339.914740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17339.916425] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17339.926137] workqueue events_long: flags=0x0 [17339.931156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17339.936623] pending: lu_objects_flush [obdclass] [17339.940885] workqueue events_unbound: flags=0x2 [17339.944115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17339.950127] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17339.957765] workqueue events_power_efficient: flags=0x80 [17339.961915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17339.968548] pending: check_lifetime [17339.974351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17339.978380] pending: neigh_periodic_work [17339.984923] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17339.989549] pending: neigh_periodic_work [17339.991876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17340.002774] pending: do_cache_clean [17340.005533] workqueue mm_percpu_wq: flags=0x8 [17340.010580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17340.013235] pending: vmstat_update [17340.019648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17340.024449] pending: vmstat_update [17340.027771] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17340.030687] pending: vmstat_update [17340.032077] workqueue writeback: flags=0x4a [17340.041194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17340.048327] pending: wb_workfn, wb_workfn [17340.051879] workqueue xprtiod: flags=0xa [17340.055531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17340.065304] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17340.070186] workqueue ipv6_addrconf: flags=0x40008 [17340.073941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17340.077510] pending: addrconf_verify_work [17340.082082] workqueue knbd0-recv: flags=0x1a [17340.084588] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17340.088937] in-flight: 651:recv_work [17340.090715] workqueue ptlrpc_pinger: flags=0x2 [17340.092798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17340.095313] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17340.102489] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15422s workers=4 idle: 37 31846 28897 [17340.107537] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17330s workers=2 idle: 653 [17344.058970] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17344.067428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17344.104985] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17344.112195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17344.123317] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17344.129455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17344.132816] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17344.144817] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17344.148392] RAX: 000000000000346e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17344.151339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17344.154559] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17344.157398] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17344.160299] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17344.162843] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17344.166741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17344.169752] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17344.174429] Call Trace: [17344.175727] [17344.176745] ? show_regs.cold.9+0x22/0x2f [17344.178716] ? watchdog_timer_fn.cold.6+0x62/0x10a [17344.180808] ? __hrtimer_run_queues+0x1c2/0x4d0 [17344.182494] ? watchdog+0x40/0x40 [17344.184465] ? hrtimer_interrupt+0x128/0x2c0 [17344.185886] ? smp_apic_timer_interrupt+0xa6/0x270 [17344.188810] ? apic_timer_interrupt+0xf/0x20 [17344.191779] [17344.192432] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17344.195815] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17344.198425] do_raw_spin_lock+0x177/0x190 [17344.201067] _raw_spin_lock+0x21/0x30 [17344.204509] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17344.207260] process_one_work+0x2c8/0x700 [17344.210764] worker_thread+0x296/0x6e0 [17344.213102] ? rescuer_thread+0x570/0x570 [17344.215757] kthread+0x1d1/0x200 [17344.218401] ? set_kthread_struct+0x70/0x70 [17344.221930] ret_from_fork+0x1f/0x30 [17356.058073] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [17356.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17356.061136] Modules linked in: lustre(O) [17356.066194] Modules linked in: [17356.066196] osp(O) ofd(O) [17356.067469] lustre(O) [17356.069246] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17356.069313] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17356.069322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17356.069325] RIP: 0010:native_safe_halt+0x2e/0x50 [17356.069338] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17356.069343] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17356.069347] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17356.069350] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17356.069352] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17356.069355] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17356.069358] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17356.069361] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17356.069364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17356.069366] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17356.069374] Call Trace: [17356.069381] [17356.069385] ? show_regs.cold.9+0x22/0x2f [17356.069396] ? watchdog_timer_fn.cold.6+0x62/0x10a [17356.069402] ? __hrtimer_run_queues+0x1c2/0x4d0 [17356.069406] ? watchdog+0x40/0x40 [17356.069410] ? hrtimer_interrupt+0x128/0x2c0 [17356.069413] ? smp_apic_timer_interrupt+0xa6/0x270 [17356.069420] ? apic_timer_interrupt+0xf/0x20 [17356.069424] [17356.069427] ? native_safe_halt+0x2e/0x50 [17356.069432] kvm_wait+0xcd/0xe0 [17356.069439] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17356.069446] do_raw_spin_lock+0x177/0x190 [17356.069450] _raw_spin_lock+0x21/0x30 [17356.069455] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17356.069717] obd_mod_free+0x46/0x140 [ptlrpc] [17356.069911] mdc_commit_open+0x91/0xb0 [mdc] [17356.069950] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17356.070162] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17356.070367] after_reply+0xe2c/0x1780 [ptlrpc] [17356.071110] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17356.071388] ? __next_timer_interrupt+0x160/0x160 [17356.071399] ptlrpcd+0xa82/0xe40 [ptlrpc] [17356.071625] ? do_wait_intr+0xf0/0xf0 [17356.071634] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17356.071919] kthread+0x1d1/0x200 [17356.071925] ? set_kthread_struct+0x70/0x70 [17356.071929] ret_from_fork+0x1f/0x30 [17356.348387] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17356.369014] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17356.373161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17356.377080] RIP: 0010:native_safe_halt+0x2e/0x50 [17356.379639] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17356.387466] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17356.390386] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17356.393168] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17356.397025] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17356.402226] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17356.407200] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17356.413220] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17356.417905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17356.424305] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17356.426100] Call Trace: [17356.426746] [17356.432679] ? show_regs.cold.9+0x22/0x2f [17356.437026] ? watchdog_timer_fn.cold.6+0x62/0x10a [17356.447499] ? __hrtimer_run_queues+0x1c2/0x4d0 [17356.450953] ? watchdog+0x40/0x40 [17356.454823] ? hrtimer_interrupt+0x128/0x2c0 [17356.457806] ? smp_apic_timer_interrupt+0xa6/0x270 [17356.463294] ? apic_timer_interrupt+0xf/0x20 [17356.465934] [17356.467635] ? native_safe_halt+0x2e/0x50 [17356.470641] kvm_wait+0xcd/0xe0 [17356.473607] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17356.483123] do_raw_spin_lock+0x177/0x190 [17356.487830] _raw_spin_lock+0x21/0x30 [17356.492217] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17356.498669] ? do_raw_spin_unlock+0x75/0x190 [17356.502055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17356.507936] ? do_wait_intr+0xf0/0xf0 [17356.510665] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17356.512371] mdc_close+0x53b/0xd70 [mdc] [17356.515002] lmv_close+0x18d/0x490 [lmv] [17356.516987] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17356.519878] ll_md_real_close+0xa5/0x2a0 [lustre] [17356.522011] ll_md_close+0x274/0x800 [lustre] [17356.524148] ll_file_release+0x128/0xc50 [lustre] [17356.528934] __fput+0x10d/0x440 [17356.530133] ____fput+0x12/0x20 [17356.532002] task_work_run+0xc6/0x110 [17356.535015] exit_to_usermode_loop+0x1dd/0x1f0 [17356.539161] do_syscall_64+0x3d6/0x440 [17356.542435] entry_SYSCALL_64_after_hwframe+0x49/0xae [17356.545512] RIP: 0033:0x7fa749552cd8 [17356.549927] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17356.565184] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17356.572481] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17356.576112] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17356.583281] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17356.588873] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17356.592223] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17368.055959] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [17368.061571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17368.103379] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17368.108837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17368.113674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17368.116544] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17368.124967] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17368.128443] RAX: 0000000000000387 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17368.131521] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17368.134321] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17368.137242] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17368.140423] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17368.143421] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17368.147041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17368.149729] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17368.152738] Call Trace: [17368.153722] [17368.154620] ? show_regs.cold.9+0x22/0x2f [17368.156311] ? watchdog_timer_fn.cold.6+0x62/0x10a [17368.158156] ? __hrtimer_run_queues+0x1c2/0x4d0 [17368.160064] ? watchdog+0x40/0x40 [17368.161737] ? hrtimer_interrupt+0x128/0x2c0 [17368.163938] ? smp_apic_timer_interrupt+0xa6/0x270 [17368.166082] ? apic_timer_interrupt+0xf/0x20 [17368.168026] [17368.168834] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17368.171051] do_raw_spin_lock+0x177/0x190 [17368.172862] _raw_spin_lock+0x21/0x30 [17368.174578] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17368.177093] ? do_raw_spin_unlock+0x75/0x190 [17368.178948] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17368.180665] ? do_wait_intr+0xf0/0xf0 [17368.182337] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17368.184971] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17368.186975] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17368.189387] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17368.191581] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17368.194273] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17368.196418] mdc_intent_lock+0x353/0x7e0 [mdc] [17368.198164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17368.200391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17368.203118] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17368.205264] lmv_intent_open+0x338/0x1150 [lmv] [17368.206997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17368.209577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17368.211902] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17368.213907] ll_intent_lock+0xc5/0xaa0 [lustre] [17368.216287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17368.218685] ll_lookup_it+0xe0e/0x2270 [lustre] [17368.221040] ? do_raw_spin_unlock+0x75/0x190 [17368.222923] ? _raw_spin_unlock+0x12/0x30 [17368.224638] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17368.227461] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17368.229966] ll_atomic_open+0x36a/0xd70 [lustre] [17368.231900] lookup_open+0x356/0xb40 [17368.232941] do_last+0x313/0xfc0 [17368.233881] ? nd_jump_root+0xe5/0x160 [17368.235238] ? path_init+0x437/0x520 [17368.237215] path_openat+0xf7/0x500 [17368.239092] ? do_raw_spin_unlock+0x75/0x190 [17368.241073] do_filp_open+0x99/0x140 [17368.242835] ? getname_flags+0x6e/0x330 [17368.244211] ? __check_object_size+0xff/0x256 [17368.245826] ? do_raw_spin_unlock+0x75/0x190 [17368.247658] ? _raw_spin_unlock+0x12/0x30 [17368.249216] do_sys_openat2+0x2b4/0x410 [17368.250757] do_sys_open+0x73/0xa0 [17368.252285] __x64_sys_openat+0x24/0x30 [17368.253894] do_syscall_64+0xc1/0x440 [17368.255201] entry_SYSCALL_64_after_hwframe+0x49/0xae [17368.257375] RIP: 0033:0x7fa749552332 [17368.258886] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17368.265756] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17368.269008] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17368.272159] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17368.274600] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17368.277189] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17368.279663] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17370.603051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15453s! [17370.608885] Showing busy workqueues and worker pools: [17370.611914] workqueue events: flags=0x0 [17370.614276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17370.628053] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17370.640430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17370.647250] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17370.659802] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.662081] pending: kfree_rcu_monitor [17370.665935] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17370.669706] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17370.672340] workqueue events_long: flags=0x0 [17370.673337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.676907] pending: lu_objects_flush [obdclass] [17370.679215] workqueue events_unbound: flags=0x2 [17370.681158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17370.686566] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17370.695183] workqueue events_power_efficient: flags=0x80 [17370.697915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.702342] pending: check_lifetime [17370.705908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.709262] pending: neigh_periodic_work [17370.711091] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.713979] pending: neigh_periodic_work [17370.715656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.718267] pending: do_cache_clean [17370.722206] workqueue mm_percpu_wq: flags=0x8 [17370.724269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.727294] pending: vmstat_update [17370.729307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.732349] pending: vmstat_update [17370.734561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17370.738101] pending: vmstat_update [17370.743397] workqueue writeback: flags=0x4a [17370.747797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17370.753040] pending: wb_workfn, wb_workfn [17370.755291] workqueue xprtiod: flags=0xa [17370.756531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17370.761351] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17370.769282] workqueue ipv6_addrconf: flags=0x40008 [17370.771506] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17370.774182] pending: addrconf_verify_work [17370.776286] workqueue knbd0-recv: flags=0x1a [17370.778154] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17370.780941] in-flight: 651:recv_work [17370.783747] workqueue ptlrpc_pinger: flags=0x2 [17370.785829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17370.788325] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17370.790833] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15453s workers=4 idle: 37 31846 28897 [17370.794067] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17360s workers=2 idle: 653 [17372.057008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17372.064885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17372.087760] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17372.093010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17372.097329] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17372.100687] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17372.103253] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17372.110082] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17372.113366] RAX: 0000000000000f23 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17372.117090] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17372.120383] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17372.125190] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17372.131679] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17372.135141] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17372.140669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17372.144051] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17372.148161] Call Trace: [17372.149138] [17372.150021] ? show_regs.cold.9+0x22/0x2f [17372.152504] ? watchdog_timer_fn.cold.6+0x62/0x10a [17372.156470] ? __hrtimer_run_queues+0x1c2/0x4d0 [17372.159107] ? watchdog+0x40/0x40 [17372.164128] ? hrtimer_interrupt+0x128/0x2c0 [17372.167184] ? smp_apic_timer_interrupt+0xa6/0x270 [17372.170403] ? apic_timer_interrupt+0xf/0x20 [17372.173231] [17372.174327] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17372.177825] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17372.182084] do_raw_spin_lock+0x177/0x190 [17372.184239] _raw_spin_lock+0x21/0x30 [17372.186267] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17372.189237] process_one_work+0x2c8/0x700 [17372.191895] worker_thread+0x296/0x6e0 [17372.195197] ? rescuer_thread+0x570/0x570 [17372.198161] kthread+0x1d1/0x200 [17372.199805] ? set_kthread_struct+0x70/0x70 [17372.201766] ret_from_fork+0x1f/0x30 [17384.058098] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [17384.061955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17384.063966] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17384.080969] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17384.080988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17384.080991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17384.081005] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17384.081010] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17384.081016] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17384.081018] RDX: 0000000000000152 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17384.081020] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17384.081023] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17384.081025] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17384.081028] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17384.081031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17384.081033] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17384.081042] Call Trace: [17384.081048] [17384.081054] ? show_regs.cold.9+0x22/0x2f [17384.081064] ? watchdog_timer_fn.cold.6+0x62/0x10a [17384.081071] ? __hrtimer_run_queues+0x1c2/0x4d0 [17384.081075] ? watchdog+0x40/0x40 [17384.081079] ? hrtimer_interrupt+0x128/0x2c0 [17384.081083] ? smp_apic_timer_interrupt+0xa6/0x270 [17384.081092] ? apic_timer_interrupt+0xf/0x20 [17384.081096] [17384.081099] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17384.081105] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17384.081110] do_raw_spin_lock+0x177/0x190 [17384.081115] _raw_spin_lock+0x21/0x30 [17384.081121] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17384.085155] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17384.085234] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17384.085242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17384.085245] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17384.085259] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17384.085263] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17384.085268] RAX: 0000000000007b30 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17384.085270] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17384.085273] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17384.085275] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17384.085278] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17384.085281] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17384.085284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17384.085287] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17384.085297] Call Trace: [17384.085302] [17384.085308] ? show_regs.cold.9+0x22/0x2f [17384.085319] ? watchdog_timer_fn.cold.6+0x62/0x10a [17384.085326] ? __hrtimer_run_queues+0x1c2/0x4d0 [17384.085330] ? watchdog+0x40/0x40 [17384.085335] ? hrtimer_interrupt+0x128/0x2c0 [17384.085340] ? smp_apic_timer_interrupt+0xa6/0x270 [17384.085349] ? apic_timer_interrupt+0xf/0x20 [17384.085354] [17384.085358] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17384.085366] do_raw_spin_lock+0x177/0x190 [17384.085371] _raw_spin_lock+0x21/0x30 [17384.085447] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17384.085725] ? do_raw_spin_unlock+0x75/0x190 [17384.085731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17384.090586] obd_mod_free+0x46/0x140 [ptlrpc] [17384.093666] ? do_wait_intr+0xf0/0xf0 [17384.093688] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17384.094174] mdc_close+0x53b/0xd70 [mdc] [17384.094288] lmv_close+0x18d/0x490 [lmv] [17384.094318] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17384.094452] ll_md_real_close+0xa5/0x2a0 [lustre] [17384.096630] mdc_commit_open+0x91/0xb0 [mdc] [17384.096689] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17384.097009] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17384.097224] after_reply+0xe2c/0x1780 [ptlrpc] [17384.097440] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17384.097652] ? __next_timer_interrupt+0x160/0x160 [17384.097662] ptlrpcd+0xa82/0xe40 [ptlrpc] [17384.097864] ? do_wait_intr+0xf0/0xf0 [17384.097874] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17384.098167] kthread+0x1d1/0x200 [17384.098173] ? set_kthread_struct+0x70/0x70 [17384.098177] ret_from_fork+0x1f/0x30 [17384.273749] ll_md_close+0x274/0x800 [lustre] [17384.275436] ll_file_release+0x128/0xc50 [lustre] [17384.277468] __fput+0x10d/0x440 [17384.278815] ____fput+0x12/0x20 [17384.280290] task_work_run+0xc6/0x110 [17384.281823] exit_to_usermode_loop+0x1dd/0x1f0 [17384.283818] do_syscall_64+0x3d6/0x440 [17384.285145] entry_SYSCALL_64_after_hwframe+0x49/0xae [17384.287340] RIP: 0033:0x7fa749552cd8 [17384.288565] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17384.294982] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17384.297855] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17384.300463] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17384.303260] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17384.305785] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17384.308684] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17396.056801] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [17396.070335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17396.118648] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17396.129317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17396.134471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17396.138751] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17396.158387] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17396.163261] RAX: 000000000000620e RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17396.167754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17396.173288] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17396.177515] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17396.182674] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17396.186697] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17396.190676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17396.193450] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17396.196567] Call Trace: [17396.197755] [17396.199214] ? show_regs.cold.9+0x22/0x2f [17396.201349] ? watchdog_timer_fn.cold.6+0x62/0x10a [17396.204312] ? __hrtimer_run_queues+0x1c2/0x4d0 [17396.206100] ? watchdog+0x40/0x40 [17396.207675] ? hrtimer_interrupt+0x128/0x2c0 [17396.209534] ? smp_apic_timer_interrupt+0xa6/0x270 [17396.211492] ? apic_timer_interrupt+0xf/0x20 [17396.213639] [17396.214456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17396.216974] do_raw_spin_lock+0x177/0x190 [17396.219077] _raw_spin_lock+0x21/0x30 [17396.220789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17396.223521] ? do_raw_spin_unlock+0x75/0x190 [17396.225650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17396.227765] ? do_wait_intr+0xf0/0xf0 [17396.229302] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17396.232061] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17396.233620] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17396.236092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17396.238025] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17396.239548] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17396.240946] mdc_intent_lock+0x353/0x7e0 [mdc] [17396.242450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17396.244218] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17396.246215] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17396.250120] lmv_intent_open+0x338/0x1150 [lmv] [17396.252655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17396.255349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17396.257058] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17396.258601] ll_intent_lock+0xc5/0xaa0 [lustre] [17396.260038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17396.261719] ll_lookup_it+0xe0e/0x2270 [lustre] [17396.263262] ? do_raw_spin_unlock+0x75/0x190 [17396.264381] ? _raw_spin_unlock+0x12/0x30 [17396.265443] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17396.266911] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17396.268336] ll_atomic_open+0x36a/0xd70 [lustre] [17396.269637] lookup_open+0x356/0xb40 [17396.270614] do_last+0x313/0xfc0 [17396.271483] ? nd_jump_root+0xe5/0x160 [17396.272488] ? path_init+0x437/0x520 [17396.273456] path_openat+0xf7/0x500 [17396.274804] ? do_raw_spin_unlock+0x75/0x190 [17396.276562] do_filp_open+0x99/0x140 [17396.277651] ? getname_flags+0x6e/0x330 [17396.279044] ? __check_object_size+0xff/0x256 [17396.280670] ? do_raw_spin_unlock+0x75/0x190 [17396.282392] ? _raw_spin_unlock+0x12/0x30 [17396.283962] do_sys_openat2+0x2b4/0x410 [17396.286269] do_sys_open+0x73/0xa0 [17396.287793] __x64_sys_openat+0x24/0x30 [17396.289622] do_syscall_64+0xc1/0x440 [17396.290937] entry_SYSCALL_64_after_hwframe+0x49/0xae [17396.293295] RIP: 0033:0x7fa749552332 [17396.295237] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17396.303518] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17396.309789] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17396.312887] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17396.319470] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17396.322674] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17396.325491] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17400.057023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [17400.061172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17400.096458] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17400.104040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17400.112792] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17400.121566] RIP: 0010:native_safe_halt+0x2e/0x50 [17400.123039] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17400.139478] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17400.144697] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17400.150499] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17400.163755] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17400.167285] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17400.172215] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17400.175393] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17400.180686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17400.184258] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17400.192365] Call Trace: [17400.193653] [17400.194249] ? show_regs.cold.9+0x22/0x2f [17400.195259] ? watchdog_timer_fn.cold.6+0x62/0x10a [17400.200782] ? __hrtimer_run_queues+0x1c2/0x4d0 [17400.203659] ? watchdog+0x40/0x40 [17400.210754] ? hrtimer_interrupt+0x128/0x2c0 [17400.215864] ? smp_apic_timer_interrupt+0xa6/0x270 [17400.218568] ? apic_timer_interrupt+0xf/0x20 [17400.222323] [17400.223326] ? native_safe_halt+0x2e/0x50 [17400.227852] kvm_wait+0xcd/0xe0 [17400.234366] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17400.238387] do_raw_spin_lock+0x177/0x190 [17400.242116] _raw_spin_lock+0x21/0x30 [17400.243816] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17400.253078] process_one_work+0x2c8/0x700 [17400.256258] worker_thread+0x296/0x6e0 [17400.257352] ? rescuer_thread+0x570/0x570 [17400.261514] kthread+0x1d1/0x200 [17400.264524] ? set_kthread_struct+0x70/0x70 [17400.268177] ret_from_fork+0x1f/0x30 [17400.801046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15483s! [17400.804467] Showing busy workqueues and worker pools: [17400.806983] workqueue events: flags=0x0 [17400.808430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17400.810511] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17400.815060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17400.817243] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17400.820761] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.824801] pending: kfree_rcu_monitor [17400.826675] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17400.831537] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17400.835901] workqueue events_long: flags=0x0 [17400.837835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.840355] pending: lu_objects_flush [obdclass] [17400.842631] workqueue events_unbound: flags=0x2 [17400.844666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17400.847271] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17400.851282] workqueue events_power_efficient: flags=0x80 [17400.853612] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.856520] pending: check_lifetime [17400.858252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.861535] pending: neigh_periodic_work [17400.863337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.866313] pending: neigh_periodic_work [17400.868205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.871306] pending: do_cache_clean [17400.873290] workqueue mm_percpu_wq: flags=0x8 [17400.876015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.880884] pending: vmstat_update [17400.883011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.888631] pending: vmstat_update [17400.891286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17400.900153] pending: vmstat_update [17400.904185] workqueue writeback: flags=0x4a [17400.908336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17400.912007] pending: wb_workfn, wb_workfn [17400.914267] workqueue xprtiod: flags=0xa [17400.915853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17400.918685] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17400.922155] workqueue ipv6_addrconf: flags=0x40008 [17400.924174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17400.926932] pending: addrconf_verify_work [17400.928877] workqueue knbd0-recv: flags=0x1a [17400.930867] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17400.933653] in-flight: 651:recv_work [17400.935525] workqueue ptlrpc_pinger: flags=0x2 [17400.937449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17400.939722] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17400.942445] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15483s workers=4 idle: 37 31846 28897 [17400.945984] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17390s workers=2 idle: 653 [17412.058039] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [17412.059028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17412.060560] Modules linked in: [17412.065774] Modules linked in: lustre(O) osp(O) [17412.068800] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17412.068945] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17412.068961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17412.069031] RIP: 0010:native_safe_halt+0x2e/0x50 [17412.069048] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17412.069053] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17412.069058] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17412.069061] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17412.069063] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17412.069065] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17412.069068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17412.069071] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17412.069074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17412.069076] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17412.069084] Call Trace: [17412.069089] [17412.069095] ? show_regs.cold.9+0x22/0x2f [17412.069104] ? watchdog_timer_fn.cold.6+0x62/0x10a [17412.069110] ? __hrtimer_run_queues+0x1c2/0x4d0 [17412.069114] ? watchdog+0x40/0x40 [17412.069119] ? hrtimer_interrupt+0x128/0x2c0 [17412.069123] ? smp_apic_timer_interrupt+0xa6/0x270 [17412.069130] ? apic_timer_interrupt+0xf/0x20 [17412.069135] [17412.069138] ? native_safe_halt+0x2e/0x50 [17412.069142] kvm_wait+0xcd/0xe0 [17412.069149] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17412.069156] do_raw_spin_lock+0x177/0x190 [17412.069161] _raw_spin_lock+0x21/0x30 [17412.069167] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17412.069587] obd_mod_free+0x46/0x140 [ptlrpc] [17412.073506] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17412.073683] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17412.073690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17412.073693] RIP: 0010:native_safe_halt+0x2e/0x50 [17412.073707] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17412.073830] RSP: 0018:ffffb893c146bae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17412.073836] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17412.073838] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17412.073841] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17412.073843] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17412.073845] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17412.073849] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17412.073852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17412.073854] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17412.073863] Call Trace: [17412.073868] [17412.073874] ? show_regs.cold.9+0x22/0x2f [17412.073885] ? watchdog_timer_fn.cold.6+0x62/0x10a [17412.073892] ? __hrtimer_run_queues+0x1c2/0x4d0 [17412.073897] ? watchdog+0x40/0x40 [17412.073902] ? hrtimer_interrupt+0x128/0x2c0 [17412.073906] ? smp_apic_timer_interrupt+0xa6/0x270 [17412.073914] ? apic_timer_interrupt+0xf/0x20 [17412.073918] [17412.073922] ? native_safe_halt+0x2e/0x50 [17412.073926] kvm_wait+0xcd/0xe0 [17412.073934] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17412.073943] do_raw_spin_lock+0x177/0x190 [17412.074136] _raw_spin_lock+0x21/0x30 [17412.074142] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17412.117355] mdc_commit_open+0x91/0xb0 [mdc] [17412.117408] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17412.117680] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17412.118124] after_reply+0xe2c/0x1780 [ptlrpc] [17412.129919] ? do_raw_spin_unlock+0x75/0x190 [17412.129939] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17412.130234] ? do_wait_intr+0xf0/0xf0 [17412.130242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17412.130596] mdc_close+0x53b/0xd70 [mdc] [17412.130636] lmv_close+0x18d/0x490 [lmv] [17412.130664] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17412.130828] ll_md_real_close+0xa5/0x2a0 [lustre] [17412.130968] ll_md_close+0x274/0x800 [lustre] [17412.131073] ll_file_release+0x128/0xc50 [lustre] [17412.131163] __fput+0x10d/0x440 [17412.131173] ____fput+0x12/0x20 [17412.131178] task_work_run+0xc6/0x110 [17412.131185] exit_to_usermode_loop+0x1dd/0x1f0 [17412.131192] do_syscall_64+0x3d6/0x440 [17412.131196] entry_SYSCALL_64_after_hwframe+0x49/0xae [17412.131205] RIP: 0033:0x7fa749552cd8 [17412.131212] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17412.131216] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17412.131221] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17412.131224] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17412.131226] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17412.131228] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17412.131231] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17412.340125] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17412.345169] ? __next_timer_interrupt+0x160/0x160 [17412.348369] ptlrpcd+0xa82/0xe40 [ptlrpc] [17412.351349] ? do_wait_intr+0xf0/0xf0 [17412.353559] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17412.356852] kthread+0x1d1/0x200 [17412.358671] ? set_kthread_struct+0x70/0x70 [17412.360810] ret_from_fork+0x1f/0x30 [17428.057155] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [17428.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [17428.067938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash [17428.076672] Modules linked in: [17428.076675] dm_log dm_mod sha512_ssse3 [17428.108744] lustre(O) [17428.109837] sha512_generic [17428.109846] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17428.109853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17428.109856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17428.109870] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17428.109874] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17428.109879] RAX: 0000000000007f83 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17428.109881] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17428.109884] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17428.109886] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17428.109889] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17428.109893] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17428.109896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17428.109898] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17428.109907] Call Trace: [17428.109912] [17428.109917] ? show_regs.cold.9+0x22/0x2f [17428.109925] ? watchdog_timer_fn.cold.6+0x62/0x10a [17428.109932] ? __hrtimer_run_queues+0x1c2/0x4d0 [17428.109936] ? watchdog+0x40/0x40 [17428.109940] ? hrtimer_interrupt+0x128/0x2c0 [17428.109943] ? smp_apic_timer_interrupt+0xa6/0x270 [17428.109950] ? apic_timer_interrupt+0xf/0x20 [17428.109955] [17428.109957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17428.109965] do_raw_spin_lock+0x177/0x190 [17428.109970] _raw_spin_lock+0x21/0x30 [17428.109975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17428.110246] ? do_raw_spin_unlock+0x75/0x190 [17428.110253] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17428.112266] osp(O) [17428.113217] ? do_wait_intr+0xf0/0xf0 [17428.113234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17428.114869] ofd(O) [17428.118967] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17428.119249] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17428.119459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17428.119565] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17428.119602] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17428.119632] mdc_intent_lock+0x353/0x7e0 [mdc] [17428.119661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17428.119762] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17428.128319] lod(O) [17428.130444] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17428.147823] mdt(O) [17428.150757] lmv_intent_open+0x338/0x1150 [lmv] [17428.150807] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17428.150922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17428.154830] mdd(O) [17428.157629] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17428.157692] ll_intent_lock+0xc5/0xaa0 [lustre] [17428.157862] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17428.157951] ll_lookup_it+0xe0e/0x2270 [lustre] [17428.158034] ? do_raw_spin_unlock+0x75/0x190 [17428.158042] ? _raw_spin_unlock+0x12/0x30 [17428.158050] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17428.158089] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17428.158113] ll_atomic_open+0x36a/0xd70 [lustre] [17428.158200] lookup_open+0x356/0xb40 [17428.158210] do_last+0x313/0xfc0 [17428.158216] ? nd_jump_root+0xe5/0x160 [17428.158221] ? path_init+0x437/0x520 [17428.158226] path_openat+0xf7/0x500 [17428.158230] ? do_raw_spin_unlock+0x75/0x190 [17428.158234] do_filp_open+0x99/0x140 [17428.158241] ? getname_flags+0x6e/0x330 [17428.158245] ? __check_object_size+0xff/0x256 [17428.158256] ? do_raw_spin_unlock+0x75/0x190 [17428.158259] ? _raw_spin_unlock+0x12/0x30 [17428.158263] do_sys_openat2+0x2b4/0x410 [17428.158269] do_sys_open+0x73/0xa0 [17428.158273] __x64_sys_openat+0x24/0x30 [17428.158278] do_syscall_64+0xc1/0x440 [17428.158285] entry_SYSCALL_64_after_hwframe+0x49/0xae [17428.158291] RIP: 0033:0x7fa749552332 [17428.158300] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17428.158304] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17428.158308] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17428.158311] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17428.158313] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17428.158315] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17428.158317] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17428.326421] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17428.349025] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17428.354497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17428.358083] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17428.361380] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17428.363785] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17428.371862] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17428.375123] RAX: 0000000000007f8b RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17428.378333] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17428.382271] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17428.384913] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17428.386610] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17428.389236] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17428.391520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17428.393057] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17428.395391] Call Trace: [17428.396216] [17428.396870] ? show_regs.cold.9+0x22/0x2f [17428.398036] ? watchdog_timer_fn.cold.6+0x62/0x10a [17428.399410] ? __hrtimer_run_queues+0x1c2/0x4d0 [17428.400814] ? watchdog+0x40/0x40 [17428.401518] ? hrtimer_interrupt+0x128/0x2c0 [17428.402632] ? smp_apic_timer_interrupt+0xa6/0x270 [17428.403880] ? apic_timer_interrupt+0xf/0x20 [17428.404910] [17428.405697] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17428.406955] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17428.408312] do_raw_spin_lock+0x177/0x190 [17428.409394] _raw_spin_lock+0x21/0x30 [17428.410273] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17428.412213] process_one_work+0x2c8/0x700 [17428.413255] worker_thread+0x296/0x6e0 [17428.414533] ? rescuer_thread+0x570/0x570 [17428.415828] kthread+0x1d1/0x200 [17428.416794] ? set_kthread_struct+0x70/0x70 [17428.418158] ret_from_fork+0x1f/0x30 [17431.011411] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15513s! [17431.020284] Showing busy workqueues and worker pools: [17431.025089] workqueue events: flags=0x0 [17431.032513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17431.044732] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17431.059082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17431.070139] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17431.078489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.090134] pending: kfree_rcu_monitor [17431.094816] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17431.106566] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17431.114876] workqueue events_long: flags=0x0 [17431.118145] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.122755] pending: lu_objects_flush [obdclass] [17431.128903] workqueue events_unbound: flags=0x2 [17431.130410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17431.135837] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17431.145087] workqueue events_power_efficient: flags=0x80 [17431.146808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.149235] pending: check_lifetime [17431.150615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.157245] pending: neigh_periodic_work [17431.163572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.169266] pending: neigh_periodic_work [17431.175074] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.178597] pending: do_cache_clean [17431.181571] workqueue mm_percpu_wq: flags=0x8 [17431.183831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.188613] pending: vmstat_update [17431.190573] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.196699] pending: vmstat_update [17431.200280] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17431.207543] pending: vmstat_update [17431.212853] workqueue writeback: flags=0x4a [17431.216429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17431.222081] pending: wb_workfn, wb_workfn [17431.224809] workqueue xprtiod: flags=0xa [17431.227216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17431.231180] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17431.235853] workqueue ipv6_addrconf: flags=0x40008 [17431.239260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17431.244572] pending: addrconf_verify_work [17431.253058] workqueue knbd0-recv: flags=0x1a [17431.258120] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17431.261844] in-flight: 651:recv_work [17431.264560] workqueue ptlrpc_pinger: flags=0x2 [17431.268653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17431.274583] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17431.280347] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15514s workers=4 idle: 37 31846 28897 [17431.289980] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17421s workers=2 idle: 653 [17440.058111] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2403] [17440.059137] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37249] [17440.061646] Modules linked in: lustre(O) osp(O) [17440.066212] Modules linked in: [17440.066214] ofd(O) [17440.069843] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17440.070091] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17440.070099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17440.070102] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17440.072637] lod(O) [17440.079129] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17440.097877] mdt(O) [17440.118242] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17440.118321] RAX: 00000000000052e4 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17440.118324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17440.118326] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17440.118329] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17440.118331] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17440.118334] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17440.118337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17440.118339] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17440.118347] Call Trace: [17440.118395] [17440.118402] ? show_regs.cold.9+0x22/0x2f [17440.118416] ? watchdog_timer_fn.cold.6+0x62/0x10a [17440.118423] ? __hrtimer_run_queues+0x1c2/0x4d0 [17440.118428] ? watchdog+0x40/0x40 [17440.118433] ? hrtimer_interrupt+0x128/0x2c0 [17440.118438] ? smp_apic_timer_interrupt+0xa6/0x270 [17440.118446] ? apic_timer_interrupt+0xf/0x20 [17440.118451] [17440.118454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17440.118464] do_raw_spin_lock+0x177/0x190 [17440.118469] _raw_spin_lock+0x21/0x30 [17440.118474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17440.119205] ? do_raw_spin_unlock+0x75/0x190 [17440.119212] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17440.119711] ? do_wait_intr+0xf0/0xf0 [17440.119722] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17440.120843] mdc_close+0x53b/0xd70 [mdc] [17440.120885] lmv_close+0x18d/0x490 [lmv] [17440.120939] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17440.124621] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17440.124724] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17440.124731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17440.124734] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17440.124747] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17440.124751] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17440.124756] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17440.124758] RDX: 0000000000002721 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17440.124760] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17440.124763] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17440.124765] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17440.124768] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17440.124771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17440.124775] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17440.128694] ll_md_real_close+0xa5/0x2a0 [lustre] [17440.129294] Call Trace: [17440.129300] [17440.129303] ? show_regs.cold.9+0x22/0x2f [17440.139740] ll_md_close+0x274/0x800 [lustre] [17440.140765] ? watchdog_timer_fn.cold.6+0x62/0x10a [17440.140775] ? __hrtimer_run_queues+0x1c2/0x4d0 [17440.140779] ? watchdog+0x40/0x40 [17440.140784] ? hrtimer_interrupt+0x128/0x2c0 [17440.140791] ? smp_apic_timer_interrupt+0xa6/0x270 [17440.140799] ? apic_timer_interrupt+0xf/0x20 [17440.140803] [17440.140807] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17440.140813] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17440.140817] do_raw_spin_lock+0x177/0x190 [17440.140873] _raw_spin_lock+0x21/0x30 [17440.140878] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17440.143548] ll_file_release+0x128/0xc50 [lustre] [17440.143663] __fput+0x10d/0x440 [17440.143672] ____fput+0x12/0x20 [17440.143676] task_work_run+0xc6/0x110 [17440.143683] exit_to_usermode_loop+0x1dd/0x1f0 [17440.143690] do_syscall_64+0x3d6/0x440 [17440.143759] entry_SYSCALL_64_after_hwframe+0x49/0xae [17440.143767] RIP: 0033:0x7fa749552cd8 [17440.143805] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17440.143808] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17440.143813] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17440.143816] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17440.143818] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17440.143820] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17440.143822] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17440.284127] obd_mod_free+0x46/0x140 [ptlrpc] [17440.285680] mdc_commit_open+0x91/0xb0 [mdc] [17440.286822] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17440.288532] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17440.290206] after_reply+0xe2c/0x1780 [ptlrpc] [17440.291748] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17440.293423] ? __next_timer_interrupt+0x160/0x160 [17440.294642] ptlrpcd+0xa82/0xe40 [ptlrpc] [17440.296055] ? do_wait_intr+0xf0/0xf0 [17440.297352] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17440.298930] kthread+0x1d1/0x200 [17440.299912] ? set_kthread_struct+0x70/0x70 [17440.301042] ret_from_fork+0x1f/0x30 [17454.027083] rcu: INFO: rcu_sched self-detected stall on CPU [17454.029822] rcu: 2-...!: (9177189 ticks this GP) idle=4aa/1/0x4000000000000002 softirq=131143/131143 fqs=645 [17454.034726] (t=15540338 jiffies g=258701 q=5694) [17454.037378] rcu: rcu_sched kthread starved for 15537133 jiffies! g258701 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [17454.043301] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17454.051692] rcu: RCU grace-period kthread stack dump: [17454.059153] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17454.067658] Call Trace: [17454.068821] __schedule+0x351/0xcb0 [17454.071762] schedule+0xc0/0x180 [17454.074696] schedule_timeout+0xb4/0x190 [17454.077373] ? do_raw_spin_unlock+0x75/0x190 [17454.081137] ? __next_timer_interrupt+0x160/0x160 [17454.084342] ? prepare_to_swait_event+0xf3/0x140 [17454.086432] rcu_gp_kthread+0x8dd/0x15b0 [17454.089725] ? force_qs_rnp+0x2c0/0x2c0 [17454.091918] kthread+0x1d1/0x200 [17454.095543] ? set_kthread_struct+0x70/0x70 [17454.097315] ret_from_fork+0x1f/0x30 [17454.098562] rcu: Stack dump where RCU GP kthread last ran: [17454.102828] Sending NMI from CPU 2 to CPUs 0: [17454.109817] NMI backtrace for cpu 0 [17454.109824] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17454.109827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17454.109829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [17454.109830] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [17454.109833] RSP: 0018:ffffb893c1493500 EFLAGS: 00000202 [17454.109835] RAX: 0000000000002c2a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17454.109836] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17454.109838] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17454.109839] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17454.109841] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17454.109842] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17454.109844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17454.109845] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17454.109846] Call Trace: [17454.109847] [17454.109848] ? show_regs.cold.9+0x22/0x2f [17454.109849] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17454.109851] ? nmi_cpu_backtrace_handler+0x15/0x40 [17454.109852] ? nmi_handle+0xbd/0x210 [17454.109853] ? default_do_nmi+0x4b/0x1b0 [17454.109854] ? do_nmi+0x280/0x380 [17454.109855] ? end_repeat_nmi+0x16/0x1f [17454.109856] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [17454.109857] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [17454.109859] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [17454.109860] [17454.109861] do_raw_spin_lock+0x177/0x190 [17454.109862] _raw_spin_lock+0x21/0x30 [17454.109863] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17454.109864] ? do_raw_spin_unlock+0x75/0x190 [17454.109865] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17454.109866] ? do_wait_intr+0xf0/0xf0 [17454.109867] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17454.109869] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17454.109870] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17454.109871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.109872] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17454.109874] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17454.109875] mdc_intent_lock+0x353/0x7e0 [mdc] [17454.109876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.109877] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17454.109879] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17454.109880] lmv_intent_open+0x338/0x1150 [lmv] [17454.109881] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.109882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.109883] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17454.109884] ll_intent_lock+0xc5/0xaa0 [lustre] [17454.109886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.109887] ll_lookup_it+0xe0e/0x2270 [lustre] [17454.109888] ? do_raw_spin_unlock+0x75/0x190 [17454.109889] ? _raw_spin_unlock+0x12/0x30 [17454.109890] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17454.109891] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17454.109892] ll_atomic_open+0x36a/0xd70 [lustre] [17454.109893] lookup_open+0x356/0xb40 [17454.109894] do_last+0x313/0xfc0 [17454.109895] ? nd_jump_root+0xe5/0x160 [17454.109896] ? path_init+0x437/0x520 [17454.109897] path_openat+0xf7/0x500 [17454.109898] ? do_raw_spin_unlock+0x75/0x190 [17454.109899] do_filp_open+0x99/0x140 [17454.109900] ? getname_flags+0x6e/0x330 [17454.109901] ? __check_object_size+0xff/0x256 [17454.109901] ? do_raw_spin_unlock+0x75/0x190 [17454.109902] ? _raw_spin_unlock+0x12/0x30 [17454.109903] do_sys_openat2+0x2b4/0x410 [17454.109904] do_sys_open+0x73/0xa0 [17454.109905] __x64_sys_openat+0x24/0x30 [17454.109907] do_syscall_64+0xc1/0x440 [17454.109907] entry_SYSCALL_64_after_hwframe+0x49/0xae [17454.109908] RIP: 0033:0x7fa749552332 [17454.109910] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17454.109912] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17454.109915] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17454.109916] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17454.109917] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17454.109919] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17454.109920] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17454.113402] NMI backtrace for cpu 2 [17454.327773] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17454.335617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17454.345925] Call Trace: [17454.347163] [17454.347679] ? dump_stack+0xbb/0x10e [17454.349064] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17454.351237] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17454.353200] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17454.355812] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17454.358525] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17454.361294] ? rcu_dump_cpu_stacks+0x11b/0x171 [17454.364565] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17454.366947] ? update_process_times+0x6d/0xc0 [17454.369100] ? tick_sched_handle.isra.18+0x27/0x90 [17454.371120] ? tick_sched_timer+0x53/0xb0 [17454.372759] ? __hrtimer_run_queues+0x1c2/0x4d0 [17454.375180] ? tick_sched_handle.isra.18+0x90/0x90 [17454.378744] ? hrtimer_interrupt+0x128/0x2c0 [17454.381325] ? smp_apic_timer_interrupt+0xa6/0x270 [17454.383422] ? apic_timer_interrupt+0xf/0x20 [17454.388176] [17454.390221] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17454.394185] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17454.397358] ? do_raw_spin_lock+0x177/0x190 [17454.398797] ? _raw_spin_lock+0x21/0x30 [17454.401640] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17454.405245] ? obd_mod_free+0x46/0x140 [ptlrpc] [17454.407046] ? mdc_commit_open+0x91/0xb0 [mdc] [17454.410771] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17454.414189] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17454.417219] ? after_reply+0xe2c/0x1780 [ptlrpc] [17454.419474] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17454.422599] ? __next_timer_interrupt+0x160/0x160 [17454.425811] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17454.427803] ? do_wait_intr+0xf0/0xf0 [17454.430305] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17454.432178] ? kthread+0x1d1/0x200 [17454.433188] ? set_kthread_struct+0x70/0x70 [17454.434718] ? ret_from_fork+0x1f/0x30 [17456.057324] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [17456.064495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17456.091378] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17456.094683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17456.097613] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17456.100152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17456.102676] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17456.108763] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17456.114286] RAX: 0000000000006f09 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17456.118370] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17456.120401] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17456.122371] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17456.124659] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17456.126352] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17456.128839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17456.130407] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17456.132928] Call Trace: [17456.133903] [17456.135375] ? show_regs.cold.9+0x22/0x2f [17456.137095] ? watchdog_timer_fn.cold.6+0x62/0x10a [17456.138102] ? __hrtimer_run_queues+0x1c2/0x4d0 [17456.139409] ? watchdog+0x40/0x40 [17456.140532] ? hrtimer_interrupt+0x128/0x2c0 [17456.141628] ? smp_apic_timer_interrupt+0xa6/0x270 [17456.143067] ? apic_timer_interrupt+0xf/0x20 [17456.143967] [17456.144416] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17456.145536] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17456.147324] do_raw_spin_lock+0x177/0x190 [17456.148404] _raw_spin_lock+0x21/0x30 [17456.149655] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17456.155385] process_one_work+0x2c8/0x700 [17456.156934] worker_thread+0x296/0x6e0 [17456.160163] ? rescuer_thread+0x570/0x570 [17456.164929] kthread+0x1d1/0x200 [17456.166249] ? set_kthread_struct+0x70/0x70 [17456.167368] ret_from_fork+0x1f/0x30 [17461.730016] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15544s! [17461.736439] Showing busy workqueues and worker pools: [17461.743067] workqueue events: flags=0x0 [17461.749099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17461.754655] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17461.763617] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17461.765930] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17461.774059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.793103] pending: kfree_rcu_monitor [17461.795883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17461.802816] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17461.810737] workqueue events_long: flags=0x0 [17461.812207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.823272] pending: lu_objects_flush [obdclass] [17461.826358] workqueue events_unbound: flags=0x2 [17461.828102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17461.835623] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17461.846680] workqueue events_power_efficient: flags=0x80 [17461.850048] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.857001] pending: check_lifetime [17461.858651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.864885] pending: neigh_periodic_work [17461.866976] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.873057] pending: neigh_periodic_work [17461.874829] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.877225] pending: do_cache_clean [17461.878504] workqueue mm_percpu_wq: flags=0x8 [17461.879849] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.881836] pending: vmstat_update [17461.883081] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.885151] pending: vmstat_update [17461.887134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17461.892374] pending: vmstat_update [17461.894352] workqueue writeback: flags=0x4a [17461.899378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17461.903094] pending: wb_workfn, wb_workfn [17461.904740] workqueue xprtiod: flags=0xa [17461.909557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17461.913571] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17461.925433] workqueue ipv6_addrconf: flags=0x40008 [17461.928981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17461.933760] pending: addrconf_verify_work [17461.935423] workqueue knbd0-recv: flags=0x1a [17461.941975] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17461.949217] in-flight: 651:recv_work [17461.952351] workqueue ptlrpc_pinger: flags=0x2 [17461.954620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17461.961696] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17461.966341] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15544s workers=4 idle: 37 31846 28897 [17461.970033] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17451s workers=2 idle: 653 [17468.059012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17468.063580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17468.091470] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17468.101176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17468.106238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17468.111677] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17468.124812] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17468.128288] RAX: 00000000000007f0 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17468.131524] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17468.135340] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17468.138238] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17468.141921] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17468.145493] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17468.148491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17468.151060] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17468.153817] Call Trace: [17468.154797] [17468.155437] ? show_regs.cold.9+0x22/0x2f [17468.157396] ? watchdog_timer_fn.cold.6+0x62/0x10a [17468.159404] ? __hrtimer_run_queues+0x1c2/0x4d0 [17468.160849] ? watchdog+0x40/0x40 [17468.161980] ? hrtimer_interrupt+0x128/0x2c0 [17468.163423] ? smp_apic_timer_interrupt+0xa6/0x270 [17468.165365] ? apic_timer_interrupt+0xf/0x20 [17468.166881] [17468.167505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17468.169358] do_raw_spin_lock+0x177/0x190 [17468.170989] _raw_spin_lock+0x21/0x30 [17468.172337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17468.174410] ? do_raw_spin_unlock+0x75/0x190 [17468.176024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17468.178061] ? do_wait_intr+0xf0/0xf0 [17468.179333] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17468.181382] mdc_close+0x53b/0xd70 [mdc] [17468.182852] lmv_close+0x18d/0x490 [lmv] [17468.184333] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17468.186406] ll_md_real_close+0xa5/0x2a0 [lustre] [17468.188167] ll_md_close+0x274/0x800 [lustre] [17468.189554] ll_file_release+0x128/0xc50 [lustre] [17468.191295] __fput+0x10d/0x440 [17468.192361] ____fput+0x12/0x20 [17468.193433] task_work_run+0xc6/0x110 [17468.194884] exit_to_usermode_loop+0x1dd/0x1f0 [17468.196361] do_syscall_64+0x3d6/0x440 [17468.198292] entry_SYSCALL_64_after_hwframe+0x49/0xae [17468.200469] RIP: 0033:0x7fa749552cd8 [17468.202257] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17468.211137] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17468.214704] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17468.217999] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17468.221240] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17468.224126] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17468.226578] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17480.058010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17480.061252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17480.088791] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17480.094346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17480.098898] RIP: 0010:native_safe_halt+0x2e/0x50 [17480.101261] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17480.111980] RSP: 0018:ffffb893c1bd3bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17480.121318] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17480.128177] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17480.139922] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17480.151024] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17480.161570] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17480.166812] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17480.172573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17480.176124] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17480.181768] Call Trace: [17480.182982] [17480.183932] ? show_regs.cold.9+0x22/0x2f [17480.187050] ? watchdog_timer_fn.cold.6+0x62/0x10a [17480.188772] ? __hrtimer_run_queues+0x1c2/0x4d0 [17480.191729] ? watchdog+0x40/0x40 [17480.193494] ? hrtimer_interrupt+0x128/0x2c0 [17480.196213] ? smp_apic_timer_interrupt+0xa6/0x270 [17480.199296] ? apic_timer_interrupt+0xf/0x20 [17480.200986] [17480.202496] ? native_safe_halt+0x2e/0x50 [17480.204381] kvm_wait+0xcd/0xe0 [17480.205599] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17480.208354] do_raw_spin_lock+0x177/0x190 [17480.210357] _raw_spin_lock+0x21/0x30 [17480.211910] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17480.214448] obd_mod_free+0x46/0x140 [ptlrpc] [17480.216761] mdc_commit_open+0x91/0xb0 [mdc] [17480.218724] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17480.221229] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17480.223902] after_reply+0xe2c/0x1780 [ptlrpc] [17480.225985] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17480.228451] ? __next_timer_interrupt+0x160/0x160 [17480.231222] ptlrpcd+0xa82/0xe40 [ptlrpc] [17480.233324] ? do_wait_intr+0xf0/0xf0 [17480.234880] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17480.237170] kthread+0x1d1/0x200 [17480.238587] ? set_kthread_struct+0x70/0x70 [17480.240781] ret_from_fork+0x1f/0x30 [17484.057018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:0:8] [17484.065401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17484.137711] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17484.156673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17484.162687] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17484.167076] RIP: 0010:native_safe_halt+0x2e/0x50 [17484.169748] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17484.198646] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17484.206766] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17484.213716] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17484.220901] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17484.226213] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17484.234358] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17484.242789] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17484.250136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17484.259848] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17484.272836] Call Trace: [17484.274866] [17484.277776] ? show_regs.cold.9+0x22/0x2f [17484.284787] ? watchdog_timer_fn.cold.6+0x62/0x10a [17484.296280] ? __hrtimer_run_queues+0x1c2/0x4d0 [17484.301406] ? watchdog+0x40/0x40 [17484.308077] ? hrtimer_interrupt+0x128/0x2c0 [17484.315505] ? smp_apic_timer_interrupt+0xa6/0x270 [17484.329977] ? apic_timer_interrupt+0xf/0x20 [17484.331793] [17484.352629] ? native_safe_halt+0x2e/0x50 [17484.354302] kvm_wait+0xcd/0xe0 [17484.355419] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17484.371912] do_raw_spin_lock+0x177/0x190 [17484.373433] _raw_spin_lock+0x21/0x30 [17484.391848] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17484.401037] process_one_work+0x2c8/0x700 [17484.404185] worker_thread+0x296/0x6e0 [17484.405562] ? rescuer_thread+0x570/0x570 [17484.422915] kthread+0x1d1/0x200 [17484.429079] ? set_kthread_struct+0x70/0x70 [17484.436076] ret_from_fork+0x1f/0x30 [17488.054626] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37250] [17488.060294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17488.091630] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17488.102229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17488.110269] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17488.116366] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17488.133450] RSP: 0018:ffffb893c1493500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17488.142004] RAX: 0000000000005e11 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17488.150136] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17488.158009] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17488.162175] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17488.172072] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17488.184998] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17488.196212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17488.203844] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17488.208223] Call Trace: [17488.210851] [17488.212800] ? show_regs.cold.9+0x22/0x2f [17488.220063] ? watchdog_timer_fn.cold.6+0x62/0x10a [17488.224287] ? __hrtimer_run_queues+0x1c2/0x4d0 [17488.232238] ? watchdog+0x40/0x40 [17488.233833] ? hrtimer_interrupt+0x128/0x2c0 [17488.235515] ? smp_apic_timer_interrupt+0xa6/0x270 [17488.237005] ? apic_timer_interrupt+0xf/0x20 [17488.238365] [17488.244123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17488.248048] do_raw_spin_lock+0x177/0x190 [17488.250749] _raw_spin_lock+0x21/0x30 [17488.253210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17488.258770] ? do_raw_spin_unlock+0x75/0x190 [17488.262975] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17488.265281] ? do_wait_intr+0xf0/0xf0 [17488.271408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17488.275775] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17488.280117] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17488.288212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17488.295275] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17488.301273] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17488.306354] mdc_intent_lock+0x353/0x7e0 [mdc] [17488.313052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17488.320946] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17488.332225] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17488.336001] lmv_intent_open+0x338/0x1150 [lmv] [17488.337857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17488.339324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17488.342215] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17488.343382] ll_intent_lock+0xc5/0xaa0 [lustre] [17488.344836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17488.354873] ll_lookup_it+0xe0e/0x2270 [lustre] [17488.356690] ? do_raw_spin_unlock+0x75/0x190 [17488.357746] ? _raw_spin_unlock+0x12/0x30 [17488.360982] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17488.362483] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17488.364027] ll_atomic_open+0x36a/0xd70 [lustre] [17488.365782] lookup_open+0x356/0xb40 [17488.367040] do_last+0x313/0xfc0 [17488.370998] ? nd_jump_root+0xe5/0x160 [17488.373844] ? path_init+0x437/0x520 [17488.375978] path_openat+0xf7/0x500 [17488.377149] ? do_raw_spin_unlock+0x75/0x190 [17488.382824] do_filp_open+0x99/0x140 [17488.384858] ? getname_flags+0x6e/0x330 [17488.390022] ? __check_object_size+0xff/0x256 [17488.391537] ? do_raw_spin_unlock+0x75/0x190 [17488.392609] ? _raw_spin_unlock+0x12/0x30 [17488.395921] do_sys_openat2+0x2b4/0x410 [17488.396887] do_sys_open+0x73/0xa0 [17488.401142] __x64_sys_openat+0x24/0x30 [17488.404231] do_syscall_64+0xc1/0x440 [17488.407139] entry_SYSCALL_64_after_hwframe+0x49/0xae [17488.412219] RIP: 0033:0x7fa749552332 [17488.413943] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17488.426193] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17488.435167] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17488.440770] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17488.448795] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17488.455828] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17488.465283] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17492.449239] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15575s! [17492.473061] Showing busy workqueues and worker pools: [17492.480173] workqueue events: flags=0x0 [17492.487814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17492.490522] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17492.502780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17492.507976] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17492.523934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.539026] pending: kfree_rcu_monitor [17492.540553] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17492.555569] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17492.574984] workqueue events_long: flags=0x0 [17492.576767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.585244] pending: lu_objects_flush [obdclass] [17492.590384] workqueue events_unbound: flags=0x2 [17492.591301] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17492.596048] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17492.605930] workqueue events_power_efficient: flags=0x80 [17492.611288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.617216] pending: check_lifetime [17492.620789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.624664] pending: neigh_periodic_work [17492.633512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.637165] pending: neigh_periodic_work [17492.642797] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.650253] pending: do_cache_clean [17492.657183] workqueue mm_percpu_wq: flags=0x8 [17492.667275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.672642] pending: vmstat_update [17492.673882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.678219] pending: vmstat_update [17492.684454] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17492.688660] pending: vmstat_update [17492.690104] workqueue writeback: flags=0x4a [17492.693686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17492.699045] pending: wb_workfn, wb_workfn [17492.704021] workqueue xprtiod: flags=0xa [17492.705182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17492.712802] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17492.721341] workqueue ipv6_addrconf: flags=0x40008 [17492.726816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17492.734248] pending: addrconf_verify_work [17492.737568] workqueue knbd0-recv: flags=0x1a [17492.739513] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17492.742423] in-flight: 651:recv_work [17492.744187] workqueue ptlrpc_pinger: flags=0x2 [17492.745818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17492.748283] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17492.750555] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15575s workers=4 idle: 37 31846 28897 [17492.753874] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17482s workers=2 idle: 653 [17496.061829] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17496.067115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17496.124679] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17496.135468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17496.141867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17496.147176] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17496.160123] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17496.168052] RAX: 0000000000006887 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17496.175199] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17496.177289] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17496.188119] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17496.198012] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17496.207935] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17496.216273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17496.227025] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17496.236047] Call Trace: [17496.237686] [17496.238656] ? show_regs.cold.9+0x22/0x2f [17496.241559] ? watchdog_timer_fn.cold.6+0x62/0x10a [17496.247980] ? __hrtimer_run_queues+0x1c2/0x4d0 [17496.255249] ? watchdog+0x40/0x40 [17496.257005] ? hrtimer_interrupt+0x128/0x2c0 [17496.259103] ? smp_apic_timer_interrupt+0xa6/0x270 [17496.260405] ? apic_timer_interrupt+0xf/0x20 [17496.261428] [17496.274517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17496.278543] do_raw_spin_lock+0x177/0x190 [17496.282888] _raw_spin_lock+0x21/0x30 [17496.285495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17496.289353] ? do_raw_spin_unlock+0x75/0x190 [17496.301453] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17496.303454] ? do_wait_intr+0xf0/0xf0 [17496.310033] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17496.315058] mdc_close+0x53b/0xd70 [mdc] [17496.317952] lmv_close+0x18d/0x490 [lmv] [17496.321036] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17496.328914] ll_md_real_close+0xa5/0x2a0 [lustre] [17496.332674] ll_md_close+0x274/0x800 [lustre] [17496.339209] ll_file_release+0x128/0xc50 [lustre] [17496.343109] __fput+0x10d/0x440 [17496.345606] ____fput+0x12/0x20 [17496.348023] task_work_run+0xc6/0x110 [17496.355146] exit_to_usermode_loop+0x1dd/0x1f0 [17496.362385] do_syscall_64+0x3d6/0x440 [17496.365556] entry_SYSCALL_64_after_hwframe+0x49/0xae [17496.367811] RIP: 0033:0x7fa749552cd8 [17496.370197] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17496.401039] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17496.411715] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17496.421217] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17496.427581] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17496.435474] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17496.448576] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17508.064572] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17508.070159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17508.118010] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17508.132474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17508.143783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17508.152500] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17508.173797] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17508.180762] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17508.189108] RDX: 0000000000007e0d RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17508.192840] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17508.197123] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17508.199214] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17508.202572] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17508.208332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17508.212307] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17508.216632] Call Trace: [17508.219239] [17508.220083] ? show_regs.cold.9+0x22/0x2f [17508.223088] ? watchdog_timer_fn.cold.6+0x62/0x10a [17508.231723] ? __hrtimer_run_queues+0x1c2/0x4d0 [17508.236742] ? watchdog+0x40/0x40 [17508.241502] ? hrtimer_interrupt+0x128/0x2c0 [17508.250373] ? smp_apic_timer_interrupt+0xa6/0x270 [17508.252046] ? apic_timer_interrupt+0xf/0x20 [17508.256864] [17508.259746] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17508.262589] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17508.266946] do_raw_spin_lock+0x177/0x190 [17508.271989] _raw_spin_lock+0x21/0x30 [17508.278633] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17508.285336] obd_mod_free+0x46/0x140 [ptlrpc] [17508.289892] mdc_commit_open+0x91/0xb0 [mdc] [17508.292978] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17508.302107] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17508.308827] after_reply+0xe2c/0x1780 [ptlrpc] [17508.315624] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17508.323472] ? __next_timer_interrupt+0x160/0x160 [17508.327955] ptlrpcd+0xa82/0xe40 [ptlrpc] [17508.331974] ? do_wait_intr+0xf0/0xf0 [17508.337256] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17508.345291] kthread+0x1d1/0x200 [17508.351236] ? set_kthread_struct+0x70/0x70 [17508.354875] ret_from_fork+0x1f/0x30 [17512.057052] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17512.064455] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17512.109573] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17512.117523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17512.123666] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17512.126684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17512.131140] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17512.148334] RSP: 0018:ffffb893c0c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17512.154178] RAX: 0000000000007d1f RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17512.158783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17512.162649] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17512.168558] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17512.174942] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17512.182283] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17512.189379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17512.194935] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17512.200554] Call Trace: [17512.202673] [17512.204638] ? show_regs.cold.9+0x22/0x2f [17512.208450] ? watchdog_timer_fn.cold.6+0x62/0x10a [17512.212633] ? __hrtimer_run_queues+0x1c2/0x4d0 [17512.217500] ? watchdog+0x40/0x40 [17512.220354] ? hrtimer_interrupt+0x128/0x2c0 [17512.222298] ? smp_apic_timer_interrupt+0xa6/0x270 [17512.225641] ? apic_timer_interrupt+0xf/0x20 [17512.229593] [17512.231486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17512.237214] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17512.245632] do_raw_spin_lock+0x177/0x190 [17512.249701] _raw_spin_lock+0x21/0x30 [17512.252703] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17512.257769] process_one_work+0x2c8/0x700 [17512.263203] worker_thread+0x296/0x6e0 [17512.265539] ? rescuer_thread+0x570/0x570 [17512.271188] kthread+0x1d1/0x200 [17512.274131] ? set_kthread_struct+0x70/0x70 [17512.278639] ret_from_fork+0x1f/0x30 [17520.054197] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17520.056929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17520.097369] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17520.108682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17520.120454] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17520.130658] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17520.156265] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17520.165696] RAX: 0000000000001547 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17520.171265] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17520.183626] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17520.188203] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17520.192808] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17520.196322] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17520.201694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17520.204464] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17520.210214] Call Trace: [17520.211667] [17520.214555] ? show_regs.cold.9+0x22/0x2f [17520.217495] ? watchdog_timer_fn.cold.6+0x62/0x10a [17520.220048] ? __hrtimer_run_queues+0x1c2/0x4d0 [17520.222024] ? watchdog+0x40/0x40 [17520.223591] ? hrtimer_interrupt+0x128/0x2c0 [17520.225823] ? smp_apic_timer_interrupt+0xa6/0x270 [17520.228023] ? apic_timer_interrupt+0xf/0x20 [17520.229947] [17520.231090] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17520.234486] do_raw_spin_lock+0x177/0x190 [17520.237682] _raw_spin_lock+0x21/0x30 [17520.239957] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17520.242627] ? do_raw_spin_unlock+0x75/0x190 [17520.244796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17520.247215] ? do_wait_intr+0xf0/0xf0 [17520.249138] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17520.251756] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17520.254595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17520.257721] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17520.265295] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17520.268014] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17520.273922] mdc_intent_lock+0x353/0x7e0 [mdc] [17520.279581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17520.285869] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17520.293912] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17520.304153] lmv_intent_open+0x338/0x1150 [lmv] [17520.310892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17520.312841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17520.316891] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17520.319528] ll_intent_lock+0xc5/0xaa0 [lustre] [17520.329472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17520.331401] ll_lookup_it+0xe0e/0x2270 [lustre] [17520.332900] ? do_raw_spin_unlock+0x75/0x190 [17520.337346] ? _raw_spin_unlock+0x12/0x30 [17520.344309] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17520.348062] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17520.359295] ll_atomic_open+0x36a/0xd70 [lustre] [17520.365089] lookup_open+0x356/0xb40 [17520.373695] do_last+0x313/0xfc0 [17520.379776] ? nd_jump_root+0xe5/0x160 [17520.382294] ? path_init+0x437/0x520 [17520.384300] path_openat+0xf7/0x500 [17520.386448] ? do_raw_spin_unlock+0x75/0x190 [17520.388531] do_filp_open+0x99/0x140 [17520.390637] ? getname_flags+0x6e/0x330 [17520.392469] ? __check_object_size+0xff/0x256 [17520.395086] ? do_raw_spin_unlock+0x75/0x190 [17520.397069] ? _raw_spin_unlock+0x12/0x30 [17520.401950] do_sys_openat2+0x2b4/0x410 [17520.403742] do_sys_open+0x73/0xa0 [17520.405805] __x64_sys_openat+0x24/0x30 [17520.411281] do_syscall_64+0xc1/0x440 [17520.413784] entry_SYSCALL_64_after_hwframe+0x49/0xae [17520.418880] RIP: 0033:0x7fa749552332 [17520.422753] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17520.438076] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17520.448529] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17520.456951] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17520.461699] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17520.465499] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17520.475529] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17523.169225] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15606s! [17523.173487] Showing busy workqueues and worker pools: [17523.176934] workqueue events: flags=0x0 [17523.180605] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17523.183725] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17523.188438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17523.191058] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17523.194695] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.199102] pending: kfree_rcu_monitor [17523.202130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17523.208214] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17523.211851] workqueue events_long: flags=0x0 [17523.213528] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.216067] pending: lu_objects_flush [obdclass] [17523.218345] workqueue events_unbound: flags=0x2 [17523.220108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17523.222761] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17523.227528] workqueue events_power_efficient: flags=0x80 [17523.229163] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.232076] pending: check_lifetime [17523.233509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.236338] pending: neigh_periodic_work [17523.237699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.240361] pending: neigh_periodic_work [17523.242709] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.245132] pending: do_cache_clean [17523.246581] workqueue mm_percpu_wq: flags=0x8 [17523.248644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.250504] pending: vmstat_update [17523.251859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.253951] pending: vmstat_update [17523.255495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17523.257445] pending: vmstat_update [17523.259099] workqueue writeback: flags=0x4a [17523.260511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17523.268648] pending: wb_workfn, wb_workfn [17523.270514] workqueue xprtiod: flags=0xa [17523.271943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17523.278465] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17523.285633] workqueue ipv6_addrconf: flags=0x40008 [17523.287610] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17523.290687] pending: addrconf_verify_work [17523.293094] workqueue knbd0-recv: flags=0x1a [17523.295733] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17523.299676] in-flight: 651:recv_work [17523.302820] workqueue ptlrpc_pinger: flags=0x2 [17523.304512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17523.306599] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17523.307950] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15606s workers=4 idle: 37 31846 28897 [17523.316531] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17513s workers=2 idle: 653 [17524.059768] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17524.064241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17524.089907] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17524.096740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17524.101508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17524.104834] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17524.115654] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17524.122704] RAX: 0000000000002b0a RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17524.138775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17524.153465] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17524.158779] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17524.164644] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17524.168827] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17524.173715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17524.177652] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17524.181077] Call Trace: [17524.182527] [17524.183438] ? show_regs.cold.9+0x22/0x2f [17524.185651] ? watchdog_timer_fn.cold.6+0x62/0x10a [17524.187914] ? __hrtimer_run_queues+0x1c2/0x4d0 [17524.190420] ? watchdog+0x40/0x40 [17524.191988] ? hrtimer_interrupt+0x128/0x2c0 [17524.194422] ? smp_apic_timer_interrupt+0xa6/0x270 [17524.196727] ? apic_timer_interrupt+0xf/0x20 [17524.198829] [17524.200078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17524.202784] do_raw_spin_lock+0x177/0x190 [17524.204605] _raw_spin_lock+0x21/0x30 [17524.206467] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17524.213230] ? do_raw_spin_unlock+0x75/0x190 [17524.218576] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17524.224006] ? do_wait_intr+0xf0/0xf0 [17524.225692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17524.228618] mdc_close+0x53b/0xd70 [mdc] [17524.230942] lmv_close+0x18d/0x490 [lmv] [17524.232951] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17524.236725] ll_md_real_close+0xa5/0x2a0 [lustre] [17524.239791] ll_md_close+0x274/0x800 [lustre] [17524.242392] ll_file_release+0x128/0xc50 [lustre] [17524.245703] __fput+0x10d/0x440 [17524.247868] ____fput+0x12/0x20 [17524.249671] task_work_run+0xc6/0x110 [17524.251662] exit_to_usermode_loop+0x1dd/0x1f0 [17524.254792] do_syscall_64+0x3d6/0x440 [17524.256614] entry_SYSCALL_64_after_hwframe+0x49/0xae [17524.259028] RIP: 0033:0x7fa749552cd8 [17524.260852] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17524.270056] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17524.275002] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17524.279537] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17524.283866] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17524.289153] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17524.293965] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17536.058038] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17536.063988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17536.115635] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17536.130213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17536.137496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17536.140388] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17536.150288] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17536.153988] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17536.160689] RDX: 0000000000001e82 RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17536.164295] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17536.168270] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17536.174879] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17536.181374] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17536.189460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17536.196041] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17536.199409] Call Trace: [17536.204040] [17536.205100] ? show_regs.cold.9+0x22/0x2f [17536.207382] ? watchdog_timer_fn.cold.6+0x62/0x10a [17536.211447] ? __hrtimer_run_queues+0x1c2/0x4d0 [17536.213637] ? watchdog+0x40/0x40 [17536.218558] ? hrtimer_interrupt+0x128/0x2c0 [17536.220985] ? smp_apic_timer_interrupt+0xa6/0x270 [17536.223449] ? apic_timer_interrupt+0xf/0x20 [17536.224991] [17536.225685] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17536.227484] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17536.230301] do_raw_spin_lock+0x177/0x190 [17536.232062] _raw_spin_lock+0x21/0x30 [17536.233783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17536.236363] obd_mod_free+0x46/0x140 [ptlrpc] [17536.238977] mdc_commit_open+0x91/0xb0 [mdc] [17536.240999] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17536.243488] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17536.246344] after_reply+0xe2c/0x1780 [ptlrpc] [17536.249274] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17536.252323] ? __next_timer_interrupt+0x160/0x160 [17536.254627] ptlrpcd+0xa82/0xe40 [ptlrpc] [17536.256910] ? do_wait_intr+0xf0/0xf0 [17536.258542] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17536.264464] kthread+0x1d1/0x200 [17536.269483] ? set_kthread_struct+0x70/0x70 [17536.271790] ret_from_fork+0x1f/0x30 [17540.057067] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:0:8] [17540.061956] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17540.112409] CPU: 1 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17540.126968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17540.135952] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17540.140460] RIP: 0010:native_safe_halt+0x2e/0x50 [17540.143394] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17540.152113] RSP: 0018:ffffb893c0c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17540.155221] RAX: 0000000000000001 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17540.158239] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d73820b4154 [17540.162526] RBP: ffff8d73820b4140 R08: 0000000000000000 R09: 0000000000000000 [17540.164554] R10: 0000000003cb3067 R11: ffff8d73586d6000 R12: 0000000000080000 [17540.174614] R13: ffff8d73821b4140 R14: 000000000001e480 R15: ffff8d738209e480 [17540.179943] FS: 0000000000000000(0000) GS:ffff8d7382080000(0000) knlGS:0000000000000000 [17540.186806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17540.190163] CR2: 00007efe8a96ea08 CR3: 0000000108e16004 CR4: 0000000000170ee0 [17540.195697] Call Trace: [17540.197108] [17540.198075] ? show_regs.cold.9+0x22/0x2f [17540.200032] ? watchdog_timer_fn.cold.6+0x62/0x10a [17540.202222] ? __hrtimer_run_queues+0x1c2/0x4d0 [17540.206304] ? watchdog+0x40/0x40 [17540.208409] ? hrtimer_interrupt+0x128/0x2c0 [17540.212231] ? smp_apic_timer_interrupt+0xa6/0x270 [17540.214396] ? apic_timer_interrupt+0xf/0x20 [17540.217087] [17540.218325] ? native_safe_halt+0x2e/0x50 [17540.220300] kvm_wait+0xcd/0xe0 [17540.221920] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17540.226172] do_raw_spin_lock+0x177/0x190 [17540.231807] _raw_spin_lock+0x21/0x30 [17540.234361] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17540.238004] process_one_work+0x2c8/0x700 [17540.242076] worker_thread+0x296/0x6e0 [17540.248512] ? rescuer_thread+0x570/0x570 [17540.251244] kthread+0x1d1/0x200 [17540.253379] ? set_kthread_struct+0x70/0x70 [17540.255071] ret_from_fork+0x1f/0x30 [17548.054008] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37250] [17548.057294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17548.074283] CPU: 0 PID: 37250 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17548.077724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17548.081433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17548.083321] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17548.089928] RSP: 0018:ffffb893c1493500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17548.093039] RAX: 00000000000036e1 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17548.096400] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d7382034154 [17548.099501] RBP: ffff8d7382034140 R08: 0000000000000000 R09: 0000000000000000 [17548.102843] R10: 0000000003cb297d R11: ffff8d73500eb000 R12: 0000000000040000 [17548.105671] R13: ffff8d7382134140 R14: 000000000001e480 R15: ffff8d738201e480 [17548.109407] FS: 00007fa749c23740(0000) GS:ffff8d7382000000(0000) knlGS:0000000000000000 [17548.118913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17548.120470] CR2: 000056281e378140 CR3: 0000000101357004 CR4: 0000000000170ef0 [17548.122229] Call Trace: [17548.122894] [17548.123428] ? show_regs.cold.9+0x22/0x2f [17548.124445] ? watchdog_timer_fn.cold.6+0x62/0x10a [17548.125807] ? __hrtimer_run_queues+0x1c2/0x4d0 [17548.126924] ? watchdog+0x40/0x40 [17548.127716] ? hrtimer_interrupt+0x128/0x2c0 [17548.129066] ? smp_apic_timer_interrupt+0xa6/0x270 [17548.130454] ? apic_timer_interrupt+0xf/0x20 [17548.131592] [17548.132125] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17548.133436] do_raw_spin_lock+0x177/0x190 [17548.134505] _raw_spin_lock+0x21/0x30 [17548.135647] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17548.137147] ? do_raw_spin_unlock+0x75/0x190 [17548.138499] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17548.140035] ? do_wait_intr+0xf0/0xf0 [17548.140913] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17548.142324] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17548.143847] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17548.145460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17548.146877] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17548.148246] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17548.149479] mdc_intent_lock+0x353/0x7e0 [mdc] [17548.150647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17548.152126] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17548.153904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17548.155575] lmv_intent_open+0x338/0x1150 [lmv] [17548.157112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17548.159152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17548.161582] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17548.163168] ll_intent_lock+0xc5/0xaa0 [lustre] [17548.164576] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17548.166876] ll_lookup_it+0xe0e/0x2270 [lustre] [17548.168499] ? do_raw_spin_unlock+0x75/0x190 [17548.170061] ? _raw_spin_unlock+0x12/0x30 [17548.172401] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17548.174884] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17548.176951] ll_atomic_open+0x36a/0xd70 [lustre] [17548.179113] lookup_open+0x356/0xb40 [17548.181414] do_last+0x313/0xfc0 [17548.182821] ? nd_jump_root+0xe5/0x160 [17548.184037] ? path_init+0x437/0x520 [17548.185223] path_openat+0xf7/0x500 [17548.186496] ? do_raw_spin_unlock+0x75/0x190 [17548.188097] do_filp_open+0x99/0x140 [17548.189667] ? getname_flags+0x6e/0x330 [17548.191052] ? __check_object_size+0xff/0x256 [17548.192702] ? do_raw_spin_unlock+0x75/0x190 [17548.194148] ? _raw_spin_unlock+0x12/0x30 [17548.195510] do_sys_openat2+0x2b4/0x410 [17548.196900] do_sys_open+0x73/0xa0 [17548.198041] __x64_sys_openat+0x24/0x30 [17548.199252] do_syscall_64+0xc1/0x440 [17548.200666] entry_SYSCALL_64_after_hwframe+0x49/0xae [17548.202561] RIP: 0033:0x7fa749552332 [17548.203904] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17548.210010] RSP: 002b:00007fff9a132280 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17548.212522] RAX: ffffffffffffffda RBX: 0000000000009fe8 RCX: 00007fa749552332 [17548.214483] RDX: 0000000000000042 RSI: 00007fff9a132350 RDI: 00000000ffffff9c [17548.216570] RBP: 00007fff9a132750 R08: 0000000000000000 R09: 0000000000000000 [17548.218736] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff9a132350 [17548.220804] R13: 00000000ffffffff R14: 000000000000000e R15: 0000000000000000 [17552.059171] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37249] [17552.062402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17552.077139] CPU: 3 PID: 37249 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17552.081004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17552.084763] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17552.087638] Code: 4c 03 2c c5 80 58 68 b4 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17552.096284] RSP: 0018:ffffb893c146baf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17552.099921] RAX: 0000000000002f86 RBX: ffff8d73433a52e0 RCX: 0000000000000001 [17552.103537] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8d73821b4154 [17552.107634] RBP: ffff8d73821b4140 R08: 0000000000000000 R09: 0000000000000000 [17552.111815] R10: 0000000003cb2a23 R11: ffff8d734b73e000 R12: 0000000000100000 [17552.116476] R13: ffff8d7382034140 R14: 000000000001e480 R15: ffff8d738219e480 [17552.119428] FS: 00007fa749c23740(0000) GS:ffff8d7382180000(0000) knlGS:0000000000000000 [17552.123559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17552.126171] CR2: 00007f55821bd040 CR3: 00000001186b0003 CR4: 0000000000170ee0 [17552.129419] Call Trace: [17552.130602] [17552.131437] ? show_regs.cold.9+0x22/0x2f [17552.133362] ? watchdog_timer_fn.cold.6+0x62/0x10a [17552.135275] ? __hrtimer_run_queues+0x1c2/0x4d0 [17552.136669] ? watchdog+0x40/0x40 [17552.137789] ? hrtimer_interrupt+0x128/0x2c0 [17552.139225] ? smp_apic_timer_interrupt+0xa6/0x270 [17552.141149] ? apic_timer_interrupt+0xf/0x20 [17552.142688] [17552.143624] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17552.145312] do_raw_spin_lock+0x177/0x190 [17552.146721] _raw_spin_lock+0x21/0x30 [17552.148346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17552.150875] ? do_raw_spin_unlock+0x75/0x190 [17552.152827] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17552.155489] ? do_wait_intr+0xf0/0xf0 [17552.156873] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17552.159417] mdc_close+0x53b/0xd70 [mdc] [17552.160985] lmv_close+0x18d/0x490 [lmv] [17552.162720] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17552.165161] ll_md_real_close+0xa5/0x2a0 [lustre] [17552.167443] ll_md_close+0x274/0x800 [lustre] [17552.169177] ll_file_release+0x128/0xc50 [lustre] [17552.171322] __fput+0x10d/0x440 [17552.172486] ____fput+0x12/0x20 [17552.173690] task_work_run+0xc6/0x110 [17552.175262] exit_to_usermode_loop+0x1dd/0x1f0 [17552.176832] do_syscall_64+0x3d6/0x440 [17552.178594] entry_SYSCALL_64_after_hwframe+0x49/0xae [17552.180413] RIP: 0033:0x7fa749552cd8 [17552.181838] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17552.189662] RSP: 002b:00007fff9a1322e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17552.192623] RAX: 0000000000000000 RBX: 0000000000001538 RCX: 00007fa749552cd8 [17552.195305] RDX: 0000000000000038 RSI: 00007fff9a132750 RDI: 0000000000000003 [17552.197757] RBP: 00007fff9a132750 R08: 00007fa7497f5234 R09: 00007fa7497f5260 [17552.200535] R10: 00007fff9a132092 R11: 0000000000000246 R12: 0000000000000003 [17552.203470] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17553.377437] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15636s! [17553.381883] Showing busy workqueues and worker pools: [17553.384884] workqueue events: flags=0x0 [17553.386740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17553.389506] pending: cpu_latency_work [ptlrpc], key_garbage_collector, osc_grant_work_handler [osc], push_to_pool [17553.393866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17553.396043] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [17553.399207] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.401354] pending: kfree_rcu_monitor [17553.402647] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17553.405073] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], sec_gc_main [ptlrpc] [17553.408288] workqueue events_long: flags=0x0 [17553.409758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.412025] pending: lu_objects_flush [obdclass] [17553.413784] workqueue events_unbound: flags=0x2 [17553.415322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [17553.417871] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17553.421119] workqueue events_power_efficient: flags=0x80 [17553.423304] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.425543] pending: check_lifetime [17553.427178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.429483] pending: neigh_periodic_work [17553.430887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.433326] pending: neigh_periodic_work [17553.435115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.437115] pending: do_cache_clean [17553.438292] workqueue mm_percpu_wq: flags=0x8 [17553.439626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.441988] pending: vmstat_update [17553.443067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.445644] pending: vmstat_update [17553.446894] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17553.449362] pending: vmstat_update [17553.450584] workqueue writeback: flags=0x4a [17553.452207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17553.454417] pending: wb_workfn, wb_workfn [17553.456192] workqueue xprtiod: flags=0xa [17553.457432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17553.459557] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [17553.462377] workqueue ipv6_addrconf: flags=0x40008 [17553.463852] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17553.466189] pending: addrconf_verify_work [17553.467593] workqueue knbd0-recv: flags=0x1a [17553.468985] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17553.471157] in-flight: 651:recv_work [17553.472418] workqueue ptlrpc_pinger: flags=0x2 [17553.474339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17553.477460] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [17553.479994] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15636s workers=4 idle: 37 31846 28897 [17553.482744] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17543s workers=2 idle: 653 [17564.058428] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2403] [17564.064863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17564.088244] CPU: 2 PID: 2403 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17564.093063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17564.096734] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17564.099902] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 4d c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17564.107554] RSP: 0018:ffffb893c1bd3bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17564.111068] RAX: 0000000000000003 RBX: ffff8d73433a52e0 RCX: ffff8d738211e480 [17564.114231] RDX: 0000000000002c0e RSI: 0000000000000003 RDI: ffff8d73433a52e0 [17564.117505] RBP: ffff8d7382134140 R08: 0000000000000008 R09: 00000000000000f0 [17564.120450] R10: ffffffffb4808b80 R11: ffffb893c1bd38f0 R12: 00000000000c0000 [17564.123266] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17564.125972] FS: 0000000000000000(0000) GS:ffff8d7382100000(0000) knlGS:0000000000000000 [17564.129429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17564.132217] CR2: 0000564c8f5100f8 CR3: 0000000108e16002 CR4: 0000000000170ee0 [17564.135375] Call Trace: [17564.136406] [17564.137108] ? show_regs.cold.9+0x22/0x2f [17564.138882] ? watchdog_timer_fn.cold.6+0x62/0x10a [17564.140920] ? __hrtimer_run_queues+0x1c2/0x4d0 [17564.142785] ? watchdog+0x40/0x40 [17564.144362] ? hrtimer_interrupt+0x128/0x2c0 [17564.146369] ? smp_apic_timer_interrupt+0xa6/0x270 [17564.148516] ? apic_timer_interrupt+0xf/0x20 [17564.152028] [17564.152839] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17564.159948] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17564.164776] do_raw_spin_lock+0x177/0x190 [17564.167518] _raw_spin_lock+0x21/0x30 [17564.170152] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17564.179600] obd_mod_free+0x46/0x140 [ptlrpc] [17564.183448] mdc_commit_open+0x91/0xb0 [mdc] [17564.184870] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17564.186898] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17564.192200] after_reply+0xe2c/0x1780 [ptlrpc] [17564.194190] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17564.197590] ? __next_timer_interrupt+0x160/0x160 [17564.199563] ptlrpcd+0xa82/0xe40 [ptlrpc] [17564.202023] ? do_wait_intr+0xf0/0xf0 [17564.203387] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17564.205780] kthread+0x1d1/0x200 [17564.206863] ? set_kthread_struct+0x70/0x70 [17564.208528] ret_from_fork+0x1f/0x30